# Table of Contents - [Nomad | HashiCorp Developer](#nomad-hashicorp-developer) - [Tutorials | Nomad | HashiCorp Developer](#tutorials-nomad-hashicorp-developer) - [HTTP API | Nomad | HashiCorp Developer](#http-api-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Tools | Nomad | HashiCorp Developer](#tools-nomad-hashicorp-developer) - [Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer](#nomad-command-line-interface-cli-reference-nomad-hashicorp-developer) - [Nomad plugins | Nomad | HashiCorp Developer](#nomad-plugins-nomad-hashicorp-developer) - [Integrations | Nomad | HashiCorp Developer](#integrations-nomad-hashicorp-developer) - [Install | Nomad | HashiCorp Developer](#install-nomad-hashicorp-developer) - [Nomad | HashiCorp Developer](#nomad-hashicorp-developer) - [What is Nomad? | Nomad | HashiCorp Developer](#what-is-nomad-nomad-hashicorp-developer) - [Quick Start | Nomad | HashiCorp Developer](#quick-start-nomad-hashicorp-developer) - [Create and submit a job | Nomad | HashiCorp Developer](#create-and-submit-a-job-nomad-hashicorp-developer) - [Create a parameterized Nomad job | Nomad | HashiCorp Developer](#create-a-parameterized-nomad-job-nomad-hashicorp-developer) - [Vault | Nomad | HashiCorp Developer](#vault-nomad-hashicorp-developer) - [Enterprise | Nomad | HashiCorp Developer](#enterprise-nomad-hashicorp-developer) - [Consul | Nomad | HashiCorp Developer](#consul-nomad-hashicorp-developer) - [ACL system overview | Nomad | HashiCorp Developer](#acl-system-overview-nomad-hashicorp-developer) - [Nomad Pack | Nomad | HashiCorp Developer](#nomad-pack-nomad-hashicorp-developer) - [Governance and policy on Nomad | Nomad | HashiCorp Developer](#governance-and-policy-on-nomad-nomad-hashicorp-developer) - [Traffic encryption overview | Nomad | HashiCorp Developer](#traffic-encryption-overview-nomad-hashicorp-developer) - [Nomad clusters | Nomad | HashiCorp Developer](#nomad-clusters-nomad-hashicorp-developer) - [Failure recovery strategies | Nomad | HashiCorp Developer](#failure-recovery-strategies-nomad-hashicorp-developer) - [Advanced job scheduling | Nomad | HashiCorp Developer](#advanced-job-scheduling-nomad-hashicorp-developer) - [Upgrade Nomad | Nomad | HashiCorp Developer](#upgrade-nomad-nomad-hashicorp-developer) - [Configure Nomad task drivers | Nomad | HashiCorp Developer](#configure-nomad-task-drivers-nomad-hashicorp-developer) - [Run a Granite AI workload on Nomad | Nomad | HashiCorp Developer](#run-a-granite-ai-workload-on-nomad-nomad-hashicorp-developer) - [Migrate a monolith to microservices | Nomad | HashiCorp Developer](#migrate-a-monolith-to-microservices-nomad-hashicorp-developer) - [Cluster Setup | Nomad | HashiCorp Developer](#cluster-setup-nomad-hashicorp-developer) - [Job Specifications | Nomad | HashiCorp Developer](#job-specifications-nomad-hashicorp-developer) - [Edge Computing | Nomad | HashiCorp Developer](#edge-computing-nomad-hashicorp-developer) - [Migrate a monolith | Nomad | HashiCorp Developer](#migrate-a-monolith-nomad-hashicorp-developer) - [Advanced Scheduling | Nomad | HashiCorp Developer](#advanced-scheduling-nomad-hashicorp-developer) - [AI workloads | Nomad | HashiCorp Developer](#ai-workloads-nomad-hashicorp-developer) - [Workload identity | Nomad | HashiCorp Developer](#workload-identity-nomad-hashicorp-developer) - [Windows | Nomad | HashiCorp Developer](#windows-nomad-hashicorp-developer) - [Nomad Variables | Nomad | HashiCorp Developer](#nomad-variables-nomad-hashicorp-developer) - [Manage Clusters | Nomad | HashiCorp Developer](#manage-clusters-nomad-hashicorp-developer) - [Autoscaling | Nomad | HashiCorp Developer](#autoscaling-nomad-hashicorp-developer) - [Service Discovery on Nomad | Nomad | HashiCorp Developer](#service-discovery-on-nomad-nomad-hashicorp-developer) - [Templates | Nomad | HashiCorp Developer](#templates-nomad-hashicorp-developer) - [Load Balancing | Nomad | HashiCorp Developer](#load-balancing-nomad-hashicorp-developer) - [Libraries and SDKs - HTTP API | Nomad | HashiCorp Developer](#libraries-and-sdks-http-api-nomad-hashicorp-developer) - [Task driver plugins | Nomad | HashiCorp Developer](#task-driver-plugins-nomad-hashicorp-developer) - [Alertmanager | Integrations | Nomad | HashiCorp Developer](#alertmanager-integrations-nomad-hashicorp-developer) - [Create custom packs | Nomad | HashiCorp Developer](#create-custom-packs-nomad-hashicorp-developer) - [Boundary | Integrations | Nomad | HashiCorp Developer](#boundary-integrations-nomad-hashicorp-developer) - [CSI OpenStack Cinder | Integrations | Nomad | HashiCorp Developer](#csi-openstack-cinder-integrations-nomad-hashicorp-developer) - [Create a secret provider plugin | Nomad | HashiCorp Developer](#create-a-secret-provider-plugin-nomad-hashicorp-developer) - [AWS EFS CSI | Integrations | Nomad | HashiCorp Developer](#aws-efs-csi-integrations-nomad-hashicorp-developer) - [AWS EBS CSI | Integrations | Nomad | HashiCorp Developer](#aws-ebs-csi-integrations-nomad-hashicorp-developer) - [Community task driver plugins | Nomad | HashiCorp Developer](#community-task-driver-plugins-nomad-hashicorp-developer) - [Nomad plugin authoring guide | Nomad | HashiCorp Developer](#nomad-plugin-authoring-guide-nomad-hashicorp-developer) - [Backstage | Integrations | Nomad | HashiCorp Developer](#backstage-integrations-nomad-hashicorp-developer) - [Nomad device driver plugins | Nomad | HashiCorp Developer](#nomad-device-driver-plugins-nomad-hashicorp-developer) - [Install the Nomad CLI | Nomad | HashiCorp Developer](#install-the-nomad-cli-nomad-hashicorp-developer) - [Schedule edge Services with native service discovery | Nomad | HashiCorp Developer](#schedule-edge-services-with-native-service-discovery-nomad-hashicorp-developer) - [Set up a Nomad cluster on AWS | Nomad | HashiCorp Developer](#set-up-a-nomad-cluster-on-aws-nomad-hashicorp-developer) - [Install a HashiCorp Enterprise license | Nomad | HashiCorp Developer](#install-a-hashicorp-enterprise-license-nomad-hashicorp-developer) - [nomad agent-info command reference | Nomad | HashiCorp Developer](#nomad-agent-info-command-reference-nomad-hashicorp-developer) - [Create a Nomad cluster | Nomad | HashiCorp Developer](#create-a-nomad-cluster-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Consul service mesh in production | Nomad | HashiCorp Developer](#consul-service-mesh-in-production-nomad-hashicorp-developer) - [Nomad Pack advanced usage | Nomad | HashiCorp Developer](#nomad-pack-advanced-usage-nomad-hashicorp-developer) - [nomad fmt command reference | Nomad | HashiCorp Developer](#nomad-fmt-command-reference-nomad-hashicorp-developer) - [nomad agent command reference | Nomad | HashiCorp Developer](#nomad-agent-command-reference-nomad-hashicorp-developer) - [Template Nomad jobspecs with Levant | Nomad | HashiCorp Developer](#template-nomad-jobspecs-with-levant-nomad-hashicorp-developer) - [Stop Nomad and clean up | Nomad | HashiCorp Developer](#stop-nomad-and-clean-up-nomad-hashicorp-developer) - [Agent - HTTP API | Nomad | HashiCorp Developer](#agent-http-api-nomad-hashicorp-developer) - [Scale your Nomad cluster with horizontal cluster autoscaling | Nomad | HashiCorp Developer](#scale-your-nomad-cluster-with-horizontal-cluster-autoscaling-nomad-hashicorp-developer) - [Autoscaling | Nomad | HashiCorp Developer](#autoscaling-nomad-hashicorp-developer) - [Generate mTLS certificates for Nomad using Vault | Nomad | HashiCorp Developer](#generate-mtls-certificates-for-nomad-using-vault-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Template abstract job specs with Levant | Nomad | HashiCorp Developer](#template-abstract-job-specs-with-levant-nomad-hashicorp-developer) - [Dynamically scale a Nomad cluster with the Nomad Autoscaler | Nomad | HashiCorp Developer](#dynamically-scale-a-nomad-cluster-with-the-nomad-autoscaler-nomad-hashicorp-developer) - [Task HTTP API | Nomad | HashiCorp Developer](#task-http-api-nomad-hashicorp-developer) - [Glossary | Nomad | HashiCorp Developer](#glossary-nomad-hashicorp-developer) - [Introduction to Nomad | Nomad | HashiCorp Developer](#introduction-to-nomad-nomad-hashicorp-developer) - [Dynamic Application Sizing concepts | Nomad | HashiCorp Developer](#dynamic-application-sizing-concepts-nomad-hashicorp-developer) - [nomad login command reference | Nomad | HashiCorp Developer](#nomad-login-command-reference-nomad-hashicorp-developer) - [Deploy and update a Nomad job | Nomad | HashiCorp Developer](#deploy-and-update-a-nomad-job-nomad-hashicorp-developer) - [Create a host volume plugin | Nomad | HashiCorp Developer](#create-a-host-volume-plugin-nomad-hashicorp-developer) - [nomad version command reference | Nomad | HashiCorp Developer](#nomad-version-command-reference-nomad-hashicorp-developer) - [nomad ui command reference | Nomad | HashiCorp Developer](#nomad-ui-command-reference-nomad-hashicorp-developer) - [Migrate a Linux-based Java application to Nomad | Nomad | HashiCorp Developer](#migrate-a-linux-based-java-application-to-nomad-nomad-hashicorp-developer) - [Migrate a Windows-based Java application to Nomad | Nomad | HashiCorp Developer](#migrate-a-windows-based-java-application-to-nomad-nomad-hashicorp-developer) - [nomad status command reference | Nomad | HashiCorp Developer](#nomad-status-command-reference-nomad-hashicorp-developer) - [Nomad quickstart | Nomad | HashiCorp Developer](#nomad-quickstart-nomad-hashicorp-developer) - [Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer](#nomad-command-line-interface-cli-reference-nomad-hashicorp-developer) - [Use Cases | Nomad | HashiCorp Developer](#use-cases-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Deploy a Nomad Enterprise cluster | Nomad | HashiCorp Developer](#deploy-a-nomad-enterprise-cluster-nomad-hashicorp-developer) - [Nomad Ecosystem | Nomad | HashiCorp Developer](#nomad-ecosystem-nomad-hashicorp-developer) - [Allocations - HTTP API | Nomad | HashiCorp Developer](#allocations-http-api-nomad-hashicorp-developer) - [Deploy a Consul API Gateway on Nomad | Nomad | HashiCorp Developer](#deploy-a-consul-api-gateway-on-nomad-nomad-hashicorp-developer) - [Secure Nomad jobs with Consul service mesh | Nomad | HashiCorp Developer](#secure-nomad-jobs-with-consul-service-mesh-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Frequently Asked Questions | Nomad | HashiCorp Developer](#frequently-asked-questions-nomad-hashicorp-developer) - [Client - HTTP API | Nomad | HashiCorp Developer](#client-http-api-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Deployments - HTTP API | Nomad | HashiCorp Developer](#deployments-http-api-nomad-hashicorp-developer) - [Install Nomad | Nomad | HashiCorp Developer](#install-nomad-nomad-hashicorp-developer) - [Use Dynamic Application Sizing | Nomad | HashiCorp Developer](#use-dynamic-application-sizing-nomad-hashicorp-developer) - [Federate multi-region clusters | Nomad | HashiCorp Developer](#federate-multi-region-clusters-nomad-hashicorp-developer) - [Evaluations - HTTP API | Nomad | HashiCorp Developer](#evaluations-http-api-nomad-hashicorp-developer) - [JSON Job Specification - HTTP API | Nomad | HashiCorp Developer](#json-job-specification-http-api-nomad-hashicorp-developer) - [What is Nomad? | Nomad | HashiCorp Developer](#what-is-nomad-nomad-hashicorp-developer) - [Connect nodes into a cluster | Nomad | HashiCorp Developer](#connect-nodes-into-a-cluster-nomad-hashicorp-developer) - [Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer](#nomad-command-line-interface-cli-reference-nomad-hashicorp-developer) - [Regions - HTTP API | Nomad | HashiCorp Developer](#regions-http-api-nomad-hashicorp-developer) - [Status - HTTP API | Nomad | HashiCorp Developer](#status-http-api-nomad-hashicorp-developer) - [Exec2 task driver plugin | Nomad | HashiCorp Developer](#exec2-task-driver-plugin-nomad-hashicorp-developer) - [Oversubscribe memory | Nomad | HashiCorp Developer](#oversubscribe-memory-nomad-hashicorp-developer) - [Commands (CLI) | Nomad | HashiCorp Developer](#commands-cli-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Documentation | Nomad | HashiCorp Developer](#documentation-nomad-hashicorp-developer) - [Namespace - HTTP API | Nomad | HashiCorp Developer](#namespace-http-api-nomad-hashicorp-developer) - [System - HTTP API | Nomad | HashiCorp Developer](#system-http-api-nomad-hashicorp-developer) - [Validate - HTTP API | Nomad | HashiCorp Developer](#validate-http-api-nomad-hashicorp-developer) - [Metrics - HTTP API | Nomad | HashiCorp Developer](#metrics-http-api-nomad-hashicorp-developer) - [Plugins - HTTP API | Nomad | HashiCorp Developer](#plugins-http-api-nomad-hashicorp-developer) - [Services - HTTP API | Nomad | HashiCorp Developer](#services-http-api-nomad-hashicorp-developer) - [Scaling Policies - HTTP API | Nomad | HashiCorp Developer](#scaling-policies-http-api-nomad-hashicorp-developer) - [Events - HTTP API | Nomad | HashiCorp Developer](#events-http-api-nomad-hashicorp-developer) - [Nomad Integration Program | Nomad | HashiCorp Developer](#nomad-integration-program-nomad-hashicorp-developer) - [Sentinel Policies - HTTP API | Nomad | HashiCorp Developer](#sentinel-policies-http-api-nomad-hashicorp-developer) - [Enable gossip encryption | Nomad | HashiCorp Developer](#enable-gossip-encryption-nomad-hashicorp-developer) - [Traffic encryption overview | Nomad | HashiCorp Developer](#traffic-encryption-overview-nomad-hashicorp-developer) --- # Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) Sign In[Sign Up](https://developer.hashicorp.com/sign-up) Theme DarkLightSystem Orchestrate, deploy, and manage containers, binaries, and batch jobs in the cloud or on-prem ============================================================================================ ![](https://www.datocms-assets.com/2885/1662767792-dev-dot-nomad-landing-hero-2x.png) *  [](https://developer.hashicorp.com/nomad/install) Install *  [](https://developer.hashicorp.com/nomad/tutorials) Tutorials *  [](https://developer.hashicorp.com/nomad/docs) Documentation *  [](https://developer.hashicorp.com/nomad/integrations) Integrations What is Nomad? -------------- A simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-prem and clouds at scale [Learn more about Nomad features](https://developer.hashicorp.com/nomad/docs/what-is-nomad) ![](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fwww.datocms-assets.com%2F2885%2F1679087719-devdot-nomad_dm.png&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn)![](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fwww.datocms-assets.com%2F2885%2F1679095153-devdot-nomad_lm.png&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Getting Started --------------- Learn how to use Nomad to schedule and orchestrate workloads. [CLI Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) Sandbox ------- *  [](https://developer.hashicorp.com/nomad/sandbox) Nomad sandbox The Nomad sandbox contains preinstalled tools and services for you to experiment with Nomad. Featured Tutorials ------------------ *  [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload) Interactive 15min Run a Granite AI workload on Nomad Create a Nomad cluster on AWS and run a Granite LLM workload with Ollama and Open WebUI. * Nomad * Terraform *  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview) 10min Migrate a monolith to microservices Migrate a monolithic application to microservices using Consul and Nomad. * Nomad * Consul Nomad Jobs ---------- *  [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job) Create and Deploy Jobs Learn the fundamentals of expressing your workload as Nomad jobs. *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized) Parameterized Jobs Nomad parameterized jobs enable dynamic runtime behaviors from a single job specification. *  [](https://developer.hashicorp.com/nomad/tools/nomad-pack) Nomad Pack Create and re-use existing job templates with Nomad Pack. HashiCorp Tools Integrations ---------------------------- *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) Integrate Consul Learn to secure jobs and add Consul health checking, service discovery, and service mesh capabilities. *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault) Integrate Vault Configure the Nomad secrets engine in Vault and use Vault to obtain secrets for workloads. Nomad in Production ------------------- *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise) Nomad Enterprise Learn about Nomad Enterprise features and how to use them at scale. *  [](https://developer.hashicorp.com/nomad/docs/secure/acl) Secure Nomad with Access Control Learn to configure a Nomad cluster for ACLs, author your first policy, and grant a token based on it. *  [](https://developer.hashicorp.com/nomad/docs/secure/traffic) Enable Traffic Encryption for Nomad Encrypt Nomad's intercluster traffic—UDP gossip and TCP API/RPC traffic. *  [](https://developer.hashicorp.com/nomad/docs/job-declare/failure) Job Resiliency Learn to keep jobs running in the event of the unexpected with local retries and node rescheduling. *  [](https://developer.hashicorp.com/nomad/docs/govern) Governance and Policy Operate Nomad securely in a multi-team environment with resource quotas and Sentinel integration. *  [](https://developer.hashicorp.com/nomad/docs/job-scheduling) Advanced Application Scheduling Learn to express placement preferences for job allocations using affinities and preemption. Cluster Management ------------------ *  [](https://developer.hashicorp.com/nomad/docs/deploy/clusters) Build Nomad Clusters Learn the features operators need to build and maintain Nomad clusters. *  [](https://developer.hashicorp.com/nomad/docs/upgrade) Upgrade a Cluster Nomad is designed to be flexible and resilient when upgrading from one Nomad version to the next. Plugins and Tools ----------------- *  [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver) Task Drivers Nomad clients use task drivers to execute a task. Extending task drivers allows Nomad to support many types of workloads. *  [](https://developer.hashicorp.com/nomad/tools) External Tools External tools provide additional capabilities and use cases to Nomad and can sometimes be deployed as a Nomad job in your cluster. Best practices for deploying infrastructure ------------------------------------------- *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/define/as-code/orchestration) Define your container orchestrator Use infrastructure as code to define your container orchestration and create consistent version-controlled specifications for Nomad. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/monitor/setup-monitoring-agents/containers) Monitor your container orchestrator Monitor container orchestrators to keep your clusters and services healthy, which ensures your application's performance and reliability. *  [](https://developer.hashicorp.com/well-architected-framework/optimize-systems/scale-resources/containers) Optimize your container orchestrator Container orchestrators quickly scale your applications and help you optimize resource utilization. *  [](https://developer.hashicorp.com/well-architected-framework/secure-systems/infrastructure/zero-trust-security) Secure your container orchestrator A zero trust security model enhances your application's security by eliminating implicit trust and continuously verifying access requests. Best practices for package management ------------------------------------- *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/define/immutable-infrastructure/containers) Create immutable containers Immutable containers package your application into an unchangeable unit that you build once and deploy one or more times. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/automate/packaging) Package applications Application packaging creates deployable artifacts that ensure consistency across environments and simplify deployment processes. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/define/centralize-packages) Centralize packages Centralize software artifacts and dependencies to streamline the build process and enhance security and governance. CI/CD automation best practices ------------------------------- *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/automate/cicd) Implement CI/CD Continuous integration and deployment pipelines ensure that delivery is consistent and reliable while reducing manual errors and deployment time. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/automate/testing) Automate testing Automated testing validates that applications and infrastructure work correctly before deployment, reducing the risk of failures in production. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/automate/deployments) Automate deployments Automated deployments eliminate manual errors to provide predictable and repeatable processes for both applications and infrastructure. *  [](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/deploy/zero-downtime-deployments) Zero downtime deployments Zero downtime deployment strategies are designed to reduce or eliminate downtime when you update your infrastructure or applications. --- # Tutorials | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Overview ======== Get Started ----------- Experience Nomad—from the CLI to the UI. *  [](https://developer.hashicorp.com/nomad/tutorials/get-started) 5 tutorials Get Started Get up and running with Nomad by learning about scheduling, setting up a cluster, and deploying an example job. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup) 4 tutorials Cluster Setup Provision a Nomad cluster in the Cloud with Consul and Access Control Lists (ACLs) enabled. * Nomad Take Nomad to Production ------------------------ Build a production quality cluster. *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise) 5 tutorials Nomad Enterprise Learn about Nomad Enterprise features and how to use them at scale. * Nomad Run Nomad on Edge Workloads --------------------------- *  [](https://developer.hashicorp.com/nomad/tutorials/edge) 1 tutorial Orchestrate Edge Services with Nomad Use Nomad to schedule edge workloads closer to your users. Connect edge services with Nomad's native service discovery. Seamlessly handle unstable Nomad client node connections. * Nomad Featured Tutorials ------------------ *  [](https://developer.hashicorp.com/nomad/tutorials/variables) 2 tutorials Nomad Variables Store encrypted configuration data in Nomad and make it accessible to tasks. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/service-discovery) 2 tutorials Service Discovery on Nomad Learn how to use and implement Nomad's Native Service Discovery. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing) 6 tutorials Load Balancer Integrations Integrate your Nomad cluster with several popular load balancers and leverage application load balancing for external traffic. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/fed-workload-identity) 1 tutorial Federated workload identity Integrate your Nomad cluster with cloud providers or any service that supports JWT/OIDC federated identities. * Nomad Integrate with HashiCorp Tools ------------------------------ Use Consul and Vault with Nomad. *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) 4 tutorials Use Nomad's Consul Integration Use Nomad’s Consul integration to secure Nomad jobs and learn more about Consul health checking, service discovery, and service mesh capabilities. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault) 1 tutorial Integrate Nomad with Vault Configure Nomad to obtain secrets from Vault for Nomad workloads. Learn how to configure the Nomad secrets engine in Vault. * Nomad ![](https://cdn.usefathom.com/?h=https%3A%2F%2Fdeveloper.hashicorp.com&p=%2Fnomad%2Ftutorials&r=&sid=ZFTCLDIZ&qs=%7B%7D&cid=35477890) --- # HTTP API | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/api-docs#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) HTTP API ======== Nomad exposes a RESTful HTTP API to control almost every aspect of the Nomad agent. The main interface to Nomad is a RESTful HTTP API. The API can query the current state of the system as well as modify the state of the system. The Nomad CLI actually invokes Nomad's HTTP for many commands. [](https://developer.hashicorp.com/nomad/api-docs#version-prefix) Version Prefix -------------------------------------------------------------------------------- All API routes are prefixed with `/v1/`. [](https://developer.hashicorp.com/nomad/api-docs#addressing-and-ports) Addressing and Ports -------------------------------------------------------------------------------------------- Nomad binds to a specific set of addresses and ports. The HTTP API is served via the `http` address and port. This `address:port` must be accessible locally. If you bind to `127.0.0.1:4646`, the API is only available _from that host_. If you bind to a private internal IP, the API will be available from within that network. If you bind to a public IP, the API will be available from the public Internet (not recommended). The default port for the Nomad HTTP API is `4646`. This can be overridden via the Nomad configuration block. Here is an example curl request to query a Nomad server with the default configuration: $ curl http://127.0.0.1:4646/v1/agent/members $ curl http://127.0.0.1:4646/v1/agent/members The conventions used in the API documentation do not list a port and use the standard URL `localhost:4646`. Be sure to replace this with your Nomad agent URL when using the examples. [](https://developer.hashicorp.com/nomad/api-docs#data-model-and-layout) Data Model and Layout ---------------------------------------------------------------------------------------------- There are five primary nouns in Nomad: * jobs * nodes * allocations * deployments * evaluations [![Nomad Data Model](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/nomad-data-model.png)](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/nomad-data-model.png) Jobs are submitted by users and represent a _desired state_. A job is a declarative description of tasks to run which are bounded by constraints and require resources. Jobs can also have affinities which are used to express placement preferences. Nodes are the servers in the clusters that tasks can be scheduled on. The mapping of tasks in a job to nodes is done using allocations. An allocation is used to declare that a set of tasks in a job should be run on a particular node. Scheduling is the process of determining the appropriate allocations and is done as part of an evaluation. Deployments are objects to track a rolling update of allocations between two versions of a job. The API is modeled closely on the underlying data model. Use the links to the left for documentation about specific endpoints. There are also "Agent" APIs which interact with a specific agent and not the broader cluster used for administration. [](https://developer.hashicorp.com/nomad/api-docs#acls) ACLs ------------------------------------------------------------ Several endpoints in Nomad use or require ACL tokens to operate. The token are used to authenticate the request and determine if the request is allowed based on the associated authorizations. Tokens are specified per-request by using the `X-Nomad-Token` request header or with the Bearer scheme in the authorization header set to the `SecretID` of an ACL Token. For more details about ACLs, please see the [ACL Guide](https://developer.hashicorp.com/nomad/docs/secure/acl) . [](https://developer.hashicorp.com/nomad/api-docs#authentication) Authentication -------------------------------------------------------------------------------- When ACLs are enabled, a Nomad token should be provided to API requests using the `X-Nomad-Token` header or with the Bearer scheme in the authorization header. When using authentication, clients should communicate via TLS. Here is an example using curl with `X-Nomad-Token`: $ curl \\ --header "X-Nomad-Token: aa534e09-6a07-0a45-2295-a7f77063d429" \\ https://localhost:4646/v1/jobs $ curl \ --header "X-Nomad-Token: aa534e09-6a07-0a45-2295-a7f77063d429" \ https://localhost:4646/v1/jobs Below is an example using `curl` with a [RFC6750](https://tools.ietf.org/html/rfc6750) Bearer token: $ curl \\ --header "Authorization: Bearer " \\ http://localhost:4646/v1/jobs $ curl \ --header "Authorization: Bearer " \ http://localhost:4646/v1/jobs [](https://developer.hashicorp.com/nomad/api-docs#namespaces) Namespaces ------------------------------------------------------------------------ Nomad has support for namespaces, which allow jobs and their associated objects to be segmented from each other and other users of the cluster. When using non-default namespace, the API request must pass the target namespace as an API query parameter. Here is an example using curl to query the `qa` namespace: $ curl 'localhost:4646/v1/jobs?namespace=qa' $ curl 'localhost:4646/v1/jobs?namespace=qa' Use a wildcard (`*`) to query all namespaces: $ curl 'localhost:4646/v1/jobs?namespace=\*' $ curl 'localhost:4646/v1/jobs?namespace=*' [](https://developer.hashicorp.com/nomad/api-docs#filtering) Filtering ---------------------------------------------------------------------- Filter expressions refine data queries for some API listing endpoints, as notated in the individual API endpoints documentation. To create a filter expression, you will write one or more expressions. Each expression has matching operators composed of selectors and values. Filtering is executed on the Nomad server, before data is returned, reducing the network load. To pass a filter expression to Nomad, use the `filter` query parameter with the URL encoded expression when sending requests to HTTP API endpoints that support it. $ curl --get https://localhost:4646/v1/ \\ --data-urlencode 'filter=' $ curl --get https://localhost:4646/v1/ \ --data-urlencode 'filter=' The filter expression can also be specified in the [`-filter`](https://developer.hashicorp.com/nomad/commands/operator/api#filter) flag of the [`nomad operator api`](https://developer.hashicorp.com/nomad/commands/operator/api) command. $ nomad operator api -filter '' /v1/ $ nomad operator api -filter '' /v1/ Some endpoints may have other query parameters that are used for filtering, but they can't be used with the `filter` query parameter. Doing so will result in a `400` status error response. These query parameters are usually backed by a database index, so they may be prefereable over an equivalent simple `filter` expression due to better resource usage and performance. ### [](https://developer.hashicorp.com/nomad/api-docs#list-stubs) List Stubs Some list endpoints return a reduced version of the resource being queried. This smaller version is called a _stub_ and may have different fields than the full resource definition. To allow more expressive filtering operations, the filter is applied to the full version, not the stub. If a request returns an error such as `error finding value in datum` the field used in filter expression may need to be adjusted. For example, filtering on node addresses should use the `HTTPAddr` field of the full node definition instead of `Address` field present in the stub. $ nomad operator api -filter 'HTTPAddr matches "10.0.0..+"' /v1/nodes $ nomad operator api -filter 'HTTPAddr matches "10.0.0..+"' /v1/nodes ### [](https://developer.hashicorp.com/nomad/api-docs#creating-expressions) Creating Expressions A single expression is a matching operator with a selector and value and they are written in plain text format. Boolean logic and parenthesization are supported. In general, whitespace is ignored, except within literal strings. #### [](https://developer.hashicorp.com/nomad/api-docs#matching-operators) Matching Operators All matching operators use a selector or value to choose what data should be matched. Each endpoint that supports filtering accepts a potentially different list of selectors and is detailed in the API documentation for those endpoints. // Equality & Inequality checks == "" != "" // Emptiness checks is empty is not empty // Contains checks or Substring Matching "" in "" not in contains "" not contains "" // Regular Expression Matching matches "" not matches "" // Equality & Inequality checks == "" != "" // Emptiness checks is empty is not empty // Contains checks or Substring Matching "" in "" not in contains "" not contains "" // Regular Expression Matching matches "" not matches "" #### [](https://developer.hashicorp.com/nomad/api-docs#selectors) Selectors Selectors are used by matching operators to create an expression. They are defined by a `.` separated list of names. Each name must start with an ASCII letter and can contain ASCII letters, numbers, and underscores. When part of the selector references a map value it may be expressed using the form `[""]` instead of `.`. This allows the possibility of using map keys that are not valid selectors in and of themselves. // selects the \`cache\` key within the \`TaskGroups\` mapping for the // /v1/deployments endpoint TaskGroups.cache // Also selects the \`cache\` key for the same endpoint TaskGroups\["cache"\] // selects the `cache` key within the `TaskGroups` mapping for the // /v1/deployments endpoint TaskGroups.cache // Also selects the `cache` key for the same endpoint TaskGroups["cache"] #### [](https://developer.hashicorp.com/nomad/api-docs#values) Values Values are used by matching operators to create an expression. Values can be any valid selector, a number, or a string. It is best practice to quote values. Numbers can be base 10 integers or floating point numbers. When quoting strings, they may either be enclosed in double quotes or backticks. When enclosed in backticks they are treated as raw strings and escape sequences such as `\n` will not be expanded. ### [](https://developer.hashicorp.com/nomad/api-docs#connecting-expressions) Connecting Expressions There are several methods for connecting expressions, including: * logical `or` * logical `and` * logical `not` * grouping with parenthesis * matching expressions // Logical Or - evaluates to true if either sub-expression does or // Logical And - evaluates to true if both sub-expressions do and // Logical Not - evaluates to true if the sub-expression does not not // Grouping - Overrides normal precedence rules ( ) // Inspects data to check for a match // Logical Or - evaluates to true if either sub-expression does or // Logical And - evaluates to true if both sub-expressions do and // Logical Not - evaluates to true if the sub-expression does not not // Grouping - Overrides normal precedence rules ( ) // Inspects data to check for a match Standard operator precedence can be expected for the various forms. For example, the following two expressions would be equivalent. and not or ( and (not )) or and not or ( and (not )) or ### [](https://developer.hashicorp.com/nomad/api-docs#filter-utilization) Filter Utilization Generally, only the main object is filtered. When filtering for an item within an array that is not at the top level, the entire array that contains the item will be returned. This is usually the outermost object of a response, but in some cases the filtering is performed on a object embedded within the results. #### [](https://developer.hashicorp.com/nomad/api-docs#performance) Performance Filters are executed on the servers and therefore will consume some amount of CPU time on the server. For non-stale queries this means that the filter is executed on the leader. #### [](https://developer.hashicorp.com/nomad/api-docs#filtering-examples) Filtering Examples ##### [](https://developer.hashicorp.com/nomad/api-docs#jobs-api) Jobs API Command (Unfiltered) $ nomad operator api /v1/jobs $ nomad operator api /v1/jobs Response (Unfiltered) \[\ {\ "CreateIndex": 52,\ "Datacenters": \[\ "dc1",\ "dc2"\ \],\ "ID": "countdash",\ "JobModifyIndex": 56,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 52,\ "JobID": "countdash",\ "ModifyIndex": 55,\ "Namespace": "default",\ "Summary": {\ "api": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ },\ "dashboard": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 56,\ "Multiregion": null,\ "Name": "countdash",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "pending",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230445788556000,\ "Type": "service"\ },\ {\ "CreateIndex": 42,\ "Datacenters": \[\ "dc1"\ \],\ "ID": "example",\ "JobModifyIndex": 42,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 42,\ "JobID": "example",\ "ModifyIndex": 46,\ "Namespace": "default",\ "Summary": {\ "cache": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 0,\ "Running": 1,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 49,\ "Multiregion": null,\ "Name": "example",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "running",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230403921889000,\ "Type": "service"\ }\ \] [\ {\ "CreateIndex": 52,\ "Datacenters": [\ "dc1",\ "dc2"\ ],\ "ID": "countdash",\ "JobModifyIndex": 56,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 52,\ "JobID": "countdash",\ "ModifyIndex": 55,\ "Namespace": "default",\ "Summary": {\ "api": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ },\ "dashboard": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 56,\ "Multiregion": null,\ "Name": "countdash",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "pending",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230445788556000,\ "Type": "service"\ },\ {\ "CreateIndex": 42,\ "Datacenters": [\ "dc1"\ ],\ "ID": "example",\ "JobModifyIndex": 42,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 42,\ "JobID": "example",\ "ModifyIndex": 46,\ "Namespace": "default",\ "Summary": {\ "cache": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 0,\ "Running": 1,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 49,\ "Multiregion": null,\ "Name": "example",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "running",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230403921889000,\ "Type": "service"\ }\ ] Command (Filtered) $ nomad operator api -filter 'Datacenters contains "dc2"' /v1/jobs $ nomad operator api -filter 'Datacenters contains "dc2"' /v1/jobs Response (Filtered) \[\ {\ "CreateIndex": 52,\ "Datacenters": \[\ "dc1",\ "dc2"\ \],\ "ID": "countdash",\ "JobModifyIndex": 56,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 52,\ "JobID": "countdash",\ "ModifyIndex": 55,\ "Namespace": "default",\ "Summary": {\ "api": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ },\ "dashboard": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 56,\ "Multiregion": null,\ "Name": "countdash",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "pending",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230445788556000,\ "Type": "service"\ }\ \] [\ {\ "CreateIndex": 52,\ "Datacenters": [\ "dc1",\ "dc2"\ ],\ "ID": "countdash",\ "JobModifyIndex": 56,\ "JobSummary": {\ "Children": {\ "Dead": 0,\ "Pending": 0,\ "Running": 0\ },\ "CreateIndex": 52,\ "JobID": "countdash",\ "ModifyIndex": 55,\ "Namespace": "default",\ "Summary": {\ "api": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ },\ "dashboard": {\ "Complete": 0,\ "Failed": 0,\ "Lost": 0,\ "Queued": 1,\ "Running": 0,\ "Starting": 0\ }\ }\ },\ "ModifyIndex": 56,\ "Multiregion": null,\ "Name": "countdash",\ "Namespace": "default",\ "ParameterizedJob": false,\ "ParentID": "",\ "Periodic": false,\ "Priority": 50,\ "Status": "pending",\ "StatusDescription": "",\ "Stop": false,\ "SubmitTime": 1645230445788556000,\ "Type": "service"\ }\ ] ##### [](https://developer.hashicorp.com/nomad/api-docs#deployments-api) Deployments API Command (Unfiltered) $ nomad operator api /v1/deployments $ nomad operator api /v1/deployments Response (Unfiltered) \[\ {\ "CreateIndex": 54,\ "EvalPriority": 50,\ "ID": "58fd0616-ce64-d14b-6917-03d0ab5af67e",\ "IsMultiregion": false,\ "JobCreateIndex": 52,\ "JobID": "countdash",\ "JobModifyIndex": 52,\ "JobSpecModifyIndex": 52,\ "JobVersion": 0,\ "ModifyIndex": 59,\ "Namespace": "default",\ "Status": "cancelled",\ "StatusDescription": "Cancelled due to newer version of job",\ "TaskGroups": {\ "dashboard": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ },\ "api": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ }\ }\ },\ {\ "CreateIndex": 43,\ "EvalPriority": 50,\ "ID": "1f18b48c-b33b-8e96-5640-71e3f3000242",\ "IsMultiregion": false,\ "JobCreateIndex": 42,\ "JobID": "example",\ "JobModifyIndex": 42,\ "JobSpecModifyIndex": 42,\ "JobVersion": 0,\ "ModifyIndex": 49,\ "Namespace": "default",\ "Status": "successful",\ "StatusDescription": "Deployment completed successfully",\ "TaskGroups": {\ "cache": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 1,\ "PlacedAllocs": 1,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": "2022-02-18T19:36:54.421823-05:00",\ "UnhealthyAllocs": 0\ }\ }\ }\ \] [\ {\ "CreateIndex": 54,\ "EvalPriority": 50,\ "ID": "58fd0616-ce64-d14b-6917-03d0ab5af67e",\ "IsMultiregion": false,\ "JobCreateIndex": 52,\ "JobID": "countdash",\ "JobModifyIndex": 52,\ "JobSpecModifyIndex": 52,\ "JobVersion": 0,\ "ModifyIndex": 59,\ "Namespace": "default",\ "Status": "cancelled",\ "StatusDescription": "Cancelled due to newer version of job",\ "TaskGroups": {\ "dashboard": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ },\ "api": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ }\ }\ },\ {\ "CreateIndex": 43,\ "EvalPriority": 50,\ "ID": "1f18b48c-b33b-8e96-5640-71e3f3000242",\ "IsMultiregion": false,\ "JobCreateIndex": 42,\ "JobID": "example",\ "JobModifyIndex": 42,\ "JobSpecModifyIndex": 42,\ "JobVersion": 0,\ "ModifyIndex": 49,\ "Namespace": "default",\ "Status": "successful",\ "StatusDescription": "Deployment completed successfully",\ "TaskGroups": {\ "cache": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 1,\ "PlacedAllocs": 1,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": "2022-02-18T19:36:54.421823-05:00",\ "UnhealthyAllocs": 0\ }\ }\ }\ ] Command (Filtered) $ nomad operator api -filter 'Status != "successful"' /v1/deployments $ nomad operator api -filter 'Status != "successful"' /v1/deployments Response (Filtered) \[\ {\ "CreateIndex": 54,\ "EvalPriority": 50,\ "ID": "58fd0616-ce64-d14b-6917-03d0ab5af67e",\ "IsMultiregion": false,\ "JobCreateIndex": 52,\ "JobID": "countdash",\ "JobModifyIndex": 52,\ "JobSpecModifyIndex": 52,\ "JobVersion": 0,\ "ModifyIndex": 59,\ "Namespace": "default",\ "Status": "cancelled",\ "StatusDescription": "Cancelled due to newer version of job",\ "TaskGroups": {\ "dashboard": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ },\ "api": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ }\ }\ }\ \] [\ {\ "CreateIndex": 54,\ "EvalPriority": 50,\ "ID": "58fd0616-ce64-d14b-6917-03d0ab5af67e",\ "IsMultiregion": false,\ "JobCreateIndex": 52,\ "JobID": "countdash",\ "JobModifyIndex": 52,\ "JobSpecModifyIndex": 52,\ "JobVersion": 0,\ "ModifyIndex": 59,\ "Namespace": "default",\ "Status": "cancelled",\ "StatusDescription": "Cancelled due to newer version of job",\ "TaskGroups": {\ "dashboard": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ },\ "api": {\ "AutoPromote": false,\ "AutoRevert": false,\ "DesiredCanaries": 0,\ "DesiredTotal": 1,\ "HealthyAllocs": 0,\ "PlacedAllocs": 0,\ "PlacedCanaries": null,\ "ProgressDeadline": 600000000000,\ "Promoted": false,\ "RequireProgressBy": null,\ "UnhealthyAllocs": 0\ }\ }\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs#pagination) Pagination ------------------------------------------------------------------------ Some list endpoints support partial results to limit the amount of data retrieved. The returned list is split into pages and the page size can be set using the `per_page` query parameter with a positive integer value. If more data is available past the page requested, the response will contain an HTTP header named `X-Nomad-Nexttoken` with the value of the next item to be retrieved. This value can then be set as a query parameter called `next_token` in a follow-up request to retrieve the next page. When the last page is reached, the `X-Nomad-Nexttoken` HTTP header will not be present in the response, indicating that there is nothing more to return. [](https://developer.hashicorp.com/nomad/api-docs#ordering) Ordering -------------------------------------------------------------------- List results are usually returned in ascending order by their internal key, such as their `ID`. Some endpoints may return data sorted by their `CreateIndex` value, which roughly corelates to their creation order. The result order may be reversed using the `reverse=true` query parameter when supported by the endpoint. [](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) Blocking Queries ------------------------------------------------------------------------------------ Many endpoints in Nomad support a feature known as "blocking queries". A blocking query is used to wait for a potential change using long polling. Not all endpoints support blocking, but each endpoint uniquely documents its support for blocking queries in the documentation. Endpoints that support blocking queries return an HTTP header named `X-Nomad-Index`. This is a unique identifier representing the current state of the requested resource. On a new Nomad cluster the value of this index starts at 1. On subsequent requests for this resource, the client can set the `index` query string parameter to the value of `X-Nomad-Index`, indicating that the client wishes to wait for any changes subsequent to that index. When this is provided, the HTTP request will "hang" until a change in the system occurs, or the maximum timeout is reached. A critical note is that the return of a blocking request is **no guarantee** of a change. It is possible that the timeout was reached or that there was an idempotent write that does not affect the result of the query. In addition to `index`, endpoints that support blocking will also honor a `wait` parameter specifying a maximum duration for the blocking request. This is limited to 10 minutes. If not set, the wait time defaults to 5 minutes. This value can be specified in the form of "10s" or "5m" (i.e., 10 seconds or 5 minutes, respectively). A small random amount of additional wait time is added to the supplied maximum `wait` time to spread out the wake up time of any concurrent requests. This adds up to `wait / 16` additional time to the maximum duration. [](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) Consistency Modes -------------------------------------------------------------------------------------- Most of the read query endpoints support multiple levels of consistency. Since no policy will suit all clients' needs, these consistency modes allow the user to have the ultimate say in how to balance the trade-offs inherent in a distributed system. The two read modes are: * [](https://developer.hashicorp.com/nomad/api-docs#) [`default`](https://developer.hashicorp.com/nomad/api-docs#default) - If not specified, the default is strongly consistent in almost all cases. However, there is a small window in which a new leader may be elected during which the old leader may service stale values. The trade-off is fast reads but potentially stale values. The condition resulting in stale reads is hard to trigger, and most clients should not need to worry about this case. Also, note that this race condition only applies to reads, not writes. * [](https://developer.hashicorp.com/nomad/api-docs#) [`stale`](https://developer.hashicorp.com/nomad/api-docs#stale) - This mode allows any server to service the read regardless of whether it is the leader. This means reads can be arbitrarily stale; however, results are generally consistent to within 50 milliseconds of the leader. The trade-off is very fast and scalable reads with a higher likelihood of stale values. Since this mode allows reads without a leader, a cluster that is unavailable will still be able to respond to queries. To switch these modes, use the `stale` query parameter on requests. To support bounding the acceptable staleness of data, responses provide the `X-Nomad-LastContact` header containing the time in milliseconds that a server was last contacted by the leader node. The `X-Nomad-KnownLeader` header also indicates if there is a known leader. These can be used by clients to gauge the staleness of a result and take appropriate action. [](https://developer.hashicorp.com/nomad/api-docs#cross-region-requests) Cross-Region Requests ---------------------------------------------------------------------------------------------- By default, any request to the HTTP API will default to the region on which the machine is servicing the request. If the agent runs in "region1", the request will query the region "region1". A target region can be explicitly request using the `?region` query parameter. The request will be transparently forwarded and serviced by a server in the requested region. [](https://developer.hashicorp.com/nomad/api-docs#compressed-responses) Compressed Responses -------------------------------------------------------------------------------------------- The HTTP API will gzip the response if the HTTP request denotes that the client accepts gzip compression. This is achieved by passing the accept encoding: $ curl \\ --header "Accept-Encoding: gzip" \\ https://localhost:4646/v1/... $ curl \ --header "Accept-Encoding: gzip" \ https://localhost:4646/v1/... [](https://developer.hashicorp.com/nomad/api-docs#formatted-json-output) Formatted JSON Output ---------------------------------------------------------------------------------------------- By default, the output of all HTTP API requests is minimized JSON. If the client passes `pretty` on the query string, formatted JSON will be returned. In general, clients should prefer a client-side parser like `jq` instead of server-formatted data. Asking the server to format the data takes away processing cycles from more important tasks. $ curl https://localhost:4646/v1/page?pretty $ curl https://localhost:4646/v1/page?pretty [](https://developer.hashicorp.com/nomad/api-docs#http-methods) HTTP Methods ---------------------------------------------------------------------------- Nomad's API aims to be RESTful, although there are some exceptions. The API responds to the standard HTTP verbs GET, PUT, and DELETE. Each API method will clearly document the verb(s) it responds to and the generated response. The same path with different verbs may trigger different behavior. For example: PUT /v1/jobs GET /v1/jobs PUT /v1/jobs GET /v1/jobs Even though these share a path, the `PUT` operation creates a new job whereas the `GET` operation reads all jobs. [](https://developer.hashicorp.com/nomad/api-docs#http-response-codes) HTTP Response Codes ------------------------------------------------------------------------------------------ Individual API's will contain further documentation in the case that more specific response codes are returned but all clients should handle the following: * 200 and 204 as success codes. * 400 indicates a validation failure and if a parameter is modified in the request, it could potentially succeed. * 403 marks that the client isn't authenticated for the request. * 404 indicates an unknown resource. * 5xx means that the client should not expect the request to succeed if retried. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Getting Started --------------- Learn how to use Nomad to schedule and orchestrate workloads. [CLI Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) Use Cases --------- ### Fundamentals Become familiar with the core concepts of Nomad. *  [](https://developer.hashicorp.com/nomad/docs/deploy) Installing Nomad Nomad is available as a pre-compiled binary, a package for several OSs, or as source code for you to build from. *  [](https://developer.hashicorp.com/nomad/docs/job-specification) Writing Job Specs The Nomad job specification or 'jobspec' is written in HCL and defines the schema for Nomad jobs. *  [](https://developer.hashicorp.com/nomad/docs/configuration) Agent Configuration The Nomad agent specification is written in HCL and defines configs like networking, plugins, and integrations. *  [](https://developer.hashicorp.com/nomad/docs/job-run) Manage Nomad Jobs Learn how to deploy and manage jobs. ### Nomad Pack Use Nomad Pack to deploy popular applications to Nomad. *  [](https://developer.hashicorp.com/nomad/tools/nomad-pack) Introduction to Nomad Pack Learn about Nomad Pack and how to use it to easily create, share, deploy, and re-use Nomad job specs. *  [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs) Write a custom Nomad Pack Learn how to create your own custom Nomad Pack. *  [](https://github.com/hashicorp/nomad-pack-community-registry) Community Repository The registry of community-maintained packs for Nomad Pack. ### Autoscaling Automatically maintain your cluster and workload instance count to respond to demand while minimizing over-provisioning cost. *  [](https://developer.hashicorp.com/nomad/tools/autoscaling) Introduction to Autoscaling The Nomad Autoscaler is a horizontal application and cluster autoscaler for Nomad. *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling) Horizontal Cluster Autoscaling Learn how to use the autoscaler to dynamically scale infrastructure up and down and handle application load spikes. *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling-on-demand-batch) Create On-demand Batch Jobs Learn how to use the autoscaler to automatically provision and decommission clients for running batch jobs. *  [](https://developer.hashicorp.com/nomad/docs/job-scheduling/affinity) Manage Job Placement and Affinities Learn how to use the affinity stanza to express job placement preferences. ### Cluster Management Learn the features operators will need to build and maintain Nomad clusters. *  [](https://developer.hashicorp.com/nomad/docs/upgrade) Upgrade a Cluster Learn about the process of upgrading a cluster including upgrading in place. *  [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions) Enable Multi-Region Federation Learn how to use federation to allow users to submit jobs or interact with the API from any region attached to the cluster. *  [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes) Connect Nodes into a Cluster Learn how to connect nodes to a cluster manually, with Cloud Auto-Join, or with Consul. *  [](https://developer.hashicorp.com/nomad/docs/manage/autopilot) Enable Autopilot Learn about the Autopilot features and how to use them to cleanup servers, monitor Raft state, and introduce servers stably. Developers ---------- *  [](https://github.com/hashicorp/nomad/tree/main/terraform) Provision Nomad with Terraform *  [](https://developer.hashicorp.com/nomad/api-docs) API Reference *  [](https://developer.hashicorp.com/nomad/tools) Nomad Ecosystem Tools *  [](https://developer.hashicorp.com/nomad/plugins/drivers/community) Community Task Drivers ![](https://cdn.usefathom.com/?h=https%3A%2F%2Fdeveloper.hashicorp.com&p=%2Fnomad%2Fdocs&r=&sid=ZFTCLDIZ&qs=%7B%7D&cid=89990650) --- # Tools | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tools#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Tools ===== Learn about tools for Nomad. External tools provide additional capabilities and use cases to Nomad. They are installed separately from Nomad itself and can sometimes be deployed as a Nomad job in your cluster. [](https://developer.hashicorp.com/nomad/tools#hashicorp-maintained-tools) HashiCorp-Maintained Tools ----------------------------------------------------------------------------------------------------- The following external tools are currently available for Nomad and maintained by HashiCorp: * [Autoscaling](https://developer.hashicorp.com/nomad/tools/autoscaling) - HashiCorp's official Nomad Autoscaler. Supports scaling allocations within Nomad and scaling nodes on AWS, Azure, GCP, or arbitrary infrastructure via plugins. * [Damon](https://github.com/hashicorp/damon) - A terminal dashboard for Nomad. * [Levant](https://github.com/hashicorp/levant) - A templating and deployment tool for HashiCorp Nomad jobs that provides realtime feedback and detailed failure messages upon deployment issues. * [Nomad Pack](https://github.com/hashicorp/nomad-pack) - An official package manager and templating tool for Nomad, currently a Tech Preview. * [Nomad Pack GitHub Action](https://github.com/marketplace/actions/setup-hashicorp-nomad-pack) - A GitHub Action for Nomad Pack. [](https://developer.hashicorp.com/nomad/tools#community-tools) Community Tools ------------------------------------------------------------------------------- The following external tools are currently available for Nomad and maintained by members of the Nomad Community: * [Caravan](https://github.com/bitrockteam/caravan/wiki) - Caravan is a tool to deploy and configure Nomad, Consul and Vault to AWS, Azure, or GCP, all with a single script. * [Chaotic](https://github.com/ngine-io/chaotic) - A Chaos Engineering tool to stop allocations, reboot or stop/start virtual machines in your cloud environment * [Deadman Check](https://github.com/sepulworld/deadman-check) - A monitoring companion for Nomad periodic jobs that alerts if periodic isn't running at the expected interval * [Hashi Up](https://github.com/jsiebens/hashi-up) - A utility to install Nomad on remote Linux hosts * [HashiBox](https://github.com/nunchistudio/hashibox) - Vagrant environment to simulate a highly-available cloud with Consul, Vault, and Nomad. * [Jenkins Nomad Cloud Plugin](https://github.com/jenkinsci/nomad-plugin) - A Jenkins plugin using Nomad to provision build workers * [Kreconciler](https://github.com/koyeb/kreconciler) - A library to build control-loops for things on Nomad (or other schedulers) * [Nelson](https://getnelson.io/) - A tool for automated, multi-region container deployment using Nomad * [Node Problem Detector](https://github.com/Roblox/nomad-node-problem-detector) - A tool used to detect problems on Nomad nodes based on user-defined health checks * [Nomad Admission Control Proxy](https://github.com/mxab/nacp) - A Nomad Proxy that mutates and validates job data on the fly via embedded OPA rules or remote webhooks. * [Nomadctx](https://github.com/mr-karan/nomctx) - A faster way to switch between clusters and namespaces in nomad * [Nomad deploy result action](https://github.com/let-sh/nomad-deploy-result-action) - A Github action to monitor deployment results * [Nomad events sink](https://github.com/mr-karan/nomad-events-sink) - An events collection agent which processes Nomad Events and dumps to external sink providers like HTTP * [Nomad External DNS](https://github.com/mr-karan/nomad-external-dns/) - Set external DNS records for Nomad services * [Nomad Firehose](https://github.com/seatgeek/nomad-firehose) - A tool to enable teams to quickly build logic around nomad task events without hooking into Nomad API * [Nomad Helper](https://github.com/seatgeek/nomad-helper) - A Nomad helper binary. Reevaluate jobs, force garbage collection, drain nodes, export/import count information * [Nomad MCP Server](https://github.com/kocierik/mcp-nomad) - An MCP server that exposes Nomad APIs to AI agents, enabling LLM-driven automation, job management, diagnostics, and cluster operations. * [Nomad Monitoring](https://github.com/mr-karan/nomad-monitoring) - Collection of jobspecs and Grafana dashboards for end to end monitoring of Nomad clusters * [Nomad Ops](https://nomad-ops.github.io/nomad-ops) - A simple operator for Nomad which reconciles the running jobs in comparison to git repos * [Nomad Pipeline](https://github.com/HyperBadger/nomad-pipeline) - A tool to make running pipeline-style workloads on Nomad * [Nomad Port Forward](https://github.com/Mongey/nomad-port-forward) - A tool for forwarding ports from a Nomad job to your local machine * [Nomad Toast](https://github.com/jrasell/nomad-toast) - A tool for receiving notifications based on HashiCorp Nomad events * [Nomad Tools](https://github.com/Kamilcuk/nomad-tools) - Set of tools and utilities to ease interacting with HashiCorp Nomad scheduling solution. * [Nomad Vector Logger](https://github.com/mr-karan/nomad-vector-logger) - A daemon which continuously watches jobs running in a Nomad cluster and templates out a Vector configuration file which can be used to collect application logs enriched with Nomad metadata. * [Nomad Watcher](https://github.com/blalor/nomad-watcher) - A simple service that watches Nomad's nodes, jobs, deployments, evaluations, allocations, and task states, and writes the events to a file * [Nomadgen](https://github.com/smintz/nomadgen) - Craft your Hashicorp's Nomad job specs in python. * [Nomadctld](https://github.com/42wim/nomadctld) - An ssh server that sits between users and the nomad cluster allowing for limited access to your nomad cluster and allow them to attach, see logs, tail logs, stop, restart, exec containers they own on the cluster * [Nomad-spk](https://github.com/numkem/nomad-spk) - A tool to install nomad into a Synology NAS * [Wander](https://github.com/robinovitch61/wander) - A terminal application for Nomad by HashiCorp [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/tools/index.mdx) --- # Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/commands#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad command-line interface (CLI) reference ============================================ The Nomad command-line interface (CLI) interacts with a Nomad cluster. This page contains reference information on how to install CLI autocomplete, configure remote usage, and specify CLI options. Review connection, access control list (ACL), mutual TLS (mTLS), and license environment variables. The Nomad CLI is a well-behaved command-line application. In erroneous cases, the CLI returns a non-zero exit status. To view a list of available commands, run the `nomad` command with no arguments, `-h`, or `--help`. To access help for any specific subcommand, run the subcommand with the `-h` argument. [](https://developer.hashicorp.com/nomad/commands#autocomplete) Autocomplete ---------------------------------------------------------------------------- Nomad's CLI supports command autocomplete. To install autocomplete, run the `nomad` command with the `-autocomplete-install` option. $ nomad -autocomplete-install $ nomad -autocomplete-install To uninstall autocomplete, run the `nomad` command with the `-autocomplete-uninstall` option. $ nomad -autocomplete-uninstall $ nomad -autocomplete-uninstall [](https://developer.hashicorp.com/nomad/commands#command-contexts) Command contexts ------------------------------------------------------------------------------------ Nomad's CLI commands have implied contexts in their naming convention. Because the CLI is most commonly used to manipulate or query jobs, you can assume that any given command is working in that context unless the command name implies otherwise. For example, the `nomad job run` command runs a new job and the `nomad status` command queries information about existing jobs. Conversely, commands with a prefix in their name likely operate in a different context. Examples include the `nomad agent-info` or `nomad node drain` commands, which operate in the agent or node contexts respectively. ### [](https://developer.hashicorp.com/nomad/commands#remote-usage) Remote usage You may use the Nomad CLI to interact with a remote Nomad cluster, even when the local machine does not have a running Nomad agent. To do so, set the `NOMAD_ADDR` environment variable. $ NOMAD\_ADDR=https://remote-address:4646 $ nomad status $ NOMAD_ADDR=https://remote-address:4646 $ nomad status Or use the `-address=` flag when running commands. $ nomad status -address=https://remote-address:4646 $ nomad status -address=https://remote-address:4646 The Nomad address must be reachable from your local machine. If the Nomad port is exposed to the public internet, we recommend configuring TLS. Refer to the [`tls` block in agent configuration](https://developer.hashicorp.com/nomad/docs/configuration/tls) for details. ### [](https://developer.hashicorp.com/nomad/commands#environment-variables) Environment variables Nomad can use environment variables to configure command-line tool options. You may override these environment variables with individual flags. Except where noted, these variables influence the behavior of the Nomad CLI and should not be set for Nomad agents. #### [](https://developer.hashicorp.com/nomad/commands#connection-environment-variables) Connection environment variables * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/commands#nomad_addr) - The address of the Nomad server. Defaults to `http://127.0.0.1:4646`. For unix sockets, as with the [task API](https://developer.hashicorp.com/nomad/api-docs/task-api) , the format is either `unix:/path/to/api.sock` or `unix:///path/to/api.sock`. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_REGION`](https://developer.hashicorp.com/nomad/commands#nomad_region) - The region of the Nomad server to forward commands to. Defaults to the Agent's local region * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_NAMESPACE`](https://developer.hashicorp.com/nomad/commands#nomad_namespace) - The target namespace for queries and actions bound to a namespace. If set to `*`, job and alloc subcommands query all namespacecs authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_HTTP_AUTH`](https://developer.hashicorp.com/nomad/commands#nomad_http_auth) - (Optional) This allows users to supply "Basic" HTTP authentication scheme ([RFC 7617](https://tools.ietf.org/html/rfc7617) ) information in environments where the Nomad API is behind an authenticating proxy server. #### [](https://developer.hashicorp.com/nomad/commands#access-control-list-acl-environment-variables) Access control list (ACL) environment variables * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/commands#nomad_token) - The SecretID of an ACL token to use to authenticate API requests with. #### [](https://developer.hashicorp.com/nomad/commands#cli-environment-variables) CLI environment variables * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CLI_NO_COLOR`](https://developer.hashicorp.com/nomad/commands#nomad_cli_no_color) - Disables colored command output. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CLI_SHOW_HINTS`](https://developer.hashicorp.com/nomad/commands#nomad_cli_show_hints) - Enables ui-hints in common CLI command output. #### [](https://developer.hashicorp.com/nomad/commands#mutual-tls-mtls-environment-variables) Mutual TLS (mTLS) environment variables * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CLIENT_CERT`](https://developer.hashicorp.com/nomad/commands#nomad_client_cert) - Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `NOMAD_CLIENT_KEY`. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CLIENT_KEY`](https://developer.hashicorp.com/nomad/commands#nomad_client_key) - Path to an unencrypted PEM encoded private key matching the client certificate from `NOMAD_CLIENT_CERT`. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CACERT`](https://developer.hashicorp.com/nomad/commands#nomad_cacert) - Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_CAPATH`](https://developer.hashicorp.com/nomad/commands#nomad_capath) - Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `NOMAD_CACERT` and `NOMAD_CAPATH` are specified, `NOMAD_CACERT` is used. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_SKIP_VERIFY`](https://developer.hashicorp.com/nomad/commands#nomad_skip_verify) - Do not verify TLS certificate. **This is highly not recommended.** * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_TLS_SERVER_NAME`](https://developer.hashicorp.com/nomad/commands#nomad_tls_server_name) - The server name to use as the SNI host when connecting via TLS. #### [](https://developer.hashicorp.com/nomad/commands#nomad-enterprise-license-environment-variables) Nomad Enterprise license environment variables These environment variables influence the Nomad Enterprise license configuration. These values are only used for Nomad Enterprise agents, not the Nomad CLI. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_LICENSE_PATH`](https://developer.hashicorp.com/nomad/commands#nomad_license_path) - An absolute path to a Nomad Enterprise license file, for example `/etc/nomad.d/license.hclic`. * [](https://developer.hashicorp.com/nomad/commands#) [`NOMAD_LICENSE`](https://developer.hashicorp.com/nomad/commands#nomad_license) - The Nomad Enterprise license file contents as a string. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/index.mdx) --- # Nomad plugins | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/plugins#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad plugins ============= Plugins extend Nomad's functionality. Plugins extend Nomad's functionality. Review configuration, installation, usage, and reference information for device and task driver plugins that are not bundled with Nomad. Consult the [Plugin authoring guide](https://developer.hashicorp.com/nomad/plugins/author) for details on how to create your own device, task driver, and host volume plugins. [](https://developer.hashicorp.com/nomad/plugins#configure-plugin-directories) Configure plugin directories ----------------------------------------------------------------------------------------------------------- You may configure the directories where you place plugin executable binaries. The following table lists the Agent configuration field as well as the default directory for each plugin type. Note that the default directories use the value of the configured [`data_dir`](https://developer.hashicorp.com/nomad/docs/configuration#data_dir) parameter, which specifies a local directory. | Type | Plugin directory configuration parameter | Default plugin directory | Plugin configuration | | --- | --- | --- | --- | | [Device driver](https://developer.hashicorp.com/nomad/plugins/devices) | [`plugin_dir`](https://developer.hashicorp.com/nomad/docs/configuration#plugin_dir) | `/plugins` | [`plugin` block](https://developer.hashicorp.com/nomad/docs/configuration/plugin) | | [Task driver](https://developer.hashicorp.com/nomad/plugins/drivers) | [`plugin_dir`](https://developer.hashicorp.com/nomad/docs/configuration#plugin_dir) | `/plugins` | [`plugin` block](https://developer.hashicorp.com/nomad/docs/configuration/plugin) | | [Dynamic host volume](https://developer.hashicorp.com/nomad/plugins/author/host-volume) | [`client.host_volume_plugin_dir`](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume_plugin_dir) | `/host_volume_plugins` | [`plugin` block](https://developer.hashicorp.com/nomad/docs/configuration/plugin) | | [Secret provider](https://developer.hashicorp.com/nomad/plugins/author/secret-provider) | [`client.common_plugin_dir`/secrets](https://developer.hashicorp.com/nomad/docs/configuration/client#common_plugin_dir) | `/common_plugins` | [`plugin` block](https://developer.hashicorp.com/nomad/docs/configuration/plugin) | | [CNI reference](https://www.cni.dev/plugins/current/) | [`client.cni_path`](https://developer.hashicorp.com/nomad/docs/configuration/client#cni_path) | `/opt/cni/bin` | [`client.cni_config_dir`](https://developer.hashicorp.com/nomad/docs/configuration/client#cni_config_dir) | [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/index.mdx) --- # Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Integrations ================== A curated collection of official, partner, and community Nomad Integrations. ![]() Tiers * Official * Community Flags * Archived * Enterprise Types * Task Drivers * Packs * Autoscaler Targets * Autoscaler Strategies * Autoscaler APM Plugins * Device Plugins 84 results found Filters No filters selected 84 results found *  [](https://developer.hashicorp.com/nomad/tools/autoscaling/plugins/target/aws-asg) ### AWS AutoScaling Group Target @hashicorp The aws-asg target plugin allows for the scaling of the Nomad cluster clients via manipulating AWS AutoScaling Groups. * Autoscaler Target * Official View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi) ### AWS EBS CSI v0.2.0 @hashicorp This pack deploys the AWS EBS CSI plugin * Pack * Community View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi) ### AWS EFS CSI v0.2.0 @hashicorp Configures a set of nodes to run the AWS EFS CSI volume plugin. * Pack * Community View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/alertmanager) ### Alertmanager v0.2.0 @hashicorp The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts. * Pack * Community View Details *  [](https://developer.hashicorp.com/nomad/tools/autoscaling/plugins/target/azure-vmss) ### Azure Virtual Machine Scale Set Target @hashicorp The azure-vmss target plugin allows for the scaling of the Nomad cluster clients via manipulating Azure Virtual Machine Scale Sets. * Autoscaler Target * Official View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage) ### Backstage v0.2.0 @hashicorp An open platform for building developer portals * Pack * Community View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/boundary) ### Boundary v0.2.0 @hashicorp Boundary is an intelligent proxy that creates granular, identity-based access controls for dynamic infrastructure. * Pack * Official View Details *  [](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder) ### CSI OpenStack Cinder v0.2.0 @hashicorp The Cinder CSI Driver is a CSI Specification compliant driver used by Container Orchestrators to manage the lifecycle of OpenStack Cinder Volumes. * Pack * Community View Details 1 - 8 of 84 Items per page481624 --- # Install | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/install#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Install Nomad ============= 1.11.1 (latest)1.11.01.10.51.10.41.10.31.10.21.10.11.10.01.9.71.9.61.9.51.9.41.9.31.9.11.9.01.8.41.8.31.8.21.8.11.8.01.7.71.7.61.7.51.7.41.7.31.7.21.7.11.7.01.6.101.6.91.6.81.6.71.6.61.6.51.6.41.6.31.6.21.6.11.6.01.5.171.5.161.5.151.5.141.5.131.5.121.5.111.5.101.5.91.5.81.5.71.5.61.5.51.5.41.5.31.5.21.5.11.5.01.4.141.4.131.4.121.4.111.4.101.4.91.4.81.4.71.4.61.4.51.4.41.4.31.4.21.4.11.4.01.3.161.3.151.3.141.3.131.3.121.3.111.3.101.3.91.3.81.3.71.3.61.3.51.3.41.3.31.3.21.3.11.3.01.2.161.2.151.2.141.2.131.2.121.2.111.2.101.2.91.2.81.2.71.2.61.2.51.2.41.2.31.2.21.2.11.2.01.1.181.1.171.1.161.1.151.1.141.1.131.1.121.1.111.1.101.1.91.1.81.1.71.1.61.1.51.1.41.1.31.1.21.1.11.1.01.0.181.0.171.0.161.0.151.0.141.0.131.0.121.0.111.0.101.0.91.0.81.0.71.0.61.0.51.0.41.0.31.0.21.0.11.0.00.12.120.12.110.12.100.12.90.12.80.12.70.12.60.12.50.12.40.12.30.12.20.12.10.12.00.11.80.11.70.11.60.11.50.11.40.11.30.11.20.11.10.11.00.10.90.10.80.10.70.10.60.10.50.10.40.10.30.10.20.10.10.10.00.9.70.9.60.9.50.9.40.9.30.9.20.9.10.9.00.8.70.8.60.8.50.8.40.8.30.8.20.8.10.8.00.7.10.7.00.6.30.6.20.6.10.6.00.5.60.5.50.5.40.5.30.5.20.5.10.5.00.4.10.4.00.3.20.3.10.3.00.2.30.2.20.2.10.2.00.1.20.1.10.1.0 macOS ----- [](https://developer.hashicorp.com/nomad/install#darwin) ### Package manager brew tap hashicorp/tap brew install hashicorp/tap/nomad brew tap hashicorp/tap brew install hashicorp/tap/nomad ### Binary download AMD64 Version: 1.11.1 [Download](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_darwin_amd64.zip) ARM64 Version: 1.11.1 [Download](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_darwin_arm64.zip) Windows ------- [](https://developer.hashicorp.com/nomad/install#windows) ### Binary download AMD64 Version: 1.11.1 [Download](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_windows_amd64.zip) Linux ----- [](https://developer.hashicorp.com/nomad/install#linux) ### Package manager Ubuntu/DebianCentOS/RHELFedora 41Fedora 42Amazon LinuxHomebrew wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg echo "deb \[arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg\] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU\_CODENAME=).\*' /etc/os-release || lsb\_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list sudo apt update && sudo apt install nomad wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list sudo apt update && sudo apt install nomad sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo sudo yum -y install nomad sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo sudo yum -y install nomad sudo dnf install -y dnf-plugins-core sudo dnf config-manager addrepo --from-repofile\=https://rpm.releases.hashicorp.com/fedora/hashicorp.repo sudo dnf -y install nomad sudo dnf install -y dnf-plugins-core sudo dnf config-manager addrepo --from-repofile=https://rpm.releases.hashicorp.com/fedora/hashicorp.repo sudo dnf -y install nomad wget -O- https://rpm.releases.hashicorp.com/fedora/hashicorp.repo | sudo tee /etc/yum.repos.d/hashicorp.repo sudo yum list available | grep hashicorp sudo dnf -y install nomad wget -O- https://rpm.releases.hashicorp.com/fedora/hashicorp.repo | sudo tee /etc/yum.repos.d/hashicorp.repo sudo yum list available | grep hashicorp sudo dnf -y install nomad sudo yum install -y yum-utils shadow-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo sudo yum install nomad sudo yum install -y yum-utils shadow-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo sudo yum install nomad brew tap hashicorp/tap brew install hashicorp/tap/nomad brew tap hashicorp/tap brew install hashicorp/tap/nomad ### Binary download AMD64 Version: 1.11.1 [Download](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_linux_amd64.zip) ARM64 Version: 1.11.1 [Download](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_linux_arm64.zip) Note Complete this [tutorial](https://developer.hashicorp.com/well-architected-framework/operational-excellence/verify-hashicorp-binary) to learn how to install and verify HashiCorp tools on any Linux distribution, and create a custom Linux container with verified HashiCorp tools. Release information ------------------- [](https://developer.hashicorp.com/nomad/install#release-information) Changelog Nomad Version: 1.11.1 [GitHub](https://github.com/hashicorp/nomad/releases/tag/v1.11.1) (opens in new tab) Official releases All officially supported HashiCorp release channels and their security guarantees. [View all](https://www.hashicorp.com/official-release-channels) (opens in new tab) Note You can find the [SHA256 checksums for Nomad 1.11.1](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_SHA256SUMS) online and you can [verify the checksums signature file](https://releases.hashicorp.com/nomad/1.11.1/nomad_1.11.1_SHA256SUMS.sig) which has been signed using [HashiCorp's GPG key](https://www.hashicorp.com/security) . Complete this [tutorial](https://developer.hashicorp.com/well-architected-framework/operational-excellence/verify-hashicorp-binary) to learn how to install and verify HashiCorp tools on any Linux distribution. Next steps ---------- [](https://developer.hashicorp.com/nomad/install#next-steps) *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license) 12min Install a HashiCorp Enterprise license Add an Enterprise license to Vault, Consul, Nomad, or Boundary with environment variables, a license file, or a configuration value. * Consul * Nomad * Vault * Boundary *  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview) 6min Introduction to Nomad Discover what HashiCorp Nomad is, become familiar with important related terms, and understand how Nomad fits into the application development workflow. * Nomad * Video *  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws) 30min Set up a Nomad cluster on AWS Create a Nomad cluster with Consul on AWS and then enable ACLs. Use this cluster as a starting point for your Nomad workloads. * Nomad * Packer * Terraform *  [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services) 25min Schedule edge Services with native service discovery Schedule a demo application on the edge, use Nomad's native service discovery to connect edge services, and simulate unstable edge connections to learn how Nomad gracefully handles disconnected clients. * Nomad * Terraform * Packer *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh) 12min Secure Nomad jobs with Consul service mesh Configure Nomad for an ACL-enabled Consul cluster and run a sample job that uses Consul service mesh. * Consul * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview) 10min Migrate a monolith to microservices Migrate a monolithic application to microservices using Consul and Nomad. * Nomad * Consul About Nomad Orchestrate, deploy, and manage containers, binaries, and batch jobs in the cloud or on-prem. Featured docs * [Linux post-installation steps](https://developer.hashicorp.com/nomad/docs/deploy) * [Bootstrap the ACL system](https://developer.hashicorp.com/nomad/docs/secure/acl/bootstrap) * [Build Nomad clusters](https://developer.hashicorp.com/nomad/docs/deploy/clusters) * [Network Nomad](https://developer.hashicorp.com/nomad/docs/networking) * [Upgrade Nomad](https://developer.hashicorp.com/nomad/docs/upgrade) --- # Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/sandbox#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Interactive Sandboxes =========================== Experiment with HashiCorp products in a safe, pre-configured environment. ![](https://developer.hashicorp.com/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fnomad.997a20b8.png&w=3840&q=75) HashiCorp Sandboxes provide interactive environments where you can experiment with HashiCorp products without any installation or setup. They're perfect for: * Learning how products work in a real environment * Testing configurations and commands without affecting your systems * Exploring product features in a safe sandbox * Following along with tutorials and documentation Each sandbox comes pre-configured with everything you need to start using the product immediately. Just click on a sandbox below to launch it in your browser. Available Nomad sandboxes ------------------------- When you launch a sandbox, you'll be presented with a terminal interface where you can interact with the pre-configured environment. The sandbox runs in your browser and doesn't require any downloads or installations. Each sandbox session lasts for up to 1 hour, giving you plenty of time to experiment. Your work isn't saved between sessions, so be sure to copy any important configurations before your session ends. * Nomad Sandbox Experiment with Nomad. This sandbox contains a Nomad cluster with one server and one client node with TLS and ACLs enabled. [Launch Sandbox](https://developer.hashicorp.com/nomad/sandbox#) Sandbox documentation --------------------- Nomad Sandbox OverviewGet startedSample jobsArchitectureConfigure and restart agentsLimitations Warning This sandbox is not a production-ready environment. It is a development and testing environment with security configurations for gossip encryption, TLS, and ACLs enabled. For more information about deploying Nomad in production, refer to [Installing Nomad for Production](https://developer.hashicorp.com/nomad/docs/install/production) . This sandbox provides a Nomad cluster that consists of one server node and one client node. Each node runs a Nomad agent and a Consul agent. Both agents are configured with ACLs, gossip encryption, and TLS encryption. The server node also runs a single-node Vault cluster that has been initialized and unsealed. The client node has access to a scoped token for reading secrets from Vault. You can also read Vault secrets from a Nomad jobspec. To interact with the Nomad, Consul, and Vault CLIs, you must load the necessary tokens. $ env | grep "NOMAD\\|CONSUL\\|VAULT" $ env | grep "NOMAD\|CONSUL\|VAULT" Nomad and Consul ACL management tokens are located in `/nomad-sandbox/configs/tokens`. You can use them to log in to the web UIs. $ cat /nomad-sandbox/configs/tokens/nomad\_mgmt.token $ cat /nomad-sandbox/configs/tokens/consul\_mgmt.token $ cat /nomad-sandbox/configs/tokens/nomad_mgmt.token $ cat /nomad-sandbox/configs/tokens/consul_mgmt.token You can find the Vault's unseal key, certificates, and root token in `/nomad-sandbox/configs/vault`. $ cat /nomad-sandbox/configs/vault/vault\_root.token $ cat /nomad-sandbox/configs/vault/vault_root.token This sandbox includes sample Nomad jobspecs that you can use to run jobs. Use the following commands to submit the jobs to Nomad: $ nomad job run /nomad-sandbox/configs/jobspecs/2048.hcl $ nomad job run /nomad-sandbox/configs/jobspecs/terramino.hcl $ nomad job run /nomad-sandbox/configs/jobspecs/hashicups.hcl $ nomad job run /nomad-sandbox/configs/jobspecs/2048.hcl $ nomad job run /nomad-sandbox/configs/jobspecs/terramino.hcl $ nomad job run /nomad-sandbox/configs/jobspecs/hashicups.hcl ### Infrastructure * Server node (node-01-server) * n1-standard-2 VM (2 vCPU, 7.5GB RAM, 128GB disk) * Nomad agent configured and running * Access pre-configured with environment variables (NOMAD\_\*) * Management token saved to NOMAD\_TOKEN * Consul agent configured and running * Access pre-configured with environment variables (CONSUL\_\*) * Management token saved to CONSUL\_TOKEN * Vault agent configured and running * Access pre-configured with environment variables (VAULT\_\*) * Root token saved to VAULT\_TOKEN * Client node (node-02-client) * n1-standard-2 VM (2 vCPU, 7.5GB RAM, 128GB disk) * Nomad agent configured and running * Consul agent configured and running * Vault access pre-configured with environment variables (VAULT\_\*) * Nomad drivers: `docker`, `exec`, `java`, `raw_exec` ### Directory structure Environment related files are in the `/nomad-sandbox` directory. /nomad-sandbox └── configs ├── agentfiles │ ├── consul\_server.hcl │ └── nomad\_server.hcl ├── env-info │ ├── ARCHITECTURE │ ├── LIMITATIONS │ └── README ├── jobspecs │ ├── 2048.hcl │ ├── hashicups.hcl │ └── terramino.hcl ├── tokens │ ├── consul\_agent\_server.token │ ├── consul\_dns.token │ ├── consul\_gossip\_key │ ├── consul\_mgmt.token │ ├── nomad\_gossip\_key │ ├── nomad\_mgmt.token │ └── nomad\_node\_join.token └── vault ├── data ├── dev-app-secrets.token ├── unseal.key ├── vault-cert.pem ├── vault-key.pem ├── vault-server.hcl └── vault\_root.token /nomad-sandbox └── configs ├── agentfiles │ ├── consul_server.hcl │ └── nomad_server.hcl ├── env-info │ ├── ARCHITECTURE │ ├── LIMITATIONS │ └── README ├── jobspecs │ ├── 2048.hcl │ ├── hashicups.hcl │ └── terramino.hcl ├── tokens │ ├── consul_agent_server.token │ ├── consul_dns.token │ ├── consul_gossip_key │ ├── consul_mgmt.token │ ├── nomad_gossip_key │ ├── nomad_mgmt.token │ └── nomad_node_join.token └── vault ├── data ├── dev-app-secrets.token ├── unseal.key ├── vault-cert.pem ├── vault-key.pem ├── vault-server.hcl └── vault_root.token Logs are output to the `/tmp` directory. $ ls /tmp ├── consul.log ├── nomad.log └── vault.log $ ls /tmp ├── consul.log ├── nomad.log └── vault.log ### HashiCorp applications The sandbox includes the following HashiCorp applications: * Nomad * One server node and one client node * ACLs enabled * Gossip encryption enabled * TLS encryption enabled * Consul * One server node and one client node * ACLs enabled * Gossip encryption enabled * TLS encryption enabled * DNS configured (.consul domain) * Vault * TLS encryption enabled * Initialized and unsealed The agent files for Nomad and Consul exist in the `/nomad-sandbox/configs/agentfiles` directory. If you make changes to the agent configuration files, you must restart the agent for the changes to take effect. Use the following commands to restart the agents. $ pkill nomad $ nomad agent \\ -config=/nomad-sandbox/configs/agentfiles/nomad\_server.hcl \\ >> /tmp/nomad.log 2>&1 & $ pkill nomad $ nomad agent \ -config=/nomad-sandbox/configs/agentfiles/nomad_server.hcl \ >> /tmp/nomad.log 2>&1 & $ pkill consul $ consul agent \\ -config-file=/nomad-sandbox/configs/agentfiles/consul\_server.hcl \\ >> /tmp/consul.log 2>&1 & $ pkill consul $ consul agent \ -config-file=/nomad-sandbox/configs/agentfiles/consul_server.hcl \ >> /tmp/consul.log 2>&1 & This cluster is made up of one server one and one client node, which is not what we recommend for production environments. The Vault cluster consists of a single node. We do not recommend this for production environments. Nomad jobspecs must be configured to use one of the ports exposed through the Instruqt web browser tabs interface: `3001`, `3333`, or `4444`. Other sandboxes --------------- Explore sandboxes for other HashiCorp products that you might find useful. *  [](https://developer.hashicorp.com/terraform/sandbox?launch=terraform-sandbox) Interactive Interactive Sandbox Terraform Sandbox Experiment with Terraform. This sandbox includes Docker and LocalStack preinstalled to test your Terraform configuration. * Terraform *  [](https://developer.hashicorp.com/vault/sandbox?launch=vault-sandbox) Interactive Interactive Sandbox Vault Sandbox Experiment with Vault. This sandbox contains five Vault Docker containers and a reverse proxy container. * Vault *  [](https://developer.hashicorp.com/boundary/sandbox?launch=boundary-sandbox) Interactive Interactive Sandbox Boundary Sandbox Experiment with Boundary. This sandbox contains a Boundary cluster (one controller server and one worker) configured with a Boundary target (Ubuntu 24.04) and a Postgres database. * Boundary *  [](https://developer.hashicorp.com/boundary/sandbox?launch=boundary-enterprise-sandbox) Interactive Interactive Sandbox Boundary Enterprise Sandbox Experiment with Boundary Enterprise. This sandbox contains a Boundary Enterprise cluster (one controller server and two workers) configured with a Boundary target (Ubuntu 24.04), a Postgres database, and Vault. * Boundary *  [](https://developer.hashicorp.com/consul/sandbox?launch=consul-service-discovery-sandbox) Interactive Interactive Sandbox Consul Sandbox (Service discovery) Experiment with Consul. This sandbox contains a Consul cluster (3 servers and 4 clients) and a Bastion host. It also runs HashiCups, a demo web application. * Consul *  [](https://developer.hashicorp.com/consul/sandbox?launch=consul-service-mesh-sandbox) Interactive Interactive Sandbox Consul Sandbox (Service mesh) Experiment with Consul. This sandbox contains a Consul cluster (3 servers and 4 clients) and a Bastion host. It also runs HashiCups, a demo web application. * Consul ![](https://cdn.usefathom.com/?h=https%3A%2F%2Fdeveloper.hashicorp.com&p=%2Fnomad%2Fsandbox&r=&sid=ZFTCLDIZ&qs=%7B%7D&cid=46381643) --- # What is Nomad? | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/what-is-nomad#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Introduction to Nomad ===================== Welcome to the intro guide to Nomad. This guide is the best place to start with Nomad. We cover what Nomad is, what problems it can solve, how it compares to existing software, and how you can get started using it. If you are familiar with the basics of Nomad, the [documentation](https://developer.hashicorp.com/nomad/docs) and [tutorials](https://developer.hashicorp.com/nomad/tutorials) provide a more detailed reference of available features. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#what-is-nomad) What is Nomad? ----------------------------------------------------------------------------------------- Nomad is a flexible workload orchestrator that enables an organization to easily deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run a diverse workload of Docker, non-containerized, microservice, and batch applications. Nomad enables developers to use declarative infrastructure-as-code for deploying applications. Nomad uses bin packing to efficiently schedule jobs and optimize for resource utilization. Nomad is supported on macOS, Windows, and Linux. Nomad is widely adopted and used in production by PagerDuty, Target, Citadel, Trivago, SAP, Pandora, Roblox, eBay, Deluxe Entertainment, and more. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#key-features) Key features -------------------------------------------------------------------------------------- * **Deploy Containers and Legacy Applications**: Nomad’s flexibility as an orchestrator enables an organization to run containers, legacy, and batch applications together on the same infrastructure. Nomad brings core orchestration benefits to legacy applications without needing to containerize via pluggable [task drivers](https://developer.hashicorp.com/nomad/docs/job-declare/task-driver) . * **Simple & Reliable**: Nomad runs as a single binary and is entirely self contained - combining resource management and scheduling into a single system. Nomad does not require any external services for storage or coordination. Nomad automatically handles application, node, and driver failures. Nomad is distributed and resilient, using leader election and state replication to provide high availability in the event of failures. * **Device Plugins & GPU Support**: Nomad offers built-in support for GPU workloads such as machine learning (ML) and artificial intelligence (AI). Nomad uses [device plugins](https://developer.hashicorp.com/nomad/plugins/devices) to automatically detect and utilize resources from hardware devices such as GPU, FPGAs, and TPUs. * **Federation for Multi-Region**: Nomad has native support for multi-region federation. This built-in capability allows multiple clusters to be linked together, which in turn enables developers to deploy jobs to any cluster in any region. Federation also enables automatic replication of ACL policies, namespaces, resource quotas and Sentinel policies across all clusters. * **Proven Scalability**: Nomad is optimistically concurrent, which increases throughput and reduces latency for workloads. Nomad has been proven to scale to clusters of 10K+ nodes in real-world production environments. * **HashiCorp Ecosystem**: Nomad integrates seamlessly with Terraform, Consul, Vault for provisioning, service discovery, and secrets management. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#how-nomad-compares-to-other-tools) How Nomad compares to other tools -------------------------------------------------------------------------------------------------------------------------------- The following characteristics generally differentiate Nomad from related products: * **Simplicity**: Nomad runs as a single process with zero external dependencies. Operators can easily provision, manage, and scale Nomad. Developers can easily define and run applications. * **Flexibility**: Nomad can run a diverse workload of containerized, legacy, microservice, and batch applications. Nomad can schedule service, batch processing and system jobs, and can run on both Linux and Windows. * **Scalability and High Performance**: Nomad can schedule thousands of containers per second, scale to thousands of nodes in a single cluster, and easily federate across regions and cloud providers. * **HashiCorp Interoperability**: Nomad elegantly integrates with Vault for secrets management and Consul for service discovery and dynamic configuration. Nomad's Consul-like architecture and Terraform-like job specification lower the barrier to entry for existing users of the HashiCorp stack. There are many relevant categories for comparison including cluster managers, resource managers, workload managers, and schedulers. There are many existing tools in each category, and the comparisons are not exhaustive of the entire space. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#nomad-versus-kubernetes) Nomad versus Kubernetes ------------------------------------------------------------------------------------------------------------ Kubernetes and Nomad support similar core use cases for application deployment and management, but they differ in a few key ways. Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, and secrets management. Nomad only aims to focus on cluster management and scheduling, and Nomad is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#simplicity) Simplicity Kubernetes is designed as a collection of more than a half-dozen interoperating services which together provide the full functionality. Coordination and storage is provided by etcd at the core. The state is wrapped by API controllers which are consumed by other services that provide higher level APIs for features like scheduling. Kubernetes supports running in a highly available configuration but is operationally complex to setup. Nomad is architecturally much simpler. Nomad is a single binary, both for clients and servers, and requires no external services for coordination or storage. Nomad combines a lightweight resource manager and a sophisticated scheduler into a single system. By default, Nomad is distributed, highly available, and operationally simple. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#flexible-workload-support) Flexible Workload Support While Kubernetes is specifically focused on Linux containers, Nomad is more general purpose. Nomad supports virtualized, containerized and standalone applications, including Docker, Java, IIS on Windows, Qemu, etc. Nomad is designed with extensible drivers and support will be extended to all common drivers. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#consistent-deployment) Consistent Deployment A full Kubernetes installation for a production environment is time consuming, operationally complex, and resource intensive. An increasing number of implementations are created by the Kubernetes community to mitigate these challenges, such as minikube, kubeadm, k3s, and more. These trimmed versions of Kubernetes offer easier adoption for development and testing, but lead to inconsistency in capabilities, configuration, and management when moving into production. In contrast to Kubernetes' fragmented distributions, Nomad as a single lightweight binary can be deployed in local dev, production, on-prem, at the edge, and in the cloud in a consistent manner, and provides the same operational ease-of-use across all environments. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#scalability) Scalability [Kubernetes documentation](https://kubernetes.io/docs/setup/best-practices/cluster-large/) states that they support clusters up to 5,000 nodes and 300,000 total containers. As the environment grows, the interoperating components with different constraints compound the operational complexity. [Even operators at Google revealed the significant challenges of managing the system at scale](https://blog.dave.tf/post/new-kubernetes/) . The lack of maturity in the Federation project and the additional overhead of managing a centralized management plane also make it a hard experience to deploy a distributed system that spans multiple clusters. Nomad has been proven to scale to cluster sizes that exceed 10,000 nodes in real-world production environments. It can be deployed across multiple availability zones, regions, and data centers with a single cluster or multiple clusters. Nomad is designed to natively handle multi-cluster deployments without the overhead of running clusters on clusters. This makes it easier to scale the application deployment across multiple datacenters, regions, and clouds with no additional complexity. Nomad has performed strenuous benchmark on scalability with [1 million container challenge](https://www.hashicorp.com/c1m) in 2016 and [2 million container challenge](https://www.hashicorp.com/c2m) in 2020. These tests are aimed to validate Nomad's architectural design and ensure that Nomad performs under the most extreme requirements. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#supplement-to-kubernetes) Supplement to Kubernetes Enterprises are comprised of multiple groups of people (business units) with different projects, infrastructure environments, technical competencies, team sizes, budgets, and SLAs. Each group has different requirements and leverages technologies based on their particular needs and constraints. Medium to large scale enterprises run into challenges when trying to standardize hundreds to thousands of software developers and administrators onto one single orchestrator (Kubernetes, Nomad, Mesos) as no scheduler today fits all applications, environments, projects, and teams. Companies in the Global 2000 today such as Intel, Autodesk and GitHub with multiple products and business units organically run Nomad and Kubernetes to supplement each other. They leverage each scheduler to its strengths with Kubernetes for its cutting edge ecosystem and Nomad for simple maintenance and flexibility in core scheduling. ![How organizations leverage Nomad and Kubernetes](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/nomad-kubernetes.png) These are the characteristics we see in teams that typically adopt self-hosted Kubernetes: * Greenfield use cases such as machine learning (ML), serverless, and big data that require the Kubernetes ecosystem and Helm chart * High budget and full-time staffing to maintain Kubernetes * High-profile projects with significant investment and long-term timeline (multi-year) * Deploying and managing new, cloud-native applications * Public cloud environment such as AWS, GCP, Azure Characteristics of teams that typically adopt Nomad: * Run a mix of containerized and non-containerized workloads (Windows, Java) * Small/medium-sized teams with limited capacity to maintain an orchestrator * Deploying and managing core, existing applications * On-premises environment, or hybrid environments * Require simplicity to move fast and fulfill business needs with hard deadlines We continue to see small enterprises continue to standardize on a single orchestrator given the natural staffing and organizational constraints. There are not enough DevOps members to maintain more than one orchestrator, not enough developers to warrant diverging workflows, or simply not enough workload diversity to require more than one orchestrator. ### [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#resources) Resources Review the following resources for in-depth comparisons between Nomad and Kubernetes: * [A Kubernetes User's Guide to HashiCorp Nomad](https://www.hashicorp.com/en/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad) * [The Kubernetes to Nomad Cheat Sheet](https://www.hashicorp.com/en/blog/the-kubernetes-to-nomad-cheat-sheet) * [A Kubernetes User's Guide to HashiCorp Nomad Secret Management](https://www.hashicorp.com/en/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad-secret-management) [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#nomad-versus-aws-ecs) Nomad versus AWS ECS ------------------------------------------------------------------------------------------------------ Amazon Web Services provides the Elastic Container Service (ECS), which is a cluster manager. The ECS service is only available within AWS and can only be used for Docker workloads. Amazon provides customers with the agent that is installed on EC2 instances, but does not provide the servers which are a hosted service of AWS. There are a number of fundamental differences between Nomad and ECS. Nomad is completely open source, including both the client and server components. By contrast, only the agent code for ECS is open and the servers are closed sourced and managed by Amazon. As a side effect of the ECS servers being managed by AWS, it is not possible to use ECS outside of AWS. Nomad is agnostic to the environment in which it is run, supporting public and private clouds, as well as bare metal datacenters. Clusters in Nomad can span multiple datacenters and regions, meaning a single cluster could be managing machines on AWS, Azure, and GCE simultaneously. The ECS service is specifically focused on containers and the Docker engine, while Nomad is more general purpose. Nomad supports virtualized, containerized, and standalone applications, including Docker. Nomad is designed with extensible drivers and support will be extended to all common drivers. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#nomad-versus-terraform) Nomad versus Terraform ---------------------------------------------------------------------------------------------------------- [Terraform](https://www.terraform.io/) is a tool for building, changing, and versioning infrastructure safely and efficiently. Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied. Nomad differs from Terraform in a number of key ways. Terraform is designed to support any type of resource including low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc. Terraform knows how to create, provision, and manage the lifecycle of these resources. Nomad runs on existing infrastructure and manages the lifecycle of applications running on that infrastructure. Another major distinction is that Terraform is an offline tool that runs to completion, while Nomad is an online system with long lived servers. Nomad allows new jobs to be submitted, existing jobs updated or deleted, and can handle node failures. This requires operating continuously instead of in a single shot like Terraform. For small infrastructures with only a handful of servers or applications, the complexity of Nomad may not outweigh simply using Terraform to statically assign applications to machines. At larger scales, Terraform should be used to provision capacity for Nomad, and Nomad used to manage scheduling applications to machines dynamically. [](https://developer.hashicorp.com/nomad/docs/what-is-nomad#next-steps) Next steps ---------------------------------------------------------------------------------- Review [use cases](https://developer.hashicorp.com/nomad/docs/use-cases) to understand the many ways Nomad is used in production today across many industries to solve critical, real-world business objectives. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/what-is-nomad.mdx) --- # Quick Start | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials/get-started#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Get Started =========== Get up and running with Nomad by learning about scheduling, setting up a cluster, and deploying an example job. [Start](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview) 5 tutorials 1.  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview) 6min Introduction to Nomad Discover what HashiCorp Nomad is, become familiar with important related terms, and understand how Nomad fits into the application development workflow. * Nomad * Video 2.  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install) 4min Install the Nomad CLI Install the Nomad CLI on Mac, Windows, or Linux. * Nomad 3.  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster) 11min Create a Nomad cluster Create a local development cluster on your machine or a cloud-based one on AWS, Azure, or GCP. * Nomad * Terraform 4.  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job) 12min Deploy and update a Nomad job Use the Nomad CLI to deploy an example application. Then update the application's jobspec and redeploy it. * Nomad 5.  [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-stop-nomad) 3min Stop Nomad and clean up Stop and clean up the application jobs in Nomad. Destroy the cluster resources and discover additional topics to continue your Nomad learning. * Nomad --- # Create and submit a job | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Create and submit a job ======================= In Nomad, the description of the job and all its requirements are maintained in a single file called the "job file". This job file resides locally on disk and it is highly recommended that you check job files into source control. The general flow for submitting a job in Nomad is: 1. Author a job file according to the job specification 2. Plan and review changes with a Nomad server 3. Submit the job file to a Nomad server 4. (Optional) Review job status and logs Here is a very basic example to get you started. [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#author-a-job-file) Author a job file --------------------------------------------------------------------------------------------------------- Nomad attempts to strike a balance between ease and expressiveness in its job specification. Nomad also provides the `nomad init` command to generate sample job files. For more detailed information about the Nomad job specification, please consult the [Nomad documentation](https://developer.hashicorp.com/nomad/docs/job-specification) . Here is a sample job file which runs a small docker container web server to start with. job "docs" { datacenters \= \["dc1"\] group "example" { network { port "http" { static \= "5678" } } task "server" { driver \= "docker" config { image \= "hashicorp/http-echo" ports \= \["http"\] args \= \[\ "-listen",\ ":5678",\ "-text",\ "hello world",\ \] } } } } job "docs" { datacenters = ["dc1"] group "example" { network { port "http" { static = "5678" } } task "server" { driver = "docker" config { image = "hashicorp/http-echo" ports = ["http"] args = [\ "-listen",\ ":5678",\ "-text",\ "hello world",\ ] } } } } This job file exists on your local workstation in plain text. When you are satisfied with this job file, you will plan and review the scheduler decision. It is generally a best practice to commit job files to source control, especially if you are working in a team. [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#plan-the-job) Plan the job ----------------------------------------------------------------------------------------------- Once the job file is authored, you probably want to preview the changes Nomad will make when it runs. The `nomad job plan` command invokes a dry-run of the scheduler and output which scheduling decisions would take place. $ nomad job plan docs.nomad.hcl \+ Job: "docs" \+ Task Group: "example" (1 create) + Task: "server" (forces create) Scheduler dry-run: \- All tasks successfully allocated. Job Modify Index: 0 To submit the job with version verification run: nomad job run -check-index 0 docs.nomad.hcl When running the job with the check-index flag, the job will only be run if the job modify index given matches the server-side version. If the index has changed, another user has modified the job and the plan's results are potentially invalid. $ nomad job plan docs.nomad.hcl + Job: "docs" + Task Group: "example" (1 create) + Task: "server" (forces create) Scheduler dry-run: - All tasks successfully allocated. Job Modify Index: 0 To submit the job with version verification run: nomad job run -check-index 0 docs.nomad.hcl When running the job with the check-index flag, the job will only be run if the job modify index given matches the server-side version. If the index has changed, another user has modified the job and the plan's results are potentially invalid. Note that no action was taken. This job is not running. This is a complete dry-run and no allocations have taken place. [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#submit-the-job) Submit the job --------------------------------------------------------------------------------------------------- Assuming the output of the plan looks acceptable, now ask Nomad to execute the job. This is done via the `nomad job run` command. You can optionally supply the modify index provided by the plan command to ensure no changes to this job have taken place between our plan and now. $ nomad job run docs.nomad.hcl \==> Monitoring evaluation "0d159869" Evaluation triggered by job "docs" Allocation "5cbf23a1" created: node "1e1aa1e0", group "example" Evaluation status changed: "pending" -> "complete" \==> Evaluation "0d159869" finished with status "complete" $ nomad job run docs.nomad.hcl ==> Monitoring evaluation "0d159869" Evaluation triggered by job "docs" Allocation "5cbf23a1" created: node "1e1aa1e0", group "example" Evaluation status changed: "pending" -> "complete" ==> Evaluation "0d159869" finished with status "complete" Now that the job is scheduled, it may or may not be running. You need to inspect the allocation status and logs to make sure the job started correctly. The next section on [inspecting state](https://developer.hashicorp.com/nomad/docs/job-run/inspect) details ways to examine this job's state. [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#update-and-plan-the-job) Update and plan the job --------------------------------------------------------------------------------------------------------------------- When making updates to the job, it is best to always run the plan command and then the run command. For example: @@ -2,6 +2,8 @@ job "docs" { datacenters = \["dc1"\] group "example" { + count = "2" + task "server" { driver = "docker" @@ -2,6 +2,8 @@ job "docs" { datacenters = ["dc1"] group "example" { + count = "2" + task "server" { driver = "docker" After saving these changes to disk, run the `nomad job plan` command: $ nomad job plan docs.nomad.hcl +/- Job: "docs" +/- Task Group: "example" (1 create, 1 in-place update) +/- Count: "1" => "2" (forces create) Task: "server" Scheduler dry-run: \- All tasks successfully allocated. Job Modify Index: 131 To submit the job with version verification run: nomad job run -check-index 131 docs.nomad.hcl When running the job with the check-index flag, the job will only be run if the job modify index given matches the server-side version. If the index has changed, another user has modified the job and the plan's results are potentially invalid. $ nomad job plan docs.nomad.hcl +/- Job: "docs" +/- Task Group: "example" (1 create, 1 in-place update) +/- Count: "1" => "2" (forces create) Task: "server" Scheduler dry-run: - All tasks successfully allocated. Job Modify Index: 131 To submit the job with version verification run: nomad job run -check-index 131 docs.nomad.hcl When running the job with the check-index flag, the job will only be run if the job modify index given matches the server-side version. If the index has changed, another user has modified the job and the plan's results are potentially invalid. #### [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#reserved-port-collisions) Reserved port collisions Because this job uses a static port, it is possible for some instances to not be placeable depending on the number of clients you have in your Nomad cluster. If your plan output contains: Dimension "network: reserved port collision" exhausted on x nodes Dimension "network: reserved port collision" exhausted on x nodes This indicates that every feasible client in your cluster has or will have something placed at the requested port, leaving no place for some of these allocations to run. To resolve this, you need to reduce the requested count, add additional clients, or migrate from static ports to dynamic ports in your job specification. [](https://developer.hashicorp.com/nomad/docs/job-declare/create-job#run-the-job) Run the job --------------------------------------------------------------------------------------------- Now, assuming the output is okay, execute the `nomad job run` command. Including the `check-index` parameter ensures that the job was not changed between the plan and run phases. $ nomad job run -check-index 131 docs.nomad.hcl \==> Monitoring evaluation "42d788a3" Evaluation triggered by job "docs" Allocation "e7b8d4f5" created: node "012ea79b", group "example" Allocation "5cbf23a1" modified: node "1e1aa1e0", group "example" Evaluation status changed: "pending" -> "complete" \==> Evaluation "42d788a3" finished with status "complete" $ nomad job run -check-index 131 docs.nomad.hcl ==> Monitoring evaluation "42d788a3" Evaluation triggered by job "docs" Allocation "e7b8d4f5" created: node "012ea79b", group "example" Allocation "5cbf23a1" modified: node "1e1aa1e0", group "example" Evaluation status changed: "pending" -> "complete" ==> Evaluation "42d788a3" finished with status "complete" For more details on advanced job updating strategies such as canary builds and build-green deployments, consult the documentation on [job update strategies](https://developer.hashicorp.com/nomad/docs/job-declare/strategy) . [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/job-declare/create-job.mdx) --- # Create a parameterized Nomad job | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) Nomad parameterized jobs enable dynamic runtime behaviors from a single job specification. These specialized batch-type jobs act less like regular Nomad jobs and more like functions. They are reusable, can accept parameters, and run only when invoked. Parameterized jobs provide three key benefits to end users. * Creating instances of the job without resubmitting the job specification. * Providing dynamic run-time values as parameters to the job during dispatch. * Proving, when the job allows, an opaque payload that Nomad will write to one or more of the job's tasks. Nomad registers, but does not run, a parameterized job when you submit it to the cluster. To run a parameterized job, end-users invoke—or **dispatch**—it with the [`nomad job dispatch` command](https://developer.hashicorp.com/nomad/commands/job/dispatch) or Nomad [Job Dispatch API](https://developer.hashicorp.com/nomad/api-docs/jobs#dispatch-job) . During dispatch, the user provides the parameter values and payload. Nomad combines these values with the registered job, creates a dispatched instance, and returns the scheduling information to the end-user. Once dispatched, the created instances are handled like regular Nomad batch jobs. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#challenge) Challenge ----------------------------------------------------------------------------------------------------------------- In this tutorial, you will learn the fundamentals of Nomad parameterized jobs. You will take a Nomad batch job that renders a template, and * Convert it to a parameterized job * Enhance it with optional and required parameters * Define default values for optional parameters * Handle dispatch payloads These fundamental practices can be used to create more complex parameterized workloads in your own environment. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#prerequisites) Prerequisites ------------------------------------------------------------------------------------------------------------------------- * Nomad v1.0 or greater * Nomad dev agent or Nomad cluster [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#configure-your-learning-environment) Configure your learning environment --------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#fetch-the-tutorial-content) Fetch the tutorial content This tutorial uses content provided in the `hashicorp/learn-nomad-jobspec` repository on GitHub. You can download a ZIP archive directly or use `git` to clone the repository. Download ZIP archiveClone repository $ wget https://github.com/hashicorp-education/learn-nomad-jobspec/archive/release.zip $ wget https://github.com/hashicorp-education/learn-nomad-jobspec/archive/release.zip Unarchive the downloaded release. $ unzip release.zip $ unzip release.zip The unzipping process creates a directory named `learn-nomad-jobspec-release`. Change into it and into the tutorial's folder. $ cd learn-nomad-jobspec-release/parameterized $ cd learn-nomad-jobspec-release/parameterized Clone the `hashicorp/learn-nomad-jobspec` repository. $ git clone https://github.com/hashicorp-education/learn-nomad-jobspec $ git clone https://github.com/hashicorp-education/learn-nomad-jobspec Change into the project directory. $ cd learn-nomad-jobspec $ cd learn-nomad-jobspec Check out the release tag. $ git checkout release $ git checkout release Change to the tutorial's folder $ cd parameterized $ cd parameterized ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#configure-tutorial-environment) Configure tutorial environment This tutorial uses HCL2 variables to reduce the amount of editing that you need to do to the job specification based on your environment. The defaults are optimized for a Linux/macOS boxes targeting a local Nomad dev agent using the default datacenter value of `dc1`. These values can be customized using environment variables before running the job file. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#override-default-task-driver-optional) Override default task driver (optional) This tutorial uses either the raw\_exec or exec task driver (raw\_exec by default) to render a text file and output it to standard output. Because Windows and macOS do not support the exec driver and the raw\_exec driver is not enabled by default on non-dev agents, this job uses an HCL2 variable to let you set your preferred driver without editing the job. The **driver** variable defaults to raw\_exec, suitable for all learners using a dev agent. If you need to use the exec driver instead, set the `NOMAD_VAR_driver` environment variable to `exec`. $ export NOMAD\_VAR\_driver\=exec $ export NOMAD_VAR_driver=exec #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#override-default-datacenter-optional) Override default datacenter (optional) The job also allows you to specify your datacenter value if it differs from the default value of `dc1`. If you are using a non-default datacenter value, set the `NOMAD_VAR_datacenter` environment variable. If you are unsure of a valid datacenter value for your target cluster or dev agent, run the `nomad node status` command to list all the datacenters available. $ nomad node status ID DC Name Class Drain Eligibility Status d715f8b4 mydc nomad-client-1.node.consul false eligible ready 14ab9290 mydc nomad-client-2.node.consul false eligible ready 0f357b26 mydc nomad-client-3.node.consul false eligible ready $ nomad node status ID DC Name Class Drain Eligibility Status d715f8b4 mydc nomad-client-1.node.consul false eligible ready 14ab9290 mydc nomad-client-2.node.consul false eligible ready 0f357b26 mydc nomad-client-3.node.consul false eligible ready In this sample output, all the clients are in a datacenter named `mydc`, based on this output you would run: $ export NOMAD\_VAR\_datacenter=mydc $ export NOMAD_VAR_datacenter=mydc ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#run-the-starting-job) Run the starting job To get you started, the tutorial repository includes a Nomad batch job specification that you will use as a base to build on. Use the `nomad job run` command to run the `start.nomad` job file. $ nomad job run start.nomad \==> Monitoring evaluation "68b8b7dc" Evaluation triggered by job "template" Allocation "cb1d80a1" created: node "66d36f06", group "renderer" \==> Monitoring evaluation "68b8b7dc" Allocation "cb1d80a1" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" \==> Evaluation "68b8b7dc" finished with status "complete" $ nomad job run start.nomad ==> Monitoring evaluation "68b8b7dc" Evaluation triggered by job "template" Allocation "cb1d80a1" created: node "66d36f06", group "renderer" ==> Monitoring evaluation "68b8b7dc" Allocation "cb1d80a1" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" ==> Evaluation "68b8b7dc" finished with status "complete" Make note of the allocation ID shown in your output. In the provided sample output, the allocation ID is `cb1d80a1`. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#fetch-the-output-of-the-job) Fetch the output of the job Since this job writes its output to standard output, you can retrieve it with the `nomad alloc logs` command. Run the `nomad alloc logs` command, supplying the allocation ID you noted in the previous step. **Pro Tip** - You only need to provide enough of the allocation ID to The `nomad alloc logs` command to point to a distinct allocation. This works with many identifiers in the Nomad command-line tool. $ nomad alloc logs cb1d80a1 Hello, Thanks for your interest. I have attached the information you requested. I look forward to talking with you soon. $ nomad alloc logs cb1d80a1 Hello, Thanks for your interest. I have attached the information you requested. I look forward to talking with you soon. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#purge-the-job) Purge the job Nomad will not allow you to convert a batch job to a parameterized job without purging the original version as a safety feature. When run with the `-purge` flag, the `nomad job stop` command deletes the job and information about any former runs of it from the Nomad server state. Note The `start.nomad` file defines a Nomad job named `template`. The filename and job name are intentionally different allowing you to keep copies of the different states of the job as you progress through the tutorial. $ nomad job stop -purge template \==> Monitoring evaluation "a0704fca" Evaluation triggered by job "template" Evaluation status changed: "pending" -> "complete" \==> Evaluation "a0704fca" finished with status "complete" $ nomad job stop -purge template ==> Monitoring evaluation "a0704fca" Evaluation triggered by job "template" Evaluation status changed: "pending" -> "complete" ==> Evaluation "a0704fca" finished with status "complete" [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#make-the-job-parameterized) Make the job parameterized --------------------------------------------------------------------------------------------------------------------------------------------------- Adding a parameterized stanza converts a batch-type job to a parameterized one. An empty `parameterized` stanza creates a parameterized job that isn't customizable at dispatch time. This is still a useful feature, since it enables an operator or application to run an instance of the job without having to have the job specification itself. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#add-parameterized-stanza) Add parameterized stanza Copy the `start.nomad` file to a new file named `parameterized.nomad`. $ cp start.nomad parameterized.nomad $ cp start.nomad parameterized.nomad Open `parameterized.nomad` in a text editor. Add an empty `parameterized` stanza inside of the `job` stanza and save the file. parameterized { } parameterized { } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#register-the-job) Register the job Run the parameterized version of the job. $ nomad job run parameterized.nomad Job registration successful $ nomad job run parameterized.nomad Job registration successful Notice that the output doesn't show any scheduling activity—no evaluation or allocation information. Parameterized jobs themselves do not run. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#if-you-get-an-error) If you get an error If you receive the following error, it indicates that you missed purging the non-parameterized version of the template job. Run `nomad job stop -purge template` and re-run `nomad job run parameterized.nomad` to resolve it. $ nomad job run template.nomad Error submitting job: Unexpected response code: 500 (cannot update non-parameterized job to being parameterized) $ nomad job run template.nomad Error submitting job: Unexpected response code: 500 (cannot update non-parameterized job to being parameterized) Run the `nomad job status` command to verify your parameterized job is available for dispatch. $ nomad job status ID Type Priority Status Submit Date template batch/parameterized 50 running 2021-04-11T22:01:45-04:00 $ nomad job status ID Type Priority Status Submit Date template batch/parameterized 50 running 2021-04-11T22:01:45-04:00 ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#dispatch-the-job) Dispatch the job Run the `nomad job dispatch` command to dispatch an instance of the parameterized job. The `nomad job dispatch` command's final argument is the **registered job ID** to dispatch, unlike the `nomad job run` command which uses the filename of the job specification. $ nomad job dispatch template Dispatched Job ID = template/dispatch-1620682960-e9dfcaf8 Evaluation ID = 897c8095 \==> Monitoring evaluation "897c8095" Evaluation triggered by job "template/dispatch-1620682960-e9dfcaf8" Allocation "4e317cb8" created: node "66d36f06", group "renderer" Evaluation status changed: "pending" -> "complete" \==> Evaluation "897c8095" finished with status "complete" $ nomad job dispatch template Dispatched Job ID = template/dispatch-1620682960-e9dfcaf8 Evaluation ID = 897c8095 ==> Monitoring evaluation "897c8095" Evaluation triggered by job "template/dispatch-1620682960-e9dfcaf8" Allocation "4e317cb8" created: node "66d36f06", group "renderer" Evaluation status changed: "pending" -> "complete" ==> Evaluation "897c8095" finished with status "complete" Examine the output of the command. There are some key differences from the typical output of the `nomad job run` command. Notice that Nomad generates a `Dispatched Job ID`. This generated job ID refers to this specific instance of the parameterized job, and it enables you to manage discrete instances of a parameterized job in the same ways you manage your other Nomad batch jobs. The output also provides scheduling information. Collect the allocation ID from your output. In the preceding output it's `4e317cb8`. Like earlier, run the `nomad alloc logs` command for your allocation ID. $ nomad alloc logs 4e317cb8 Hello, Thanks for your interest. I have attached the information you requested. I look forward to talking with you soon. $ nomad alloc logs 4e317cb8 Hello, Thanks for your interest. I have attached the information you requested. I look forward to talking with you soon. Parameterized jobs without variables can be used to provide a means for running a batch job without having to supply the job specification. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#use-dispatch-variables) Use dispatch variables ------------------------------------------------------------------------------------------------------------------------------------------- Parameterized jobs also provide the ability to submit run-specific variables while dispatching the job. The job specification defines the variable names. They can be optional or required. Nomad provides the dispatched variable values as environment variables. The environment variable names that Nomad creates for these variables are prepended with `NOMAD_META_`, for example `NOMAD_META_customer_email`. Nomad will not create environment variables for unset optional variables; your workload should handle this case. You may also define defaults for optional variables as part of the job specification. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#update-the-job-to-use-email-tmpl) Update the job to use email.tmpl Copy the `parameterized.nomad` file to a new file named `variables.nomad`. $ cp parameterized.nomad variables.nomad $ cp parameterized.nomad variables.nomad Open the `variables.nomad` file in a text editor and change the HCL2 `file` function to read in the `email.tmpl` file. This template generates an email-like output. Change ${file("./templates/template.tmpl")} ${file("./templates/template.tmpl")} to ${file("./templates/email.tmpl")} ${file("./templates/email.tmpl")} ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#define-job-variables) Define job variables Required and optional variables are defined in the `meta_required` and `meta_optional` attributes of the `parameterized` stanza, respectively. These attributes take a list of quoted variable names the value. The `email.tmpl` template needs more information to render properly: * **required** - `customer_email`, `customer_name` * **optional** - `rep_name`, `rep_email`, `rep_title`, `product_name` Define the variables by adding these attributes to the `variables.nomad` file inside of the `parameterized` stanza. meta\_required \= \["customer\_name","customer\_email"\] meta\_optional \= \["rep\_name","rep\_email","rep\_title","product\_name"\] meta_required = ["customer_name","customer_email"] meta_optional = ["rep_name","rep_email","rep_title","product_name"] ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#provide-defaults-for-optional-values) Provide defaults for optional values You can define default values for your optional variables using a job-level `meta` stanza. Use the variable name as the attribute name and set the value to the default. For this example, create a default for the `rep_name` and the `rep_email` values to provide generic values when the `rep_name` and `rep_email` are not provided. In the `variables.nomad` file, add this `meta` stanza inside of the `job` stanza. meta { rep\_name \= "BabbageCorp" rep\_email \= "hello@mechanicalcomputing.com" } meta { rep_name = "BabbageCorp" rep_email = "hello@mechanicalcomputing.com" } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#deploy-and-dispatch-the-job) Deploy and dispatch the job Save the `variables.nomad` file and submit it using the `nomad job run` command. $ nomad job run variables.nomad Job registration successful $ nomad job run variables.nomad Job registration successful [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#dispatch-the-job-with-variables) Dispatch the job with variables ------------------------------------------------------------------------------------------------------------------------------------------------------------- Now, dispatch a copy with the `nomad job dispatch` command. Use the `-meta` flag to set the `customer_email` to `alovelace@programmers.net` and `customer_name` to `Ada Lovelace`. $ nomad job dispatch \\ -meta customer\_email="alovelace@programmers.net" \\ -meta customer\_name="Ada Lovelace" \\ template $ nomad job dispatch \ -meta customer_email="alovelace@programmers.net" \ -meta customer_name="Ada Lovelace" \ template The command will output scheduling information. Dispatched Job ID = template/dispatch-1620750077-463bca71 Evaluation ID = 1b407851 \==> Monitoring evaluation "1b407851" Evaluation triggered by job "template/dispatch-1620750077-463bca71" \==> Monitoring evaluation "1b407851" Allocation "391fc0c0" created: node "f7bc1f2d", group "renderer" Evaluation status changed: "pending" -> "complete" \==> Evaluation "1b407851" finished with status "complete" Dispatched Job ID = template/dispatch-1620750077-463bca71 Evaluation ID = 1b407851 ==> Monitoring evaluation "1b407851" Evaluation triggered by job "template/dispatch-1620750077-463bca71" ==> Monitoring evaluation "1b407851" Allocation "391fc0c0" created: node "f7bc1f2d", group "renderer" Evaluation status changed: "pending" -> "complete" ==> Evaluation "1b407851" finished with status "complete" Fetch the template output using the `nomad alloc logs` command for the allocation ID output when you dispatched the job. $ nomad alloc logs 391fc0c0 TO: Ada Lovelace FROM: BabbageCorp SUBJ: Thanks for your interest. \------ Hello Ada, Thanks for your interest. I look forward to talking with you soon. Sincerely, BabbageCorp $ nomad alloc logs 391fc0c0 TO: Ada Lovelace FROM: BabbageCorp SUBJ: Thanks for your interest. ------ Hello Ada, Thanks for your interest. I look forward to talking with you soon. Sincerely, BabbageCorp ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#test-the-required-variables) Test the required variables Nomad validates that the dispatch request contains all required variables. The operator or application dispatching the job must provide all required variables when dispatching the job. If not, Nomad will return an error. Observe this by dispatching the job with no variables set. $ nomad job dispatch template Failed to dispatch job: Unexpected response code: 500 (rpc error: Dispatch did not provide required meta keys: \[customer\_email customer\_name\]) $ nomad job dispatch template Failed to dispatch job: Unexpected response code: 500 (rpc error: Dispatch did not provide required meta keys: [customer_email customer_name]) ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#use-the-optional-variables) Use the optional variables Now, use the optional variables to personalize the email. Dispatch the job again with the following command. $ nomad job dispatch \\ -meta customer\_email="alovelace@programmers.net" \\ -meta customer\_name="Ada Lovelace" \\ -meta rep\_name="Charles Babbage" \\ -meta rep\_email="cbabbage@mechanicalcomputing.com" \\ -meta rep\_title="Chief Innovation Officer" \\ -meta product\_name="Analytical Engine" \\ template $ nomad job dispatch \ -meta customer_email="alovelace@programmers.net" \ -meta customer_name="Ada Lovelace" \ -meta rep_name="Charles Babbage" \ -meta rep_email="cbabbage@mechanicalcomputing.com" \ -meta rep_title="Chief Innovation Officer" \ -meta product_name="Analytical Engine" \ template The command outputs the scheduling information. Dispatched Job ID = template/dispatch-1620750369-9431cc3b Evaluation ID = 6312c1e3 \==> Monitoring evaluation "6312c1e3" Evaluation triggered by job "template/dispatch-1620750369-9431cc3b" Allocation "902a3d43" created: node "f7bc1f2d", group "renderer" \==> Monitoring evaluation "6312c1e3" Allocation "902a3d43" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" \==> Evaluation "6312c1e3" finished with status "complete" Dispatched Job ID = template/dispatch-1620750369-9431cc3b Evaluation ID = 6312c1e3 ==> Monitoring evaluation "6312c1e3" Evaluation triggered by job "template/dispatch-1620750369-9431cc3b" Allocation "902a3d43" created: node "f7bc1f2d", group "renderer" ==> Monitoring evaluation "6312c1e3" Allocation "902a3d43" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" ==> Evaluation "6312c1e3" finished with status "complete" Fetch the output from the generated allocation using the `nomad alloc logs` command. $ nomad alloc logs 902a3d43 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. \------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer $ nomad alloc logs 902a3d43 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. ------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#consume-a-dispatched-payload) Consume a dispatched payload ------------------------------------------------------------------------------------------------------------------------------------------------------- When defining a Nomad parameterized job, you have the option to use a `payload`. A payload is a means to supply up to 16Kib of opaque data to instances of your parameterized job during dispatch. Note These payloads are included in the dispatched version's job definition data and will consume a corresponding amount of server memory until the dispatched instance of the job is garbage collected. The `parameterized` stanza's `payload` value determines if a payload is `required`, `optional`, or `forbidden` when the job is dispatched. Nomad writes the dispatch payload into any task containing a `dispatch_payload` stanza. The `path` attribute specifies where to write the data relative to the [task's local directory](https://developer.hashicorp.com/nomad/docs/reference/runtime-environment-settings#local) . ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#enable-a-mandatory-payload) Enable a mandatory payload Copy the `variables.nomad` file to a new file named `payload.nomad`. $ cp variables.nomad payload.nomad $ cp variables.nomad payload.nomad Open `payload.nomad` in your text editor. Add a `payload = "required` attribute inside the `parameterized` stanza. payload \= "required" payload = "required" ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#write-the-payload-into-the-task) Write the payload into the task Also, add a `dispatch_payload` stanza inside the `output` task stanza. dispatch\_payload { file \= "payload.txt" } dispatch_payload { file = "payload.txt" } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#submit-the-updated-job) Submit the updated job Save the `payload.nomad` file and submit the job to Nomad. $ nomad job run payload.nomad Job registration successful $ nomad job run payload.nomad Job registration successful [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#dispatch-the-job-with-the-payload) Dispatch the job with the payload ----------------------------------------------------------------------------------------------------------------------------------------------------------------- There are two ways to provide a payload to the `nomad job dispatch` command * from a file * from standard input ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#get-payload-from-a-file) Get payload from a file The tutorial repository contains a sample text file for you to use as the payload—engine.txt Use the `nomad job dispatch` command to submit the payload directly $ nomad job dispatch \\ -meta customer\_email="alovelace@programmers.net" \\ -meta customer\_name="Ada Lovelace" \\ -meta rep\_name="Charles Babbage" \\ -meta rep\_email="cbabbage@mechanicalcomputing.com" \\ -meta rep\_title="Chief Innovation Officer" \\ -meta product\_name="Analytical Engine" \\ template payloads/engine.txt $ nomad job dispatch \ -meta customer_email="alovelace@programmers.net" \ -meta customer_name="Ada Lovelace" \ -meta rep_name="Charles Babbage" \ -meta rep_email="cbabbage@mechanicalcomputing.com" \ -meta rep_title="Chief Innovation Officer" \ -meta product_name="Analytical Engine" \ template payloads/engine.txt The command outputs the scheduling information. Dispatched Job ID = template/dispatch-1620751626-eefe8be3 Evaluation ID = 297238d6 \==> Monitoring evaluation "297238d6" Evaluation triggered by job "template/dispatch-1620751626-eefe8be3" Allocation "a2971fa2" created: node "f7bc1f2d", group "renderer" \==> Monitoring evaluation "297238d6" Allocation "a2971fa2" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" \==> Evaluation "297238d6" finished with status "complete" Dispatched Job ID = template/dispatch-1620751626-eefe8be3 Evaluation ID = 297238d6 ==> Monitoring evaluation "297238d6" Evaluation triggered by job "template/dispatch-1620751626-eefe8be3" Allocation "a2971fa2" created: node "f7bc1f2d", group "renderer" ==> Monitoring evaluation "297238d6" Allocation "a2971fa2" status changed: "pending" -> "complete" (All tasks have completed) Evaluation status changed: "pending" -> "complete" ==> Evaluation "297238d6" finished with status "complete" Note the allocation ID; retrieve the output of the allocation using the `nomad alloc logs` command. $ nomad alloc logs a2971fa2 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. \------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer \---- CHAPTER VIII OF THE ANALYTICAL ENGINE The circular arrangement of the axes of the Difference Engine round large central wheels led to the most extended prospects. The whole of arithmetic now appeared within the grasp of mechanism. A vague glimpse even of an Analytical ... $ nomad alloc logs a2971fa2 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. ------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer ---- CHAPTER VIII OF THE ANALYTICAL ENGINE The circular arrangement of the axes of the Difference Engine round large central wheels led to the most extended prospects. The whole of arithmetic now appeared within the grasp of mechanism. A vague glimpse even of an Analytical ... ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#get-payload-from-standard-input) Get payload from standard input Using standard input allows you submit dynamic payloads to your parameterized jobs. Use `-` as the filename in the `nomad job dispatch` command to collect the payload from standard input. This example uses the `date` command to generate a dynamic payload. $ date | nomad job dispatch \\ -meta customer\_email="alovelace@programmers.net" \\ -meta customer\_name="Ada Lovelace" \\ -meta rep\_name="Charles Babbage" \\ -meta rep\_email="cbabbage@mechanicalcomputing.com" \\ -meta rep\_title="Chief Innovation Officer" \\ -meta product\_name="Analytical Engine" \\ template - $ date | nomad job dispatch \ -meta customer_email="alovelace@programmers.net" \ -meta customer_name="Ada Lovelace" \ -meta rep_name="Charles Babbage" \ -meta rep_email="cbabbage@mechanicalcomputing.com" \ -meta rep_title="Chief Innovation Officer" \ -meta product_name="Analytical Engine" \ template - Dispatched Job ID = template/dispatch-1620751993-b9397497 Evaluation ID = dc32bd93 ==> Monitoring evaluation "dc32bd93" Evaluation triggered by job "template/dispatch-1620751993-b9397497" ==> Monitoring evaluation "dc32bd93" Allocation "b22837a8" created: node "f7bc1f2d", group "renderer" Evaluation status changed: "pending" -> "complete" ==> Evaluation "dc32bd93" finished with status "complete" Dispatched Job ID = template/dispatch-1620751993-b9397497 Evaluation ID = dc32bd93   ==> Monitoring evaluation "dc32bd93" Evaluation triggered by job "template/dispatch-1620751993-b9397497" ==> Monitoring evaluation "dc32bd93" Allocation "b22837a8" created: node "f7bc1f2d", group "renderer" Evaluation status changed: "pending" -> "complete" ==> Evaluation "dc32bd93" finished with status "complete" Fetch the output of the dispatched allocation using the `nomad alloc logs` command. $nomad alloc logs b22837a8 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. \------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer Wed May 12 11:59:30 EDT 2021 $nomad alloc logs b22837a8 TO: Ada Lovelace FROM: Charles Babbage SUBJ: Thanks for your interest. ------ Hello Ada, Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon. Sincerely, Charles Chief Innovation Officer Wed May 12 11:59:30 EDT 2021 [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#clean-up) Clean up --------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#clean-up-the-target-environment) Clean up the target environment The cleanup instructions vary depending on your target infrastructure. Select the one that applies. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#nomad-dev-agent) Nomad dev agent Stop the Nomad dev agent with `Ctrl-C`. Nomad dev agents use ephemeral storage and require no further cleanup. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#nomad-cluster) Nomad cluster Since Nomad clusters maintain persistent state, you will need to perform extra steps to remove all the tutorial artifacts from the cluster. ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#remove-the-parameterized-job) Remove the parameterized job Stop the `template` job. You can optionally pass the `-purge` flag to tell Nomad to immediately remove it from the server state. Since no scheduling activity is required to remove the job, there is no output generated by the command. $ nomad job stop -purge template $ nomad job stop -purge template ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#remove-dispatched-instances-optional) Remove dispatched instances (optional) Each time you dispatch a parameterized job, Nomad creates a distinct but related job. These jobs will naturally be removed in time by Nomad's garbage collection process. You can see a list of these instances by using the `nomad job status` command referencing an ambiguous prefix, like `template\`. These IDs will be specific to your environment. **Pro Tip** - The Nomad command-line tool displays a list of matching IDs whenever an incomplete ID matches more than one item. $ nomad job status template/ Prefix matched multiple jobs ID Type Priority Status Submit Date template/dispatch-1620850743-2d3e0743 batch 50 dead 2021-05-12T16:19:03-04:00 template/dispatch-1620850966-1ba39df6 batch 50 dead 2021-05-12T16:22:46-04:00 template/dispatch-1620851039-1584df83 batch 50 dead 2021-05-12T16:23:59-04:00 template/dispatch-1620851914-68ee4baf batch 50 dead 2021-05-12T16:38:34-04:00 template/dispatch-1620851954-e0c9f384 batch 50 dead 2021-05-12T16:39:14-04:00 $ nomad job status template/ Prefix matched multiple jobs ID Type Priority Status Submit Date template/dispatch-1620850743-2d3e0743 batch 50 dead 2021-05-12T16:19:03-04:00 template/dispatch-1620850966-1ba39df6 batch 50 dead 2021-05-12T16:22:46-04:00 template/dispatch-1620851039-1584df83 batch 50 dead 2021-05-12T16:23:59-04:00 template/dispatch-1620851914-68ee4baf batch 50 dead 2021-05-12T16:38:34-04:00 template/dispatch-1620851954-e0c9f384 batch 50 dead 2021-05-12T16:39:14-04:00 These jobs will naturally be removed in time by Nomad's garbage collection process. You can also use the `nomad job stop` command with the `-purge` flag set to remove individual dispatched instances immediately. For example: $ nomad job stop -purge template/dispatch-1620748308-41f86ed3 \==> Monitoring evaluation "d771b1fc" Evaluation triggered by job "template/dispatch-1620748308-41f86ed3" \==> Monitoring evaluation "d771b1fc" Evaluation status changed: "pending" -> "complete" \==> Evaluation "d771b1fc" finished with status "complete" $ nomad job stop -purge template/dispatch-1620748308-41f86ed3 ==> Monitoring evaluation "d771b1fc" Evaluation triggered by job "template/dispatch-1620748308-41f86ed3" ==> Monitoring evaluation "d771b1fc" Evaluation status changed: "pending" -> "complete" ==> Evaluation "d771b1fc" finished with status "complete" ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#remove-the-tutorial-s-working-directory) Remove the tutorial's working directory Ensure that you have closed any file editors that might have files in the working directory open. Change to your home directory and recursively remove the learning environment you created in the first step. From ZIPFrom git Remove the unarchived learning environment. $ rm -rf learn-nomad-jobspec-release $ rm -rf learn-nomad-jobspec-release Delete the downloaded archive. $ rm release.zip $ rm release.zip Remove the cloned learning environment. $ rm -rf learn-nomad-jobspec $ rm -rf learn-nomad-jobspec [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#review-what-you-learned) Review what you learned --------------------------------------------------------------------------------------------------------------------------------------------- Question 1: How do you create a parameterized job in Nomad? You create a parameterized job by adding the `parameterized` stanza to a batch-type Nomad job specification. Question 2: How do you run an instance of a parameterized job? You dispatch an instance of a parameterized job by running then`nomad job dispatch` command or using the Job Dispatch API. Question 3: How does Nomad supply dispatch variables to the workload? Nomad dispatch variables are supplied to the workload as environment variables. The environment variable's names are generated by prepending `NOMAD_META_` to the variable's name, like `NOMAD_META_customer_email`. Question 4: How large can a dispatch payload be? What is an important consideration when using payloads? Dispatch payloads can be up to 16KiB. Payloads are written directly into the dispatched job's definition, so they use that same amount of server RAM until the job is complete and garbage collected. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized#next-steps) Next steps ------------------------------------------------------------------------------------------------------------------- Nomad parameterized jobs are a powerful tool, encouraging reuse of a job specification by allowing users to provide runtime specific values when dispatching the job. Parameterized jobs enable job creators to make consistent and reusable experiences by encapsulating the specifics of the job and only requiring users to provide the run-time specific information. This ultimately simplifies the user experience for both end-users and API consumers. In this tutorial, you performed these job specification development actions: * Converted a batch job to a parameterized job. * Added required and optional variables to the job specification. * Defined default values for optional variables * Configured the job to use a dispatch payload. You performed these end-user operations: * Dispatched instances of a parameterized job * Provided variables during dispatch * Submitted a static and dynamic payload during dispatch You performed these cluster-operator tasks: * Retrieved logs for an allocation using the `nomad alloc logs` command Learn more about the template language used in the `.tmpl` files in the [Learn Go Template Syntax](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax) and in the [consul-template documentation](https://github.com/hashicorp/consul-template/blob/master/docs/templating-language.md) . You can also learn more about parameterized jobs in the [Nomad documentation](https://developer.hashicorp.com/nomad/docs/job-specification/parameterized) . **Was this tutorial helpful?** YesNo [Collection Overview\ \ Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) [Next\ \ Migrate a Linux Java app](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux) --- # Vault | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials/integrate-vault#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Integrate Nomad with Vault ========================== Configure Nomad to obtain secrets from Vault for Nomad workloads. Learn how to configure the Nomad secrets engine in Vault. 1 tutorial *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad) 12min Generate mTLS certificates for Nomad using Vault Create and configure Vault-managed mTLS certificates for Nomad's API and RPC traffic with Vault and consul-template. * Vault * Nomad --- # Enterprise | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials/enterprise#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Enterprise ================ Learn about Nomad Enterprise features and how to use them at scale. 5 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license) 12min Install a HashiCorp Enterprise license Add an Enterprise license to Vault, Consul, Nomad, or Boundary with environment variables, a license file, or a configuration value. * Consul * Nomad * Vault * Boundary *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) 8min Discover Nomad reference architecture Review the recommended compute and networking resources for provisioning a Nomad Enterprise cluster in a production environment. * Nomad * Consul *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul) 8min Deploy a Nomad Enterprise cluster Download, install, and configure Nomad on your infrastructure to create a Nomad Enterprise cluster. * Nomad * Consul *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts) 11min Dynamic Application Sizing concepts Discover the fundamental concepts used by the Nomad Autoscaler to enable dynamic application sizing suggestions for Nomad workloads. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing) 18min Use Dynamic Application Sizing Use the Nomad Autoscaler to enable dynamic application sizing and get suggestions on how to make your Nomad workloads more efficient. * Vagrant * Nomad * Video --- # Consul | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tutorials/integrate-consul#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Use Nomad's Consul Integration ============================== Use Nomad’s Consul integration to secure Nomad jobs and learn more about Consul health checking, service discovery, and service mesh capabilities. 4 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-acl) Interactive 19min Configure Consul ACL with Nomad Workload Identities Configure Consul ACL with Nomad Workload Identities and then run a Nomad job to verify workload access to the Consul key/value (KV) store. * Nomad * Consul *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh) 12min Secure Nomad jobs with Consul service mesh Configure Nomad for an ACL-enabled Consul cluster and run a sample job that uses Consul service mesh. * Nomad * Consul *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist) 12min Consul service mesh in production Understand best practices for running Consul service mesh in production. * Consul *  [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad) 3min Deploy a Consul API Gateway on Nomad Configure and deploy a Consul API Gateway to direct traffic to Nomad workloads. * Nomad * Consul --- # ACL system overview | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/secure/acl#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) ACL system overview =================== The ACL system is designed to be intuitive, high-performance, and to provide administrative insight. At the highest level, there are three major components to the ACL system: tokens, policies, and capabilities. The components are illustrated in the diagram below. [![Image showing that ACL Tokens refer to one or more associated policies and\ that those policies encapsulate capabilities like "Allow Submit Job" or "Allow\ Query Nodes"](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/secure/acl.jpg)](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/secure/acl.jpg) [](https://developer.hashicorp.com/nomad/docs/secure/acl#policies) Policies --------------------------------------------------------------------------- Policies consist of a set of rules defining the capabilities or actions to be granted. For example, a `readonly` policy might only grant the ability to list and inspect running jobs, but not to submit new ones. No permissions are granted by default, making Nomad a default-deny system. Each policy must have a unique name, an optional description, and a rule set. The rules define the capabilities of a Nomad ACL token for accessing the objects in a Nomad cluster, objects like namespaces, node, agent, operator, quota, etc. A client ACL token can be associated with multiple policies; a request is allowed if _any_ of the associated policies grant the capability. Management tokens cannot be associated with policies because they are granted all capabilities. The special `anonymous` policy can be defined to grant capabilities to anonymous requests. An anonymous request is a request made to Nomad without the `X-Nomad-Token` header specified. This can be used to allow anonymous users to list jobs and view their status, while requiring authenticated requests to submit new jobs or modify existing jobs. By default, there is no `anonymous` policy set meaning all anonymous requests are denied. Warning All unauthenticated requests to Nomad receive permissions from the anonymous token. To ensure clusters remain secure, we recommend using tokens with specific policies instead of an overly permissive anonymous token. [](https://developer.hashicorp.com/nomad/docs/secure/acl#roles) Roles --------------------------------------------------------------------- Roles group one or more ACL policies into a container which can then be used to generate ACL tokens for authorisation. This abstraction allows easier control and updating of ACL permissions, particularly in larger, more diverse clusters. [](https://developer.hashicorp.com/nomad/docs/secure/acl#capabilities) Capabilities ----------------------------------------------------------------------------------- Capabilities are the set of actions that can be performed. This includes listing jobs, submitting jobs, querying nodes, etc. A management token is granted all capabilities, while client tokens are granted specific capabilities via ACL policies. The full set of capabilities is discussed in the rule specifications. [](https://developer.hashicorp.com/nomad/docs/secure/acl#binding-rules) Binding rules ------------------------------------------------------------------------------------- Policies are comprised of one or more rules. The rules define the capabilities of a Nomad ACL token for accessing the objects in a Nomad cluster, objects like namespaces, node, agent, operator, quota, etc. Binding rules provide a mapping between a Nomad user's SSO authorisation claims and internal Nomad objects such as ACL Roles and ACL Policies. A binding rule is directly related to a single auth method, and therefore only evaluated by login attempts using that method. All binding rules mapped to an auth method are evaluated during each login attempt. Note Binding rules are evaluated in no specific order, and should there be an overlap in their selectors or scope, a "sum" of all the binding rules will be applied, thus the least granular binding rules will always override the more granular ones, as long as they apply to the same auth method and identity. A successful selector match between an SSO provider claim and a binding rule will result in the generated ACL token having the identified ACL role or policy assigned to it. If the `BindType` parameter is `management`, the ACL token generated will be a `management` token, rather than a client token. This matcher supersedes role or policy assignments, and therefore should be used with caution. ### [](https://developer.hashicorp.com/nomad/docs/secure/acl#rules-and-scope) Rules and scope The following table summarizes the rules that are available for constructing ACL policies: | Rule | Scope | | --- | --- | | [agent](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#agent-rules) | Utility operations in the Agent API | | [host\_volume](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#host-volume-rules) | Host Volume related operations | | [namespace](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#namespace-rules) | Job related operations by namespace | | [node](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#node-rules) | Node-level catalog operations | | [operator](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#operator-rules) | Cluster-level operations in the Operator API | | [plugin](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#plugin-rules) | CSI Plugin related operations | | [quota](https://developer.hashicorp.com/nomad/docs/secure/acl/policies#quota-rules) | Quota specification related operations | Constructing policies from these rules is covered in detail in the [Nomad ACL Policy Concepts guide](https://developer.hashicorp.com/nomad/docs/secure/acl/policies) . [](https://developer.hashicorp.com/nomad/docs/secure/acl#tokens) Tokens ----------------------------------------------------------------------- ACL tokens are used to authenticate requests and determine if the caller is authorized to perform an action. Each ACL token has a public Accessor ID which is used to name a token and a Secret ID which is used to make requests to Nomad. The Secret ID is provided using a request header (`X-Nomad-Token`) and is used to authenticate the caller. Tokens are either management or client types. The `management` tokens are effectively "root" in the system and can perform any operation. The `client` tokens are associated with one or more ACL policies or roles which grant specific capabilities. When ACL tokens are created, they can be optionally marked as `Global`. This causes them to be created in the authoritative region and replicated to all other regions. Otherwise, tokens are created locally in the region the request was made and not replicated. Local tokens cannot be used for cross-region requests since they are not replicated between regions. [](https://developer.hashicorp.com/nomad/docs/secure/acl#workload-identity) Workload Identity --------------------------------------------------------------------------------------------- Nomad allocations can receive workload identities in the form of a [JSON Web Token (JWT)](https://datatracker.ietf.org/doc/html/rfc7519) . The [Workload Identity concept page](https://developer.hashicorp.com/nomad/docs/concepts/workload-identity) has more information on this topic. [](https://developer.hashicorp.com/nomad/docs/secure/acl#authentication-methods) Authentication methods ------------------------------------------------------------------------------------------------------- Authentication methods dictate how Nomad should talk to SSO providers when a user requests to authenticate using one. Currently, Nomad supports the [OpenID Connect (OIDC)](https://openid.net/connect/) SSO workflow which allows users to log in to Nomad via applications such as [Auth0](https://auth0.com/) , [Okta](https://www.okta.com/) , and \[Vault\]\[vault\], and non-interactive login via externally-issued [JSON Web Tokens (JWT)](https://datatracker.ietf.org/doc/html/rfc7519) . Since both the `oidc` and `jwt` auth methods ultimately operate on JWTs as bearer tokens, use the following to determine which method fits your use case: * **JWT** * Ideal for machine-oriented, headless login where an operator may have already arranged for a valid JWT to be dropped on a VM or provided to a container. * User or application performing the Nomad login must have a valid JWT to begin login. * Does not require browser interaction. * **OIDC** * Ideal for human-oriented, interactive login where an operator or administrator may have deployed SSO widely and doesn't want to distribute Nomad ACL tokens to every authorized user. * User performing the Nomad login does not need a JWT. * Requires browser interaction. [](https://developer.hashicorp.com/nomad/docs/secure/acl#multi-region-configuration) Multi-Region configuration --------------------------------------------------------------------------------------------------------------- Nomad supports multi-datacenter and multi-region configurations. A single region is able to service multiple datacenters, and all servers in a region replicate their state between each other. In a multi-region configuration, there is a set of servers per region. Each region operates independently and is loosely coupled to allow jobs to be scheduled in any region and requests to flow transparently to the correct region. When ACLs are enabled, Nomad depends on an "authoritative region" to act as a single source of truth for ACL policies and global ACL tokens. The authoritative region is configured in the [`server` stanza](https://developer.hashicorp.com/nomad/docs/configuration/server) of agents, and all regions must share a single authoritative source. Any ACL policies or global ACL tokens are created in the authoritative region first. All other regions replicate ACL policies and global ACL tokens to act as local mirrors. This allows policies to be administered centrally, and for enforcement to be local to each region for low latency. Global ACL tokens are used to allow cross-region requests. Standard ACL tokens are created in a single target region and not replicated. This means if a request takes place between regions, global tokens must be used so that both regions will have the token registered. [](https://developer.hashicorp.com/nomad/docs/secure/acl#replication) Replication --------------------------------------------------------------------------------- Multi-region federated clusters run replication process to replicate ACL objects from the [authoritative region](https://developer.hashicorp.com/nomad/docs/configuration/server#authoritative_region) . The replication processes run on each federated leader and replicate ACL policies, roles, auth methods, binding rules, and token marked as `Global`. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/secure/acl/index.mdx) --- # Nomad Pack | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/tools/nomad-pack#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Pack ========== This guide will walk you through basic usage of [Nomad Pack](https://github.com/hashicorp/nomad-pack) , a package manager and templating tool for Nomad. By the end of this guide, you will know what Nomad Pack does, be able to deploy applications to Nomad using Nomad Pack, and discover packs built by the Nomad community. [](https://developer.hashicorp.com/nomad/tools/nomad-pack#what-is-nomad-pack) What is Nomad Pack? ------------------------------------------------------------------------------------------------- Nomad Pack is a templating and packaging tool used with [HashiCorp Nomad](https://developer.hashicorp.com/nomad) . Nomad Pack is used to: * Easily deploy popular applications to Nomad * Re-use common patterns across internal applications * Find and share job specifications with the Nomad community Nomad Pack can be thought of as a templating and deployment tool like [Levant](https://github.com/hashicorp/levant) with the ability to pull from remote registries and deploy multiple resources together, like [Helm](https://helm.sh/) . [](https://developer.hashicorp.com/nomad/tools/nomad-pack#requirements) Requirements ------------------------------------------------------------------------------------ * A Nomad cluster available * Nomad cluster address defined in the `NOMAD_ADDR` environment variable. Note If Nomad ACLs are enabled, a token with proper permissions must be defined in the `NOMAD_TOKEN` environment variable. [](https://developer.hashicorp.com/nomad/tools/nomad-pack#installing-nomad-pack) Installing Nomad Pack ------------------------------------------------------------------------------------------------------ To use Nomad Pack, download the binary for your system from [HashiCorp Releases](https://releases.hashicorp.com/nomad-pack) . After downloading Nomad Pack, unzip the package, and make sure that the `nomad-pack` binary is available on your PATH. Alternatively, you can install it with Homebrew if you are on MacOS. $ brew tap hashicorp/tap $ brew install hashicorp/tap/nomad-pack $ brew tap hashicorp/tap $ brew install hashicorp/tap/nomad-pack You can now run `nomad-pack`. [](https://developer.hashicorp.com/nomad/tools/nomad-pack#basic-use) Basic use ------------------------------------------------------------------------------ To get started, run the `list` command to see which packs are available to deploy. $ nomad-pack list PACK NAME | METADATA VERSION | REGISTRY NAME \-----------------------------+------------------+--------------------------------------------------- alertmanager | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 aws\_ebs\_csi | 0.1.0 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 aws\_efs\_csi | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 backstage | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 boundary | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 caddy | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 ceph | 0.1.0 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 \[...\] $ nomad-pack list PACK NAME | METADATA VERSION | REGISTRY NAME -----------------------------+------------------+--------------------------------------------------- alertmanager | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 aws_ebs_csi | 0.1.0 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 aws_efs_csi | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 backstage | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 boundary | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 caddy | 0.0.1 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 ceph | 0.1.0 | default@6bac78a905b7966ed93078cba1604fafe0fa2334 [...] The first time you run the `list` command, Nomad Pack will add a `nomad/packs` directory to your desktop user's cache directory—`$XDG_CACHE_DIR` on Linux, `~/Library/Caches` on macOS, `%AppData%` on Windows, etc. This folder stores information about cloned registries and their available packs. During initializing, Nomad Pack downloads a default registry of packs from the [Nomad Pack community registry](https://github.com/hashicorp/nomad-pack-community-registry) . To deploy one of these packs, use the `run` command. This deploys each job defined in the pack to Nomad. To deploy the `hello_world` pack, you would run the following command: $ nomad-pack run hello\_world Evaluation ID: 67835384-763b-62b0-7c41-eb98a5417e9c Job 'hello\_world' in pack deployment 'hello\_world@latest' registered successfully Pack successfully deployed. Use --name=hello\_world@latest to manage this this deployed instance with run, plan, or destroy Congrats! You deployed a simple service on Nomad. $ nomad-pack run hello_world Evaluation ID: 67835384-763b-62b0-7c41-eb98a5417e9c Job 'hello_world' in pack deployment 'hello_world@latest' registered successfully Pack successfully deployed. Use --name=hello_world@latest to manage this this deployed instance with run, plan, or destroy Congrats! You deployed a simple service on Nomad. Note The syntax for Nomad Pack is different in version 0.1. To run packs written with the old syntax, provide the `--parser-v1` flag. Each pack defines a set of variables that can be provided by the user. To get information on the pack and to see which variables can be passed in, run the `info` command. $ nomad-pack info hello\_world Pack Name hello\_world Description This deploys a simple application as a service with an optional associated consul service. Application URL https://learn.hashicorp.com/tutorials/nomad/get-started-run?in=nomad/get-started Application Author HashiCorp Pack "hello\_world" Variables: - "message" (string) - The message your application will render - "register\_consul\_service" (bool) - If you want to register a consul service for the job - "consul\_service\_name" (string) - The consul service name for the hello-world application - "consul\_service\_tags" (list of string) - The consul service name for the hello-world application - "job\_name" (string) - The name to use as the job name which overrides using the pack name - "region" (string) - The region where jobs will be deployed - "datacenters" (list of string) - A list of datacenters in the region which are eligible for task placement - "count" (number) - The number of app instances to deploy $ nomad-pack info hello_world Pack Name hello_world Description This deploys a simple application as a service with an optional associated consul service. Application URL https://learn.hashicorp.com/tutorials/nomad/get-started-run?in=nomad/get-started Application Author HashiCorp Pack "hello_world" Variables: - "message" (string) - The message your application will render - "register_consul_service" (bool) - If you want to register a consul service for the job - "consul_service_name" (string) - The consul service name for the hello-world application - "consul_service_tags" (list of string) - The consul service name for the hello-world application - "job_name" (string) - The name to use as the job name which overrides using the pack name - "region" (string) - The region where jobs will be deployed - "datacenters" (list of string) - A list of datacenters in the region which are eligible for task placement - "count" (number) - The number of app instances to deploy Values for these variables are provided using the `--var` flag. Update your pack using the following command: $ nomad-pack run hello\_world --var message=hola $ nomad-pack run hello_world --var message=hola Values can also be provided by passing in a variables file with the `-f` flag. $ tee -a ./my-variables.hcl << END message=bonjour END $ tee -a ./my-variables.hcl << END message=bonjour END $ nomad-pack run hello\_world -f ./my-variables.hcl $ nomad-pack run hello_world -f ./my-variables.hcl To see a list of deployed packs, run the `status` command $ nomad-pack status PACK NAME | REGISTRY NAME \--------------+---------------- hello\_world | default $ nomad-pack status PACK NAME | REGISTRY NAME --------------+---------------- hello_world | default To see the status of the jobs deployed by a pack, run the `status` command with the pack name. $ nomad-pack status hello\_world PACK NAME | REGISTRY NAME | DEPLOYMENT NAME | JOB NAME | STATUS \--------------+---------------+--------------------+-------------+---------- hello\_world | default | hello\_world@latest | hello\_world | pending $ nomad-pack status hello_world PACK NAME | REGISTRY NAME | DEPLOYMENT NAME | JOB NAME | STATUS --------------+---------------+--------------------+-------------+---------- hello_world | default | hello_world@latest | hello_world | pending To remove all the resources deployed by a pack, run the `destroy` command with the pack name. $ nomad-pack destroy hello\_world $ nomad-pack destroy hello_world To stop the jobs without removing them from Nomad, use the `stop` command: $ nomad-pack stop hello\_world $ nomad-pack stop hello_world ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack#adding-non-default-pack-registries) Adding non-default Pack registries When using Nomad Pack, the default registry for packs is [the Nomad Pack Community Registry](https://github.com/hashicorp/nomad-pack-community-registry) . Packs from this registry will be made automatically available. You can add more registries by using the `registry add` command. For instance, if you want to add a registry from GitLab with the alias `my_packs`, you can run the following command to download the registry and its contents. $ nomad-pack registry add my\_packs gitlab.com/mikenomitch/pack-registry $ nomad-pack registry add my_packs gitlab.com/mikenomitch/pack-registry go-getter URL is gitlab.com/mikenomitch/pack-registry Registry successfully cloned at /Users/arusso/Library/Caches/nomad/packs/nomad-pack-tmp Processing pack entries at /Users/arusso/Library/Caches/nomad/packs/nomad-pack-tmp found pack entry fabio Processing pack fabio@latest Updating pack Removing previous latest Writing pack to /Users/arusso/Library/Caches/nomad/packs/my\_packs/fabio@latest Loading cloned pack from /Users/arusso/Library/Caches/nomad/packs/my\_packs/fabio@latest Calculating SHA for latest found pack entry grafana Processing pack grafana@latest Updating pack Removing previous latest Writing pack to /Users/arusso/Library/Caches/nomad/packs/my\_packs/grafana@latest Loading cloned pack from /Users/arusso/Library/Caches/nomad/packs/my\_packs/grafana@latest Calculating SHA for latest \[...\] Try running one the packs you just added liked this nomad-pack run fabio --registry=my\_packs --ref=latest go-getter URL is gitlab.com/mikenomitch/pack-registry Registry successfully cloned at /Users/arusso/Library/Caches/nomad/packs/nomad-pack-tmp Processing pack entries at /Users/arusso/Library/Caches/nomad/packs/nomad-pack-tmp found pack entry fabio Processing pack fabio@latest Updating pack Removing previous latest Writing pack to /Users/arusso/Library/Caches/nomad/packs/my_packs/fabio@latest Loading cloned pack from /Users/arusso/Library/Caches/nomad/packs/my_packs/fabio@latest Calculating SHA for latest found pack entry grafana Processing pack grafana@latest Updating pack Removing previous latest Writing pack to /Users/arusso/Library/Caches/nomad/packs/my_packs/grafana@latest Loading cloned pack from /Users/arusso/Library/Caches/nomad/packs/my_packs/grafana@latest Calculating SHA for latest [...] Try running one the packs you just added liked this nomad-pack run fabio --registry=my_packs --ref=latest To view the available registries, including yours, run the `registry list` command. $ nomad-pack registry list REGISTRY NAME | REF | LOCAL REF | REGISTRY URL \----------------+--------+-----------+----------------------------------------------------- default | latest | 6bac78a9 | github.com/hashicorp/nomad-pack-community-registry my\_packs | latest | 7ca313c7 | gitlab.com/mikenomitch/pack-registry $ nomad-pack registry list REGISTRY NAME | REF | LOCAL REF | REGISTRY URL ----------------+--------+-----------+----------------------------------------------------- default | latest | 6bac78a9 | github.com/hashicorp/nomad-pack-community-registry my_packs | latest | 7ca313c7 | gitlab.com/mikenomitch/pack-registry To view the available packs, including yours, run the `list` command. $ nomad-pack list \[...\] fabio | 0.0.1 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 grafana | 0.1.0 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 haproxy | 0.0.1 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 hello\_world | 0.0.1 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 loki | 0.0.1 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 nginx | 0.0.1 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 nomad\_autoscaler | 0.1.0 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 traefik | 0.1.0 | my\_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 $ nomad-pack list [...] fabio | 0.0.1 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 grafana | 0.1.0 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 haproxy | 0.0.1 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 hello_world | 0.0.1 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 loki | 0.0.1 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 nginx | 0.0.1 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 nomad_autoscaler | 0.1.0 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 traefik | 0.1.0 | my_packs@7ca313c705b7966ed93078cba1604fafe0fa2334 You can deploy packs from this registry with the `run` command and the alias given to the registry, in this case `my_packs`. $ nomad-pack run nginx --registry=my\_packs Evaluation ID: aaa3c319-1928-7c35-54b0-2358841c0e96 Job 'nginx' in pack deployment 'nginx@latest' registered successfully Pack successfully deployed. Use nginx with --ref=latest to manage this this deployed instance with plan, stop, destroy, or info Nginx successfully deployed. See the Load Balancing with Nginx tutorial for more information: https://learn.hashicorp.com/tutorials/nomad/load-balancing-nginx $ nomad-pack run nginx --registry=my_packs Evaluation ID: aaa3c319-1928-7c35-54b0-2358841c0e96 Job 'nginx' in pack deployment 'nginx@latest' registered successfully Pack successfully deployed. Use nginx with --ref=latest to manage this this deployed instance with plan, stop, destroy, or info Nginx successfully deployed. See the Load Balancing with Nginx tutorial for more information: https://learn.hashicorp.com/tutorials/nomad/load-balancing-nginx [](https://developer.hashicorp.com/nomad/tools/nomad-pack#next-steps) Next steps -------------------------------------------------------------------------------- In this tutorial you learned what Nomad Pack does, how to deploy applications to Nomad using Nomad Pack, and how to discover packs built by the Nomad community. Continue on to the [Usage guide](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage) for more detailed information about how to use Nomad Pack. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/tools/nomad-pack/index.mdx) --- # Governance and policy on Nomad | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/govern#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Governance and policy on Nomad ============================== Nomad Enterprise is aimed at teams and organizations and addresses the organizational complexity of multi-team and multi-cluster deployments with collaboration and governance features. This section provides best practices and guidance for operating Nomad securely in a multi-team setting through features such as resource quotas, node pools, and Sentinel policies. [](https://developer.hashicorp.com/nomad/docs/govern#resource-quotas) Resource quotas ------------------------------------------------------------------------------------- Enterprise When many teams or users are sharing Nomad clusters, there is the concern that a single user could use more than their fair share of resources. Resource quotas provide a mechanism for cluster administrators to restrict the resources within a namespace. Quota specifications are first class objects in Nomad. A quota specification has a unique name, an optional human readable description, and a set of quota limits. The quota limits define the allowed resource usage within a region. Quota objects are shareable among namespaces. This allows an operator to define higher level quota specifications, such as a `prod-api` quota, and multiple namespaces can apply the `prod-api` quota specification. [](https://developer.hashicorp.com/nomad/docs/govern#sentinel) Sentinel ----------------------------------------------------------------------- Enterprise [![Sentinel Overview](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/govern/sentinel.jpg)](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/govern/sentinel.jpg) * **Sentinel Policies** - Policies are able to introspect on request arguments and use complex logic to determine if the request meets policy requirements. For example, a Sentinel policy may restrict Nomad jobs to only using the "docker" driver or prevent jobs from being modified outside of business hours. * **Policy Scope** - Sentinel policies declare a "scope", which determines when the policies apply. Currently the only supported scope is "submit-job", which applies to any new jobs being submitted, or existing jobs being updated. * **Enforcement Level** - Sentinel policies support multiple enforcement levels. The `advisory` level emits a warning when the policy fails, while `soft-mandatory` and `hard-mandatory` will prevent the operation. A `soft-mandatory` policy can be overridden if the user has necessary permissions. ### [](https://developer.hashicorp.com/nomad/docs/govern#sentinel-policies) Sentinel policies Each Sentinel policy has a unique name, an optional description, applicable scope, enforcement level, and a Sentinel rule definition. If multiple policies are installed for the same scope, all of them are enforced and must pass. Sentinel policies _cannot_ be used unless the ACL system is enabled. ### [](https://developer.hashicorp.com/nomad/docs/govern#policy-scope) Policy scope Sentinel policies specify an applicable scope, which limits when the policy is enforced. This allows policies to govern various aspects of the system. The following table summarizes the scopes that are available for Sentinel policies: | Scope | Description | | --- | --- | | submit-job | Applies to any jobs (new or updated) being registered | ### [](https://developer.hashicorp.com/nomad/docs/govern#enforcement-level) Enforcement level Sentinel policies specify an enforcement level which changes how a policy is enforced. This allows for more flexibility in policy enforcement. The following table summarizes the enforcement levels that are available: | Enforcement Level | Description | | --- | --- | | advisory | Issues a warning when a policy fails | | soft-mandatory | Prevents operation when a policy fails, issues a warning if overridden | | hard-mandatory | Prevents operation when a policy fails | The [`sentinel-override` capability](https://developer.hashicorp.com/nomad/docs/secure/acl#sentinel-override) is required to override a `soft-mandatory` policy. This allows a restricted set of users to have override capability when necessary. ### [](https://developer.hashicorp.com/nomad/docs/govern#multi-region-configuration) Multi-region configuration Nomad supports multi-datacenter and multi-region configurations. A single region is able to service multiple datacenters, and all servers in a region replicate their state between each other. In a multi-region configuration, there is a set of servers per region. Each region operates independently and is loosely coupled to allow jobs to be scheduled in any region and requests to flow transparently to the correct region. When ACLs are enabled, Nomad depends on an "authoritative region" to act as a single source of truth for ACL policies, global ACL tokens, and Sentinel policies. The authoritative region is configured in the [`server` stanza](https://developer.hashicorp.com/nomad/docs/configuration/server) of agents, and all regions must share a single authoritative source. Any Sentinel policies are created in the authoritative region first. All other regions replicate Sentinel policies, ACL policies, and global ACL tokens to act as local mirrors. This allows policies to be administered centrally, and for enforcement to be local to each region for low latency. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/govern/index.mdx) --- # Traffic encryption overview | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/secure/traffic#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Traffic encryption overview =========================== The Nomad agent supports encrypting all of its network traffic. There are two separate encryption systems, one for gossip traffic, and one for HTTP and RPC. Once you complete this collection you will have configured transport encryption for all of Nomad's protocols. RPC and HTTP communication is secured with mTLS. Nomad servers also communicate with a gossip protocol, Serf, which is encrypted using symmetric encryption and preshared keys. To review Nomad uses two different encryption schemes for its traffic. * HTTP - Used to communicate between CLI and Nomad agents. Secured by mTLS. * RPC - Used to communicate between Nomad agents. Secured by mTLS. * Serf - Used to communicate between Nomad servers. Frequently referred to as "gossip". Secured by a shared key. Use the [Enable TLS Encryption for Nomad](https://developer.hashicorp.com/nomad/docs/secure/traffic/tls) guide to configure Nomad for mTLS and the [Enable Gossip Encryption for Nomad](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption) to configure gossip encryption using a pre-shared key. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/secure/traffic/index.mdx) --- # Nomad clusters | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/deploy/clusters#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad clusters ============== This section contains content for creating and federating Nomad clusters. Connect nodes into a cluster, federate multi-region clusters, and configure a web UI reverse proxy. Review multi-region federation considerations and failure scenarios. Implement load balancing using Fabio, HAProxy, NGINX, or Traefik. Manage external traffic with application load balancing. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/deploy/clusters/index.mdx) --- # Failure recovery strategies | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/job-declare/failure#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Failure recovery strategies =========================== Most applications deployed in Nomad are either long running services or one time batch jobs. They can fail for various reasons like: * A temporary error in the service that resolves when its restarted. * An upstream dependency might not be available, leading to a health check failure. * Disk, Memory or CPU contention on the node that the application is running on. * The application uses Docker and the Docker daemon on that node is unresponsive. Nomad provides configurable options to enable recovering failed tasks to avoid downtime. Nomad will try to restart a failed task on the node it is running on, and also try to reschedule it on another node. Please start with one of the guides below or use the navigation on the left for details on each option: * [Local restarts](https://developer.hashicorp.com/nomad/docs/job-declare/failure/restart) * [Health check restarts](https://developer.hashicorp.com/nomad/docs/job-declare/failure/check-restart) * [Reschedule](https://developer.hashicorp.com/nomad/docs/job-declare/failure/reschedule) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/job-declare/failure/index.mdx) --- # Advanced job scheduling | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/job-scheduling#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Advanced job scheduling ======================= The Nomad [scheduler](https://developer.hashicorp.com/nomad/docs/concepts/scheduling/how-scheduling-works) uses a bin-packing algorithm to optimize the resource utilization and density of applications in your Nomad cluster. Nomad 0.9 adds new features to allow operators more fine-grained control over allocation placement. This enables use cases similar to the following: * Expressing preference for a certain class of nodes for a specific application via the [affinity stanza](https://developer.hashicorp.com/nomad/docs/job-specification/affinity) . * Spreading allocations across a datacenter, rack or any other node attribute or metadata with the [spread stanza](https://developer.hashicorp.com/nomad/docs/job-specification/spread) . Please refer to the tutorials below for using affinity and spread in Nomad 0.9. * [Preemption](https://developer.hashicorp.com/nomad/docs/job-scheduling/preemption) * [Affinity](https://developer.hashicorp.com/nomad/docs/job-scheduling/affinity) * [Spread](https://developer.hashicorp.com/nomad/docs/job-scheduling/spread) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/job-scheduling/index.mdx) --- # Upgrade Nomad | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/upgrade#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Upgrade Nomad ============= This page provides guidance on upgrading HashiCorp Nomad without causing an outage. Review the upgrade process for a new version, for moving from Nomad Community to Nomad Enterprise, and for Raft protocol version three. [](https://developer.hashicorp.com/nomad/docs/upgrade#introduction) Introduction -------------------------------------------------------------------------------- Nomad is designed to be flexible and resilient when upgrading from one Nomad version to the next. Upgrades should cause neither a Nomad nor a service outage. However, there are some restrictions to be aware of before upgrading: * Nomad strives to be backward compatible for at least 2 point release, so Nomad v1.7.x works with v1.5.x. * Nomad does _not_ support downgrading at this time. Downgrading clients requires draining allocations and removing the [data directory](https://developer.hashicorp.com/nomad/docs/configuration#data_dir) . Downgrading servers safely requires re-provisioning the cluster. * New features are unlikely to work correctly until all nodes have been upgraded. * Check the [version upgrade details page](https://developer.hashicorp.com/nomad/docs/upgrade/upgrade-specific) for important changes and backward incompatibilities. * When upgrading a Nomad Client, if it takes longer than the [`heartbeat_grace`](https://developer.hashicorp.com/nomad/docs/configuration/server#heartbeat_grace) (10s by default) period to restart, all allocations on that node may be rescheduled. Nomad supports upgrading in place or by rolling in new servers: * In Place: The Nomad binary can be updated on existing hosts. Running allocations will continue running uninterrupted. * Rolling: New hosts containing the new Nomad version may be added followed by the removal of old hosts. The old nodes must be drained to migrate running allocations to the new nodes. This guide describes both approaches. [](https://developer.hashicorp.com/nomad/docs/upgrade#upgrade-process) Upgrade Process -------------------------------------------------------------------------------------- Once you have checked the [upgrade details for the new version](https://developer.hashicorp.com/nomad/docs/upgrade/upgrade-specific) , the upgrade process is as simple as updating the binary on each host and restarting the Nomad service. At a high level we complete the following steps to upgrade Nomad: * **Add the new version** * **Check cluster health** * **Remove the old version** * **Check cluster health** * **Upgrade clients** ### [](https://developer.hashicorp.com/nomad/docs/upgrade#1-add-the-new-version-to-the-existing-cluster) 1\. Add the new version to the existing cluster While it is possible to upgrade Nomad client nodes before servers, this guide recommends upgrading servers first as many new client features will not work until servers are upgraded. In a [federated cluster](https://nomad/docs/deploy/clusters/federate-regions) , new features are not guaranteed to work until all agents in a region and the server nodes in the authoritative region are upgraded. Whether you are replacing Nomad in place on existing systems or bringing up new servers you should make changes incrementally, verifying cluster health at each step of the upgrade. On a single server, install the new version of Nomad. You can do this by joining a new server to the cluster or by replacing or upgrading the binary locally and restarting the Nomad service. Note that if you have [`leave_on_terminate`](https://developer.hashicorp.com/nomad/docs/configuration#leave_on_terminate) or [`leave_on_interrupt`](https://developer.hashicorp.com/nomad/docs/configuration#leave_on_interrupt) set, you should ensure you're using the expected signal for your upgrade process. For example, if you have `leave_on_terminate` set and you intend on updating a server in-place, you should `SIGINT` and not `SIGTERM` when shutting down the server before restarting it. ### [](https://developer.hashicorp.com/nomad/docs/upgrade#2-check-cluster-health) 2\. Check cluster health [Monitor the Nomad logs](https://developer.hashicorp.com/nomad/commands/monitor) on the remaining servers to check that the new server has joined the cluster correctly. Run `nomad agent-info` on the new servers and check that the `last_log_index` is of a similar value to the other servers. This step ensures that changes have been replicated to the new server. ubuntu@nomad-server-10-1-1-4:~$ nomad agent-info nomad bootstrap = false known\_regions = 1 leader = false server = true raft applied\_index = 53460 commit\_index = 53460 fsm\_pending = 0 last\_contact = 54.512216ms last\_log\_index = 53460 last\_log\_term = 1 last\_snapshot\_index = 49511 last\_snapshot\_term = 1 num\_peers = 2 ... ubuntu@nomad-server-10-1-1-4:~$ nomad agent-info nomad bootstrap = false known_regions = 1 leader = false server = true raft applied_index = 53460 commit_index = 53460 fsm_pending = 0 last_contact = 54.512216ms last_log_index = 53460 last_log_term = 1 last_snapshot_index = 49511 last_snapshot_term = 1 num_peers = 2 ... Continue with the upgrades across the servers making sure to do a single Nomad server at a time. You can check state of the servers with [`nomad server members`](https://developer.hashicorp.com/nomad/commands/server/members) , and the state of the client nodes with [`nomad node status`](https://developer.hashicorp.com/nomad/commands/node/status) . ### [](https://developer.hashicorp.com/nomad/docs/upgrade#3-remove-the-old-versions-from-servers) 3\. Remove the old versions from servers If you are doing an in place upgrade on existing servers this step is not necessary as the version was changed in place. If you are doing an upgrade by adding new servers and removing old servers from the fleet you need to ensure that the server has left the fleet safely. 1. Stop the service on the existing host 2. On another server issue a `nomad server members` and check the status, if the server is now in a left state you are safe to continue. 3. If the server is not in a left state, issue a `nomad server force-leave ` to remove the server from the cluster. Monitor the logs of the other hosts in the Nomad cluster over this period. ### [](https://developer.hashicorp.com/nomad/docs/upgrade#4-check-cluster-health) 4\. Check cluster health Use the same actions in step #2 above to confirm cluster health. ### [](https://developer.hashicorp.com/nomad/docs/upgrade#5-upgrade-clients) 5\. Upgrade clients Following the successful upgrade of the servers you can now update your clients using a similar process as the servers. You may either upgrade clients in-place or start new nodes on the new version. See the [Workload Migration Guide](https://developer.hashicorp.com/nomad/docs/manage/migrate-workloads) for instructions on how to migrate running allocations from the old nodes to the new nodes with the [`nomad node drain`](https://developer.hashicorp.com/nomad/commands/node/drain) command. [](https://developer.hashicorp.com/nomad/docs/upgrade#done) Done ---------------------------------------------------------------- You are now running the latest Nomad version. You can verify all Clients joined by running `nomad node status` and checking all the clients are in a `ready` state. [](https://developer.hashicorp.com/nomad/docs/upgrade#upgrading-to-nomad-enterprise) Upgrading to Nomad Enterprise ------------------------------------------------------------------------------------------------------------------ Before upgrading servers to Nomad Enterprise versions 1.6.0 and later, you should validate your enterprise license with the [`nomad license inspect` command](https://developer.hashicorp.com/nomad/commands/license/inspect) using the binary that you are upgrading to. See the [licensing FAQ](https://developer.hashicorp.com/nomad/docs/enterprise/license/faq) for more information. After that, the process of upgrading to a Nomad Enterprise version is identical to upgrading between versions of open source Nomad. The same guidance above should be followed and as always, prior to starting the upgrade please check the [specific version details](https://developer.hashicorp.com/nomad/docs/upgrade/upgrade-specific) page as some version differences may require specific steps. [](https://developer.hashicorp.com/nomad/docs/upgrade#upgrading-to-raft-protocol-3) Upgrading to Raft Protocol 3 ---------------------------------------------------------------------------------------------------------------- This section provides details on upgrading to Raft Protocol 3. Raft protocol version 3 requires Nomad running 0.8.0 or newer on all servers in order to work. Raft protocol version 2 will be removed in Nomad 1.4.0. To see the version of the Raft protocol in use on each server, use the `nomad operator raft list-peers` command. Note that the format of `peers.json` used for outage recovery is different when running with the latest Raft protocol. See [Manual Recovery Using peers.json](https://developer.hashicorp.com/nomad/docs/manage/outage-recovery#manual-recovery-using-peers-json) for a description of the required format. When using Raft protocol version 3, servers are identified by their `node-id` instead of their IP address when Nomad makes changes to its internal Raft quorum configuration. This means that once a cluster has been upgraded with servers all running Raft protocol version 3, it will no longer allow servers running any older Raft protocol versions to be added. ### [](https://developer.hashicorp.com/nomad/docs/upgrade#upgrading-a-production-cluster-to-raft-version-3) Upgrading a Production Cluster to Raft Version 3 For production raft clusters with 3 or more members, the easiest way to upgrade servers is to have each server leave the cluster, upgrade its [`raft_protocol`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_protocol) version in the `server` block (if upgrading to a version lower than v1.3.0), and then add it back. Make sure the new server joins successfully and that the cluster is stable before rolling the upgrade forward to the next server. It's also possible to stand up a new set of servers, and then slowly stand down each of the older servers in a similar fashion. For in-place raft protocol upgrades, perform the following for each server, leaving the leader until last to reduce the chance of leader elections that will slow down the process: * Stop the server. * Run `nomad server force-leave $server_name`. * If the upgrade is for a Nomad version lower than v1.3.0, update the [`raft_protocol`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_protocol) in the server's configuration file to `3`. * Restart the server. * Run `nomad operator raft list-peers` to verify that the `RaftProtocol` for the server is now `3`. * On the server, run `nomad agent-info` and check that the `last_log_index` is of a similar value to the other servers. This step ensures that raft is healthy and changes are replicating to the new server. ### [](https://developer.hashicorp.com/nomad/docs/upgrade#upgrading-a-single-server-cluster-to-raft-version-3) Upgrading a Single Server Cluster to Raft Version 3 If you are running a single Nomad server, restarting it in-place will result in that server not being able to elect itself as a leader. To avoid this, create a new [`peers.json`](https://developer.hashicorp.com/nomad/docs/manage/outage-recovery#manual-recovery-using-peers-json) file before restarting the server with the new configuration. If you have `jq` installed you can run the following script on the server's host to write the correct `peers.json` file: #!/usr/bin/env bash NOMAD\_DATA\_DIR=$(nomad agent-info -json | jq -r '.config.DataDir') NOMAD\_ADDR=$(nomad agent-info -json | jq -r '.stats.nomad.leader\_addr') NODE\_ID=$(cat "$NOMAD\_DATA\_DIR/server/node-id") cat < "$NOMAD\_DATA\_DIR/server/raft/peers.json" \[\ {\ "id": "$NODE\_ID",\ "address": "$NOMAD\_ADDR",\ "non\_voter": false\ }\ \] EOF #!/usr/bin/env bash   NOMAD_DATA_DIR=$(nomad agent-info -json | jq -r '.config.DataDir') NOMAD_ADDR=$(nomad agent-info -json | jq -r '.stats.nomad.leader_addr') NODE_ID=$(cat "$NOMAD_DATA_DIR/server/node-id")   cat < "$NOMAD_DATA_DIR/server/raft/peers.json" [\ {\ "id": "$NODE_ID",\ "address": "$NOMAD_ADDR",\ "non_voter": false\ }\ ] EOF After running this script, if the upgrade is for a Nomad version lower than v1.3.0, update the [`raft_protocol`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_protocol) in the server's configuration to `3` and restart the server. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/upgrade/index.mdx) --- # Configure Nomad task drivers | Nomad | HashiCorp Developer [Skip to main content](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#main) [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Configure Nomad task drivers ============================ Nomad's bundled task drivers integrate with the host OS to run job tasks in isolation. Review driver capabilities, client configuration, and reference information for the Docker, Isolated Fork/Exec, Java, QEMU, and Raw Fork/Exec task drivers. [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#introduction) Introduction ------------------------------------------------------------------------------------------- Nomad clients use task drivers to execute a task and provide resource isolation. Extensible task drivers provide Nomad the flexibility to support a broad set of workloads across all major operating systems. Task driver resource isolation provides a degree of separation for Nomad's client CPU, memory, and storage between tasks. Resource isolation effectiveness depends upon individual task driver implementations and underlying client operating systems. Task drivers include various security-related controls but do not use the Nomad client-to-task interface as a security boundary. Refer to the [access control guide](https://developer.hashicorp.com/nomad/docs/secure/acl) for more information on how to protect Nomad cluster operations. [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#configuration) Configuration --------------------------------------------------------------------------------------------- Refer to the [plugin block documentation](https://developer.hashicorp.com/nomad/docs/configuration/plugin) for examples on how to use the plugin block in Nomad's client configuration. Review the [Docker driver's client requirements section](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker#client-requirements) for a detailed example. [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#nomad-task-drivers) Nomad task drivers ------------------------------------------------------------------------------------------------------- The Nomad binary contains several bundled task drivers. We also support additional task driver plugins that you may install separately. | Bundled with Nomad | Plugins | | --- | --- | | [Docker](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker) | [Exec2](https://developer.hashicorp.com/nomad/plugins/drivers/exec2) | | [Isolated Fork/Exec](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/exec) | [Podman](https://developer.hashicorp.com/nomad/plugins/drivers/podman) | | [Java](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java) | [Virt](https://developer.hashicorp.com/nomad/plugins/drivers/virt) | | [QEMU](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/qemu) | | | [Raw Fork/Exec](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/raw_exec) | | [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#community-task-drivers) Community task drivers --------------------------------------------------------------------------------------------------------------- You may also use [community-supported task driver plugins](https://developer.hashicorp.com/nomad/plugins/drivers/community) . [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#use-task-drivers-in-jobs) Use task drivers in jobs ------------------------------------------------------------------------------------------------------------------- Refer to [Use Nomad task drivers in jobs](https://developer.hashicorp.com/nomad/docs/job-declare/task-driver) for usage information. [](https://developer.hashicorp.com/nomad/docs/deploy/task-driver#create-task-drivers) Create task drivers --------------------------------------------------------------------------------------------------------- Nomad's task driver architecture is pluggable, which gives you the flexibility to create your own drivers without having to recompile Nomad. Refer to the [plugin authoring guide](https://developer.hashicorp.com/nomad/plugins/author/task-driver) for details. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/deploy/task-driver/index.mdx) --- # Run a Granite AI workload on Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [AI workloads](https://developer.hashicorp.com/nomad/tutorials/ai-workloads) In this tutorial, you will use Terraform to create the underlying infrastructure on AWS and then start the Nomad cluster. Once the cluster is running, you will submit the [jobspec for an AI workload described in the previous tutorial](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/configure-ai-workload) and then generate text using the Granite LLM. Finally, you will create a new custom model by fine tuning the original. An Instruqt track is available that allows you to complete this tutorial using a hosted web-based session. The only requirement is a [compatible web browser](https://docs.instruqt.com/reference/platform/requirements) , so you do not need to install additional software on your local machine. This tutorial also describes how to run this scenario on your local machine with your own AWS account using [this tutorial's companion code repository on GitHub](https://github.com/hashicorp-education/learn-nomad-ai-workload) . Select the version you would like to run. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#create-the-nomad-cluster) Create the Nomad cluster ---------------------------------------------------------------------------------------------------------------------------------- InstruqtLocal ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#start-the-instruqt-track) Start the Instruqt track Click on the **Start interactive lab** button below to open the Instruqt track. A page overlay appears that loads the track next to these instructions. We call this overlay "the lab." You can resize the lab at any time using the two horizontal lines at the top. At the top right, click **Launch** to start the track. The track will take about six minutes to load. Click the **Start** button in the bottom right corner after the track loads. ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#view-the-lab-layout) View the lab layout The lab has two tabs visible at the top: * **CLI**, a terminal session * **Editor**, a visual code editor. During this tutorial, you will run commands in the **CLI** tab and edit code in the **Editor** tab. This tutorial uses Terraform to create the infrastructure in your AWS account and to install Nomad on the instances. ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#prerequisites) Prerequisites Running this tutorial locally requires the following software and credentials: * [AWS account](https://portal.aws.amazon.com/billing/signup?nc2=h_ct&src=default&redirect_url=https%3A%2F%2Faws.amazon.com%2Fregistration-confirmation#/start) with [credentials environment variables set](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html) locally * Terraform CLI installed locally * Nomad CLI installed locally ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#set-up-aws-access) Set up AWS access Export the default AWS region. $ export AWS_DEFAULT_REGION=us-east-1 Export your AWS credentials as Terraform variables. This action makes the credentials available to the Terraform configuration and writes them to Nomad Variables during the apply process. They are part of the application's configuration and give the application access to S3 for storing data. $ export TF_VAR_aws_access_key=$AWS_ACCESS_KEY_ID && \ export TF_VAR_aws_secret_access_key=$AWS_SECRET_ACCESS_KEY && \ export TF_VAR_aws_default_region=$AWS_DEFAULT_REGION ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#download-the-source-repository) Download the source repository This tutorial has [companion repository on GitHub](https://github.com/hashicorp-education/learn-nomad-ai-workload) that contains the Terraform configuration files to create the Nomad cluster infrastructure in AWS. Clone the repository. $ git clone https://github.com/hashicorp-education/learn-nomad-ai-workload Change to the directory. $ cd learn-nomad-ai-workload ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#set-deployment-variables) Set deployment variables The repository automates the processes for deploying infrastructure and configuring Nomad according to a customizable set of variables. Change to the `nomad` directory. $ cd nomad Create the variables file. $ touch defaults.auto.tfvars Copy the following configuration and paste it into the variables file. Then save the file. nomad/defaults.auto.tfvars aws_region = "us-east-1" aws_server_count = "1" aws_small_private_client_count = "0" aws_small_public_client_count = "1" aws_medium_private_client_count = "0" aws_medium_public_client_count = "0" aws_large_private_client_count = "1" aws_large_public_client_count = "0" ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#run-terraform) Run Terraform Go to the **CLI** tab and initialize Terraform. $ terraform init Initializing the backend... Initializing modules... Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 5.13.0 for vpc... - vpc in .terraform/modules/vpc Initializing provider plugins... - Finding hashicorp/nomad versions matching "2.3.1"... - Finding hashicorp/aws versions matching ">= 5.46.0, ~> 5.46.0"... - Finding hashicorp/random versions matching ">= 2.0.0"... - Finding hashicorp/tls versions matching "4.0.5"... - Finding hashicorp/local versions matching "2.5.1"... - Installing hashicorp/tls v4.0.5... - Installed hashicorp/tls v4.0.5 (signed by HashiCorp) - Installing hashicorp/local v2.5.1... - Installed hashicorp/local v2.5.1 (signed by HashiCorp) - Installing hashicorp/nomad v2.3.1... - Installed hashicorp/nomad v2.3.1 (signed by HashiCorp) - Installing hashicorp/aws v5.46.0... - Installed hashicorp/aws v5.46.0 (signed by HashiCorp) - Installing hashicorp/random v3.7.2... - Installed hashicorp/random v3.7.2 (signed by HashiCorp) # ... Terraform has been successfully initialized! # ... Apply the Terraform configuration. When prompted, type `yes` and then press `Enter` to confirm. Terraform automatically loads the variables file because it ends in `.auto.tfvars`. This deployment process will take a few minutes to complete. $ terraform apply # ... Plan: 56 to add, 0 to change, 0 to destroy. Changes to Outputs: + nomad_UI = (known after apply) + nomad_management_token = (known after apply) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: # ... aws_instance.server[0]: Creation complete after 3m27s [id=i-086488c9b580862f5] nomad_acl_policy.nomad_user_policy: Creating... nomad_variable.aws_configs: Creating... nomad_acl_token.nomad_user_token: Creating... nomad_acl_policy.nomad_user_policy: Creation complete after 1s [id=nomad-user] nomad_variable.aws_configs: Creation complete after 1s [id=nomad/jobs/open-webui@default] nomad_acl_token.nomad_user_token: Creation complete after 1s [id=70a59b6b-8d7a-74ed-f856-d45b224ce52b] Apply complete! Resources: 56 added, 0 changed, 0 destroyed. Outputs: nomad_UI = "https://123.123.123.123:4646" nomad_management_token = "TOKEN_VALUE" [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#cluster-architecture) Cluster architecture -------------------------------------------------------------------------------------------------------------------------- The following diagram illustrates the infrastructure currently deployed in your environment. InstruqtLocal ![Cloud diagram showing Nomad infrastructure in Instruqt deployment](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgranite-llm%252Fcloud-diagram-instruqt-light.jpg%26width%3D1312%26height%3D738%23light-theme-only&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Cloud diagram showing Nomad infrastructure in Instruqt deployment](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgranite-llm%252Fcloud-diagram-instruqt-dark.jpg%26width%3D1312%26height%3D738%23dark-theme-only&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The Nomad cluster is running on AWS and consists of four nodes: one server node and three client nodes. One client node is publicly accessible on port 80, which is the port that Open WebUI is configured to listen on. The other two client nodes are not externally accessible and can only be reached from within the cluster. The S3 bucket is for Open WebUI to store its application data. ![Cloud diagram showing Nomad infrastructure in local deployment](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgranite-llm%252Fcloud-diagram-light.jpg%26width%3D1312%26height%3D738%23light-theme-only&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Cloud diagram showing Nomad infrastructure in local deployment](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgranite-llm%252Fcloud-diagram-dark.jpg%26width%3D1312%26height%3D738%23dark-theme-only&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The Nomad cluster is running on AWS and consists of three nodes: one server node and two client nodes. One client node is publicly accessible on port 80, which is the port that Open WebUI is configured to listen on. The other client node is not externally accessible and can only be reached from within the cluster. The S3 bucket is for Open WebUI to store its application data. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#log-in-to-the-nomad-ui) Log in to the Nomad UI ------------------------------------------------------------------------------------------------------------------------------ InstruqtLocal Print the Nomad address and token values. $ cat cluster-info Open your web browser to the value of the `nomad_UI` variable. The cluster has a self-signed certificate, so your browser will likely display a warning. Accept the certificate and then proceed to the Nomad UI. Copy the token value from the `nomad_management_token` variable. Copy the token value from the `nomad_management_token` variable in the Terraform output. Click on the `nomad_UI` address in the Terraform output to open the Nomad UI in your web browser. The cluster has a self-signed certificate, so your browser will likely show a warning. Accept the certificate and then proceed to the Nomad UI. At the top right, click the profile button that displays **Anonymous Token** and then click **Sign out**. Click the profile button again to open the **Sign In** page. On the **Sign In** page, paste the token value in the **Secret ID** field and then click **Sign in with secret**. Once you are logged in, click **Clients** from the left navigation. Confirm that the AWS client nodes are listed. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#deploy-the-ollama-job) Deploy the Ollama job ---------------------------------------------------------------------------------------------------------------------------- InstruqtLocal Export the Nomad environment variables to grant the Nomad CLI permission. $ export $(cat nomad-env-vars | xargs) Export the Nomad environment variables to grant the Nomad CLI permission. $ export NOMAD_ADDR="$(terraform output -raw nomad_UI)" \ export NOMAD_TOKEN="$(terraform output -raw nomad_management_token)" \ export NOMAD_SKIP_VERIFY=true Verify that the Nomad CLI works. $ nomad status No running jobs ==> View and manage Nomad jobs in the Web UI: https://NOMAD_ADDR:4646/ui/jobs Change to the jobs directory. $ cd jobs Submit the Ollama job to Nomad. $ nomad job run ollama.nomad.hcl ==> View this job in the Web UI: https://NOMAD_ADDR:4646/ui/jobs/ollama@default ==> 2025-08-13T15:14:12Z: Monitoring evaluation "c9732cf4" 2025-08-13T15:14:12Z: Evaluation triggered by job "ollama" 2025-08-13T15:14:12Z: Evaluation within deployment: "e73b496c" 2025-08-13T15:14:12Z: Allocation "0aa44a61" created: node "b99e0618", group "ollama" 2025-08-13T15:14:12Z: Evaluation status changed: "pending" -> "complete" ==> 2025-08-13T15:14:12Z: Evaluation "c9732cf4" finished with status "complete" ==> 2025-08-13T15:14:12Z: Monitoring deployment "e73b496c" ✓ Deployment "e73b496c" successful 2025-08-13T15:15:10Z ID = e73b496c Job ID = ollama Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline ollama 1 1 1 0 2025-08-13T15:25:08Z The job appears in the list on the Nomad UI's **Jobs** page, with the `Deploying` badge listed next to the job's name. Once the job is running, the green `Healthy` badge appears next to the job. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#deploy-the-open-webui-job) Deploy the Open WebUI job ------------------------------------------------------------------------------------------------------------------------------------ InstruqtLocal Click on the **Editor** tab. Open the jobspec at the path `/nomad/openwebui.nomad.hcl`. Open the Open WebUI jobspec in a text editor. In a new browser tab, open [Bcrypt Generator](https://bcrypt-generator.com/) to generate a password and its hash value. You will use the hash value to configure the job, and the password to log in to OpenWebUI. Copy the hash value. Then, in the jobspec, replace the placeholder `BCRYPTED_PASSWORD` text with the hash value. Save the file. $ nomad job run openwebui.nomad.hcl ==> View this job in the Web UI: https://NOMAD_ADDR:4646/ui/jobs/open-webui@default ==> 2025-08-13T15:18:06Z: Monitoring evaluation "4c3a485f" 2025-08-13T15:18:06Z: Evaluation triggered by job "open-webui" 2025-08-13T15:18:06Z: Evaluation within deployment: "2d47b283" 2025-08-13T15:18:06Z: Allocation "88e42b6e" created: node "3caf3c68", group "open-webui" 2025-08-13T15:18:06Z: Evaluation status changed: "pending" -> "complete" ==> 2025-08-13T15:18:06Z: Evaluation "4c3a485f" finished with status "complete" ==> 2025-08-13T15:18:06Z: Monitoring deployment "2d47b283" ✓ Deployment "2d47b283" successful 2025-08-13T15:20:48Z ID = 2d47b283 Job ID = open-webui Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline open-webui 1 1 1 0 2025-08-13T15:30:46Z This job also appears in the Nomad UI. Wait until the job is running before you continue the tutorial. ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#use-nomad-actions-to-create-the-admin-user) Use Nomad Actions to create the admin user From the **Jobs** page in the Nomad UI, select **open-webui**. In the top right corner of the job details page, **Actions** and then select **create-admin-user**. This action registers the admin user with the password you chose and hashed with Bcrypt Generator. ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#open-the-open-webui-application) Open the Open WebUI application In your terminal, retrieve the IP address of your small client. $ nomad node status -verbose ID Node Pool DC Name Class Address Version Drain Eligibility Status 5e64f731-4400-3e51-df60-8290a17f4c69 large dc1 aws-large-private-client-0 13.217.36.111 1.10.1 false eligible ready 1b47fa05-1ca5-1b52-9996-dbda775f7ac3 small dc1 aws-small-public-client-0 3.88.150.209 1.10.1 false eligible ready In this example, the client is available at `https://3.88.150.209`. Open this address in a new tab in your browser. If your browser returns an error, try manually visiting the `http` address instead of the `https` address. The address is also available in the custom metadata attribute `externalAddress` at the bottom of the client details page in the Nomad UI. Log in to Open WebUI. Use the password you chose to generate the hash value. If you attempt to log in using the hash value instead of the password, you will encounter a log in error. User email: `admin@local.local` Password: YOUR\_CHOSEN\_PASSWORD [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#interact-with-the-granite-model) Interact with the Granite model ------------------------------------------------------------------------------------------------------------------------------------------------ The `granite-3.3` model is installed on Ollama, and it is pre-selected as the default. In the text field, type a prompt such as `Why is the ocean blue?`. Then click the **Send Message** button. After submitting the prompt, return to the Nomad UI. Open the **Jobs** page and then click on the **ollama** job. Click on the **ollama** task group in the **Recent Allocations** list. Observe the CPU and memory resource increase under the **Resource Utilization** heading while the `ollama-task` runs. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#customize-your-model) Customize your model -------------------------------------------------------------------------------------------------------------------------- Ollama supports model customization with an [Ollama model file](https://ollama.readthedocs.io/en/modelfile/) . This configuration file allows you to create new models from existing ones and share them with others. Ollama also has [its own CLI for submitting and processing model files](https://github.com/ollama/ollama?tab=readme-ov-file#create-a-model) . Use the `nomad exec` command to connect to the allocation so that you can use the Ollama CLI. This command targets the allocation in the `ollama-task` task and the `ollama` job, as defined in the jobspec. $ nomad exec -i -t -task ollama-task -job ollama /bin/bash root@72cee5b9d64f:/# ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#create-the-model-file) Create the model file Your new model will be functionally similar to the `granite3.3` model, but responses will be phrased as though a pirate is responding. Create the model file. $ cat > pirate-granite3.3.modelfile << EOF FROM granite3.3:2b # sets the temperature to 1 [higher is more creative, lower is more coherent] PARAMETER temperature 1 # sets the context window size to 4096, this controls how many tokens the LLM can use as context to generate the next token PARAMETER num_ctx 4096 # sets a custom system message to specify the behavior of the chat assistant SYSTEM You are a pirate, acting as an assistant. EOF ### [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#submit-the-model-file-to-ollama) Submit the model file to Ollama Submit the model file to Ollama to create the new model. This will take some time as Ollama downloads the base model. $ ollama create pirate-granite-3.3 -f pirate-granite3.3.modelfile gathering model components pulling manifest pulling 77bcee066a76 100% ▕███████████████████████████████████████████████████████████████████▏ 4.9 GB pulling 3da071a01bbe 100% ▕███████████████████████████████████████████████████████████████████▏ 6.6 KB pulling 4a99a6dd617d 100% ▕███████████████████████████████████████████████████████████████████▏ 11 KB pulling 122661774644 100% ▕███████████████████████████████████████████████████████████████████▏ 417 B verifying sha256 digest writing manifest success using existing layer sha256:77bcee066a76dcdd10d0d123c87e32c8ec2c74e31b6ffd87ebee49c9ac215dca using existing layer sha256:3da071a01bbe5a1aa1e9766149ff67ed2b232f63d55e6ed50e3777b74536a67f using existing layer sha256:4a99a6dd617d9f901f29fe91925d5032600fcd78f315a9fa78c1667c950a3a5f creating new layer sha256:8df85c83aa19f2d31977bdf9c3d0b725199dd3eefdf9443089f5d16d65734b00 creating new layer sha256:584e6cd273eda8359ff0d82570f61d00f0760a81545b4f7d80f8618a304b2f4f writing manifest success [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#interact-with-the-new-model) Interact with the new model ---------------------------------------------------------------------------------------------------------------------------------------- In your browser, refresh the Open WebUI web page. At the top left, click the model dropdown, which is still set to `granite3.3`. Select the `pirate-granite` model and then start a new chat. In the text field, enter the prompt `Why is the ocean blue?` and then click the **Send Message** button. The response contains the same content but is phrased as though a pirate is speaking. [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#train-ai-models-with-custom-datasets) Train AI models with custom datasets ---------------------------------------------------------------------------------------------------------------------------------------------------------- One way of extending a model's functionality is retraining it with new data. This process is more involved than fine-tuning with Ollama. It requires more time and consumes more billable cloud resources, and it is typically accomplished by scripting with another language, such as Python. For example, HuggingFace provides a [guide on training a model with a script](https://huggingface.co/docs/transformers/v4.40.1/run_scripts) . You can run these training scripts as Nomad jobs using one of the [`exec` task drivers](https://developer.hashicorp.com/nomad/docs/deploy/task-driver) . [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#clean-up) Clean up -------------------------------------------------------------------------------------------------- This scenario continues into the next tutorial, [Scale node pools to run more AI models](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/scale-ai-workload) . If you want to continue the scenario, do not clean up your cluster and instead open the [Scale node pools to run more AI models](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/scale-ai-workload) tutorial. Otherwise, follow the instructions below to clean up your cluster resources. Stop the Ollama job from the Nomad UI. Navigate to the **Jobs** page, and then click the **ollama** job. On the right side of the page, click **Stop job**. InstruqtLocal In the bottom right corner of the lab, click on the **Check** button to complete the scenario. Change back to the directory with the Terraform configurations. $ cd ../ Destroy the infrastructure with Terraform. When prompted, type `yes` and then press `Enter` to confirm. $ terraform destroy [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload#next-steps) Next Steps ------------------------------------------------------------------------------------------------------ In this tutorial you created a Nomad cluster on AWS, ran a Granite model with Ollama and Open WebUI, interacted with the Granite model to generate text, and created a new model based on Granite 3.3. In the next tutorial, you will run another Granite AI model in a different node pool and use Nomad's service discovery to enable multiple private Ollama backends for your application’s public Open WebUI frontend. **Was this tutorial helpful?** YesNo [Previous\ \ Configure a Granite AI workload](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/configure-ai-workload) [Next\ \ Scale node pools](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/scale-ai-workload) --- # Migrate a monolith to microservices | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Migrate a monolith](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith) Monolithic applications contain all of the components they need to function in a single, indivisible unit. They can be easier to develop and test because all of the application code is in the same location and it is easier to secure communication between components that are contained within the same application package. However, issues come up as monolithic applications get larger and more complex, more developers join the project, or certain components need to be scaled individually. Moving towards a microservice architecture can help solve these issues. When you are ready to transition to a microservices architecture, Consul and Nomad provide the functionality to help you deploy, connect, secure, monitor, and scale your application. In this tutorial, you will clone the code repository to your local workstation and learn about the cloud infrastructure required to complete the scenarios in this collection. [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview#collection-overview) Collection overview ---------------------------------------------------------------------------------------------------------------------------------------- This collection is composed of six different tutorials: * a general overview, provided by this tutorial, that helps you navigate across the code used in the collection; * [Set up the cluster](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-cluster-setup) guides you through the setup of a Consul and Nomad cluster, and its infrastructure is used as a prerequisite for the remaining tutorials; * four scenario tutorials, each showing the deployment of HashiCups, a demo application, on the Nomad and Consul cluster at different levels of integration: * [Deploy HashiCups](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-hashicups) demonstrates how to convert a Docker Compose configuration, used to deploy a monolithic application locally, into a Nomad job configuration file, or [jobspec](https://developer.hashicorp.com/nomad/docs/job-specification) , that deploys the same application, as a monolith, into the Nomad cluster. This scenario does not integrate Consul for the deployment. * [Integrate service discovery](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-service-discovery) demonstrates how to convert a Nomad job configuration for a monolithic application into an application deployed using Consul service discovery. The tutorial shows two different scenarios: the first, in which the application is deployed on a single Nomad node, the second, in which the application is deployed on multiple Nomad nodes, taking advantage of Nomad scheduling capabilities. * [Integrate service mesh and API gateway](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-service-mesh-gateway) demonstrates how to set up your Consul and Nomad cluster to use Consul service mesh. The tutorial includes configuration for Consul API gateway and Consul intentions that are required for application security. * [Scale a service](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-autoscale) demonstrates how to use Nomad Autoscaler to automatically scale part of the HashiCups application in response to a spike in traffic. The architectural diagrams below provide you with a visual representation of the configurations you will learn about and use in the different steps of the collection. Prerequisite: Set up the clusterScenario 1: Deploy HashiCupsScenario 2a: Integrate service discovery on a single VMScenario 2b: Integrate service discovery on multiple VMsScenario 3: Integrate service mesh and API gatewayScenario 4: Scale a service ![Architectural diagram. Consul and Nomad cluster with one public client and three private clients.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_00_nomad-and-consul-cluster_initial-state.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. Consul and Nomad cluster with one public client and three private clients.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_00_nomad-and-consul-cluster_initial-state_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The cluster consists of three server nodes, three private client nodes, and one publicly accessible client node. Each node runs the [Consul agent](https://developer.hashicorp.com/consul/docs/fundamentals/agent) and [Nomad agent](https://developer.hashicorp.com/nomad/commands/agent) . The agents run in either server or client mode depending on the role of the node. ![Architectural diagram. Hashicups, deployed as a monolith, running on the public node of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_01_hashicups-monolith.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. Hashicups, deployed as a monolith, running on the public node of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_01_hashicups-monolith_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The initial version of HashiCups represents a conversion from a Docker Compose file to a Nomad jobspec. It has the following attributes: * All services run on the same node: the Nomad public client node * Services are configured to use the Nomad client IP address or `localhost` * No service health monitoring * No scaling of services * No secure connection (HTTPS) ![Architectural diagram. Hashicups, deployed as a monolith, running on the public node of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_02_hashicups-sd-single-vm.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. Hashicups, deployed as a monolith, running on the public node of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_02_hashicups-sd-single-vm_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) This version of HashiCups integrates Consul and uses the service discovery features. It has the following attributes: * All services run on the same node: the Nomad public client node * Services are registered in Consul * Health checks have been implemented * Services use Consul DNS (`.service..`) * No scaling of services * No secure connection (HTTPS) ![Architectural diagram. Hashicups, deployed as a monolith, running on multiple nodes of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_03_hashicups-sd-multi-vm.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. Hashicups, deployed as a monolith, running on multiple nodes of the cluster.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_03_hashicups-sd-multi-vm_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) This version of HashiCups modifies the job so that every service except **nginx** runs on one of the private client nodes. It has the following attributes: * Job has been split into multiple groups with each one containing one application service * Services are configured with `constraints` to now run on either the public Nomad node or private nodes * Services still use Consul DNS (`.service..`) * No scaling of services * No secure connection (HTTPS) ![Architectural diagram. HashiCups running in Consul service mesh and accessible using Consul API gateway.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_05_hashicups-sm-api-gw.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. HashiCups running in Consul service mesh and accessible using Consul API gateway.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_05_hashicups-sm-api-gw_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) This version of HashiCups has the following attributes: * All services now run on Nomad private client nodes * Consul service mesh integration with upstream service configurations * An API gateway running on the Nomad public client node * Nomad Workload Identity to automatically generate ACL tokens for the API gateway * Consul service intentions * Secure connections with custom TLS certificates and mTLS with Envoy proxies. Services in Consul service mesh connect to upstreams using the `localhost` address. ![Architectural diagram. HashiCups running in Consul service mesh automatically scaling with Nomad autoscaler.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_08_hashicups-sm-autoscaler-traffic.png%26width%3D1024%26height%3D768%23light-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Architectural diagram. HashiCups running in Consul service mesh automatically scaling with Nomad autoscaler.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmonolith-migration%252Farch_08_hashicups-sm-autoscaler-traffic_dark.png%26width%3D1024%26height%3D768%23dark-theme-only&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) This version of HashiCups has the following attributes: * All services now run on Nomad private client nodes * Consul service mesh integration with upstream service configurations * An API gateway running on the Nomad public client node * Nomad Workload Identity to automatically generate ACL tokens for the API gateway * Consul service intentions * Secure connections with custom TLS certificates and mTLS with Envoy proxies. Services in Consul service mesh connect to upstreams using the `localhost` address. * Service **frontend** configuration includes a scaling section that defines desired behavior for the service * A Nomad Autoscaler running in the Nomad cluster Note With the exception of the prerequisite tutorial that sets up the Nomad and Consul cluster, none of the other tutorials are mandatory. They are intended to show the progression of deployment maturity and the different integrations available between Consul and Nomad. You can decide which of the deployments suit your current scenario better and learn how to perform it without having to follow the other scenario tutorials. [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview#review-the-code-repository) Review the code repository ------------------------------------------------------------------------------------------------------------------------------------------------------ The infrastructure creation flow consists of three steps: 1. Create the Amazon Machine Image (AMI) with Packer. 2. Provision the infrastructure with Terraform. 3. Set up access to the CLI and UI for both Consul and Nomad. Clone the [`hashicorp-education/learn-consul-nomad-vm` code repository](https://github.com/hashicorp-education/learn-consul-nomad-vm) to your local workstation. $ git clone https://github.com/hashicorp-education/learn-consul-nomad-vm.git Change to the directory of the local repository. $ cd learn-consul-nomad-vm ### [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview#the-aws-directory) The `aws` directory View the structure of the `aws` directory. It contains the configuration files for creating the AMI and cluster infrastructure. $ tree aws aws ├── aws-ec2-control_plane.tf ├── aws-ec2-data_plane.tf ├── aws_base.tf ├── consul_configuration.tf ├── image.pkr.hcl ├── outputs.tf ├── providers.tf ├── secrets.tf ├── variables.hcl.example └── variables.tf 1 directory, 10 files * The `aws-ec2-control_plane.tf` file contains configuration for creating the servers while `aws-ec2-data_plane.tf` contains configuration for creating the clients. Both are structured similarly. Click here to review the aws-ec2-control\_plane.tf file. aws-ec2-control\_plane.tf resource "aws_instance" "server" { depends_on = [module.vpc] count = var.server_count ami = var.ami instance_type = var.server_instance_type key_name = aws_key_pair.vm_ssh_key-pair.key_name associate_public_ip_address = true vpc_security_group_ids = [\ aws_security_group.consul_nomad_ui_ingress.id, \ aws_security_group.ssh_ingress.id, \ aws_security_group.allow_all_internal.id\ ] subnet_id = module.vpc.public_subnets[0] # instance tags # ConsulAutoJoin is necessary for nodes to automatically join the cluster tags = { Name = "${local.name}-server-${count.index}", ConsulJoinTag = "auto-join-${random_string.suffix.result}", NomadType = "server" } # ... user_data = templatefile("${path.module}/../shared/data-scripts/user-data-server.sh", { domain = var.domain, datacenter = var.datacenter, server_count = "${var.server_count}", consul_node_name = "consul-server-${count.index}", cloud_env = "aws", retry_join = local.retry_join_consul, consul_encryption_key = random_id.consul_gossip_key.b64_std, consul_management_token = random_uuid.consul_mgmt_token.result, nomad_node_name = "nomad-server-${count.index}", nomad_encryption_key = random_id.nomad_gossip_key.b64_std, nomad_management_token = random_uuid.nomad_mgmt_token.result, ca_certificate = base64gzip("${tls_self_signed_cert.datacenter_ca.cert_pem}"), agent_certificate = base64gzip("${tls_locally_signed_cert.server_cert[count.index].cert_pem}"), agent_key = base64gzip("${tls_private_key.server_key[count.index].private_key_pem}") }) # ... # Waits for cloud-init to complete. Needed for ACL creation. provisioner "remote-exec" { inline = [\ "echo 'Waiting for user data script to finish'",\ "cloud-init status --wait > /dev/null"\ ] } iam_instance_profile = aws_iam_instance_profile.instance_profile.name # ... } * The `aws_base.tf` file contains configuration for creating the Virtual Private Cloud (VPC), security groups, and IAM configurations. This file defines the ingress ports for Consul, Nomad, and the HashiCups application. Click here to review the aws\_base.tf file. aws\_base.tf # ... resource "aws_security_group" "consul_nomad_ui_ingress" { name = "${local.name}-ui-ingress" vpc_id = module.vpc.vpc_id # Nomad UI ingress { from_port = 4646 to_port = 4646 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } # Consul UI ingress { from_port = 8443 to_port = 8443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } # ... resource "aws_security_group" "clients_ingress" { name = "${local.name}-clients-ingress" vpc_id = module.vpc.vpc_id # ... # Add application ingress rules here # These rules are applied only to the client nodes # HTTP ingress ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } # HTTPS ingress ingress { from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } # HTTPS ingress ingress { from_port = 8443 to_port = 8443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } } * The `image.pkr.hcl` file contains the configuration to create an AMI using a Ubuntu 22.04 base image. Packer copies the `shared` directory from the root of the code repository to the machine image and runs the `shared/scripts/setup.sh` script. Click here to review the image.pkr.hcl file. image.pkr.hcl # ... data "amazon-ami" "hashistack" { filters = { architecture = "x86_64" "block-device-mapping.volume-type" = "gp2" name = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["099720109477"] region = var.region } # ... build { # ... provisioner "shell" { inline = ["sudo mkdir -p /ops/shared", "sudo chmod 777 -R /ops"] } provisioner "file" { destination = "/ops" source = "../shared" } provisioner "shell" { environment_vars = ["INSTALL_NVIDIA_DOCKER=false", "CLOUD_ENV=aws"] script = "../shared/scripts/setup.sh" } } * The `secrets.tf` file contains configuration for creating gossip encryption keys, TLS certificates for the server and client nodes, and ACL policies and tokens for both Consul and Nomad. * The `variables.hcl.example` file is the configuration file template used by Packer when building the AMI and Terraform when provisioning infrastructure. A copy is made of this file during cluster creation and updated with the AWS region and AMI ID after Packer builds the image. It also contains configurable variables for the cluster and their default values. * The `variables.tf` file defines the variables used by Terraform and includes resource naming configurations, node types and counts, and Consul configurations for [cluster auto-joining](https://developer.hashicorp.com/consul/docs/install/cloud-auto-join#amazon-ec2-and-ecs) and additional cluster configuration. Click here to review the variables.tf file. variables.tf # Random suffix for Auto-join and resource naming resource "random_string" "suffix" { length = 4 special = false upper = false } # Prefix for resource names variable "prefix" { description = "The prefix used for all resources in this plan" default = "learn-consul-nomad-vms" } # Random prefix for resource names locals { name = "${var.prefix}-${random_string.suffix.result}" } # Random Auto-Join for Consul servers # Nomad servers will use Consul to join the cluster locals { retry_join_consul = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-${random_string.suffix.result}" } # ... ### [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview#the-shared-directory) The `shared` directory Next, view the structure of the `shared` directory. It contains the configuration files for creating the server and client nodes, the Nomad job specification files for HashiCups, and additional scripts. $ tree shared shared ├── conf │   ├── agent-config-consul_client.hcl │   ├── agent-config-consul_server.hcl │   ├── agent-config-consul_server_tokens.hcl │   ├── agent-config-consul_server_tokens_bootstrap.hcl │   ├── agent-config-consul_template.hcl │   ├── agent-config-nomad_client.hcl │   ├── agent-config-nomad_server.hcl │   ├── agent-config-vault.hcl │   ├── systemd-service-config-resolved.conf │   └── systemd-service-consul_template.service ├── data-scripts │   ├── user-data-client.sh │   └── user-data-server.sh ├── jobs │   ├── 01.hashicups.nomad.hcl │   ├── 02.hashicups.nomad.hcl │   ├── 03.hashicups.nomad.hcl │   ├── 04.api-gateway.config.sh │   ├── 04.api-gateway.nomad.hcl │   ├── 04.hashicups.nomad.hcl │   ├── 04.intentions.consul.sh │   ├── 05.autoscaler.config.sh │   ├── 05.autoscaler.nomad.hcl │   ├── 05.hashicups.nomad.hcl │   └── 05.load-test.sh └── scripts ├── setup.sh └── unset_env_variables.sh 5 directories, 25 files The `shared/conf` directory contains the agent configuration files for the Consul and Nomad server and client nodes. It also contains `systemd` configurations for setting up Consul as the DNS. * The `shared/data-scripts/user-data-server.sh` and `shared/data-scripts/user-data-client.sh` scripts are run by Terraform during the provisioning process for the server and client nodes respectively once the virtual machine's initial setup is complete. The scripts configure and start the Consul and Nomad agents by retrieving certificates, exporting environment variables, and starting the agent services. The `user-data-server.sh` script additionally bootstraps the Nomad ACL system. Click here to review the user-data-server.sh file. user-data-server.sh # ... # Copy template into Consul configuration directory sudo cp $CONFIG_DIR/agent-config-consul_server.hcl $CONSUL_CONFIG_DIR/consul.hcl set -x # Populate the file with values from the variables sudo sed -i "s/_CONSUL_DATACENTER/$CONSUL_DATACENTER/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s/_CONSUL_DOMAIN/$CONSUL_DOMAIN/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s/_CONSUL_NODE_NAME/$CONSUL_NODE_NAME/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s/_CONSUL_SERVER_COUNT/$CONSUL_SERVER_COUNT/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s/_CONSUL_BIND_ADDR/$CONSUL_BIND_ADDR/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s/_CONSUL_RETRY_JOIN/$CONSUL_RETRY_JOIN/g" $CONSUL_CONFIG_DIR/consul.hcl sudo sed -i "s#_CONSUL_ENCRYPTION_KEY#$CONSUL_ENCRYPTION_KEY#g" $CONSUL_CONFIG_DIR/consul.hcl # ... # Start Consul echo "Start Consul" sudo systemctl enable consul.service sudo systemctl start consul.service # ... # Create Nomad server token to interact with Consul OUTPUT=$(CONSUL_HTTP_TOKEN=$CONSUL_MANAGEMENT_TOKEN consul acl token create -description="Nomad server auto-join token for $CONSUL_NODE_NAME" --format json -templated-policy="builtin/nomad-server") CONSUL_AGENT_TOKEN=$(echo "$OUTPUT" | jq -r ".SecretID") # Copy template into Nomad configuration directory sudo cp $CONFIG_DIR/agent-config-nomad_server.hcl $NOMAD_CONFIG_DIR/nomad.hcl # Populate the file with values from the variables sudo sed -i "s/_NOMAD_DATACENTER/$NOMAD_DATACENTER/g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s/_NOMAD_DOMAIN/$NOMAD_DOMAIN/g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s/_NOMAD_NODE_NAME/$NOMAD_NODE_NAME/g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s/_NOMAD_SERVER_COUNT/$NOMAD_SERVER_COUNT/g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s#_NOMAD_ENCRYPTION_KEY#$NOMAD_ENCRYPTION_KEY#g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s/_CONSUL_IP_ADDRESS/$CONSUL_PUBLIC_BIND_ADDR/g" $NOMAD_CONFIG_DIR/nomad.hcl sudo sed -i "s/_CONSUL_AGENT_TOKEN/$CONSUL_AGENT_TOKEN/g" $NOMAD_CONFIG_DIR/nomad.hcl echo "Start Nomad" sudo systemctl enable nomad.service sudo systemctl start nomad.service # ... * The `shared/jobs` folder contains all of the HashiCups jobspecs and any associated script files for additional components like the API gateway and the Nomad Autoscaler. The other tutorials in this collection will explain each of them. * The `shared/scripts/setup.sh` file is the script run by Packer during the image creation process. This script installs the Docker and Java dependencies as well as the Consul and Nomad binaries. Click here to review the setup.sh file. setup.sh # ... # Docker distro=$(lsb_release -si | tr '[:upper:]' '[:lower:]') sudo apt-get install -y apt-transport-https ca-certificates gnupg2 curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/${distro} $(lsb_release -cs) stable" sudo apt-get update sudo apt-get install -y docker-ce # Java sudo add-apt-repository -y ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install -y openjdk-8-jdk JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::") # Install HashiCorp Apt Repository wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list # Install HashiStack Packages sudo apt-get update && sudo apt-get -y install \ consul=$CONSULVERSION* \ nomad=$NOMADVERSION* \ vault=$VAULTVERSION* \ consul-template=$CONSULTEMPLATEVERSION* # ... * The `shared/scripts/unset_env_variables.sh` script unsets local environment variables in your CLI before the infrastructure destruction process with Terraform. [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview#next-steps) Next steps ---------------------------------------------------------------------------------------------------------------------- In this tutorial, you became familiar with the infrastructure set up process for the cluster. In the next tutorial, you will create the cluster running Consul and Nomad and set up access to each of their command line and user interfaces. **Was this tutorial helpful?** YesNo [Collection Overview\ \ Migrate a monolith](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith) [Next\ \ Set up the cluster](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-cluster-setup) --- # Cluster Setup | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Cluster Setup ============= Provision a Nomad cluster in the Cloud with Consul and Access Control Lists (ACLs) enabled. [Start](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-overview) 4 tutorials 1.  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-overview) 11min Nomad clusters on the cloud Discover how to create a Nomad cluster with ACLs enabled and use it as a starting point for your Nomad workloads on AWS, Azure, or GCP. 2.  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws) 30min Set up a Nomad cluster on AWS Create a Nomad cluster with Consul on AWS and then enable ACLs. Use this cluster as a starting point for your Nomad workloads. * Nomad * Packer * Terraform 3.  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-gcp) 30min Set up a Nomad cluster on GCP Create a Nomad cluster with Consul on GCP and then enable ACLs. Use this cluster as a starting point for your Nomad workloads. * Nomad * Packer * Terraform 4.  [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-azure) 30min Set up a Nomad cluster on Azure Create a Nomad cluster with Consul on Azure and then enable ACLs. Use this cluster as a starting point for your Nomad workloads. * Nomad * Packer * Terraform --- # Job Specifications | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Create Nomad Job Specifications =============================== Use HCL to declaratively specify your jobs and their dependencies for deployment into a Nomad cluster. 5 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized) 19min Create a parameterized Nomad job Create a parameterized job and use dispatch variables to provide jobs with run-specific metadata values. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux) 15min Migrate a Linux-based Java application to Nomad Configure and deploy a Java workload from a Linux or macOS machine using the Java task driver in Nomad. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows) 15min Migrate a Windows-based Java application to Nomad Configure and deploy a Java workload from a Windows machine using the Java task driver in Nomad. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant) 30min Template Nomad jobspecs with Levant Create Nomad job specifications, or jobspecs, with Levant. Create a templated jobspec, optimize the jobspec, and submit it to Nomad. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs) 45min Template abstract job specs with Levant Template a Nomad job with Levant and create an abstract jobspec that you can use to render jobs based on user-supplied values. * Nomad --- # Edge Computing | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Orchestrate Edge Services with Nomad ==================================== Use Nomad to schedule edge workloads closer to your users. Connect edge services with Nomad's native service discovery. Seamlessly handle unstable Nomad client node connections. 1 tutorial *  [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services) 25min Schedule edge Services with native service discovery Schedule a demo application on the edge, use Nomad's native service discovery to connect edge services, and simulate unstable edge connections to learn how Nomad gracefully handles disconnected clients. * Nomad * Terraform * Packer --- # Migrate a monolith | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Migrate a monolithic application ================================ Migrate a monolithic application to microservices, integrate Consul, and run the application on Nomad. [Start](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview) 6 tutorials 1.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-overview) 10min Migrate a monolith to microservices Migrate a monolithic application to microservices using Consul and Nomad. * Nomad * Consul 2.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-cluster-setup) 15min Set up the cluster with Consul and Nomad Set up a cluster that runs both Consul and Nomad. * Nomad * Consul * Terraform * Packer 3.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-hashicups) 10min Deploy HashiCups Deploy the containerized version of HashiCups. * Nomad * Consul 4.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-service-discovery) 10min Integrate service discovery Integrate Consul service discovery into HashiCups. * Nomad * Consul 5.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-service-mesh-gateway) 10min Integrate service mesh and API gateway Integrate Consul service mesh into HashiCups and deploy an API gateway for external access. * Nomad * Consul 6.  [](https://developer.hashicorp.com/nomad/tutorials/migrate-monolith/monolith-migration-autoscale) 10min Use the Nomad Autoscaler to scale a service Use the Nomad Autoscaler to scale a HashiCups service based on incoming traffic. * Nomad * Consul --- # Advanced Scheduling | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Define Application Placement Preferences ======================================== Express placement preferences for job allocations within your cluster using affinities and spread. Prevent priority inversion with preemption. 1 tutorial *  [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription) Interactive 13min Oversubscribe memory Enable and configure a job for memory oversubscription to better allocate jobs with inconsistent load spikes or front-loaded memory footprints. * Nomad --- # AI workloads | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) AI workloads on Nomad ===================== Learn how to use Nomad to schedule AI workloads by running Ollama and Open WebUI with the Granite LLM. 3 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/configure-ai-workload) 10min AI workloads on Nomad - Overview Configure a Granite AI workload with Ollama and Open WebUI. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/run-ai-workload) Interactive 15min Run a Granite AI workload on Nomad Create a Nomad cluster on AWS and run a Granite LLM workload with Ollama and Open WebUI. * Nomad * Terraform *  [](https://developer.hashicorp.com/nomad/tutorials/ai-workloads/scale-ai-workload) 10min Scale node pools to run more AI models Scale the cluster to accommodate different AI model sizes. Use Nomad node pools to more efficiently place jobs and better leverage available cluster resources. * Nomad * Terraform --- # Workload identity | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Federated workload identity =========================== Integrate your Nomad cluster with cloud providers or any service that supports JWT/OIDC federated identities. 1 tutorial *  [](https://developer.hashicorp.com/nomad/tutorials/fed-workload-identity/integration-gcp) 9min Federate access to GCP with Nomad Workload Identity Federate access to Google Cloud Platform (GCP) services. Use Nomad's Workload Identity to upload a file to a private Google Cloud Storage bucket. * Nomad --- # Windows | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad on Windows ================ Nomad schedules both container and non-container workloads in Windows. Learn more in this collection. [Start](https://developer.hashicorp.com/nomad/tutorials/windows/windows-agent) 1 tutorial 1.  [](https://developer.hashicorp.com/nomad/tutorials/windows/windows-agent) 4min Run Nomad as a Windows service Install and run Nomad as a service on a Windows server. * Nomad --- # Nomad Variables | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Variables =============== Store encrypted configuration data in Nomad and make it accessible to tasks. [Start](https://developer.hashicorp.com/nomad/tutorials/variables/variables-create) 2 tutorials 1.  [](https://developer.hashicorp.com/nomad/tutorials/variables/variables-create) 6min Create and update Nomad Variables Create, store, and update Nomad Variables with the Nomad CLI. * Nomad 2.  [](https://developer.hashicorp.com/nomad/tutorials/variables/variables-acls) 5min Configure access control for Nomad Variables Create namespaces, ACL policies, and tokens to configure the ACL system for access to Nomad Variables stored in Nomad's state store. * Nomad --- # Manage Clusters | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Operating Nomad Clusters ======================== Learn the features operators will need to build and maintain Nomad clusters. 2 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-metrics) 11min Use Prometheus to monitor Nomad metrics Enable telemetry on your Nomad server and client nodes and use Prometheus to collect cluster metrics. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-service-mesh-metrics) 9min Monitor job service metrics with Prometheus, Grafana, and Consul Deploy Prometheus and Grafana within the Consul service mesh. Then set up service intentions, configure ingress, and use Consul service discovery to collect job service metrics. * Nomad * Consul --- # Autoscaling | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Dynamically Resize with Nomad Autoscaler ======================================== Automatically maintain your cluster and workload instance count to respond to demand while minimizing over-provisioning cost. 5 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/autoscaler-vagrant-demo) 12min Scale an application with the Nomad Autoscaler Set up a local Nomad cluster with Vagrant and then use the Nomad Autoscaler to automatically scale an application horizontally in response to increased application load. * Vagrant * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling) 40min Scale your Nomad cluster with horizontal cluster autoscaling Deploy an application and apply load so that the Nomad Autoscaler automatically adds new client nodes to scale the cluster horizontally. * Nomad * Terraform * Packer *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling-on-demand-batch) 40min Dynamically scale a Nomad cluster with the Nomad Autoscaler When high resource usage batch jobs need to run, use the Nomad Autoscaler to dynamically resize your Nomad cluster. * Terraform * Packer * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts) 11min Dynamic Application Sizing concepts Discover the fundamental concepts used by the Nomad Autoscaler to enable dynamic application sizing suggestions for Nomad workloads. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing) 18min Use Dynamic Application Sizing Use the Nomad Autoscaler to enable dynamic application sizing and get suggestions on how to make your Nomad workloads more efficient. * Vagrant * Nomad * Video --- # Service Discovery on Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Service Discovery on Nomad ========================== Learn how to use and implement Nomad's Native Service Discovery. [Start](https://developer.hashicorp.com/nomad/tutorials/service-discovery/service-discovery-app-deployment) 2 tutorials 1.  [](https://developer.hashicorp.com/nomad/tutorials/service-discovery/service-discovery-app-deployment) 14min Deploy an app with Nomad service discovery Deploy the HashiCups example application with Nomad service discovery and then enable load balancing with Nomad's service mesh. * Nomad 2.  [](https://developer.hashicorp.com/nomad/tutorials/service-discovery/service-discovery-consul-conversion) 6min Convert from Nomad to Consul service discovery Convert the HashiCups jobspec to use Consul service discovery instead of Nomad's native service discovery. * Nomad * Consul --- # Templates | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Use Templating with Nomad ========================= Create dynamic Nomad jobs with templating. 2 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/templates/dry-jobs-levant) 30min Template Nomad jobspecs with Levant Create Nomad job specifications, or jobspecs, with Levant. Create a templated jobspec, optimize the jobspec, and submit it to Nomad. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/templates/levant-abstract-jobs) 45min Template abstract job specs with Levant Template a Nomad job with Levant and create an abstract jobspec that you can use to render jobs based on user-supplied values. * Nomad --- # Load Balancing | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Load Balancer Integrations ========================== Integrate your Nomad cluster with several popular load balancers and leverage application load balancing for external traffic. 6 tutorials *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing) 3min Load balancer deployment considerations Discover the available load balancing applications to deploy and best practices for cluster performance and high availability. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-fabio) 20min Set up load balancing with Fabio Create and deploy a load balancing Fabio job and use the native Consul integration. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-haproxy) 5min Set up load balancing with HAProxy Create and deploy an HAProxy job and use the native Consul integration to load balance your application. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-nginx) 6min Set up load balancing with NGINX Create and deploy an NGINX job, generate a config with the template stanza, and then dynamically reload it. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-traefik) 5min Set up load balancing with Traefik Create and deploy a load balancing Traefik job that natively integrates with the Consul catalog provider. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/load-balancing/external-application-load-balancing) 23min Manage external traffic with application load balancing Deploy an Application Load Balancer on AWS to allow external Internet traffic to your internally load balanced applications running in Nomad. * Nomad * Terraform * Packer --- # Libraries and SDKs - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Client Libraries & SDKs ======================= The programming libraries listed on this page can be used to consume the API more conveniently. Some are officially maintained while others are provided by the community. [](https://developer.hashicorp.com/nomad/api-docs/libraries-and-sdks#official-libraries) Official Libraries ----------------------------------------------------------------------------------------------------------- * [`api`](https://github.com/hashicorp/nomad/tree/main/api) - Official Golang client for the Nomad HTTP API ([GoDoc](https://pkg.go.dev/github.com/hashicorp/nomad/api) ) [](https://developer.hashicorp.com/nomad/api-docs/libraries-and-sdks#third-party-and-unsupported-libraries) Third-Party and Unsupported Libraries ------------------------------------------------------------------------------------------------------------------------------------------------- * [`python-nomad`](https://github.com/jrxfive/python-nomad) - Python client for the Nomad HTTP API. * [`nomad-java-sdk`](https://github.com/hashicorp/nomad-java-sdk) - Official Java client for the Nomad HTTP API. * [`nomad-scala-sdk`](https://github.com/hashicorp/nomad-scala-sdk) - Official Scala client for the Nomad HTTP API. * [`nomad-ruby`](https://github.com/hashicorp/nomad-ruby) - Ruby client for the Nomad HTTP API. * [`nomad-client`](https://github.com/barakb/nomad-client) - A non-blocking Kotlin Nomad client. * [`hostel`](https://github.com/LKNSI/hostel) - Nomad Client for NodeJS. Includes property checking, and an Object-based API. * [`nix-nomad`](https://github.com/tristanpemble/nix-nomad) - HashiCorp Nomad job definitions in Nix _Want to see your library here? [Submit a Pull Request](https://github.com/hashicorp/nomad) ._ [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/libraries-and-sdks.mdx) --- # Task driver plugins | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Task driver plugins =================== This section provides reference information for task driver plugins. [](https://developer.hashicorp.com/nomad/plugins/drivers#introduction) Introduction ----------------------------------------------------------------------------------- Nomad clients use task drivers to execute a task and provide resource isolation. Extensible task drivers provide Nomad the flexibility to support a broad set of workloads across all major operating systems. Task driver resource isolation provides a degree of separation for Nomad's client CPU, memory, and storage between tasks. Resource isolation effectiveness depends upon individual task driver implementations and underlying client operating systems. Task drivers include various security-related controls but do not use the Nomad client-to-task interface as a security boundary. Refer to the [access control guide](https://developer.hashicorp.com/nomad/docs/secure/acl) for more information on how to protect Nomad cluster operations. [](https://developer.hashicorp.com/nomad/plugins/drivers#configuration) Configuration ------------------------------------------------------------------------------------- Refer to individual task driver documentation for agent configuration and job specification usage. [](https://developer.hashicorp.com/nomad/plugins/drivers#create-task-drivers) Create task drivers ------------------------------------------------------------------------------------------------- Nomad's task driver architecture is pluggable, which gives you the flexibility to create your own drivers without having to recompile Nomad. Refer to the [plugin authoring guide](https://nomadproject.io/nomad/plugins/author/task-driver) for details. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/drivers/index.mdx) --- # Alertmanager | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/alertmanager/v0.0.1) Alertmanager @hashicorp The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts. v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/alertmanager/v0.0.1) * Community * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/alertmanager) (opens in new tab) Alertmanager ============ The [Alertmanager](https://developer.hashicorp.com/nomad/integrations/hashicorp/alertmanager#) handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts. This pack deploys a single instance of an alertmanager application using the `prom/alertmanager` Docker image and Consul Service named "alertmanager". [](https://developer.hashicorp.com/nomad/integrations/hashicorp/alertmanager#dependencies) Dependencies ------------------------------------------------------------------------------------------------------- This pack requires Linux clients to run properly. --- # Create custom packs | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Create custom packs =================== This guide will walk you through the steps involved in writing your own packs and registries for [Nomad Pack](https://github.com/hashicorp/nomad-pack) . In this guide, you will learn: * how packs and pack registries are structured * how to write a custom pack * how to test your pack locally * how to deploy a custom pack [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#create-a-custom-registry) Create a custom registry ------------------------------------------------------------------------------------------------------------------------- First, you need to create a pack registry. This will be a repository that provides the structure, templates, and metadata that define your custom packs. To get started, use the `generate` command to create a new registry. Then, move into the directory of the registry. $ nomad-pack generate registry my_nomad_packs $ cd my_nomad_packs/ Each registry should have a `README.md` file that describes the packs in it, and top-level directories for each pack. Conventionally, the directory name matches the pack name. The top level of a pack registry looks like the following: . └── README.md └── CHANGELOG.md └── packs └── └── ...pack contents... └── └── ...pack contents... └── ...packs... [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#add-a-new-pack) Add a new pack ----------------------------------------------------------------------------------------------------- To add a new pack to your registry, create a new directory in the `packs` subdirectory. $ mkdir -p packs/hello_pack && cd packs/hello_pack/ The directory should have the following contents: * A `README.md` file containing a human-readable description of the pack, often including any dependency information. * A `metadata.hcl` file containing information about the pack. * A `variables.hcl` file that defines the variables in a pack. * An optional, but _highly encouraged_ `CHANGELOG.md` file that lists changes for each version of the pack. * An optional `outputs.tpl` file that defines an output to be printed when a pack is deployed. * A `templates` subdirectory containing the HCL templates used to render one or more Nomad job specifications. To streamline pack creation, you can use the `generate pack` command to scaffold the above files with boilerplate and example data: $ nomad-pack generate pack hello_pack Creating "hello_pack" Pack in "."... Next, you will create each of these files for your custom pack. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#metadata-hcl) metadata.hcl The `metadata.hcl` file contains important key value information about the pack. It contains the following blocks and their associated fields: * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`app`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#app) - Information about the application that the pack deploys * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`url`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#url) - The HTTP(S) URL of the homepage of the application. This attribute can also be used to provide a reference to the documentation and help pages. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`pack`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#pack) - Metadata about the pack itself * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`name`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#name) - The name of the pack. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`description`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#description) - A small overview of the application that is deployed by the pack. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`version`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#version) - The version of the pack. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`dependency`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#dependency) - The dependencies that the pack has on other packs. Multiple dependencies can be supplied. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`alias`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#alias) - Name to refer to this specific dependency instance. Helpful when importing multiple versions of the same dependency. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`source`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#source) - The source URL for this dependency. Dependencies with different names can refer to the same source pack. Add a `metadata.hcl` file with the following contents: metadata.hcl app { url = "https://learn.hashicorp.com/tutorials/nomad/nomad-pack-writing-packs" } pack { name = "hello_pack" description = "This is an example pack created to learn about Nomad Pack" version = "0.0.1" } #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#variables-hcl) variables.hcl The `variables.hcl` file defines the variables required to fully render and deploy all the templates found within the "templates" directory. Add a `variables.hcl` file with the following contents: variables.hcl variable "datacenters" { description = "A list of datacenters in the region which are eligible for task placement." type = list(string) default = ["dc1"] } variable "region" { description = "The region where the job should be placed." type = string default = "global" } variable "app_count" { description = "The number of instances to deploy" type = number default = 3 } variable "resources" { description = "The resource to assign to the application." type = object({ cpu = number memory = number }) default = { cpu = 500, memory = 256 } } #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#outputs-tpl) outputs.tpl The `outputs.tpl` is an optional file that defines an output to be printed when a pack is deployed. Output files have access to the pack variables defined in `variables.hcl`, metadata, and any helper templates (see below). A simple example: Congrats on deploying [[ meta "pack.name" . ]].   There are [[ var "count" . ]] instances of your job now running on Nomad. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#readme-and-changelog) README and CHANGELOG No specific format is required for the `README.md` or `CHANGELOG.md` files. Create a simple `README.md` and empty `CHANGELOG.md` for now: $ touch CHANGELOG && echo "#Hello Packs" >> README.md [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#write-the-templates) Write the templates --------------------------------------------------------------------------------------------------------------- Each file at the top level of the `templates` directory that uses the extension ".nomad.tpl" defines a resource (such as a job) that will be applied to Nomad. These files can use any UTF-8 encoded prefix as the name. Helper templates, which can be included within larger templates, have names prefixed with an underscore “\_” and use a ".tpl" extension. In a deployment, Nomad Pack will render each job template using the variables provided and apply it to Nomad. Nomad Pack will render any files ending in `.tpl` but without `.nomad` when calling the `render` command. This can be useful for templatizing configuration files for other non-jobspec files for a job. Nomad Pack will not do anything with these files other than render them. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#template-basics) Template basics Templates are written using [Go Template Syntax](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax) . This enables templates to have complex logic where necessary. Unlike default Go Template syntax, Nomad Pack uses `"[["` and `"]]"` as delimiters. Go ahead make your first template at `./templates/hello_pack.nomad.tpl` with the content below. This defines a job called "hello\_pack" and allows you to provide variable values for `region`, `datacenters`, `app_count`, and `resources`. hello\_pack.nomad.tpl job "hello_pack" { type = "service" region = "[[ var "region" . ]]" datacenters = [ [[ range $idx, $dc := (var "datacenters" .) ]][[if $idx]],[[end]][[ $dc | quote ]][[ end ]] ] group "app" { count = [[ var "count" . ]] network { port "http" { static = 80 } } [[/* this is a go template comment */]] task "server" { driver = "docker" config { image = "mnomitch/hello_world_server" network_mode = "host" ports = ["http"] } resources { cpu = [[ var "resources.cpu" . ]] memory = [[ var "resources.memory" . ]] } } } } The `datacenters` value shows a more complex usage of the Go Template, which allows for [control structures](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax#control-structure-list) like `range` and [pipelines](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax#pipelines) . Note A simpler (but less informative) option of `datacenters = [[ var "datacenters" . | toStringList ]]` is possible as well. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#template-functions) Template functions The [`masterminds/sprig`](https://github.com/Masterminds/sprig) library supplements the standard Go Template set of template functions. This adds helper functions for various use cases such as string manipulation, cryptography, and data conversion (for instance to and from JSON). Custom Nomad-specific and debugging functions are also provided: * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`nomadRegions`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#nomadregions) returns the API object from `/v1/regions`. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`nomadNamespaces`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#nomadnamespaces) returns the API object from `/v1/namespaces`. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`nomadNamespace`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#nomadnamespace) takes a single string parameter of a namespace ID which will be read via `/v1/namespace/:namespace`. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`spewDump`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#spewdump) dumps the entirety of the passed object as a string. The output includes the content types and values. This uses the `spew.SDump` function. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`spewPrintf`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#spewprintf) dumps the supplied arguments into a string according to the supplied format. This uses the `spew.Printf` function. * [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#) [`fileContents`](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#filecontents) takes an argument to a file of the local host, reads its contents and provides this as a string. A custom function within a template is called like any other: [[ nomadRegions ]] [[ nomadRegions | spewDump ]] You will not use any of the helper functions in this tutorial, but they are available to help you write custom packs in the future. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#helper-templates) Helper templates For more complex packs, you may want to reuse template snippets across multiple resources. For instance, suppose you have two jobs defined in your pack and both jobs reuse the same `region` logic. You can create a helper template to centralize that logic. Helper template names are prepended with an underscore `_` and end in `.tpl`. Go ahead and define your first helper template at `./templates/_region.tpl`. \_region.tp [[ define "region" -]] [[- if var "region" . -]] region = [[ (var "region" .) | quote ]] [[- end -]] [[- end -]] This template will only specify the "region" value on the job if the `region` variable has been passed into Nomad Pack. You can now use this helper template in your job file. hello\_pack.nomad.tpl job "hello_pack" { type = "service" [[ template "region" . ]] datacenters = [[ var "datacenters" . | toStringList ]] ... } If this pack defined multiple jobs, this logic could now be reused throughout the pack. #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#conditional-jobs) Conditional jobs Some packs will have multiple jobs, and occasionally some of these jobs should only be run if certain variable values are provided. If a template is empty, Nomad Pack will not run any job. This allows for conditional jobs: [[ if var "use-own-database" . ]] job "postgres" { ... } [[ end ]] #### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#pack-dependencies) Pack dependencies Packs can depend on content from other packs. You must place copies of dependent packs into the `deps` directory. This process is known as vendoring. The file structure for dependent packs looks like the following: └── ...Pack A's contents... └── deps └── └── ...Pack B's contents... This allows Pack A to use any helper templates defined in Pack B, and Pack A will automatically deploy any jobs defined in Pack B when it deploys. In addition to the filesystem, packs must define their dependencies in `metadata.hcl`. An example pack block with a dependency looks like the following. metadata.hcl app { url = "https://some-url-for-the-application.dev" } pack { name = "hello_pack_with_deps" description = "This pack contains a simple service job, and depends on another pack." version = "0.2.1" } dependency "demo_dep" { source = "git::https://github.com/org-name/repo-name.git//packs/demo_dep" } Nomad Pack provides a helper command to take packs defined as dependencies and vendor them. Running this command from a pack's root directory will download the files associated with any pack dependency and put them in the `deps` directory. $ nomad-pack deps vendor Note While developing a dependent pack alongside a parent pack, it can be helpful to symlink a reference to the dependent pack in the `deps` directory. This allows templates of `hello_pack_with_deps` to use `demo_dep`'s helper templates, and deploy any jobs in demo\_dep when running `hello_pack_with_deps`. [[ template "helper_data" . ]] You can pass in variables for a dependent pack with a reference to their name or alias value. $ nomad-pack run hello_pack_with_deps --var demo_dep.message="example" [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#testing-your-pack) Testing your pack ----------------------------------------------------------------------------------------------------------- As you write packs, you may want to test them. To do this, pass the directory path as the name of the pack to the `run`, `plan`, `render`, `info`, `stop`, or `destroy` commands. Relative paths are supported. $ nomad-pack info . $ nomad-pack render . $ nomad-pack run . [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#publish-and-find-your-custom-repository) Publish and find your custom repository ------------------------------------------------------------------------------------------------------------------------------------------------------- To use and share your new pack, push the git repository to a URL accessible by your command line tool. In this demo you will push to a GitHub repository. If you wish to share your packs with the Nomad community, please consider adding them to the [Nomad Pack Community Registry](https://github.com/hashicorp/nomad-pack-community-registry) . [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#deploy-your-custom-pack-from-a-custom-registry) Deploy your custom pack from a custom registry --------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you have added your own registry to GitHub, add it to your local Nomad Pack using the `nomad-pack registry add` command. $ nomad-pack registry add my_packs git@github.com// This will download the packs defined in the GitHub repository to your local filesystem. They will be found using the registry value "my\_packs". Deploy your custom pack. $ nomad-pack run hello_pack --var app_count=1 --registry=my_packs [](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs#next-steps) Next steps --------------------------------------------------------------------------------------------- In this tutorial you learned: * how packs and pack registries are structured * how to write a custom pack * how to test your pack locally * how to deploy a custom pack As you write packs, consider contributing them to the [Nomad Pack Community Registry](https://github.com/hashicorp/nomad-pack-community-registry) . This is a great source of feedback, best-practices, and shared-knowledge. To help speed up your pack development and testing, check out the [Setup HashiCorp Nomad Pack](https://github.com/marketplace/actions/setup-hashicorp-nomad-pack) GitHub Action - it takes care of setting up the Nomad Pack binary and making it available to your GitHub Actions workflow. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/tools/nomad-pack/create-packs.mdx) --- # Boundary | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/boundary/v0.0.1) Boundary @hashicorp Boundary is an intelligent proxy that creates granular, identity-based access controls for dynamic infrastructure. v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/boundary/v0.0.1) * Official * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/boundary) (opens in new tab) Boundary ======== Boundary is an intelligent proxy that creates granular, identity-based access controls for dynamic infrastructure. Boundary’s workflow layers security controls and integrations at multiple levels to monitor and manage user access through: * Tightly scoped identity-based permissions * Just-in-time network and credential access for sessions via HashiCorp Vault * Single sign-on to target services and applications via external identity providers * Access-as-code to automate the configuration of user permissions * Automated discovery of target systems * Session monitoring and management for access created via Boundary. [Source](https://www.boundaryproject.io/docs/what-is-boundary) [](https://developer.hashicorp.com/nomad/integrations/hashicorp/boundary#dependencies) Dependencies --------------------------------------------------------------------------------------------------- This pack requires an existing, unused instance of Postgres DB to be running, and a credential for this instance to be supplied to Boundary. The Boundary container itself, which will be scheduled by Nomad, must run on a Nomad client whose Docker driver has the IPC\_LOCK capability allowed on the Nomad client. Alternatively, the Docker driver could instead be allowed to run privileged Docker containers. --- # CSI OpenStack Cinder | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder/v0.0.1) CSI OpenStack Cinder @hashicorp The Cinder CSI Driver is a CSI Specification compliant driver used by Container Orchestrators to manage the lifecycle of OpenStack Cinder Volumes. v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder/v0.0.1) * Community * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/csi_openstack_cinder) (opens in new tab) CSI OpenStack Cinder ==================== This pack deploys the Openstack Cinder CSI container as a system job to all eligible Nomad Clients. See the Openstack Cinder CSI documentation for more information: [https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) [](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#dependencies) Dependencies --------------------------------------------------------------------------------------------------------------- This pack requires the Nomad Client(s) be deployed on an Openstack VM that can have Cinder Volumes attached to it. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#container-privilege) Container Privilege ----------------------------------------------------------------------------------------------------------------------------- The Cinder Node task containers run as **_Privileged_** containers. The Cinder Controller tasks do not require privileged mode. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#variables) Variables --------------------------------------------------------------------------------------------------------- | Name | Description | Type | Default | Required | | --- | --- | --- | --- | --- | | [job\_name](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_job_name) | The name to use as the job name which overrides using the pack name | `string` | `""` | no | | [datacenters](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_datacenters) | A list of datacenters in the region which are eligible for task placement | `list(string)` | \[ "dc1"\] | no | | [region](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_region) | The region where the job should be placed | `string` | `"global"` | no | | [constraints](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_constraints) | Constraints to apply to the entire job. | list(object({ attribute = string operator = string value = string })) | \[ { "attribute": "${attr.platform.aws.placement.availability-zone}", "operator": "", "value": "nova" }\] | no | | [job\_restart\_config](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_job_restart_config) | n/a | object({ attempts = number delay = string mode = string interval = string }) | { "attempts": 5, "delay": "15s", "interval": "5m", "mode": "delay"} | no | | [cloud\_conf\_file](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_cloud_conf_file) | \[REQUIRED\] Path to custom cloud.conf file to be mounted to the CSI containers. For reference, see [https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#global](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#global) | `string` | n/a | yes | | [csi\_plugin\_id](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_csi_plugin_id) | The ID to register the CSI plugin with. | `string` | `"csi-cinder"` | no | | [version\_tag](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_version_tag) | The docker image version. For options, see [https://github.com/kubernetes/cloud-provider-openstack/releases](https://github.com/kubernetes/cloud-provider-openstack/releases) | `string` | `"latest"` | no | | [cinder\_log\_level](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_cinder_log_level) | The log level to run the csi driver at. Valid values 1 through 5 | `string` | `"3"` | no | | [vault\_config](https://developer.hashicorp.com/nomad/integrations/hashicorp/csi-openstack-cinder#input_vault_config) | Nomad Job Vault Configuration. Set `enabled = true` to configure the job to use vault See: [https://www.nomadproject.io/docs/job-specification/vault](https://www.nomadproject.io/docs/job-specification/vault) | object({ enabled = bool policies = list(string) change\_mode = string change\_signal = string env = bool namespace = string }) | { "change\_mode": "restart", "change\_signal": "", "enabled": false, "env": true, "namespace": "", "policies": \[\]} | no | --- # Create a secret provider plugin | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Create a secret provider plugin =============================== This page describes the plugin specification for [custom secret providers](https://developer.hashicorp.com/nomad/docs/job-specification/secret#custom-providers) , with examples, so you can write your own plugin to utilize secrets from any secrets store. The full [specification](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#specification) follows the examples, and is followed by a list of [general considerations](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#considerations) . Follow this workflow to create and use a custom secret provider: 1. Develop the plugin code based on the [specification](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#specification) . 2. Compile your plugin into an executable binary. 3. Place the binary in the [`client.common_plugin_dir/secrets/`](https://developer.hashicorp.com/nomad/docs/configuration/client#common_plugin_dir) directory on each Nomad client node. 4. Configure the `secret` block in your job specification. [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#example) Example ---------------------------------------------------------------------------------------- We wrote the example in Go, but the specification is lean enough to be readily fulfilled in any language. The example code is not meant to be inclusive of all error handling. In this example code, the `aws` secret provider plugin fetches a secret from AWS Secret Manager. Refer to the external [AWS Secrets Manager User Guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) for details on AWS Secrets Manager and [how to fetch secrets in various languages](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) . ### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#plugin-code) Plugin code aws.go package main import ( "context" "errors" "fmt" "os" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/secretsmanager" ) func main() { if len(os.Args) < 2 { outputErr("internal error not enough args") return } switch os.Args[1] { case "fingerprint": fingerprint() case "fetch": // fetch should be called with the correct secret path if len(os.Args) < 3 { outputErr("internal error no path specified") return } if res, err := fetch(os.Args[2]); err != nil { outputErr(err.Error()) } else { fmt.Printf(`{"result": "%s"}`, res) } default: outputErr("internal error incorrect function") } } func fingerprint() { fmt.Println(`{"type": "secrets", "version": "0.0.1"}`) } // resource: https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets-go-sdk.html func fetch(secretName string) (string, error) { ctx := context.Background() config, err := config.LoadDefaultConfig(ctx, config.WithRegion(os.Getenv("AWS_REGION"))) if err != nil { return "", errors.New("error loading default aws config") } // Create Secrets Manager client svc := secretsmanager.NewFromConfig(config) input := &secretsmanager.GetSecretValueInput{ SecretId: aws.String(secretName), VersionStage: aws.String("AWSCURRENT"), } result, err := svc.GetSecretValue(ctx, input) if err != nil { return "", err } // Decrypted secret value var secretString string = *result.SecretString return secretString, nil } func outputErr(err string) { fmt.Printf(`{"error": "%s"}`, err) } ### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#task-configuration) Task configuration Configure your task to fetch the secret with your customer `aws` secret provider. Make sure the `provider` value matches the name of your plugin executable. In this example job specification, the task uses your custom `aws` secret provider to fetch an AWS secret named `my-aws-secret` from the AWS Secrets Manager in region `us-east-2`. job "docs" { group "example" { task "server" { secret "my_secret" { provider = "aws" path = "my-aws-secret" env { AWS_REGION = "us-east-2" } } } } } [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#specification) Specification ---------------------------------------------------------------------------------------------------- Nomad registers a secret provider plugin if the plugin meets all of the following criteria: * Is an executable file, such as a script or binary * Is located in the [`client.common_plugin_dir/secrets/`](https://developer.hashicorp.com/nomad/docs/configuration/client#common_plugin_dir) directory on Nomad client nodes * Responds appropriately to a `fingerprint` call ### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#operations) Operations A secret provider plugin must fulfill all of the following operations: * [fingerprint](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#fingerprint) * [fetch](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#fetch) Nomad passes the operation as the first positional argument to the plugin and as an environment variable prefixed with `CPI_`. #### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#fingerprint) fingerprint > Nomad calls `fingerprint` to discover valid plugins when the client agent starts or is reloaded with a SIGHUP. The version it returns is used to register the plugin on the Nomad node. > > **CLI arguments:** `$1=fingerprint` > > **Environment variables:** > > CPI_OPERATION=fingerprint > > > **Expected stdout:** > > {"type": "secrets", "version": "0.0.1"} > > > **Requirements:** > > * Must complete within 10 seconds, or Nomad kills it. It should be much faster, as no actual work should be done. > * "type" registers this as a secret provider plugin. > * "version" value must be valid per the [hashicorp/go-version](https://pkg.go.dev/github.com/hashicorp/go-version#pkg-constants) > golang package. #### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#fetch) fetch > Nomad calls `fetch` when a Nomad job includes a secret block with a provider registered as a plugin. When calling fetch, Nomad includes the secret path as the second CLI argument. If the `env{}` block was specified, any key/values are supplied as environment variables to the plugin process. Nomad automatically provides the `NOMAD_JOB_ID` and `NOMAD_NAMESPACE` environment variables to plugins. > > **CLI Arguments:** `$1=fetch $2=path` > > **Environment variables:** > > CPI_OPERATION=fetch > > > **Expected stdout:** > > {"result": {"key1": "value1", "key2": "value2"}} > > > **Expected stdout on error:** > > {"result": {}, "error": "error message"} > > > Returning an error message is optional. Nomad returns the error message in any error returned to the user. > > **Requirements:** > > * Must complete within 10 seconds, or Nomad kills it. > * Must return secret contents as a key/value object, even if the secret was not originally in key/value format. [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#considerations) Considerations ------------------------------------------------------------------------------------------------------ ### [](https://developer.hashicorp.com/nomad/plugins/author/secret-provider#execution) Execution * The plugin is executed as the same user as the Nomad agent (likely root). [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/author/secret-provider.mdx) --- # AWS EFS CSI | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi/v0.0.1) AWS EFS CSI @hashicorp Configures a set of nodes to run the AWS EFS CSI volume plugin. v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi/v0.0.1) * Community * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/aws_efs_csi) (opens in new tab) AWS EFS CSI =========== This pack contains a single system job that runs the AWS EFS CSI plugin. It will run the nodes in monolith modes, which means they will run as both nodes and controllers. The job can only be run on Nomad hosts which have enabled privileged mode for Docker. In addition, the hosts will need to have to be provided with access to AWS EFS through AWS IAM. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi#variables) Variables ------------------------------------------------------------------------------------------------ * `job_name` (string "aws-efs-csi-nodes") - The name to use as the job name. * `datacenters` (list(string) \["dc1"\]) - A list of datacenters in the region which are eligible for task placement. * `region` (string "global") - The region where the job should be placed. * `constraints` (list(object)) - Constraints to apply to the entire job. * `resources` (object) - The resource to assign to the plugin task. * `image` (string "amazon/aws-efs-csi-driver:master") - The Docker image to run on the plugin tasks. * `csi_id` (string "aws-efs") - The CSI ID to use for this plugin. ### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi#constraints-list-of-objects) `constraints` List of Objects [Nomad job specification constraints](https://www.nomadproject.io/docs/job-specification/constraint) allows restricting the set of eligible nodes on node task will run. * `attribute` (string) - Specifies the name or reference of the attribute to examine for the constraint. * `operator` (string) - Specifies the comparison operator. The ordering is compared lexically. * `value` (string) - Specifies the value to compare the attribute against using the specified operation. By default, the job will run on hosts running linux and having Docker privileged mode enabled. The HCL variable list of objects for the default configuration is shown below and uses a double dollar sign for escaping: [\ {\ attribute = "$${attr.kernel.name}",\ value = "linux",\ operator = null,\ },\ {\ attribute = "$${attr.driver.docker.privileged.enabled}",\ value = true,\ operator = null,\ }\ ] Below is also an example of how to pass `constraints` to the CLI with the -var argument. nomad-pack run -var 'constraints=[{"attribute":"$${meta.my_custom_value}","operator":">","value":"3"}]' packs/aws_efs_csi When setting the `constrains` variable manually its default value will be overriden. ### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi#resources-object) `resources` Object * `cpu` (number 100) - Specifies the CPU required to run this task in MHz. * `memory` (number 128) - Specifies the memory required in MB. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi#volume-creation-example) Volume creation example ---------------------------------------------------------------------------------------------------------------------------- The plugin currently only by creating new access points to existing EFS file systems. So you'll first have to provision a new EFS file system. The capacity in the volume spec is not used, but is required by the CSI Volume API. #### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-efs-csi#volume.hcl) **`volume.hcl`** id = "test" name = "Test" type = "csi" plugin_id = "aws-efs" capacity_max = "1G" capacity_min = "1M" capability { access_mode = "single-node-writer" attachment_mode = "file-system" } parameters { provisioningMode = "efs-ap" fileSystemId = "" directoryPerms = "700" gidRangeStart = "1000" gidRangeEnd = "2000" basePath = "/test" } Run the command: nomad volume create volume.hcl --- # AWS EBS CSI | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.1.0](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi/v0.1.0) AWS EBS CSI @hashicorp This pack deploys the AWS EBS CSI plugin v0.2.0 (latest) * [v0.1.0](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi/v0.1.0) * Community * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/aws_ebs_csi) (opens in new tab) AWS EBS CSI =========== This pack deploys two jobs that run the [AWS EBS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver) CSI plugin. The node plugin tasks will be run as a system job, and the controller tasks will be run as a service job. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#client-requirements) Client Requirements -------------------------------------------------------------------------------------------------------------------- This pack can only be run on Nomad clients that have enabled volumes and privileged mode for the Docker task driver. In addition, clients will need to be deployed in every availability zone where you intend to create volumes. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#variables) Variables ------------------------------------------------------------------------------------------------ * `job_name` (string "democratic\_csi") - The prefix to use as the job name for the plugins. For exmaple, if `job_name = "democratic_csi"`, the plugin job will be named `democratic_csi_controller`. * `datacenters` (list(string) \["dc1"\]) - A list of datacenters in the region which are eligible for task placement. * `region` (string "global") - The region where the job should be placed. * `plugin_id` (string "org.democratic-csi.nfs") - The ID to register in Nomad for the plugin. * `plugin_namespace` (string "default") - The namespace for the plugin job. * `plugin_image` (string "public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.5.1") - The container image for the plugin. * `plugin_csi_spec_version` (string "1.5.0") - The CSI spec version that democratic-csi will comply with. * `plugin_log_level` (string "debug") - The log level for the plugin. * `availability_zones` (list(string) \["us-east-1b"\]) - AWS availability zones for the node plugins and example volume output. * `controller_count` (number 2) - The number of controller instances to be deployed (at least 2 recommended). * `volume_id` (string "myvolume") - ID for the example volume spec to output. * `volume_namespace` (string "default") - Namespace for the example volume spec to output. * `volume_min_capacity` (string "10GiB") - Minimum capacity for the example volume spec to output. * `volume_max_capacity` (string "30GiB") - Maximum capacity for the example volume spec to output. * `volume_type` (string "gp2") - AWS EBS volume type. #### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#constraints-list-of-objects) `constraints` List of Objects [Nomad job specification constraints](https://www.nomadproject.io/docs/job-specification/constraint) allow restricting the set of eligible nodes on which the tasks will run. This pack automatically configures the following required constraints: * Plugin tasks will run on Linux hosts only * The node plugin tasks will run on hosts with the Docker driver's [`volumes`](https://www.nomadproject.io/docs/drivers/docker#volumes-1) enabled and [`allow_privileged`](https://www.nomadproject.io/docs/drivers/docker#allow_privileged) set to `true`. * The controller plugin tasks will be deployed on distinct hosts. You can set additional constraints with the `constraints` variable, which takes a list of objects with the following fields: * `attribute` (string) - Specifies the name or reference of the attribute to examine for the constraint. * `operator` (string) - Specifies the comparison operator. The ordering is compared lexically. * `value` (string) - Specifies the value to compare the attribute against using the specified operation. Below is also an example of how to pass `constraints` to the CLI with the `-var` argument. nomad-pack run -var 'constraints=[{"attribute":"$${meta.my_custom_value}","operator":">","value":"3"}]' packs/aws_ebs_csi #### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#resources-object) `resources` Object * `cpu` (number 500) - Specifies the CPU required to run this task in MHz. * `memory` (number 256) - Specifies the memory required in MB. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#volume-creation) Volume creation ------------------------------------------------------------------------------------------------------------ This pack outputs an example volume specification based on the plugin variables. #### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/aws-ebs-csi#volume.hcl) **`volume.hcl`** type = "csi" id = "myvolume" namespace = "default" plugin_id = "ebs.csi.aws.com" # this is used as the AWS EBS volume's CSIVolumeName tag, and # must be unique per region name = "eecede36-6de0-4de1-9a06-6d201c29e2a2" capacity_min = "10GiB" capacity_max = "30GiB" capability { access_mode = "single-node-writer" attachment_mode = "file-system" } capability { access_mode = "single-node-writer" attachment_mode = "block-device" } parameters { type = "gp2" } topology_request { required { topology { segments { "topology.ebs.csi.aws.com/zone" = "us-east-1a" "topology.ebs.csi.aws.com/zone" = "us-east-1b" } } } } Create this volume with the following command: nomad volume create volume.hcl --- # Community task driver plugins | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Community task driver plugins ============================= Task driver plugins authored and maintained by the Nomad community are listed on the [Nomad integrations page](https://developer.hashicorp.com/nomad/integrations) . On that page, select **Community** from the **Tiers** list and **Task Drivers** from the **Types** list to review Community-authored task driver plugins. If you have authored a task driver plugin that you believe is useful to the broader Nomad community and are committed to maintaining the plugin, create a [Nomad GitHub issue](https://github.com/hashicorp/nomad/issues) so we can add your plugin to the Nomad Integrations page. For details on authoring a task driver plugin, refer to the [plugin authoring guide](https://developer.hashicorp.com/nomad/docs/concepts/plugins) . [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/drivers/community/index.mdx) --- # Nomad plugin authoring guide | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad plugin authoring guide ============================ Nomad implements a plugin framework that lets you extend the functionality of some components within Nomad. The design of the plugin system is inspired by the lessons learned from plugin systems implemented in other HashiCorp products such as Terraform and Vault. The following components are currently pluggable within Nomad: * [Device](https://developer.hashicorp.com/nomad/plugins/author/device) : Extend workload functionality with a custom hardware device. * [Host volume](https://developer.hashicorp.com/nomad/plugins/author/host-volume) : Extend workload storage functionality with a plugin specific to your local storage environment. * [Secret provider](https://developer.hashicorp.com/nomad/plugins/author/secret-provider) : Extend task execution functionality with a custom secret provider. * [Task driver](https://developer.hashicorp.com/nomad/plugins/author/task-driver) : Extend task execution functionality with a custom task driver. [](https://developer.hashicorp.com/nomad/plugins/author#architecture) Architecture ---------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/plugins/author#go-plugin) Go-plugin The Nomad task driver and device plugin framework uses the [go-plugin](https://github.com/hashicorp/go-plugin) project to expose a language-independent plugin interface. Plugins implement a set of gRPC services and methods that Nomad manages by running the plugin and calling the implemented RPCs. As a result, plugins are free to be implemented in the author's language of choice. To facilitate plugin development, a set of Go interfaces and structs exist for each plugin type that abstracts away go-plugin and the gRPC interface. The guides in this documentation reference these abstractions for ease of use. ### [](https://developer.hashicorp.com/nomad/plugins/author#common-plugin-interface) Common plugin interface The Nomad secret provider formalized a pattern first introduced with dynamic host volume plugins into a common plugin interface. These plugins differ from Go-plugins in that they generally perform a simple task and exit, not needing to run for extended periods of time or needing to manage other resources similar to task and device drivers. Common plugins must implement a `fingerprint` command and implementation-specific commands. The parent directory of all common plugin implementations is managed via the [common\_plugin\_dir](https://developer.hashicorp.com/nomad/docs/configuration/client#common_plugin_dir) , with each implementation residing in a specific subdirectory. Check the specific plugin's documentation for directory details. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/author/index.mdx) --- # Backstage | Integrations | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Integrations](https://developer.hashicorp.com/nomad/integrations) v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage/v0.0.1) Backstage @hashicorp An open platform for building developer portals v0.2.0 (latest) * [v0.0.1](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage/v0.0.1) * Community * Pack Updated 3 years ago * [GitHub](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/backstage) (opens in new tab) Backstage ========= This pack contains a service job that runs [Backstage](https://backstage.io/) in a single Nomad client. It currently supports being run by the [Docker](https://www.nomadproject.io/docs/drivers/docker) driver. It has 2 tasks: * **Backstage:** [_(reference)_](https://backstage.io/) the open platform for building developer portals; * **PostgreSQL:** [_(reference)_](https://www.postgresql.org/) the persistent database used by Backstage. Setup: * Service-to-service communication is handled by Nomad; * PostgreSQL's state is persisted with Nomad Host Volumes. [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#requirements) Requirements ---------------------------------------------------------------------------------------------------- Clients that expect to run this job require: * Nomad >= 1.4.0 (because the pack use [Nomad Variables](https://developer.hashicorp.com/nomad/docs/concepts/variables) ) * [Docker volumes](https://www.nomadproject.io/docs/drivers/docker "Docker volumes") to be enabled within their Docker plugin stanza, due to the usage of Nomad's host volume: plugin "docker" { config { volumes { enabled = true } } } * [Host volume](https://www.nomadproject.io/docs/configuration/client#host_volume-stanza "Host volume") to be enabled in the client configuration (the host volume directory - /var/lib/postgres - must be created in advance): client { host_volume "backstage-postgres" { path = "/var/lib/postgres" read_only = false } } [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#customizing-the-docker-images) Customizing the Docker images -------------------------------------------------------------------------------------------------------------------------------------- The 2 docker images can be replaced by using their variable names: * postgresql\_task\_image * backstage\_task\_image Example: $ nomad-pack run backstage --var backstage_task_image="ghcr.io/backstage/backstage:1.7.1" [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#pack-usage) Pack Usage ------------------------------------------------------------------------------------------------ ### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#prerequisite) Prerequisite Create a variable specification file: # spec.nv.hcl path = "nomad/jobs/backstage" items { # Mandatory variables postgres_user = "your_postgres_username" postgres_password = "your_postgres_password" #Optional variables github_token = "your_github_token" } The default docker image for Backstage is configured to use GitHub in order to locate entities (see [GitHub integration](https://backstage.io/docs/integrations/github/locations) ), so you will need to define a variable to store your GitHub token. If you use another integration in your custom image, Azure DevOps for instance, you will need to define a variable for it (e.g. `azure_token = "your_ado_token"`). To set your variables in your Nomad instance, execute the following command: $ nomad var put @spec.nv.hcl [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#variables) Variables ---------------------------------------------------------------------------------------------- * `job_name` (string) - The name to use as the job name which overrides using the pack name * `datacenters` (list of strings) - A list of datacenters in the region which are eligible for task placement * `region` (string) - The region where jobs will be deployed * `postgresql_group_nomad_service_name` (string) - The nomad service name for the PostgreSQL application * `postgresql_task_image` (string) - PostgreSQL's Docker image (must include the tag) * `postgresql_task_volume_path` (string) - The volume's absolute path in the host to be used by PostgreSQL * `postgresql_task_resources` (object, number number) - The resources to assign to the PostgreSQL service * `backstage_group_nomad_service_name` (string) - The nomad service name for the Backstage application * `backstage_task_image` (string) - Backstage's Docker image (must include the tag) * `backstage_task_nomad_vars` (map of string) - Backstage's nomad variables (see below for the details) * `backstage_task_resources` (object, number number) - The resources to assign to the Backstage service ### [](https://developer.hashicorp.com/nomad/integrations/hashicorp/backstage#custom-nomad-variables) Custom Nomad Variables If you need to use other Nomad variables than `postgres_user` and `postgres_password` with this pack, you will need to pass them to `backstage_task_nomad_vars`. backstage_task_nomad_vars = [\ {\ key = "AZURE_TOKEN"\ value = "azure_token"\ },\ {\ key = "YOUR_SECRET_ENV_VAR"\ value = "your_nomad_var_key"\ }\ ] --- # Nomad device driver plugins | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad device driver plugins =========================== Use device driver plugins to detect physical hardware devices, such as graph processing units (GPUs), a field-programmable gate arrays (FPGAs), and universal serial buses (USBs), on your Nomad clients. You can then use those devices in your Nomad workloads. By having extensible device plugins, Nomad has the flexibility to support a broad set of devices and allows the community to build additional device plugins as needed. [](https://developer.hashicorp.com/nomad/plugins/devices#nomad-device-drivers) Nomad device drivers --------------------------------------------------------------------------------------------------- We support the [NVIDIA](https://developer.hashicorp.com/nomad/plugins/devices/nvidia) device driver plugin, which you must install separately. Refer to the [NVIDIA](https://developer.hashicorp.com/nomad/plugins/devices/nvidia) documentation for instructions. [](https://developer.hashicorp.com/nomad/plugins/devices#community-device-drivers) Community Device Drivers ----------------------------------------------------------------------------------------------------------- The community supports the [USB](https://developer.hashicorp.com/nomad/plugins/devices/usb) device driver plugin. [](https://developer.hashicorp.com/nomad/plugins/devices#create-device-drivers) Create device drivers ----------------------------------------------------------------------------------------------------- Nomad's device driver architecture is pluggable, which gives you the flexibility to create your own drivers without having to recompile Nomad. Refer to the [plugin authoring guide](https://developer.hashicorp.com/nomad/plugins/author/device) for details. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/devices/index.mdx) --- # Install the Nomad CLI | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) In this collection of tutorials, you will become familiar with Nomad by installing the CLI tool, creating a cluster, and deploying an application. In this tutorial, you will install the Nomad CLI on your local machine and verify its functionality. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install#install-the-nomad-cli) Install the Nomad CLI ---------------------------------------------------------------------------------------------------------------------- Nomad is available as a pre-compiled binary or as a package for several operating systems. Refer to the [Nomad downloads page](https://developer.hashicorp.com/nomad/install) for the most up-to-date instructions. Note This collection of tutorials requires Nomad version 1.5.0 or later. Manual installationHomebrew on macOSChocolatey on WindowsLinux To simplify the getting started experience, you can download a [precompiled binary](https://developer.hashicorp.com/nomad/install) and run it on your machine locally. After downloading Nomad, unzip the package. The `nomad` binary file needs to be moved to a location on your system's path. Inspect the locations available on your path. $ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin The output is a list of locations separated by colons and yours may be different from the example output. You can move the Nomad binary to a directory in your path or to a directory of your choice and add that directory to your path. In this example, the binary has been unzipped to `/opt/hashicorp`. Add the Nomad binary directory to your system path by updating your shell's profile. This ensures that every time a new shell starts, the Nomad binary will be available on the path. Depending on the shell your system uses, this file could be `.profile`, `.bash_profile`, or similar but usually resides in your user's home directory. If your shell profile already exports `PATH`, add the directory to the same line by first adding a colon (`:`) separator and then the directory. ~/.bash\_profile export PATH=/usr/local:/opt/bin:/opt/hashicorp If your shell profile does not already export `PATH`, add the export command with the directory. ~/.bash\_profile export PATH=/opt/hashicorp Save the file and reload your current shell with the `source` command and the shell profile you modified. $ source ~/.bash_profile [Homebrew](https://brew.sh/) is a free and open source package management system for macOS. Install the official [Nomad formula](https://github.com/hashicorp/homebrew-tap) from the terminal. First, install the HashiCorp tap, a repository of all of the HashiCorp Homebrew packages. $ brew tap hashicorp/tap Now, install Nomad with `hashicorp/tap/nomad`. This installs a signed binary of the newest available official release. $ brew install hashicorp/tap/nomad To update to the latest, run: $ brew upgrade hashicorp/tap/nomad [Chocolatey](https://chocolatey.org/) is a free and open source package management system for Windows. Install the [Nomad package](https://chocolatey.org/packages/nomad) from the command-line. $ choco install nomad HashiCorp does _not_ directly maintain Chocolatey or the Nomad package it installs. The latest version of Nomad is always available by manual installation. HashiCorp officially maintains and signs packages for the following Linux distributions. Ubuntu/DebianCentOS/RHELFedoraAmazon Linux Install the required packages. $ sudo apt-get update && \ sudo apt-get install wget gpg coreutils Add the HashiCorp [GPG key](https://apt.releases.hashicorp.com/gpg "HashiCorp GPG key") . $ wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg Add the official HashiCorp Linux repository. $ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list Update and install Nomad. $ sudo apt-get update && sudo apt-get install nomad Install `yum-config-manager` to manage your repositories. $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. $ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo Install Nomad. $ sudo yum -y install nomad Install `dnf config-manager` to manage your repositories. $ sudo dnf install -y dnf-plugins-core Use `dnf config-manager` to add the official HashiCorp Linux repository. $ sudo dnf config-manager addrepo --from-repofile=https://rpm.releases.hashicorp.com/fedora/hashicorp.repo Install Nomad. $ sudo dnf -y install nomad Install `yum-config-manager` to manage your repositories. $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. $ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo Install Nomad. $ sudo yum -y install nomad Note that you can also install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) , [Vault](https://developer.hashicorp.com/vault/install) , [Consul](https://developer.hashicorp.com/consul/tutorials/get-started-vms/virtual-machine-gs-deploy) , and [Packer](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli) with the same command now that the HashiCorp repository exists in your package manager. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install#verify-the-installation) Verify the installation -------------------------------------------------------------------------------------------------------------------------- Verify Nomad is functioning by checking the version. $ nomad -v Nomad v1.5.0 BuildDate 2023-03-01T10:11:42Z Revision fc40c491cacec3d8ec3f2f98cd82b9068a50797c [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install#next-steps) Next steps ------------------------------------------------------------------------------------------------ In this tutorial you installed Nomad on your local machine. Continue on to the next tutorial by clicking on the **Next** button below and learn how to start a Nomad cluster. **Was this tutorial helpful?** YesNo [Previous\ \ Nomad introduction](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview) [Next\ \ Create a cluster](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster) --- # Schedule edge Services with native service discovery | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Edge Computing](https://developer.hashicorp.com/nomad/tutorials/edge) Edge computing lets organizations run workloads closer to their users. This proximity unlocks several benefits: * **Decreased latency.** Data does not need to travel to distant data centers for processing. This decreases network latency which provides a better user experience. This benefit is crucial for CDN providers and online game servers. * **Privacy and Compliance.** Edge computing increases privacy by storing and processing user data close to the user, ensuring data doesn't leave the geographic region. This benefit is especially important for regulated industries like healthcare and financial services and with regulations like GDPR. * **Smart device fleet management.** Edge computing lets you collect data, monitor, and control internet of things (IoT) devices and sensors. This benefit is useful for any industries that need to manage fleets of remote devices, like agriculture, manufacturers, and more. However, when organizations adopt edge computing, they run into challenges like managing heterogeneous devices (different processors, operating systems, etc), resource constrained devices, and intermittent connectivity. Nomad addresses these challenges, making it an attractive edge orchestrator. The Nomad client agent is a single binary with a small footprint, limited resource consumption, and the ability to run on different types of devices. In addition, Nomad supports geographically distant clients, which means a Nomad server cluster does not need to run near the client. Since Nomad 1.3, native service discovery simplifies connecting Nomad tasks where you cannot use a single service mesh and removes the need to manage a separate Consul cluster. Nomad's native service discovery also removes the need to install a Consul agent on each edge device. This reduces Nomad's resource footprint even further, so you can run and support more workloads on the edge. Additionally, disconnected client allocations reconnect gracefully, handling situations when edge devices experience network latency or temporary connectivity loss. In this tutorial, you will deploy a single Nomad server cluster with distant clients edge architecture in two AWS regions. One region, representing an on-premise data center, will host the Nomad server cluster and one client. The other region, representing the edge data center, will host two Nomad clients. Then, you will schedule HashiCups, a demo application, on both on-prem and edge data centers, connecting its services with Nomad's native service discovery. Finally, you will simulate unstable network connectivity between the Nomad clients and the server to test how Nomad handles client disconnection and reconnection. In the process, you will learn how these features make Nomad an ideal edge scheduler. ![Nomad Edge single server cluster and distant client\ architecture.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fdiagram_edge-nomad-servers-clients.png%26width%3D1900%26height%3D390&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#hashicups-overview) HashiCups overview --------------------------------------------------------------------------------------------------------------------- HashiCups is a demo application that lets you view and order customized HashiCorp branded coffee. The HashiCups application consists of a frontend React application and multiple backend services. The HashiCups backend consists of a GraphQL backend (`public-api`), products API (`product-api`), a Postgres database, and a payments API (`payment-api`).The `product-api` connects to both the `public-api` and database to store and return information about HashiCups coffees, users, and orders. ![HashiCups frontend and backend\ services](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fdiagram_hashicups-services.png%26width%3D1684%26height%3D762&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) You will deploy the HashiCups application to two Nomad data centers. The primary data center will host the HashiCups database and product API. The edge data center will host the remaining HashiCups backend (public API, payments API) and the frontend (frontend and NGINX reverse proxy). This architecture decreases latency for users by placing the frontend services closer to them. In addition, sensitive payment information remains on the edge — HashiCups does not need to send this data to the primary data center, reducing potential attack surfaces. ![HashiCups services deployed in primary and edge data\ centers.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fdiagram_edge-with-hashicups-tasks.png%26width%3D1999%26height%3D446&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#prerequisites) Prerequisites ----------------------------------------------------------------------------------------------------------- The tutorial assumes that you are familiar with Nomad. If you are new to Nomad itself, refer first to the [Get Started tutorials](https://developer.hashicorp.com/nomad/tutorials/get-started) . For this tutorial, you will need: * [Packer 1.8 or later installed locally](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli) . * [Terraform 1.1.7 or later installed locally](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) . * [Nomad 1.3 or later installed locally](https://developer.hashicorp.com/nomad/tutorials/get-started/get-started-install) . * An [AWS account](https://aws.amazon.com/account/) with credentials set as local environment variables Note This tutorial creates AWS resources that may not qualify as part of the [AWS free tier](https://aws.amazon.com/free) . Be sure to follow the [Cleanup](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#clean-up-resources) process at the end so you don't incur any additional unnecessary charges. [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#clone-the-example-repository) Clone the example repository ----------------------------------------------------------------------------------------------------------------------------------------- In your terminal, clone the [example](https://github.com/hashicorp-education/learn-nomad-edge) repository. This repository contains all the Terraform, Packer, and Nomad configuration files you will need to complete this tutorial. $ git clone https://github.com/hashicorp-education/learn-nomad-edge Navigate to the cloned repository. $ cd learn-nomad-edge Now, checkout the tagged version verified for this tutorial. $ git checkout tags/v1.0.0 [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#create-ssh-key) Create SSH key ------------------------------------------------------------------------------------------------------------- Later in this tutorial, you will need to connect to your Nomad agent to bootstrap ACLs. Create a local SSH key to pair with the `terraform` user so you can securely connect to your Nomad agents. Mac or Linux command-lineWindows with PuTTY Generate a new SSH key called `learn-nomad-edge`. The argument provided with the `-f` flag creates the key in the current directory and creates two files called `learn-nomad-edge` and `learn-nomad-edge.pub`. Change the placeholder email address to your email address. $ ssh-keygen -t rsa -C "your_email@example.com" -f ./learn-nomad-edge When prompted, press enter to leave the passphrase blank on this key. If you are on a Windows machine, use Putty to generate SSH keys by following the instructions [here](https://www.ssh.com/ssh/putty/windows/puttygen) . [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#review-and-build-nomad-images) Review and build Nomad images ------------------------------------------------------------------------------------------------------------------------------------------- Navigate to the `packer` directory. $ cd packer This directory contains all the files used to build AMIs in the `us-east-2` and `us-west-1` AWS regions that contain the Nomad 1.5.3 binary and your previously created SSH public key. $ tree . ├── config │ ├── nomad.hcl │ ├── nomad-acl-user.hcl │ ├── nomad-client.hcl │ └── nomad.service └── scripts │ ├── client.sh │ ├── server.sh │ └── setup.sh └── nomad.pkr.hcl 1. The `config` directory contains configuration files for the Nomad agents. * The `nomad.hcl` file configures the Nomad servers. Since the primary and edge data centers are on different networks, the server must advertise its public IP address so the Nomad clients can successfully connect to the server cluster. packer/config/nomad.hcl advertise { http = "IP_ADDRESS:4646" rpc = "IP_ADDRESS:4647" serf = "IP_ADDRESS:4648" } The `scripts/server.sh` script will replace the placeholders (`IP_ADDRESS`, `SERVER_COUNT`, and `RETRY_JOIN`) when the server starts. The Nomad servers also have ACL enabled. * The `nomad-acl-user.hcl` file defines the ACL policies. * The `nomad-client.hcl` file configures the Nomad clients. Since the primary and edge data centers are on different networks, the client must advertise its public IP address so the Nomad clients can successfully connect to the other Nomad clients. packer/config/nomad.hcl advertise { http = "IP_ADDRESS:4646" rpc = "IP_ADDRESS:4647" serf = "IP_ADDRESS:4648" } * The `scripts/client.sh` script will replace the placeholders (`DATACENTER`, `SERVER_NAME`, and `RETRY_JOIN`) when the client starts. The Nomad clients also have ACL enabled. * The `nomad.service` defines a systemd process. This makes it easier to start, stop, and restart Nomad on the agents. 2. The `scripts` directory contains helper scripts. The `setup.sh` script creates the `terraform` user, adds the public SSH key, and installs Nomad 1.5.3 and Docker. The `client.sh` and `server.sh` scripts configure their respective Nomad agents. 3. The `nomad.pkr.hcl` Packer template file defines the AMIs. It uses the `scripts/setup.sh` to set up Nomad agents on an Ubuntu 20.04 image. ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#build-nomad-images) Build Nomad images Initialize Packer to retrieve the required plugins. $ packer init nomad.pkr.hcl Build the image. $ packer build nomad.pkr.hcl ## ... Build 'amazon-ebs.nomad-secondary' finished after 3 minutes 42 seconds. Build 'amazon-ebs.nomad-primary' finished after 8 minutes 52 seconds. ==> Wait completed after 8 minutes 52 seconds ==> Builds finished. The artifacts of successful builds are: --> amazon-ebs.nomad-secondary: AMIs were created: us-west-1: ami-0183bb1e3ab40da53 --> amazon-ebs.nomad-primary: AMIs were created: us-east-2: ami-08a59e91d881df603 Packer will display the two AMIs. You will use these AMIs in the next section to deploy the Nomad server cluster and clients. [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#review-and-deploy-nomad-cluster-and-clients) Review and deploy Nomad cluster and clients ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Navigate to the cloned repository's root directory. This directory contains Terraform configuration to deploy all the resources you will use in this tutorial. $ cd .. Open `main.tf`. This file contains the Terraform configuration to deploy the underlying shared resources and Nomad agents to the two AWS regions through the single server cluster and distant client (SCDC) edge architecture. As opposed to deploying a Nomad server cluster at every edge location, this edge architecture is simpler, scalable, has a smaller resource consumption footprint, and avoids server federation. However, it requires more client to server connection configuration, especially around heartbeats and unstable connectivity. ![Nomad Edge single server cluster and distant client\ architecture.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fdiagram_edge-nomad-servers-clients.png%26width%3D1900%26height%3D390&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) 1. The `primary_shared_resources` and `edge_shared_resources` modules use the `shared-resources` module to deploy a VPC, security groups, and IAM roles into their respective regions. 2. The `primary_nomad_servers` module uses the `nomad-server` module to deploy a three node Nomad server cluster in the primary data center (`us-east-2`). Notice that it uses `var.primary_ami` for its AMI. main.tf module "primary_nomad_servers" { source = "./nomad-server" region = "us-east-2" ## ... ami = var.primary_ami server_instance_type = "t2.micro" server_count = 3 } 3. The `primary_nomad_clients` module uses the `nomad-client` module to deploy two Nomad clients in the primary data center (`us-east-2`). Notice that it uses the same AMI (`var.primary_ami`) as the server agent — the user script (`nomad-client/data-scripts/user-data-client.sh`) configures the Nomad agent as a client — and defines `nomad_dc` as `dc1`. main.tf module "primary_nomad_clients" { source = "./nomad-client" region = "us-east-2" ## ... ami = var.primary_ami client_instance_type = "t2.small" client_count = 1 nomad_dc = "dc1" } 4. The `edge_nomad_clients` module uses the `nomad-client` module to deploy one Nomad client in the edge data center (`us-west-1`). Notice that it uses `var.edge_ami` for its AMI and defines `nomad_dc` as `dc2`. main.tf module "edge_nomad_clients" { source = "./nomad-client" region = "us-west-1" ## ... ami = var.edge_ami client_instance_type = "t2.small" client_count = 2 nomad_dc = "dc2" } ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#define-ami-ids) Define AMI IDs Update `terraform.tfvars` to reflect the AMI IDs you built with Packer. The `primary_ami` should reference the AMI created in `us-east-2`; the `edge_ami` should reference the AMI created in `us-west-1`. terraform.tfvars primary_ami = "REPLACE_WITH_BUILD_AMI_ID" edge_ami = "REPLACE_WITH_BUILD_EDGE_AMI_ID" ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#deploy-nomad-cluster-and-clients) Deploy Nomad cluster and clients Initialize your Terraform configuration. $ terraform init Initializing modules... - edge_nomad_clients in nomad-client - edge_shared_resources in shared-resources Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.13.0 for edge_shared_resources.vpc... - edge_shared_resources.vpc in .terraform/modules/edge_shared_resources.vpc - primary_nomad_clients in nomad-client - primary_nomad_servers in nomad-server - primary_shared_resources in shared-resources Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.13.0 for primary_shared_resources.vpc... - primary_shared_resources.vpc in .terraform/modules/primary_shared_resources.vpc Initializing the backend... Initializing provider plugins... - Reusing previous version of hashicorp/aws from the dependency lock file - Reusing previous version of hashicorp/template from the dependency lock file - Installing hashicorp/aws v4.6.0... - Installed hashicorp/aws v4.6.0 (signed by HashiCorp) - Installing hashicorp/template v2.2.0... - Installed hashicorp/template v2.2.0 (signed by HashiCorp) Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. Then, apply your configuration to create the resources. Respond `yes` to the prompt to confirm the apply. $ terraform apply ## ... Plan: 41 to add, 0 to change, 0 to destroy. ## ... Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes ## ... Apply complete! Resources: 41 added, 0 changed, 0 destroyed. Outputs: edge_dc_nomad_client = "184.169.204.238" nomad_lb_address = "http://learn-nomad-edge-server-lb-1934725976.us-east-2.elb.amazonaws.com:4646" nomad_primary_dc_clients = [\ "3.15.5.228",\ ] nomad_server = "18.191.0.46" nomad_server_1 = "3.145.196.167" nomad_server_2 = "3.144.15.124" nomad_servers = [\ "18.191.0.46",\ "3.145.196.167",\ "3.144.15.124",\ ] primary_dc_nomad_client = "3.15.5.228" Once Terraform finishes provisioning the resources, display the `nomad_lb_address` Terraform output. $ terraform output -raw nomad_lb_address http://learn-nomad-edge-server-lb-1934725976.us-east-2.elb.amazonaws.com:4646 Open the link in your web browser to go to the Nomad UI. It should show an unauthorized page, since you have not provided the ACL bootstrap token. [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#bootstrap-nomad-acl) Bootstrap Nomad ACL ----------------------------------------------------------------------------------------------------------------------- Connect to one of your Nomad servers via SSH. $ ssh terraform@$(terraform output -raw nomad_server) -i ./learn-nomad-edge Run the following command to bootstrap the initial ACL token, parse the bootstrap token, and export it as an environment variable. Inside Nomad server $ export NOMAD_BOOTSTRAP_TOKEN=$(nomad acl bootstrap | grep -i secret | awk -F '=' '{print $2}') Then, apply the ACL policy. This is the ACL policy defined in [`packer/config/nomad-acl-user.hcl`](https://github.com/hashicorp-education/learn-nomad-edge/blob/main/packer/config/nomad-acl-user.hcl) . Inside Nomad server $ nomad acl policy apply -token $NOMAD_BOOTSTRAP_TOKEN -description "Policy to allow reading of agents and nodes and listing and submitting jobs in all namespaces." node-read-job-submit /ops/config/nomad-acl-user.hcl Successfully wrote "node-read-job-submit" ACL policy! Finally, create an ACL token for that policy. Keep this token in a safe place, you will use it in the next section to authenticate the Nomad UI to view the Nomad agents and jobs. Inside Nomad server $ nomad acl token create -token $NOMAD_BOOTSTRAP_TOKEN -name "read-token" -policy node-read-job-submit | grep -i secret | awk -F "=" '{print $2}' xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Create a management token. Unlike the previous ACL token, this management token can perform all operations. You will use this in future sections to authenticate the Nomad CLI to deploy jobs. Inside Nomad server $ nomad acl token create -token $NOMAD_BOOTSTRAP_TOKEN -type="management" -global=true -name="Replication Token" | grep -i secret | awk -F "=" '{print $2}' xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Close the SSH connection. Inside Nomad server $ exit ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#verify-nomad-cluster-and-clients) Verify Nomad cluster and clients Go to the Nomad UI and click on **ACL Tokens** in the top right corner. Enter the management ACL token in the **Secret ID** field and click on **Set Token**. You now have read permissions in the Nomad UI. ![Set ACL token in Nomad UI](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fset-acl-token.png%26width%3D1999%26height%3D941&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Click on **Servers** to confirm there are three nodes in your Nomad server cluster. ![Nomad UI shows three nodes in Nomad server cluster](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fthree-node-server-cluster-edge.png%26width%3D1999%26height%3D681&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Click on **Clients** to confirm there are three clients — one in the primary data center (`dc1`) and two in the edge data center (`dc2`). ![Nomad UI shows three clients across two Nomad data\ centers](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fthree-clients-edge.png%26width%3D1999%26height%3D825&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#connect-to-nomad-servers) Connect to Nomad servers --------------------------------------------------------------------------------------------------------------------------------- You need to set the `NOMAD_ADDR` and `NOMAD_TOKEN` environment variables so your local Nomad binary can connect to the Nomad cluster. First, set the `NOMAD_ADDR` environment variable to one of your Nomad servers. $ export NOMAD_ADDR="http://$(terraform output -raw nomad_server):4646" Then, set the `NOMAD_TOKEN` environment variable to the management token you created in the previous step. $ export NOMAD_TOKEN= List the Nomad server members to verify you successfully configured your Nomad binary. $ nomad server members Name Address Port Status Leader Raft Version Build Datacenter Region ip-10-0-101-18.global 18.191.0.46 4648 alive false 3 1.5.3 dc1 global ip-10-0-101-57.global 3.145.196.167 4648 alive true 3 1.5.3 dc1 global ip-10-0-101-69.global 3.144.15.124 4648 alive false 3 1.5.3 dc1 global [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#review-hashicups-jobs) Review HashiCups jobs --------------------------------------------------------------------------------------------------------------------------- The `jobs` directory contains the HashiCups jobs you will schedule in the primary and edge data centers. ![HashiCups services deployed in primary and edge data\ centers.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fdiagram_edge-with-hashicups-tasks.png%26width%3D1999%26height%3D446&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#review-the-hashicups-job) Review the HashiCups job Open `jobs/hashicups.nomad.hcl`. This Nomad job file will deploy the HashiCups database and `product-api` to the primary data center. The `hashicups` job contains a `hashicups` group which defines the HashiCups database and `product-api` tasks. Nomad will only deploy this job in the primary datacenter (`var.datacenters`). jobs/hashicups.nomad.hcl ## ... # Begin Job Spec job "hashicups" { type = "service" region = var.region datacenters = var.datacenters ## ... } In the `db` task, find the `service` stanza. jobs/hashicups.nomad.hcl ## ... job "hashicups" { ## ... group "hashicups" { ## ... task "db" { driver = "docker" meta { service = "database" } service { port = "db" tags = ["hashicups", "backend"] provider = "nomad" address = attr.unique.platform.aws.public-ipv4 } ## ... } } Since this job file defines the service provider as `nomad`, Nomad will register the service in its built-in service discovery. This will enable other Nomad tasks to query and connect to the service. Nomad's native service discovery lets you register and query services. Unlike Consul, it does not provide a service mesh and route traffic. This is preferable for edge computing where unstable connectivity could impact service mesh. In addition, it reduces resource consumption since you do not need to run a Consul agent on each edge device. Notice that the service stanza defines the `address` to the attribute associated with the EC2 instance's public IP address. Since the EC2 instance's kernel is unaware of its public IP address, Nomad cannot advertise the public IP address by default. For edge workloads that want to communicate with each other over the public Internet (like the HashiCups demo application), you must set the `address` to the attribute associated with the EC2 instance's public IP address for Nomad's native service discovery to list the correct address to connect to. jobs/hashicups.nomad.hcl service { port = "db" tags = ["hashicups", "backend"] provider = "nomad" address = attr.unique.platform.aws.public-ipv4 } The `product-api` task has a similar service stanza. This advertises the `product-api`'s address and port number, letting the `public-api` query Nomad's service discovery to connect to the `product-api` service. In the `product-api` task, find the `template` stanza. jobs/hashicups.nomad.hcl ## ... job "hashicups" { ## ... group "hashicups" { ## ... task "product-api" { driver = "docker" meta { service = "product-api" } template { data = < 2022-04-24T14:47:17-07:00: Monitoring evaluation "2e20a4db" 2022-04-24T14:47:17-07:00: Evaluation triggered by job "hashicups" ==> 2022-04-24T14:47:18-07:00: Monitoring evaluation "2e20a4db" 2022-04-24T14:47:18-07:00: Evaluation within deployment: "9b7f3d50" 2022-04-24T14:47:18-07:00: Allocation "3442ce01" created: node "0643838d", group "hashicups" 2022-04-24T14:47:18-07:00: Evaluation status changed: "pending" -> "complete" ==> 2022-04-24T14:47:18-07:00: Evaluation "2e20a4db" finished with status "complete" ==> 2022-04-24T14:47:18-07:00: Monitoring deployment "9b7f3d50" ✓ Deployment "9b7f3d50" successful 2022-04-24T14:47:30-07:00 ID = 9b7f3d50 Job ID = hashicups Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline hashicups 1 1 1 0 2022-04-24T21:57:28Z Submit the `hashicups-edge` job to deploy the tasks to the edge data center. $ nomad job run jobs/hashicups-edge.nomad.hcl ==> 2022-04-24T14:47:47-07:00: Monitoring evaluation "8756c237" 2022-04-24T14:47:47-07:00: Evaluation triggered by job "hashicups-edge" 2022-04-24T14:47:47-07:00: Evaluation within deployment: "66d4779e" 2022-04-24T14:47:47-07:00: Allocation "48af7a5e" created: node "6ba84888", group "hashicups-edge" 2022-04-24T14:47:47-07:00: Evaluation status changed: "pending" -> "complete" ==> 2022-04-24T14:47:47-07:00: Evaluation "8756c237" finished with status "complete" ==> 2022-04-24T14:47:47-07:00: Monitoring deployment "66d4779e" ✓ Deployment "66d4779e" successful 2022-04-24T14:48:29-07:00 ID = 66d4779e Job ID = hashicups-edge Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline hashicups-edge 1 1 1 0 2022-04-24T21:58:28Z ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#verify-hashicups-jobs) Verify HashiCups jobs List the Nomad services. Notice the service name contains the job name, group name, and task name, separated by a dash (`-`). $ nomad service list Service Name Tags hashicups-edge-hashicups-edge-frontend [frontend,hashicups] hashicups-edge-hashicups-edge-nginx [frontend,hashicups] hashicups-edge-hashicups-edge-payments-api [backend,hashicups] hashicups-edge-hashicups-edge-public-api [backend,hashicups] hashicups-hashicups-db [backend,hashicups] hashicups-hashicups-product-api [backend,hashicups] Retrieve detailed information about the `nginx` service. Since there are two Nomad clients on the edge datacenter, this command is useful to locate which client the service is running on. Notice that the `nginx` service's address reflects the address defined by the `advertise` stanza — the client's public IP address. $ nomad service info hashicups-edge-hashicups-edge-nginx Job ID Address Tags Node ID Alloc ID hashicups-edge 184.169.204.238:80 [hashicups,frontend] 6ba84888 e3b69fc2 Open the `nginx`'s address in your web browser to go to HashiCups. ![HashiCups frontend connected with the `product-api` to display an array of\ coffee images](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fhashicups-ui.png%26width%3D1999%26height%3D1389&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#simulate-client-disconnect) Simulate client disconnect ------------------------------------------------------------------------------------------------------------------------------------- When running and managing edge services, the network connection between your Nomad servers and edge services may be unstable. In this step, you will simulate the client running the `hashicups-edge` job disconnecting from the Nomad servers to learn how Nomad reacts to disconnected clients. Retrieve the `nginx` service's client IP address. For the example below, the client IP address is `184.169.204.238`. $ nomad service info hashicups-edge-hashicups-edge-nginx Job ID Address Tags Node ID Alloc ID hashicups-edge 184.169.204.238:80 [hashicups,frontend] 6ba84888 e3b69fc2 Export the client IP address as an environment variable named `CLIENT_IP`. Do not include the port. For example, the client IP address for this example would be `184.169.204.238`. $ export CLIENT_IP= Run the following command to drop all packets from the Nomad servers to the Nomad client that is currently hosting the `hashicups-edge` job. $ ssh terraform@$CLIENT_IP -i ./learn-nomad-edge \ 'sudo iptables -I INPUT -s '$(terraform output -raw nomad_server)' -j DROP && \ sudo iptables -I INPUT -s '$(terraform output -raw nomad_server_1)' -j DROP && \ sudo iptables -I INPUT -s '$(terraform output -raw nomad_server_2)' -j DROP' ### [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#verify-disconnected-client) Verify disconnected client Retrieve the `hashicups-edge` job's status. Notice that one of the allocations's status is now `unknown` and Nomad rescheduled the allocation onto a different client. Tip If the allocation status does not change, wait a couple of seconds before retrieving the job's status. If it does not change, verify that you dropped packets on the correct client. $ nomad status hashicups-edge ## ... Allocations ID Node ID Task Group Version Desired Status Created Modified 40f52550 da109b44 hashicups-edge 0 run pending 9s ago 8s ago 48af7a5e 6ba84888 hashicups-edge 0 run unknown 2m39s ago 9s ago This is the preferred behavior as the client instance is still up but could not connect to the Nomad status, like an edge network's unstable network connection. List the `nginx` service. Notice that Nomad lists _both_ services. This is because even though the original client cannot connect to the Nomad servers, it does not necessarily mean that the client is unavailable. As a result, Nomad continues to list the original client as available. $ nomad service info hashicups-edge-hashicups-edge-nginx Job ID Address Tags Node ID Alloc ID hashicups-edge 13.57.34.53:80 [hashicups,frontend] da109b44 40f52550 hashicups-edge 184.169.204.238:80 [hashicups,frontend] 6ba84888 48af7a5e Visit both addresses to find the HashiCups dashboard. [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#re-enable-client-connection) Re-enable client connection --------------------------------------------------------------------------------------------------------------------------------------- Run the following command to re-accept packets from the Nomad servers. $ ssh terraform@$CLIENT_IP -i ./learn-nomad-edge \ 'sudo iptables -D INPUT -s '$(terraform output -raw nomad_server)' -j DROP && \ sudo iptables -D INPUT -s '$(terraform output -raw nomad_server_1)' -j DROP && \ sudo iptables -D INPUT -s '$(terraform output -raw nomad_server_2)' -j DROP' Retrieve the `hashicups-edge` job's status. Notice that the original client status is now `running` and rescheduled allocation on the new client is now `complete`. Tip If the allocation status does not change, wait a couple of seconds before retrieving the job's status. If it does not change, verify that you re-accepted packets on the correct client. $ nomad status hashicups-edge ## ... Allocations ID Node ID Task Group Version Desired Status Created Modified 40f52550 da109b44 hashicups-edge 0 stop complete 3m42s ago 3s ago 48af7a5e 6ba84888 hashicups-edge 0 run running 6m12s ago 4s ago Since the original client reconnected _and_ the node rank on the rescheduled allocation is equal to or worse than the original client, Nomad resumed the original allocation and stopped the new one. Retrieve the re-connected allocation's status to find the reconnect event, replacing `ALLOC_ID` with your re-connected allocation ID. In this example, it is `48af7a5e`. $ nomad alloc status ALLOC_ID ## ... Recent Events: Time Type Description 2022-04-24T14:53:55-07:00 Reconnected Client reconnected 2022-04-24T14:48:01-07:00 Started Task started by client 2022-04-24T14:47:48-07:00 Driver Downloading image 2022-04-24T14:47:47-07:00 Task Setup Building Task Directory 2022-04-24T14:47:47-07:00 Received Task received by client List the `nginx` service. Notice that Nomad removed the completed job – it only lists the original service. $ nomad service info hashicups-edge-hashicups-edge-nginx Job ID Address Tags Node ID Alloc ID hashicups-edge 184.169.204.238:80 [hashicups,frontend] 6ba84888 48af7a5e [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#clean-up-resources) Clean up resources --------------------------------------------------------------------------------------------------------------------- Run `terraform destroy` to clean up your provisioned infrastructure. Respond `yes` to the prompt to confirm the operation. $ terraform destroy ## ... Plan: 0 to add, 0 to change, 20 to destroy. ## ... Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes ## ... Destroy complete! Resources: 20 destroyed. Your AWS account still has the AMI and its S3-stored snapshots, which you may be charged for depending on your other usage. Delete the AMI and snapshots stored in your S3 buckets. Note Remember to delete the AMI images and snapshots in both regions where you created them. If you didn't update the `region` variable in the `terraform.tfvars` file, they will be in the `us-east-2` and `us-west-1` regions. In your `us-east-2` AWS account, deregister the [AMI](https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Images:visibility=owned-by-me;v=3;tag:Name=learn-nomad-edge;sort=desc:creationDate) by selecting it, clicking on the **Actions** button, then the **Deregister AMI** option, and finally confirm by clicking the **Deregister AMI** button in the confirmation dialog. Delete the [snapshots](https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Snapshots:visibility=owned-by-me;v=3;tag:Name=learn-nomad-edge;sort=desc:creationDate) by selecting the snapshots, clicking on the **Actions** button, then the **Delete snapshot** option, and finally confirm by clicking the **Delete** button in the confirmation dialog. Then, delete the AMI images and snapshots in the `us-west-1` region. In your `us-west-1` AWS account, deregister the [AMI](https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#Images:visibility=owned-by-me;v=3;tag:Name=learn-nomad-edge;sort=desc:creationDate) by selecting it, clicking on the **Actions** button, then the **Deregister AMI** option, and finally confirm by clicking the **Deregister AMI** button in the confirmation dialog. Delete the [snapshots](https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#Snapshots:visibility=owned-by-me;v=3;tag:Name=learn-nomad-edge;sort=desc:creationDate) by selecting the snapshots, clicking on the **Actions** button, then the **Delete snapshot** option, and finally confirm by clicking the **Delete** button in the confirmation dialog. [](https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#next-steps) Next steps ----------------------------------------------------------------------------------------------------- In this tutorial, you deployed a single server cluster and distant client edge architecture. Then, you scheduled HashiCups on both on-prem and edge data centers, connecting its services with Nomad's native service discovery. Finally, you tested the disconnected client allocation by simulating unstable network connectivity between the Nomad clients and the server. For more information, check out the following resources. * Learn more about Nomad's [native service discovery](https://developer.hashicorp.com/nomad/docs/networking/consul#service-discovery) by visiting the Nomad documentation * Read more about [disconnected client allocation handling](https://developer.hashicorp.com/nomad/docs/job-specification/group#max-client-disconnect) by visiting the Nomad documentation * Complete the tutorials in the [Nomad ACL System Fundamentals](https://developer.hashicorp.com/nomad/docs/secure/acl) collection to configure a Nomad cluster for ACLs, bootstrap the ACL system, author your first policy, and grant a token based on the policy. **Was this tutorial helpful?** YesNo [Collection Overview\ \ Edge Computing](https://developer.hashicorp.com/nomad/tutorials/edge) [Next Collection\ \ Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) --- # Set up a Nomad cluster on AWS | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Cluster Setup](https://developer.hashicorp.com/nomad/tutorials/cluster-setup) This tutorial will guide you through deploying a Nomad cluster with [access control lists (ACLs)](https://developer.hashicorp.com/nomad/docs/secure/acl) enabled on AWS. Consider checking out the [cluster setup overview](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-overview) first as it covers the contents of the code repository used in this tutorial. [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#prerequisites) Prerequisites --------------------------------------------------------------------------------------------------------------- For this tutorial, you need: * [Packer 1.9.4 or later](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli) * [Terraform 1.2.0 or later](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) * [Nomad 1.7.7 or later](https://developer.hashicorp.com/nomad/tutorials/get-started) * An [AWS account](https://aws.amazon.com/account/) configured for [use with Terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication) Note This tutorial creates AWS resources that may not qualify as part of the [AWS free tier](https://aws.amazon.com/free) . Be sure to follow the [Cleanup](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#cleanup) process at the end of this tutorial so you don't incur any additional unnecessary charges. [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#clone-the-code-repository) Clone the code repository --------------------------------------------------------------------------------------------------------------------------------------- The cluster setup [code repository](https://github.com/hashicorp-education/learn-nomad-cluster-setup) contains configuration files for creating a Nomad cluster on AWS. It uses Consul for the initial setup of the Nomad servers and clients and enables ACLs for both Consul and Nomad. Clone the code repository. $ git clone https://github.com/hashicorp-education/learn-nomad-cluster-setup Navigate to the cloned repository folder. $ cd learn-nomad-cluster-setup Navigate to the `aws` folder. $ cd aws [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#create-the-nomad-cluster) Create the Nomad cluster ------------------------------------------------------------------------------------------------------------------------------------- There are two main steps to creating the cluster: building an Amazon Machine Image (AMI) with Packer and provisioning the cluster infrastructure with Terraform. Both Packer and Terraform require that some configurations be set before they run and these configuration variables are defined in the `variables.hcl.example` file. ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#update-the-variables-file-for-packer) Update the variables file for Packer Rename `variables.hcl.example` to `variables.hcl` and open it in your text editor. $ mv variables.hcl.example variables.hcl Update the `region` variable with your preferred AWS region. In this example, the region is `us-east-1`. The remaining variables are for Terraform and you update them after building the AMI. aws/variables.hcl # Packer variables (all are required) region = "us-east-1" ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#build-the-ami) Build the AMI Note Make sure that your AWS access credentials are set as environment variables as Packer uses them to build and register the AMI in AWS. Initialize Packer to download the required plugins. Tip `packer init` returns no output when it finishes successfully. $ packer init image.pkr.hcl Then, build the image and provide the variables file with the `-var-file` flag. Tip Packer will print out a `Warning: Undefined variable` message notifying you that some variables were set in `variables.hcl` but not used, this is only a warning. The build will still complete successfully. $ packer build -var-file=variables.hcl image.pkr.hcl # ... Build 'amazon-ebs' finished after 14 minutes 32 seconds. ==> Wait completed after 14 minutes 32 seconds ==> Builds finished. The artifacts of successful builds are: --> amazon-ebs: AMIs were created: us-east-1: ami-0445eeea5e1406960 Packer outputs the specific `ami id` once it finishes building the image. In this example, the value for the `ami id` would be `ami-0445eeea5e1406960`. ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#update-the-variables-file-for-terraform) Update the variables file for Terraform Open `variables.hcl` in your text editor and update the `ami` variable with the value output from the Packer build. In this example, the value is `ami-0445eeea5e1406960`. aws/variables.hcl # ... ami = "ami-0445eeea5e1406960" # These variables default to the values shown # and do not need to be updated unless you want to # change them # allowlist_ip = "0.0.0.0/0" # name = "nomad" # server_instance_type = "t2.micro" # server_count = "3" # client_instance_type = "t2.micro" # client_count = "3" The remaining variables in `variables.hcl` are optional. *  `allowlist_ip` is a CIDR range specifying which IP addresses are allowed to access the Consul and Nomad UIs on ports `8500` and `4646` as well as SSH on port `22`. The default value of `0.0.0.0/0` allows traffic from everywhere. *  `name` is a prefix for naming the AWS resources. *  `server_instance_type` and `client_instance_type` are the virtual machine instance types for the cluster server and client nodes, respectively. *  `server_count` and `client_count` are the number of nodes to create for the servers and clients, respectively. ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#deploy-the-nomad-cluster) Deploy the Nomad cluster Before you deploy the Nomad cluster, initialize the Terraform configuration to download the necessary providers and modules. $ terraform init Initializing the backend... # ... Initializing provider plugins... # ... Terraform has been successfully initialized! # ... Provision the resources and provide the variables file with the `-var-file` flag. Respond `yes` to the prompt to confirm the operation. The provisioning takes several minutes. Once complete, the Consul and Nomad web interfaces will become available. $ terraform apply -var-file=variables.hcl # ... Plan: 18 to add, 0 to change, 0 to destroy. # ... Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: Yes # ... Apply complete! Resources: 18 added, 0 changed, 0 destroyed. Outputs: IP_Addresses = < false eligible ready 6f5076b1 default dc1 ip-172-31-16-246 false eligible ready 5fc1e22c default dc1 ip-172-31-17-43 false eligible ready Navigate to the Nomad UI in your web browser with the URL in the `post-setup.sh` script output. Click on **Sign In** in the top right corner and log in with the bootstrap token saved in the `NOMAD_TOKEN` environment variable. Set the **Secret ID** to the token's value and click **Sign in with secret**. ![Nomad UI Access Control Tokens page](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fnomad-cluster%252Fnomad_ui_acl_token_page.jpg%26width%3D1163%26height%3D808&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Click on the **Clients** page from the sidebar navigation to explore the UI. ![Nomad Clients page](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fnomad-cluster%252Fnomad_ui_clients_page.jpg%26width%3D1312%26height%3D567&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#cleanup) Cleanup --------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#destroy-infrastructure) Destroy infrastructure Use `terraform destroy` to remove the provisioned infrastructure. Respond `yes` to the prompt to confirm removal. $ terraform destroy -var-file=variables.hcl # ... aws_instance.server[0]: Destruction complete after 30s aws_instance.server[1]: Still destroying... [id=i-017defd36b45408c1, 30s elapsed] aws_instance.server[1]: Destruction complete after 30s aws_iam_instance_profile.instance_profile: Destroying... [id=nomad20220613201917520400000002] aws_security_group.primary: Destroying... [id=sg-0ffdf8214d5fc85b2] aws_iam_instance_profile.instance_profile: Destruction complete after 0s aws_iam_role.instance_role: Destroying... [id=nomad20220613201916761200000001] aws_iam_role.instance_role: Destruction complete after 0s aws_security_group.primary: Destruction complete after 0s aws_security_group.server_lb: Destroying... [id=sg-016a74cc79f3f2826] aws_security_group.server_lb: Destruction complete after 1s Destroy complete! Resources: 18 destroyed. ### [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#delete-ami-and-s3-store-snapshots) Delete AMI and S3-store snapshots Your AWS account still has the AMI and its S3-stored snapshots, which you may be charged for depending on your other usage. Delete the AMI and snapshots stored in your S3 buckets. Note Remember to delete the AMI images and snapshots in the region where you created them. If you don’t update the `region` variable in the `terraform.tfvars` file, they are created in the `us-east-1` region. AWS CLIAWS UI Delete the stored AMI built using packer using the `deregister-image` command. $ aws ec2 deregister-image --image-id ami-0445eeea5e1406960 To delete stored snapshots, first query for the snapshot using the `describe-snapshots` command. $ aws ec2 describe-snapshots \ --owner-ids self \ --query "Snapshots[*].{ID:SnapshotId,Time:StartTime}" Next, delete the stored snapshot using the `delete-snapshot` command by specifying the `snapshot-id` value. $ aws ec2 delete-snapshot --snapshot-id snap-1234567890abcdef0 In your `us-east-1` AWS account, deregister the [AMI](https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=owned-by-me;v=3;tag:Name=nomad-alb;sort=desc:creationDate) : 1. Select the AMI 2. Click the **Actions** button 3. Click the **Deregister AMI** option 4. Click the **Deregister AMI** button to confirm that you want to deregister the AMI when prompted ![Deregistering the AMI in AWS](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fnomad-cluster%252Faws_deregister_ami.jpg%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Delete the [snapshots](https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=owned-by-me;v=3;tag:Name=nomad-alb;sort=desc:creationDate) : 1. Select the snapshot 2. Click the **Actions** button 3. Click the **Delete Snapshot** option 4. Click the **Delete** button to confirm that you want to delete the snapshot when prompted ![Deleting the Snapshots in AWS](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fnomad-cluster%252Faws_delete_snapshots.jpg%26width%3D2048%26height%3D1376&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-aws#next-steps) Next steps --------------------------------------------------------------------------------------------------------- In this tutorial you created a Nomad cluster on AWS with Consul and ACLs enabled. From here, you may want to: * Run a job with [a Nomad spec file](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job) or with [Nomad Pack](https://developer.hashicorp.com/nomad/tools/nomad-pack) * Test out [native service discovery](https://developer.hashicorp.com/nomad/tutorials/service-discovery/service-discovery-app-deployment) in Nomad For more information, check out the following resources. * Learn more about [managing your Nomad cluster](https://developer.hashicorp.com/nomad/tutorials/manage-clusters) * Read more about the [ACL stanza](https://developer.hashicorp.com/nomad/docs/configuration/acl) and using [ACLs](https://developer.hashicorp.com/nomad/docs/secure/acl) **Was this tutorial helpful?** YesNo [Previous\ \ Nomad on the cloud](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-overview) [Next\ \ Nomad on GCP](https://developer.hashicorp.com/nomad/tutorials/cluster-setup/cluster-setup-gcp) --- # Install a HashiCorp Enterprise license | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) Note This guidance for enabling an enterprise license applies to both production and development environments. To use a HashiCorp Enterprise product, you need a valid HashiCorp license. This tutorial will guide you on the steps to enable your enterprise license to start the server. By the end of this tutorial, you will have a server instance with enterprise features enabled. If you don't have a license, see the [request a license](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#request-a-trial-license) section. Tip This tutorial is applicable to the following HashiCorp enterprise product versions or greater. Nomad v1.1.0, Consul v1.10.0, Vault v1.8.0, and Boundary v0.13.0. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#prerequisites) Prerequisites ----------------------------------------------------------------------------------------------------------------------- * [Enterprise binary](https://releases.hashicorp.com/) Each server needs to register the enterprise license as described below. In earlier versions of HashiCorp enterprise products, one server could distribute a license to other servers via the Raft protocol. This will no longer work since each server must be able to find a valid license during the startup process. The tutorial below describes the most basic steps needed to register a license. You may need to design other steps to integrate this process into your build pipelines (such as building private disk images with [Packer](https://developer.hashicorp.com/packer/tutorials) or using a configuration management system to distribute and install the enterprise binary and license file to your compute instances). [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#install-the-enterprise-binary) Install the enterprise binary ------------------------------------------------------------------------------------------------------------------------------------------------------- Install the enterprise binary before continuing with the tutorial. All HashiCorp binaries are in [releases.hashicorp.com](https://releases.hashicorp.com/) , look for a binary that has `+ent` in the suffix of the filename. Review the installation tutorials for each HashiCorp product: * [Install Consul tutorial](https://developer.hashicorp.com/consul/tutorials/get-started-vms/virtual-machine-gs-deploy) * [Install Nomad tutorial](https://developer.hashicorp.com/nomad/tutorials/get-started/get-started-install) * [Install Vault guide](https://developer.hashicorp.com/vault/install) * [Install Boundary tutorial](https://developer.hashicorp.com/boundary/tutorials/get-started-community/community-get-started-install) [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#request-a-trial-license) Request a trial license ------------------------------------------------------------------------------------------------------------------------------------------- Your customer support contact can generate a trial license for any HashiCorp enterprise product. If you are an existing HashiCorp enterprise customer, you may contact your organization's [customer success manager](https://support.hashicorp.com/hc/en-us) (CSM) for information on how to get your organization's enterprise license. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#enable-the-license) Enable the license --------------------------------------------------------------------------------------------------------------------------------- You have three options for enabling an enterprise license. * Provide the enterprise license as a string in an environment variable. * Save the license string to a file and reference the path with an environment variable. * Save the license string in a file and specify the path to the file in the server's configuration file. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#add-an-environment-variable-optional) Add an environment variable (optional) ConsulNomadVaultBoundary You can set the enterprise license for Consul by using an environment variable. Use the enterprise license you received in the previous step and set the environment variable `CONSUL_LICENSE` to the license key value. $ export CONSUL_LICENSE=02MV4UU43BK5.... You can set the enterprise license for Nomad by using an environment variable. The environment variable name is `NOMAD_LICENSE`. Use the enterprise license you received in the previous step and set the environment variable `NOMAD_LICENSE` to the license key value. $ export NOMAD_LICENSE=02MV4UU43BK5.... You can set the enterprise license for Vault by using an environment variable. Use the enterprise license you received in the previous step and set the environment variable `VAULT_LICENSE` to the license key value. $ export VAULT_LICENSE=02MV4UU43BK5.... You can set the enterprise license for Boundary by using an environment variable. Use the enterprise license you received in the previous step and set the environment variable `BOUNDARY_LICENSE` to the license key value. $ export BOUNDARY_LICENSE=02MV4UU43BK5.... ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#read-from-a-file-optional) Read from a file (optional) The enterprise license can be read from a file. If you add the enterprise license to a file, you have two options for how to point the server instance to the file location. The two options are an environment variable or a server instance configuration file. Use the enterprise license key you received in the previous step and create a file that will hold the enterprise license. You can use the `echo` command to copy the content from your system's clipboard into a file. $ echo "02MV4UU43BK5..." >> license.hclic Verify the content copied correctly to the specified file before you move onto the next step. $ cat license.hclic 02MV4UU43BK5... Note You must provide an absolute path. You can use shell variables like `$HOME`, but not relative paths like `~/`. ConsulNomadVaultBoundary Environment VariableConfiguration File The environment variable name is `CONSUL_LICENSE_PATH`. Use the license file you created earlier and set the environment variable `CONSUL_LICENSE_PATH` to the license key path. $ export CONSUL_LICENSE_PATH=/etc/consul.d/license.hclic The Consul server instance accepts a configuration file. In the server instance configuration file, you may specify the enterprise license key file path through the `license_path` attribute. { "server": true, "license_path": "/etc/consul.d/license.hclic" } #### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#validate-the-license) Validate the license Run the following command to validate the contents of the license file on disk. $ consul license inspect /etc/consul.d/license.hclic Source: /etc/consul.d/license.hclic Product: consul License ID: 00000000-0000-0000-0000-000000000000 Customer ID: 00000000-0000-0000-0000-000000000000 Installation ID: * Issue Time: 2021-04-27 10:17:21.653194654 +0000 UTC Start Time: 2021-04-27 00:00:00 +0000 UTC Expiration Time: 2022-04-27 00:00:00 +0000 UTC Termination Time: 2023-04-27 00:00:00 +0000 UTC Modules: Global Visibility, Routing and Scale Governance and Policy Features: Automated Backups Automated Upgrades Enhanced Read Scalability Network Segments Redundancy Zone Advanced Network Federation Namespaces SSO Audit Logging Admin Partitions License is valid Environment VariableConfiguration File The environment variable name is `NOMAD_LICENSE_PATH`. Use the license file you created earlier and set the environment variable `NOMAD_LICENSE_PATH` to the license key path. $ export NOMAD_LICENSE_PATH=/etc/nomad.d/license.hclic The Nomad server instance accepts a configuration file. In the server instance configuration file, you may also specify the enterprise license key file path in the `server` stanza block. server { enabled = true license_path = "/etc/nomad.d/license.hclic" } Environment VariableConfiguration File The environment variable name is `VAULT_LICENSE_PATH`. Use the license file you created earlier and set the environment variable `VAULT_LICENSE_PATH` to the license key path. $ export VAULT_LICENSE_PATH=/etc/vault.d/license.hclic Use the `license` subcommand to validate the license. Note The `license` subcommand is only available with the Vault Enterprise binary. $ vault license inspect $VAULT_LICENSE_PATH Source: /etc/vault.d/license.hclic Product: vault License ID: S3sAm3st-8c10-9db0-2e8f-aB34Ef78 Customer ID: aB34Ef78-dc55-cb2c-ca2d-ef7ae4c2e080 Installation ID: * Issue Time: 2023-02-15 13:57:46.677406153 +0000 UTC Start Time: 2023-02-15 00:00:00 +0000 UTC Expiration Time: 2024-02-15 23:59:59.999 +0000 UTC Termination Time: 2024-02-15 23:59:59.999 +0000 UTC {"license_id":"S3sAm3st-8c10-9db0-2e8f-aB34Ef78","customer_id":...snip...} License is valid You can specify the enterprise license key file path in the Vault server configuration using the `license_path` attribute. license_path = "/etc/vault.d/license.hclic" #### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#license-setting-priority) License setting priority When both `VAULT_LICENSE` and `VAULT_LICENSE_PATH` environment variables exist, `VAULT_LICENSE` takes precedence. In addition, if the `license_path` is defined in the server configuration file and `VAULT_LICENSE` or `VAULT_LICENSE_PATH` exists, the environment variables take precedence over `license_path`. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#license-file-reload) License file reload Use the [`sys/config/reload/license`](https://developer.hashicorp.com/vault/api-docs/system/config-reload) endpoint to reload your Vault Enterprise license if needed. Vault reads the license from a file if specified using the `license_path` configuration parameter or the `VAULT_LICENSE_PATH` environment variable. $ vault write -force sys/config/reload/license Once the updated license is applied, Vault will enable or disable licensed features if the features allowed by the license are different. Environment VariableConfiguration File The environment variable name is `BOUNDARY_LICENSE` with `file://` prepended to the license path. Use the license file you created earlier and set the environment variable `BOUNDARY_LICENSE` to the license key path. $ export BOUNDARY_LICENSE=file:///etc/boundary.d/license.hclic You can specify the enterprise license key file path in the Boundary server configuration using the `license_path` attribute with `file://` prepended to the license path. license_path = "file:///etc/boundary.d/license.hclic" ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#validating-the-server) Validating the server ConsulNomadVaultBoundary Start the Consul server instance. $ consul agent -dev ==> Starting Consul agent... Version: '1.10.0+ent' Node ID: '93a230f3-888e-7fd7-81ea-04f6d43ab199' Node name: 'defaultNode' Datacenter: 'dc1' (Segment: '') Server: true (Bootstrap: false) Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600) Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: If the server instance started successfully, then you will receive a message stating the Consul agent started. Look for the lines `Server: true` and `Version: 1.10.0+ent` to verify that an enterprise server is running. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#error-message) Error message The server instance will notify you if the enterprise license key is not found. If you encounter the error message below or similar, please revisit the steps above. $ consul agent -dev ... 2021-06-03T07:29:30.220-0700 [ERROR] agent: Error starting agent: error="server agents require a license to be loaded via the configuration" 2021-06-03T07:29:30.220-0700 [INFO] agent: Exit code: code=1 Start the Nomad server instance. $ sudo nomad agent -dev ==> No configuration files loaded ==> Starting Nomad agent... ==> Nomad agent configuration: Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648 Bind Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648 Client: true Log Level: DEBUG Region: global (DC: dc1) Server: true Version: 1.1.0+ent ==> Nomad agent started! Log data will stream in below: If the server instance started successfully, then you will receive a message stating the Nomad agent started. Look for the lines `Server: true` and `Version: 1.1.0+ent` to verify that an enterprise server is running. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#error-message-1) Error message The server instance will notify you if the enterprise license key is not found. If you encounter the error message below or similar, please revisit the steps above. $ sudo nomad agent -dev ==> No configuration files loaded ==> Starting Nomad agent... ==> Error starting agent: server setup failed: failed to initialize enterprise licensing: failed to read license: license is missing. To add a license, configure "license_path" in your server configuration file, use the NOMAD_LICENSE environment variable, or use the NOMAD_LICENSE_PATH environment variable. Start the Vault server in dev mode for testing. $ vault server -dev -dev-root-token-id=root ==> Vault server configuration: Api Address: http://127.0.0.1:8200 Cgo: disabled Cluster Address: https://127.0.0.1:8201 Go Version: go1.16.5 Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") Log Level: info Mlock: supported: false, enabled: false Recovery Mode: false Storage: inmem Version: Vault v1.8.0+rc1 Version Sha: ce481f21f5a76bfc5717f43ac7362f1240e1aaff ==> Vault server started! Log data will stream in below: If the server instance started successfully, you will receive a message stating the **Vault server started**. Now, you should be able to login. $ vault login root Read the license status information. $ vault license get **Example output:** Key Value --- ----- expiration_time 2024-02-15T23:59:59.999Z features [HSM Performance Replication DR Replication MFA Sentinel Seal Wrapping Control Groups Performance Standby Namespaces KMIP Entropy Augmentation Transform Secrets Engine Lease Count Quotas Key Management Secrets Engine Automated Snapshots] license_id S3sAm3st-8c10-9db0-2e8f-aB34Ef78 performance_standby_count 9999 start_time 2023-02-15T00:00:00Z termination_time 2024-02-15T23:59:59.999Z ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#error-message-2) Error message When the enterprise license is missing, the following error will be displayed. Error initializing core: no autoloaded license provided and storage is not initialized. Licenses can be obtained at https://vaultproject.io/trial [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy="" [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set [INFO] core: security barrier not initialized [WARN] core: A valid license is required to be loaded but none was found. Licenses can be obtained at https://vaultproject.io/trial Be sure to set your enterprise license to be [read from an environment variable](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#add-an-environment-variable-optional) or [read from a file](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#read-from-a-file-optional) . Tip Refer to the [Vault Enterprise License Frequently Asked Questions (FAQs)](https://developer.hashicorp.com/vault/docs/enterprise/license/faq) to learn more about the changes introduced in Vault 1.8. Start the Boundary server instance. $ boundary dev ==> Boundary server configuration: [Bsr] AEAD Key Bytes: iL6unSVX3YryBrpd/hL1pjRBsXpJKuMX [Controller] AEAD Key Bytes: /tFhIn0aA3GOyTcK1GDpVhmyGKq51ueS [Recovery] AEAD Key Bytes: 0aQfmA1/wrgYsDjKCWJmD21FWSJIvcpM [Worker-Auth] AEAD Key Bytes: 0cSBAGCiZXtZw68H1+KIxh3lRJJo702J [Bsr] AEAD Type: aes-gcm [Recovery] AEAD Type: aes-gcm [Root] AEAD Type: aes-gcm [Worker-Auth-Storage] AEAD Type: aes-gcm [Worker-Auth] AEAD Type: aes-gcm Cgo: disabled Controller Public Cluster Addr: 127.0.0.1:9201 Dev Database Container: unruffled_bhaskara Dev Database Url: postgres://postgres:password@localhost:32770/boundary?sslmode=disable Generated Admin Login Name: admin Generated Admin Password: password Generated Host Catalog Id: hcst_1234567890 Generated Host Id: hst_1234567890 Generated Host Set Id: hsst_1234567890 Generated Ldap Auth Method Base Search DNs: users="ou=people,dc=example,dc=org" groups="ou=groups,dc=example,dc=org" Generated Ldap Auth Method Host:Port: 127.0.0.1:63906 (does not have a root DSE; use simple bind) Generated Ldap Auth Method Id: amldap_1234567890 Generated Oidc Auth Method Id: amoidc_1234567890 Generated Org Scope Id: o_1234567890 Generated Password Auth Method Id: ampw_1234567890 Generated Project Scope Id: p_1234567890 Generated Target With Address Id: ttcp_1234567890 Generated Target With Host Source Id: ttcp_0987654321 Generated Unprivileged Login Name: user Generated Unprivileged Password: password Listener 1: tcp (addr: "127.0.0.1:9200", cors_allowed_headers: "[]", cors_allowed_origins: "[*]", cors_enabled: "true", max_request_duration: "1m30s", purpose: "api") Listener 2: tcp (addr: "127.0.0.1:9201", max_request_duration: "1m30s", purpose: "cluster") Listener 3: tcp (addr: "127.0.0.1:9203", max_request_duration: "1m30s", purpose: "ops") Listener 4: tcp (addr: "127.0.0.1:9202", max_request_duration: "1m30s", purpose: "proxy") Log Level: info Mlock: supported: false, enabled: false Version: Boundary v0.13.0+ent Version Sha: e38ecf04f5a2a60850be2190b0607f93f0ae61a7 Worker Auth Current Key Id: disorder-eardrum-impeach-race-caregiver-remix-freestyle-immodest Worker Auth Storage Path: /var/folders/lx/dq871x7x3rv2ghc0fwg_z1y40000gq/T/nodeenrollment837482576 Worker Public Proxy Addr: 127.0.0.1:9202 ==> Boundary server started! Log data will stream in below: Look for the line `Version: Boundary v0.13.0+ent` to verify that an enterprise server is running. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#error-message-3) Error message The server instance will notify you if the enterprise license key is not found. If you encounter the error message below or similar, please revisit the steps above. $ boundary dev Error creating controller dev config: error parsing dev config: No license supplied in controller block or BOUNDARY_LICENSE env var [](https://developer.hashicorp.com/nomad/tutorials/enterprise/hashicorp-enterprise-license#next-steps) Next steps ----------------------------------------------------------------------------------------------------------------- In this tutorial, you deployed a server instance for an enterprise HashiCorp product using a valid enterprise license key. You may now continue exploring other enterprise tutorials for Consul, Nomad, or Vault. [Consul Enterprise Tutorials](https://developer.hashicorp.com/consul/tutorials/enterprise) [Nomad Enterprise Tutorials](https://developer.hashicorp.com/nomad/tutorials/enterprise) [Vault Enterprise Tutorials](https://developer.hashicorp.com/vault/tutorials/enterprise) **Was this tutorial helpful?** YesNo [Collection Overview\ \ Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) [Next\ \ Discover reference architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) This tutorial also appears in: ------------------------------ *  [](https://developer.hashicorp.com/boundary/tutorials/enterprise) 5 tutorials Boundary Enterprise Learn operational tasks specific to creating and operating a Boundary Enterprise environment. * Boundary *  [](https://developer.hashicorp.com/vault/tutorials/enterprise) 23 tutorials Vault Enterprise Learn features that are only available to Vault Enterprise. * Vault --- # nomad agent-info command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad agent-info command reference ================================== The `nomad agent-info` command retrieves metrics and status information for a running agent. The information returned pertains to the specific agent the CLI is connected to. This is useful for troubleshooting and performance monitoring. [](https://developer.hashicorp.com/nomad/commands/agent-info#usage) Usage ------------------------------------------------------------------------- nomad agent-info [options] When ACLs are enabled, this command requires a token with the `agent:read` capability. [](https://developer.hashicorp.com/nomad/commands/agent-info#options) Options ----------------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-json`](https://developer.hashicorp.com/nomad/commands/agent-info#json) : Output agent info in its JSON format. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-t`](https://developer.hashicorp.com/nomad/commands/agent-info#t) : Format and display agent info using a Go template. [](https://developer.hashicorp.com/nomad/commands/agent-info#output) Output --------------------------------------------------------------------------- Depending on the agent queried, the `nomad agent-info` command retrieves information from the following subsystems: * Client: Status of the local Nomad client * Nomad: Status of the local Nomad server * Serf: Gossip protocol metrics and information * Raft: Status information about the Raft consensus protocol * Runtime: Various metrics from the runtime environment [](https://developer.hashicorp.com/nomad/commands/agent-info#example) Example ----------------------------------------------------------------------------- $ nomad agent-info raft commit_index = 0 fsm_pending = 0 last_contact = never last_snapshot_term = 0 state = Follower term = 0 applied_index = 0 last_log_index = 0 last_log_term = 0 last_snapshot_index = 0 num_peers = 0 runtime cpu_count = 4 goroutines = 43 kernel.name = darwin max_procs = 4 version = go1.5 arch = amd64 serf intent_queue = 0 member_time = 1 query_queue = 0 event_time = 1 event_queue = 0 failed = 0 left = 0 members = 1 query_time = 1 encrypted = false client heartbeat_ttl = 0 known_servers = 0 last_heartbeat = 9223372036854775807 num_allocations = 0 nomad bootstrap = false known_regions = 1 leader = false server = true [](https://developer.hashicorp.com/nomad/commands/agent-info#general-options) General options --------------------------------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-address=`](https://developer.hashicorp.com/nomad/commands/agent-info#address) : The address of the Nomad server. Overrides the `NOMAD_ADDR` environment variable if set. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-region=`](https://developer.hashicorp.com/nomad/commands/agent-info#region) : The region of the Nomad server to forward commands to. Overrides the `NOMAD_REGION` environment variable if set. Defaults to the Agent's local region. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-no-color`](https://developer.hashicorp.com/nomad/commands/agent-info#no-color) : Disables colored command output. Alternatively, `NOMAD_CLI_NO_COLOR` may be set. This option takes precedence over `-force-color`. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-force-color`](https://developer.hashicorp.com/nomad/commands/agent-info#force-color) : Forces colored command output. This can be used in cases where the usual terminal detection fails. Alternatively, `NOMAD_CLI_FORCE_COLOR` may be set. This option has no effect if `-no-color` is also used. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-ca-cert=`](https://developer.hashicorp.com/nomad/commands/agent-info#ca-cert) : Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. Overrides the `NOMAD_CACERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-ca-path=`](https://developer.hashicorp.com/nomad/commands/agent-info#ca-path) : Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `-ca-cert` and `-ca-path` are specified, `-ca-cert` is used. Overrides the `NOMAD_CAPATH` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-client-cert=`](https://developer.hashicorp.com/nomad/commands/agent-info#client-cert) : Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `-client-key`. Overrides the `NOMAD_CLIENT_CERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-client-key=`](https://developer.hashicorp.com/nomad/commands/agent-info#client-key) : Path to an unencrypted PEM encoded private key matching the client certificate from `-client-cert`. Overrides the `NOMAD_CLIENT_KEY` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-tls-server-name=`](https://developer.hashicorp.com/nomad/commands/agent-info#tls-server-name) : The server name to use as the SNI host when connecting via TLS. Overrides the `NOMAD_TLS_SERVER_NAME` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-tls-skip-verify`](https://developer.hashicorp.com/nomad/commands/agent-info#tls-skip-verify) : Do not verify TLS certificate. This is highly not recommended. Verification will also be skipped if `NOMAD_SKIP_VERIFY` is set. * [](https://developer.hashicorp.com/nomad/commands/agent-info#) [`-token`](https://developer.hashicorp.com/nomad/commands/agent-info#token) : The SecretID of an ACL token to use to authenticate API requests with. Overrides the `NOMAD_TOKEN` environment variable if set. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/agent-info.mdx) --- # Create a Nomad cluster | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) Nomad is a flexible scheduler that can run on your local machine as a single node cluster or as a multi-node cluster on virtualized or physical infrastructure. In this tutorial, you will create a cluster locally on your machine or on one of the major cloud providers, verify connectivity to the cluster, and interact with Nomad via the UI. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#prerequisites) Prerequisites -------------------------------------------------------------------------------------------------------------- * Docker Desktop [installed locally](https://docs.docker.com/get-docker/) Nomad supports many [task drivers](https://developer.hashicorp.com/nomad/docs/drivers) but the example application in this tutorial uses Docker. Docker Desktop provides the Docker engine, the CLI client, and a graphical interface. Other third-party options are available for running Docker on your machine but we recommend Docker Desktop for the simplicity and ease of use. Nomad uses a socket file to communicate with Docker and [by default](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker#endpoint) , this file is `/var/run/docker.sock` on Linux, Mac, and other Unix platforms, and `/./pipe/docker_engine` on Windows. Docker Desktop creates a symlink to this socket during installation. If the Nomad client fails to detect the Docker driver with the error `Constraint "missing drivers": 1 nodes excluded by filter`, the issue may be this missing symlink. Additional information about this symlink for [Mac](https://docs.docker.com/desktop/mac/permission-requirements/#installing-symlinks) and [Windows](https://docs.docker.com/desktop/windows/permission-requirements/) is available. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#clone-the-code-repository) Clone the code repository -------------------------------------------------------------------------------------------------------------------------------------- The [example repository](https://github.com/hashicorp-education/learn-nomad-getting-started) includes Terraform configurations for starting a cluster on the cloud as well as the jobspec files for the example application. Clone the code repository. $ git clone https://github.com/hashicorp-education/learn-nomad-getting-started Navigate to the repository folder. $ cd learn-nomad-getting-started Check out the `v1.1` tag of the repository as a local branch named `nomad-getting-started`. $ git checkout -b nomad-getting-started v1.1 Switched to a new branch 'nomad-getting-started' [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#create-the-cluster) Create the cluster ------------------------------------------------------------------------------------------------------------------------ Start a Nomad cluster. You will use the Nomad CLI to access your cluster regardless of its location. Select where you want to create and run your cluster. LocalCloud Open your terminal and start the development agent. This creates a Nomad cluster of one node that acts as both the server and client. Warning This tutorial uses the network interface attached to the default route on your machine and as a result, workloads will be accessible to other machines on the same network. This is for development and illustration purposes. We recommend using a firewall for production environments or when connected to public networks to limit unauthorized workload access. The `-bind` flag set to `0.0.0.0` instructs Nomad to listen on all network interfaces present on the machine. The [`-network-interface` flag](https://developer.hashicorp.com/nomad/docs/configuration/client#network_interface) instructs Nomad to use a network interface other than the default loopback interface (`localhost`). In this example, Nomad automatically gets the network interface attached to the default route on the machine with the [`GetDefaultInterfaces` function](https://github.com/hashicorp/go-sockaddr) but you can also provide the name of an interface. This flag is necessary for the example application as each service runs in a different container and cannot reach the other services on the loopback address. Note You may encounter a permission denied error when creating the allocation directory. If so, run the nomad agent command without `sudo`. Leave Nomad running in this terminal session. $ sudo nomad agent -dev \ -bind 0.0.0.0 \ -network-interface='{{ GetDefaultInterfaces | attr "name" }}' In a second terminal session, export the cluster address. $ export NOMAD_ADDR=http://localhost:4646 Each of the following cloud setups use Terraform to create and configure the server and client nodes. During this process, Terraform installs the dependencies and the Nomad binary, and configures Nomad with user startup scripts. Be sure that you have [Terraform](https://developer.hashicorp.com/terraform/install) installed and configured. This process will take about 5 minutes depending on the cloud provider. Warning We do not recommend this guide for a production deployment of Nomad. It is a method for quickly creating a multi-node cluster for learning, development, and testing purposes. When you are ready to move on to a production deployment, check out the [cluster setup](https://developer.hashicorp.com/nomad/tutorials/cluster-setup) collection and the [reference architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) guide for best practices. Choose the cloud provider below that you want to use. AWSGCPAzure Make sure that you have your [AWS access credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) set as environment variables. Navigate to the `aws` folder. $ cd aws Rename the example variables file to `terraform.tfvars`. $ mv terraform.tfvars.example terraform.tfvars Update the `region` variable in `terraform.tfvars` with your AWS region preference. Save the file. aws/terraform.tfvars region = "us-east-2" Make sure that you have the [`gcloud` CLI tool](https://cloud.google.com/sdk/docs/install) installed on your machine. Navigate to the `gcp` folder. $ cd gcp Rename the example variables file to `terraform.tfvars`. $ mv terraform.tfvars.example terraform.tfvars Log in to GCP with `gcloud` in your terminal and follow the prompts to complete the login process. $ gcloud auth login Your browser has been opened to visit: https://accounts.google.com/o/oauth2/auth?response_type=code[...] You are now logged in as [YOUR_GCP_ACCOUNT]. Your current project is [YOUR_CURRENT_PROJECT]. You can change this setting by running: $ gcloud config set project PROJECT_ID Set the `project`, `region`, and `zone` configurations in `gcloud`. Note If you already have a project in your GCP account, these configurations will be set for you as part of the login step. If not, first [create a project](https://developers.google.com/workspace/guides/create-project) . Set `project` to the project ID. $ gcloud config set project Updated property [core/project]. Then, set `region` to the associated region. $ gcloud config set compute/region Updated property [compute/region]. Finally, set `zone` to the associated zone. Note that the zone must be in the region set above. $ gcloud config set compute/zone Updated property [compute/zone]. List the configurations with `gcloud`. $ gcloud config list [compute] region = us-east1 zone = us-east1-b [core] account = [GCP_ACCOUNT] disable_usage_reporting = True project = hc-3ff63253e6a54756b207e4d4727 Copy the values for `project`, `region`, and `zone` into `terraform.tfvars`. In this example, those would be `hc-3ff63253e6a54756b207e4d4727`, `us-east1`, and `us-east1-b`. Save the file. gcp/terraform.tfvars region = "us-east1" zone = "us-east1-b" project = "hc-3ff63253e6a54756b207e4d4727" Make sure that you have the [`az` CLI tool](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed on your machine. Navigate to the `azure` folder. $ cd azure Rename the example variables file to `terraform.tfvars`. $ mv terraform.tfvars.example terraform.tfvars Open your terminal, log in to Azure with `az`, and follow the prompts to complete the login process. $ az login A web browser has been opened at https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`. [\ {\ "cloudName": "AzureCloud",\ "homeTenantId": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29",\ "id": "0e3e2e88-47a3-4107-a2b2-f325314dfb67",\ "isDefault": true,\ "managedByTenants": [\ {\ "tenantId": "c9ed8610-2016-4bf5-b919-437a07bf2464"\ }\ ],\ "name": "[SUBSCRIPTION_NAME]",\ "state": "Enabled",\ "tenantId": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29",\ "user": {\ "name": "[USER_EMAIL]",\ "type": "user"\ }\ }\ ] Copy the values for `id` and `tenantId` and paste them into the `terraform.tfvars` file as values for `subscription_id` and `tenant_id`. For this example, the value for `subscription_id` would be `0e3e2e88-47a3-4107-a2b2-f325314dfb67` and `tenant_id` would be `1e472a2a-7ab3-9bd1-2016-a32fd04dfb29`. Save the file. azure/terraform.tfvars location = "LOCATION" subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67" tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29" client_id = "CLIENT_ID" client_secret = "CLIENT_SECRET" Next, create an [Azure service principal](https://learn.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) by providing the value of `subscription_id` to the `--scopes` argument. $ az ad sp create-for-rbac \ --role="Contributor" \ --scopes="/subscriptions/0e3e2e88-47a3-4107-a2b2-f325314dfb67" Creating 'Contributor' role assignment under scope '/subscriptions/0e3e2e88-47a3-4107-a2b2-f325314dfb67' The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli { "appId": "ab3cb7b2-c932-4eb7-89ce-a369de998a37", "displayName": "azure-cli-2022-12-02-15-40-24", "password": "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS", "tenant": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29" } Copy the values for `appId` and `password` and paste them into the `terraform.tfvars` file as values for `client_id` and `client_secret`. For this example, the value for `client_id` would be `ab3cb7b2-c932-4eb7-89ce-a369de998a37` and `client_secret` would be `UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS`. Save the file. azure/terraform.tfvars location = "LOCATION" subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67" tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29" client_id = "ab3cb7b2-c932-4eb7-89ce-a369de998a37" client_secret = "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS" Update the `location` variable with your [Azure location](https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/#choose-your-region) preference. Save the file. azure/terraform.tfvars location = "eastus" subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67" tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29" client_id = "ab3cb7b2-c932-4eb7-89ce-a369de998a37" client_secret = "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS" ### [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#deploy-the-nomad-cluster) Deploy the Nomad cluster Initialize Terraform to download required plugins and set up the workspace. $ terraform init Provision the resources. Respond `yes` to the prompt to confirm the operation. The provisioning takes a couple of minutes. $ terraform apply # ... Apply complete! Outputs: IP_Addresses = < nomad-management.token Export the token as the `NOMAD_TOKEN` environment variable. $ export NOMAD_TOKEN=$(cat nomad-management.token) Your CLI is now set up to interact with the cluster. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#verify-connectivity) Verify connectivity -------------------------------------------------------------------------------------------------------------------------- Verify connectivity to the cluster by checking the status of the nodes. The output you see may be different than the example output depending on where your cluster is running and how many nodes it has. LocalCloud $ nomad node status ID DC Name Class Drain Eligibility Status 13416cb7 dc1 user-C05G17CLKD false eligible ready $ nomad node status ID DC Name Class Drain Eligibility Status 83a1527e dc1 ip-172-31-36-72 false eligible ready 4aeff813 dc1 ip-172-31-32-8 false eligible ready [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#browse-the-web-ui) Browse the web UI ---------------------------------------------------------------------------------------------------------------------- LocalCloud Navigate to the Nomad UI in your web browser by visiting [`http://localhost:4646/ui`](http://localhost:4646/ui) (opens in new tab). Open the Nomad UI from your terminal. This [`ui` command](https://developer.hashicorp.com/nomad/commands/ui) reads the `NOMAD_ADDR` and `NOMAD_TOKEN` environment variables, opens the UI in your browser, and logs in. $ nomad ui -authenticate Opening URL "http://3.14.8.189:4646/ui" with one-time token Alternatively, you can navigate to the Nomad UI in your web browser, click the **Sign In** link in the top right of the page, and log in with the bootstrap token saved in the `NOMAD_TOKEN` variable or the `nomad-management.token` file by setting the **Secret ID** to the token's value. ![A screenshot of the Nomad UI displaying the sign in page with the Secret ID field](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_ui_sign_in_with_token.jpg%26width%3D1163%26height%3D820&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Visit the **Servers** page from the left navigation to see the server nodes in the cluster and the **Clients** page to see the client nodes. These pages will show only one node each if you are running a local development agent. ![A screenshot of the Nomad UI displaying the list of servers in the cluster](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_ui_servers.jpg%26width%3D1180%26height%3D820&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![A screenshot of the Nomad UI displaying the list of clients in the cluster](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_ui_clients.jpg%26width%3D1180%26height%3D820&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Visit the **Topology** page to see an overview of the clients, the resources available on each, and the total resources available to the cluster. ![A screenshot of the Nomad UI displaying the cluster topology](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_ui_topology.jpg%26width%3D1180%26height%3D820&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster#next-steps) Next steps -------------------------------------------------------------------------------------------------------- In this tutorial you created a Nomad cluster. Continue on to the next tutorial by clicking on the **Next** button below and learn how to submit and run a job. **Was this tutorial helpful?** YesNo [Previous\ \ Install Nomad](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install) [Next\ \ Deploy and update a job](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v0.12.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation! This documentation is a reference guide for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [introduction and getting started guide](https://nomadproject.io/intro/v0.12.x) instead. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v0.12.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v0.11.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation! This documentation is a reference guide for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [introduction and getting started guide](https://nomadproject.io/intro/v0.11.x) instead. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v0.11.x/content/docs/index.mdx) --- # Consul service mesh in production | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Consul](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) Consul service mesh can secure all inter-service communication with mutual TLS. It's designed to work with minimal configuration out of the box on platforms including [Kubernetes](https://developer.hashicorp.com/consul/tutorials/kubernetes-features/service-mesh-deploy) or [VMs](https://developer.hashicorp.com/consul/tutorials/developer-mesh/service-mesh-deploy-vms) . However, completing the [security checklist](https://developer.hashicorp.com/consul/docs/secure-mesh/best-practice) and understanding the [Consul security model](https://developer.hashicorp.com/consul/docs/secure) are prerequisites for production deployments. After completing this tutorial, you will be able to configure Consul service mesh to secure services. First, you will secure your Consul datacenter with ACLs and TLS encryption. Next, you will enable Consul service mesh on the servers and host. Finally, you will configure your services to be part of Consul service mesh. Note To complete this tutorial you should already have a Consul datacenter with an appropriate number of servers and clients deployed according to the other reference material including the [deployment](https://developer.hashicorp.com/consul/tutorials/production-deploy/deployment-guide) and [performance](https://developer.hashicorp.com/consul/docs/reference/architecture/server) tutorials. The steps you need to get to a secure Consul datacenter with service mesh enabled are: * [Configure ACLs](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-acls) * [Configure agent transport encryption](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-agent-transport-encryption) * [Bootstrap Certificate Authority for Consul service mesh](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#bootstrap-certificate-authority-for-consul-service-mesh) * [Setup host firewall](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#setup-host-firewall) * [Configure service instances](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-service-instances) * [Next steps](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#next-steps) For existing Consul deployments, it may be necessary to incrementally adopt Consul service mesh service-by-service. In this case, step one and two should already be complete. However, we recommend reviewing all steps since the final deployment goal is to be compliant with all the security recommendations in this tutorial. Note Previously, Consul service mesh was known as Consul Connect. This term is now deprecated and references are being removed from the docs. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-acls) Configure ACLs ------------------------------------------------------------------------------------------------------------------------------------ Consul service mesh security is based on service identity. In practice, the identity of the service is only enforceable with sufficiently restrictive ACLs. This section will not replace reading the full [Secure Consul with ACLs tutorial](https://developer.hashicorp.com/consul/tutorials/security/access-control-setup-production) but will highlight the specific requirements Consul service mesh relies on to ensure its security properties. A service's identity, in the form of an x.509 certificate, will only be issued to an API client that has `service:write` permission for that service. In other words, any client that has permission to _register_ an instance of a service will be able to identify as that service and access all of the resources that that service is allowed to access. A secure ACL setup must meet the following criteria. 1. **[ACL default policy](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/acl#acl_default_policy) must be `deny`.** If for any reason you cannot use the default policy of `deny`, you must add an explicit ACL denying anonymous `service:write`. Note, in this case the intention graph will also default to `allow` and explicit `deny` intentions will be needed to restrict service access. Also note that explicit rules to limit who can manage intentions are necessary in this case. It is assumed for the remainder of this tutorial that ACL policy defaults to `deny`. 2. **Each service must have a unique ACL token** that is restricted to `service:write` only for the named service. You can review the [Securing Consul with ACLs](https://developer.hashicorp.com/consul/tutorials/security/access-control-setup-production#apply-individual-tokens-to-the-services) tutorial for a service token example. Note, it is best practices for each instance to get a unique token as described below. Individual Service Tokens: It is best practice to create a unique ACL token per service _instance_ because it limits the blast radius of a compromise. However, since intentions manage access based only on service identity, it is possible to create only one ACL token per _service_ and share it between instances. In practice, managing per-instance tokens requires automated ACL provisioning, for example using [HashiCorp's Vault](https://developer.hashicorp.com/vault/docs/secrets/consul) . [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-agent-transport-encryption) Configure agent transport encryption -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consul's gossip (UDP) and RPC (TCP) communications need to be encrypted otherwise attackers may be able to see ACL tokens while in flight between the server and client agents (RPC) or between client agent and application (HTTP). Certificate private keys never leave the host they are used on but are delivered to the application or proxy over local HTTP so local agent traffic should be encrypted where potentially untrusted parties might be able to observe localhost agent API traffic. Follow the [Gossip Encryption](https://developer.hashicorp.com/consul/tutorials/security/gossip-encryption-secure) tutorial to ensure both gossip encryption is enabled in your Consul datacenter and the [Secure Agent Communication with TLS Encryption](https://developer.hashicorp.com/consul/tutorials/security/tls-encryption-secure) tutorial to ensure RPC, HTTP and consensus communications are configured securely. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#bootstrap-certificate-authority-for-consul-service-mesh) Bootstrap Certificate Authority for Consul service mesh ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consul service mesh comes with a built-in Certificate Authority (CA) that will bootstrap by default when you first [enable](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/service-mesh#connect_enabled) Consul service mesh on your servers. To use the built-in CA, enable it in the server's configuration. connect { enabled = true } This configuration change requires a Consul server restart, which you can perform one server at a time to maintain availability in an existing datacenter. When a server is enabled with Consul Service mesh and becomes the leader, it will bootstrap a new CA and generate it's own private key which is written to the Raft state. Alternatively, an external private key can be provided via the [CA configuration](https://developer.hashicorp.com/consul/docs/connect/ca/consul#specifying-a-custom-private-key-and-root-certificate) . External CAs: Consul has been designed with a pluggable CA component so external CAs can be integrated. For production workloads we recommend using [Vault or another external CA](https://developer.hashicorp.com/consul/docs/secure-mesh/certificate/vault) once available such that the root key is not stored within Consul state at all. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#setup-host-firewall) Setup host firewall ---------------------------------------------------------------------------------------------------------------------------------------------- In order to enable inbound connections to Consul agents and the sidecar proxies, you may need to configure host or network firewalls to allow incoming connections to necessary ports. In addition to Consul agent's [communication ports](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#ports) any [proxies](https://developer.hashicorp.com/consul/docs/connect/proxy) will need to have ports open to accept incoming connections. If using [sidecar service registration](https://developer.hashicorp.com/consul/docs/connect/registration/sidecar-service) Consul will by default assign ports from [a configurable range](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#sidecar_min_port) . The default range is 21000 - 21255. If this feature is used, the agent assumes all ports in that range are both free to use (no other processes listening on them) and are exposed in the firewall to accept connections from other service hosts. It is possible to prevent automated port selection by [configuring `sidecar_min_port` and `sidecar_max_port`](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#sidecar_min_port) to both be `0`, forcing any sidecar service registrations to need an explicit port configured. It then becomes the same problem as opening ports necessary for any other application and might be managed by configuration management or a scheduler. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-service-instances) Configure service instances -------------------------------------------------------------------------------------------------------------------------------------------------------------- With [necessary ACL tokens](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#configure-acls) in place, all service registrations need to have an appropriate ACL token present. For on-disk configuration the `token` parameter of the service definition must be set. HCLJSON service { name = "cassandra_db" port = 9002 token = "" } { "service": { "name": "cassandra_db", "port": 9002, "token": "" } } For registration via the API the token is passed in the [request header](https://developer.hashicorp.com/consul/api-docs#authentication) , `X-Consul-Token`, or by using the [Go client configuration](https://godoc.org/github.com/hashicorp/consul/api#Config) . To avoid the overhead of a proxy, applications may [natively integrate](https://developer.hashicorp.com/consul/docs/automate/native) with Consul service mesh. Protect Application Listener: If using any kind of proxy for connect, the application must ensure no untrusted connections can be made to it's unprotected listening port. This is typically done by binding to `localhost` and only allowing loopback traffic, but may also be achieved using firewall rules or network namespacing. For examples of proxy service definitions see the [proxy documentation](https://developer.hashicorp.com/consul/docs/connect/proxy) . [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist#next-steps) Next steps ---------------------------------------------------------------------------------------------------------------------------- After securing your Consul datacenter with ACLs and TLS encryption, you can use Consul service mesh to secure service-to-service communication. If you encounter any issues while setting up Consul, there are many [community](https://www.consul.io/community) resources where you can find help. **Was this tutorial helpful?** YesNo [Previous\ \ Secure jobs with service mesh](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh) [Next\ \ Consul API Gateway on Nomad](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad) --- # Nomad Pack advanced usage | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Pack advanced usage ========================= In the [Nomad Pack overview guide](https://developer.hashicorp.com/nomad/tools/nomad-pack) you learned about Nomad Pack, how to deploy applications with it, and how to discover community packs. In this guide, you will learn about more advanced usage including: * Generating a variable file * Rendering a Pack * Advanced `run` options * Planning with `plan` * Registry management [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#generating-a-variable-file) Generating a variable file ------------------------------------------------------------------------------------------------------------------------------- You can pass in variables to a pack with a variables file. $ nomad-pack run hello_world -f ./my-variables.hcl The `generate var-file` command can generate variable files from a pack. $ nomad-pack generate var-file hello_world -o ./my-variables.hcl You can also use the path to a pack instead of a pack name. $ nomad-pack generate var-file ./my-local-pack -o ./my-variables.hcl [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#rendering-a-pack) Rendering a Pack ----------------------------------------------------------------------------------------------------------- At times, you may wish to use Nomad Pack to render jobspecs, but you will not want to immediately deploy these to Nomad. This can be useful when writing a pack, debugging deployments, integrating Nomad Pack into a CI/CD environment, or if you have another mechanism for handling Nomad deploys. The `render` command takes the same `--var` and `--var-file` flags that `run` takes. The `--to-dir` flag determines the directory where the rendered templates will be written. You can pass the `--render-output-template` flag to additionally render the output template. Some output templates rely on a deployment for information. In these cases, the output template may not render with all the necessary information. $ nomad-pack render hello_world --to-dir ./tmp --var greeting=hola --render-output-template [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#advanced-run-options) Advanced `run` options --------------------------------------------------------------------------------------------------------------------- To deploy the resources in a pack to Nomad, use the `run` command. $ nomad-pack run hello_world By passing a `--name` value into `run`, Nomad Pack deploys each resource in the pack with a metadata value for pack name. If no name is given, the pack name is used by default. This allows Nomad Pack to manage multiple deployments of the same pack. $ nomad-pack run hello_world --name hola-mundo It is also possible to run a local pack directly from the pack directory by passing in the directory instead of the pack name. This can be helpful while developing a pack. nomad pack run . [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#planning-with-plan) Planning with `plan` ----------------------------------------------------------------------------------------------------------------- If you want see details on how Nomad Pack will deploy a pack but are not ready to immediately deploy the pack, run the `plan` command. This invokes Nomad in a dry-run mode using the [Nomad Plan](https://developer.hashicorp.com/nomad/api-docs/jobs#create-job-plan) API endpoint. nomad-pack plan hello_world Similar to `run`, `plan` takes the `--name` flag to look for packs deployments with that name. Nomad Pack uses the pack name by default. nomad-pack plan hello_world --name hola-mundo The `plan` command also takes the `--var` and `-f` flags like the `run` command. nomad-pack plan hello_world -f ./my-variables.hcl --var greeting=hallo [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#registry-management) Registry management ----------------------------------------------------------------------------------------------------------------- The [Introduction to Nomad Pack guide](https://developer.hashicorp.com/nomad/tools/nomad-pack) explains the basics of adding to and listing packs in the registry. The following section provides additional details on registry management. ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#initialization-and-directory-structure) Initialization and directory structure The first time you run `list`, Nomad Pack will add a `nomad/packs` directory to your desktop user's cache directory—`$XDG_CACHE_DIR` on Linux, `~/Library/Caches` on macOS, `%AppData%` on Windows, etc. This folder stores information about cloned registries and their available packs. During initialization, Nomad Pack downloads a default registry of packs from the [Nomad Pack community registry](https://github.com/hashicorp/nomad-pack-community-registry) . The directory structure is as follows: parent-directories (see above) └── nomad └── packs ├── ├── ├── ├── ...files containing pack contents... Nomad Pack requires the contents of the pack cache directory to work properly, but you should not manually manage or change these files. Instead, you can use the `registry` commands. ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#adding-new-registries) Adding new registries The `registry` command includes several sub-commands for interacting with registries. Add custom registries with the `registry add` command. Any `git` based registry supported by [`go-getter`](https://github.com/hashicorp/go-getter) should work. For example, to add the entire [Nomad Pack Community Registry](https://github.com/hashicorp/nomad-pack-community-registry) , use the `registry add` command to download the registry. $ nomad-pack registry add community github.com/hashicorp/nomad-pack-community-registry ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#adding-an-individual-pack-from-a-registry) Adding an individual pack from a registry To add a single pack from the registry, use the `--target` flag. $ nomad-pack registry add community github.com/hashicorp/nomad-pack-community-registry --target=nginx ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#adding-a-registry-at-a-specific-commit) Adding a registry at a specific commit To download a single pack or an entire registry at a specific version/SHA, use the `--ref` flag. $ nomad-pack registry add community github.com/hashicorp/nomad-pack-community-registry --ref=v0.0.1 ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#removing-a-registry) Removing a registry To remove a registry or pack from your local cache, use the `registry delete` command. This command also supports the `--target` and `--ref` flags. $ nomad-pack registry delete community ### [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#updating-a-registry) Updating a registry To update a registry, use the `add` command and Nomad Pack will re-download the registry and replace the contents. $ nomad-pack registry add default github.com/hashicorp/nomad-pack-community-registry [](https://developer.hashicorp.com/nomad/tools/nomad-pack/advanced-usage#next-steps) Next steps ----------------------------------------------------------------------------------------------- In this guide you learned how to interact with Nomad Pack in an advanced way. You learned how to generate a variable file, render a pack, use additional `run` options, use the `plan` command, and interact with Nomad Pack registries. The official and community packs available to Nomad Pack are valuable because it allows you to quickly deploy apps using the best practices and leverage the knowledge of the Nomad community. To learn how to write your own Nomad Packs or convert your existing Nomad job specifications into reusable packs, continue on to the [Creating packs guide](https://developer.hashicorp.com/nomad/tools/nomad-pack/create-packs) . [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/tools/nomad-pack/advanced-usage.mdx) --- # nomad fmt command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad fmt command reference =========================== The `nomad fmt` command rewrites Nomad configuration and job specification files to canonical format. Use this command to improve readability and enforce consistency of style in Nomad files. [](https://developer.hashicorp.com/nomad/commands/fmt#usage) Usage ------------------------------------------------------------------ nomad fmt [options] paths ... If a path is a directory, the command recursively formats all files with `.nomad` and `.hcl` extensions in the directory. If you provide a single dash (-) as an argument, the `nomad fmt` command reads from standard input and writes the processed text to standard output. The `nomad fmt` command checks the syntax against [HCL2](https://github.com/hashicorp/hcl/blob/hcl2/README.md) , the second generation of Hashicorp Configuration Language. Running this command with deprecated HCL1 job specification results in errors. [](https://developer.hashicorp.com/nomad/commands/fmt#options) Options ---------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/fmt#) [`-check`](https://developer.hashicorp.com/nomad/commands/fmt#check) : Check if the files are valid HCL files. If not, exit status of the command will be 1 and the incorrect files will not be formatted. This flag overrides any `-write` flag value. * [](https://developer.hashicorp.com/nomad/commands/fmt#) [`-list`](https://developer.hashicorp.com/nomad/commands/fmt#list) : List the files which contain formatting inconsistencies. Defaults to `-list=true`. * [](https://developer.hashicorp.com/nomad/commands/fmt#) [`-recursive`](https://developer.hashicorp.com/nomad/commands/fmt#recursive) : Process files in subdirectories. By default, only the given (or current) directory is processed. * [](https://developer.hashicorp.com/nomad/commands/fmt#) [`-write`](https://developer.hashicorp.com/nomad/commands/fmt#write) : Overwrite the input files. Defaults to `-write=true`. [](https://developer.hashicorp.com/nomad/commands/fmt#examples) Examples ------------------------------------------------------------------------ $ cat agent.hcl server { enabled = true bootstrap_expect = 1 } client { enabled = true } $ nomad fmt agent.hcl $ cat agent.hcl server { enabled = true bootstrap_expect = 1 } client { enabled = true } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/fmt.mdx) --- # nomad agent command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad agent command reference ============================= The `nomad agent` command starts the Nomad agent, which handles client or server functionality, including exposing interfaces for client consumption and running jobs. The agent runs until it receives an interrupt signal. Refer to [Operating Nomad agents](https://developer.hashicorp.com/nomad/docs/deploy/nomad-agent) and [Nomad agent configuration](https://developer.hashicorp.com/nomad/docs/configuration) for more information on how to use this command and the options it has. [](https://developer.hashicorp.com/nomad/commands/agent#usage) Usage -------------------------------------------------------------------- If you are running Nomad on Linux, you need to run client agents as root, or with `sudo`, so that cpuset accounting and network namespaces work correctly. nomad agent [options] [](https://developer.hashicorp.com/nomad/commands/agent#options) Options ------------------------------------------------------------------------ We recommend configuring a Nomad agent with configuration files. Refer to [Nomad agent configuration](https://developer.hashicorp.com/nomad/docs/configuration) for details. You may, however, may pass the following configuration options as CLI arguments: * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-alloc-dir=`](https://developer.hashicorp.com/nomad/commands/agent#alloc-dir) : Equivalent to the Client [alloc\_dir](https://developer.hashicorp.com/nomad/docs/configuration/client#alloc_dir) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-acl-enabled`](https://developer.hashicorp.com/nomad/commands/agent#acl-enabled) : Equivalent to the ACL [enabled](https://developer.hashicorp.com/nomad/docs/configuration/acl#enabled) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-acl-replication-token`](https://developer.hashicorp.com/nomad/commands/agent#acl-replication-token) : Equivalent to the ACL [replication\_token](https://developer.hashicorp.com/nomad/docs/configuration/acl#replication_token) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-bind=
`](https://developer.hashicorp.com/nomad/commands/agent#bind) : Equivalent to the [bind\_addr](https://developer.hashicorp.com/nomad/docs/configuration#bind_addr) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-bootstrap-expect=`](https://developer.hashicorp.com/nomad/commands/agent#bootstrap-expect) : Equivalent to the [bootstrap\_expect](https://developer.hashicorp.com/nomad/docs/configuration/server#bootstrap_expect) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-client`](https://developer.hashicorp.com/nomad/commands/agent#client) : Enable client mode on the local agent. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-client-intro-token`](https://developer.hashicorp.com/nomad/commands/agent#client-intro-token) : The JWT token used to authenticate with servers during the client's initial registration. You may also set the token via the `NOMAD_CLIENT_INTRO_TOKEN` environment variable, which overrides this flag. If neither is set, the agent looks for an `intro_token.jwt` file within the [`client_state_dir`](https://developer.hashicorp.com/nomad/docs/configuration/client#state_dir) . It is not possible to set the intro token via a configuration file. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-config=`](https://developer.hashicorp.com/nomad/commands/agent#config) : Specifies the path to a configuration file or a directory of configuration files to load. Can be specified multiple times. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-address=`](https://developer.hashicorp.com/nomad/commands/agent#consul-address) : Equivalent to the [address](https://developer.hashicorp.com/nomad/docs/configuration/consul#address) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-auth=`](https://developer.hashicorp.com/nomad/commands/agent#consul-auth) : Equivalent to the [auth](https://developer.hashicorp.com/nomad/docs/configuration/consul#auth) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-auto-advertise`](https://developer.hashicorp.com/nomad/commands/agent#consul-auto-advertise) : Equivalent to the [auto\_advertise](https://developer.hashicorp.com/nomad/docs/configuration/consul#auto_advertise) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-ca-file=`](https://developer.hashicorp.com/nomad/commands/agent#consul-ca-file) : Equivalent to the [ca\_file](https://developer.hashicorp.com/nomad/docs/configuration/consul#ca_file) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-cert-file=`](https://developer.hashicorp.com/nomad/commands/agent#consul-cert-file) : Equivalent to the [cert\_file](https://developer.hashicorp.com/nomad/docs/configuration/consul#cert_file) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-checks-use-advertise`](https://developer.hashicorp.com/nomad/commands/agent#consul-checks-use-advertise) : Equivalent to the [checks\_use\_advertise](https://developer.hashicorp.com/nomad/docs/configuration/consul#checks_use_advertise) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-client-auto-join`](https://developer.hashicorp.com/nomad/commands/agent#consul-client-auto-join) : Equivalent to the [client\_auto\_join](https://developer.hashicorp.com/nomad/docs/configuration/consul#client_auto_join) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-client-service-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-client-service-name) : Equivalent to the [client\_service\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#client_service_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-client-http-check-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-client-http-check-name) : Equivalent to the [client\_http\_check\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#client_http_check_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-client-failures-before-critical=`](https://developer.hashicorp.com/nomad/commands/agent#consul-client-failures-before-critical) : Equivalent to the [client\_failures\_before\_critical](https://developer.hashicorp.com/nomad/docs/configuration/consul#client_failures_before_critical) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-client-failures-before-warning=`](https://developer.hashicorp.com/nomad/commands/agent#consul-client-failures-before-warning) : Equivalent to the [client\_failures\_before\_warning](https://developer.hashicorp.com/nomad/docs/configuration/consul#client_failures_before_warning) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-key-file=`](https://developer.hashicorp.com/nomad/commands/agent#consul-key-file) : Equivalent to the [key\_file](https://developer.hashicorp.com/nomad/docs/configuration/consul#key_file) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-service-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-service-name) : Equivalent to the [server\_service\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_service_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-http-check-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-http-check-name) : Equivalent to the [server\_http\_check\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_http_check_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-serf-check-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-serf-check-name) : Equivalent to the [server\_serf\_check\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_serf_check_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-rpc-check-name=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-rpc-check-name) : Equivalent to the [server\_rpc\_check\_name](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_rpc_check_name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-auto-join`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-auto-join) : Equivalent to the [server\_auto\_join](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_auto_join) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-failures-before-critical=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-failures-before-critical) : Equivalent to the [server\_failures\_before\_critical](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_failures_before_critical) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-server-failures-before-warning=`](https://developer.hashicorp.com/nomad/commands/agent#consul-server-failures-before-warning) : Equivalent to the [server\_failures\_before\_warning](https://developer.hashicorp.com/nomad/docs/configuration/consul#server_failures_before_warning) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-ssl`](https://developer.hashicorp.com/nomad/commands/agent#consul-ssl) : Equivalent to the [ssl](https://developer.hashicorp.com/nomad/docs/configuration/consul#ssl) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-token=`](https://developer.hashicorp.com/nomad/commands/agent#consul-token) : Equivalent to the [token](https://developer.hashicorp.com/nomad/docs/configuration/consul#token) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-consul-verify-ssl`](https://developer.hashicorp.com/nomad/commands/agent#consul-verify-ssl) : Equivalent to the [verify\_ssl](https://developer.hashicorp.com/nomad/docs/configuration/consul#verify_ssl) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-data-dir=`](https://developer.hashicorp.com/nomad/commands/agent#data-dir) : Equivalent to the [data\_dir](https://developer.hashicorp.com/nomad/docs/configuration#data_dir) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-dc=`](https://developer.hashicorp.com/nomad/commands/agent#dc) : Equivalent to the [datacenter](https://developer.hashicorp.com/nomad/docs/configuration#datacenter) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-dev`](https://developer.hashicorp.com/nomad/commands/agent#dev) : Start the agent in development mode. This enables a pre-configured dual-role agent (client + server) which is useful for developing or testing Nomad. No other configuration is required to start the agent in this mode, but you may pass an optional comma-separated list of mode configurations: * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-dev-connect`](https://developer.hashicorp.com/nomad/commands/agent#dev-connect) : Start the agent in development mode, but bind to a public network interface rather than localhost for using Consul service mesh. It may be used with `-dev-consul` to configure default workload identities for Consul. This mode is supported only on Linux as root. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-dev-consul`](https://developer.hashicorp.com/nomad/commands/agent#dev-consul) : Starts the agent in development mode with a default Consul configuration for Nomad workload identity. It may be used with `-dev-connect` to configure the agent for Consul Service Mesh. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-dev-vault`](https://developer.hashicorp.com/nomad/commands/agent#dev-vault) : Starts the agent in development mode with a default Vault configuration for Nomad workload identity. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-encrypt`](https://developer.hashicorp.com/nomad/commands/agent#encrypt) : Set the Serf encryption key. See the [Encryption Overview](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption) for more details. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-eventlog`](https://developer.hashicorp.com/nomad/commands/agent#eventlog) : Equivalent to the [eventlog.enabled](https://developer.hashicorp.com/nomad/docs/configuration#eventlog_enabled) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-eventlog-level`](https://developer.hashicorp.com/nomad/commands/agent#eventlog-level) : Equivalent to the [eventlog.level](https://developer.hashicorp.com/nomad/docs/configuration#eventlog_level) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-join=
`](https://developer.hashicorp.com/nomad/commands/agent#join) : Address of another agent to join upon starting up. This can be specified multiple times to specify multiple agents to join. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-log-level=`](https://developer.hashicorp.com/nomad/commands/agent#log-level) : Equivalent to the [log\_level](https://developer.hashicorp.com/nomad/docs/configuration#log_level) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-log-include-location`](https://developer.hashicorp.com/nomad/commands/agent#log-include-location) : Equivalent to the [log\_include\_location](https://developer.hashicorp.com/nomad/docs/configuration#log_include_location) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-log-json`](https://developer.hashicorp.com/nomad/commands/agent#log-json) : Equivalent to the [log\_json](https://developer.hashicorp.com/nomad/docs/configuration#log_json) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-meta=`](https://developer.hashicorp.com/nomad/commands/agent#meta) : Equivalent to the Client [meta](https://developer.hashicorp.com/nomad/docs/configuration/client#meta) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-network-interface=`](https://developer.hashicorp.com/nomad/commands/agent#network-interface) : Equivalent to the Client [network\_interface](https://developer.hashicorp.com/nomad/docs/configuration/client#network_interface) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-node=`](https://developer.hashicorp.com/nomad/commands/agent#node) : Equivalent to the [name](https://developer.hashicorp.com/nomad/docs/configuration#name) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-node-class=`](https://developer.hashicorp.com/nomad/commands/agent#node-class) : Equivalent to the Client [node\_class](https://developer.hashicorp.com/nomad/docs/configuration/client#node_class) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-node-pool=`](https://developer.hashicorp.com/nomad/commands/agent#node-pool) : Equivalent to the Client [node\_pool](https://developer.hashicorp.com/nomad/docs/configuration/client#node_pool) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-plugin-dir=`](https://developer.hashicorp.com/nomad/commands/agent#plugin-dir) : Equivalent to the [plugin\_dir](https://developer.hashicorp.com/nomad/docs/configuration#plugin_dir) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-region=`](https://developer.hashicorp.com/nomad/commands/agent#region) : Equivalent to the [region](https://developer.hashicorp.com/nomad/docs/configuration#region) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-rejoin`](https://developer.hashicorp.com/nomad/commands/agent#rejoin) : Equivalent to the [rejoin\_after\_leave](https://developer.hashicorp.com/nomad/docs/configuration/server#rejoin_after_leave) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-retry-interval`](https://developer.hashicorp.com/nomad/commands/agent#retry-interval) : Equivalent to the [retry\_interval](https://developer.hashicorp.com/nomad/docs/configuration/server#retry_interval) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-retry-join`](https://developer.hashicorp.com/nomad/commands/agent#retry-join) : Similar to `-join` but allows retrying a join if the first attempt fails. $ nomad agent -retry-join "127.0.0.1:4648" `retry-join` can be defined as a command line flag only for servers. Clients can configure `retry-join` only in configuration files. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-retry-max`](https://developer.hashicorp.com/nomad/commands/agent#retry-max) : Similar to the [retry\_max](https://developer.hashicorp.com/nomad/docs/configuration/server#retry_max) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-server`](https://developer.hashicorp.com/nomad/commands/agent#server) : Enable server mode on the local agent. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-servers=`](https://developer.hashicorp.com/nomad/commands/agent#servers) : Equivalent to the Client [servers](https://developer.hashicorp.com/nomad/docs/configuration/client#servers) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-state-dir=`](https://developer.hashicorp.com/nomad/commands/agent#state-dir) : Equivalent to the Client [state\_dir](https://developer.hashicorp.com/nomad/docs/configuration/client#state_dir) config option. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-enabled`](https://developer.hashicorp.com/nomad/commands/agent#vault-enabled) : Whether to enable or disabled Vault integration. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-address=`](https://developer.hashicorp.com/nomad/commands/agent#vault-address) : The address to communicate with Vault. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-token=`](https://developer.hashicorp.com/nomad/commands/agent#vault-token) : The Vault token used to derive tokens. Only needs to be set on Servers. Overrides the Vault token read from the VAULT\_TOKEN environment variable. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-create-from-role=`](https://developer.hashicorp.com/nomad/commands/agent#vault-create-from-role) : The role name to create tokens for tasks from. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-ca-file=`](https://developer.hashicorp.com/nomad/commands/agent#vault-ca-file) : Path to a PEM-encoded CA cert file used to verify the Vault server SSL certificate. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`-vault-ca-path=`](https://developer.hashicorp.com/nomad/commands/agent#vault-ca-path) : Path to a directory of PEM-encoded CA cert files used to verify the Vault server SSL certificate.Whether to enable or disabled Vault integration. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`vault-cert-file=`](https://developer.hashicorp.com/nomad/commands/agent#vault-cert-file) : The path to the certificate for Vault communication. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`vault-key-file=`](https://developer.hashicorp.com/nomad/commands/agent#vault-key-file) : The path to the private key for Vault communication. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`vault-namespace=`](https://developer.hashicorp.com/nomad/commands/agent#vault-namespace) : The Vault namespace used for the integration. Required for servers and clients. Overrides the Vault namespace read from the VAULT\_NAMESPACE environment variable. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`vault-tls-skip-verify`](https://developer.hashicorp.com/nomad/commands/agent#vault-tls-skip-verify) : A boolean that determines whether to skip SSL certificate verification. * [](https://developer.hashicorp.com/nomad/commands/agent#) [`vault-tls-server-name=`](https://developer.hashicorp.com/nomad/commands/agent#vault-tls-server-name) : Used to set the SNI host when connecting to Vault over TLS. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/agent.mdx) --- # Template Nomad jobspecs with Levant | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) [Nomad Pack](https://developer.hashicorp.com/nomad/tools/nomad-pack) is a new package manager and templating tool that can be used instead of Levant. Nomad Pack is currently in Tech Preview and may change during development. With Levant you can create and submit Nomad job specifications from template job specifications. These template job specifications reduce the overall amount of boilerplate that you have to manage for job files that contain a significant amount of repeated code. In this tutorial, you iteratively modify a template for rendering with Levant with the goal of deploying a Zookeeper node while reducing the overall job specification size and increasing modularity of job elements. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#prerequisites) Prerequisites ------------------------------------------------------------------------------------------------------------------ You need: * A Nomad cluster with: * Consul integrated * Docker installed and available as a task driver * a [Nomad host volume](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume-stanza) named `zk1` configured on a Nomad client agent to persist the Zookeeper data You should be: * Familiar with Go's text/template syntax. You can learn more about it in the [Learn Go Template Syntax](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax) tutorial. * Comfortable in your shell of choice, specifically adding executables to the path, editing text files, and managing directories. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#download-levant) Download Levant ---------------------------------------------------------------------------------------------------------------------- Install Levant using the instructions found in the [README](https://github.com/hashicorp/levant/blob/master/README.md) of its GitHub repository. Use one of the methods that provides you with a binary, rather than the Docker image. Verify that you have installed it to your executable path by running `levant version`. $ levant version Levant v0.3.0-dev (d7d77077+CHANGES) [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#build-the-starting-job-file) Build the starting job file ---------------------------------------------------------------------------------------------------------------------------------------------- Create a text file called **zookeeper.nomad** with the following content. job "zookeeper" { datacenters = ["dc1"] type = "service" update { max_parallel = 1 } group "zk1" { volume "zk" { type = "host" read_only = false source = "zk1" } count = 1 restart { attempts = 10 interval = "5m" delay = "25s" mode = "delay" } network { port "client" { to = -1 } port "peer" { to = -1 } port "election" { to = -1 } port "admin" { to = 8080 } } service { tags = ["client","zk1"] name = "zookeeper" port = "client" meta { ZK_ID = "1" } address_mode = "host" } service { tags = ["peer"] name = "zookeeper" port = "peer" meta { ZK_ID = "1" } address_mode = "host" } service { tags = ["election"] name = "zookeeper" port = "election" meta { ZK_ID = "1" } address_mode = "host" } service { tags = ["zk1-admin"] name = "zookeeper" port = "admin" meta { ZK_ID = "1" } address_mode = "host" } task "zookeeper" { driver = "docker" template { data=< Monitoring evaluation "498d1a25" Evaluation triggered by job "zookeeper" Allocation "b1111699" created: node "023f7896", group "zk1" Evaluation within deployment: "f8dcf6f3" Evaluation status changed: "pending" -> "complete" ==> Evaluation "498d1a25" finished with status "complete" ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#verify-zookeeper) Verify Zookeeper To verify your Zookeeper is up and responsive, connect to its admin port and run the `srvr` four letter command. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#get-zookeeper-s-admin-port) Get Zookeeper's admin port You can use one of several ways to get the Zookeeper admin port. Click on one of these techniques for more details. Using Consul DNS queries $ dig @10.0.2.21 -p8600 SRV zk1-admin.zookeeper.service.consul ; <<>> DiG 9.10.6 <<>> @10.0.2.21 -p8600 SRV zk1-admin.zookeeper.service.consul ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22461 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;zk1-admin.zookeeper.service.consul. IN SRV ;; ANSWER SECTION: zk1-admin.zookeeper.service.consul. 0 IN SRV 1 1 26284 0a000233.addr.dc1.consul. ;; ADDITIONAL SECTION: 0a000233.addr.dc1.consul. 0 IN A 10.0.2.51 ;; Query time: 54 msec ;; SERVER: 10.0.2.21#8600(10.0.2.21) ;; WHEN: Mon Nov 09 18:11:43 EST 2020 ;; MSG SIZE rcvd: 123 This output indicates that the service is accessible at 10.0.2.51:26284 Using the Nomad CLI Run `nomad job status zookeeper` and make a note of the running allocation's ID. $ nomad job status zookeeper ID = zookeeper Name = zookeeper Submit Date = 2020-11-09T17:40:17-05:00 Type = service Priority = 50 Datacenters = dc1 Namespace = default Status = running Periodic = false Parameterized = false Summary Task Group Queued Starting Running Failed Complete Lost zk1 0 0 1 0 0 0 Latest Deployment ID = f8dcf6f3 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline zk1 1 1 1 0 2020-11-09T17:53:30-05:00 Allocations ID Node ID Task Group Version Desired Status Created Modified b1111699 023f7896 zk1 0 run running 26m40s ago 26m29s ago In this case the `ID` is `b1111699`. Run `nomad alloc status` and supply the allocation ID. $ nomad alloc status b1111699 ID = b1111699-fb35-cd46-3dda-dae3233e2373 Eval ID = 498d1a25 Name = zookeeper.zk1[0] Node ID = 023f7896 Node Name = nomad-client-1.node.consul Job ID = zookeeper Job Version = 0 Client Status = running Client Description = Tasks are running Desired Status = run Desired Description = Created = 28m47s ago Modified = 28m36s ago Deployment ID = f8dcf6f3 Deployment Health = healthy Allocation Addresses Label Dynamic Address *client yes 10.0.2.51:23977 -> 2181 *peer yes 10.0.2.51:26262 -> 26262 *election yes 10.0.2.51:22589 -> 22589 *admin yes 10.0.2.51:26284 -> 8080 Task "zookeeper" is "running" ... Consult the **Allocation Addresses** section of the output. In this example, the admin address is at 10.0.2.51:26284. Using the Nomad UI Open the Nomad UI in your browser. Select the running **zookeeper** job. From the job details screen, select the **zk1** task group, select the running allocation's ID. Once you are on the allocation detail page, you can get the port mappings by scrolling down to the **Ports** table. ![Image of the Nomad UI scrolled down to the ports table, showing the ports for the zookeeper allocation](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Flevant%2Fnomad-ui-alloc-ports.png) #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#run-srvr-command-via-zookeeper-admin) Run `srvr` command via Zookeeper admin Now that you have located the port that the admin server is running on, make a request to run the `srvr` command. The following is an example using curl. $ curl zk1-admin.zookeeper.service.consul:26284/commands/srvr { "version" : "3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on 04/21/2020 15:01 GMT", "read_only" : false, "server_stats" : { // ... more json ... // } } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#stop-and-purge-the-job) Stop and purge the job Run `nomad stop -purge zookeeper` to stop the Zookeeper instance. Using the `-purge` flag while stopping reduces the amount of data you have to consult if you have to debug a non-starting instance. $ nomad stop -purge zookeeper ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#make-a-template) Make a template Back up the original template by creating a copy named **zookeeper.nomad.orig**. $ cp zookeeper.nomad zookeeper.nomad.orig [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#optimize-the-template) Optimize the template ---------------------------------------------------------------------------------------------------------------------------------- Open **zookeeper.nomad** in a text editor. There is some file-like content that you might want to extract from the template for ease of maintenance, like the template used to generate the Zookeeper configuration in the zoo.cfg file. Also, notice that there are many repeated elements in the services and network ports. The following steps cover techniques for removing repeated or reusable material from your job specification. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant#include-zoo-cfg-rather-than-inline-it) Include zoo.cfg rather than inline it One improvement involves making zoo.cfg external to the job template. For a lengthy configuration file, it's better to keep the template content separate from the rest of the configuration when possible. Navigate to the template stanza that creates the zoo.cfg file in the job. template { destination = "config/zoo.cfg" data = < 2023-03-10T12:24:45-05:00: Monitoring evaluation "d4079a21" 2023-03-10T12:24:45-05:00: Evaluation triggered by job "pytechco-web" 2023-03-10T12:24:45-05:00: Evaluation within deployment: "cdb7d282" 2023-03-10T12:24:45-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:24:45-05:00: Evaluation "d4079a21" finished with status "complete" ==> 2023-03-10T12:24:45-05:00: Monitoring deployment "cdb7d282" ✓ Deployment "cdb7d282" successful 2023-03-10T12:24:45-05:00 ID = cdb7d282 Job ID = pytechco-web Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline ptc-web 1 1 1 0 2023-03-10T17:26:57Z Then, the database job. $ nomad job stop -purge pytechco-redis ==> 2023-03-10T12:25:00-05:00: Monitoring evaluation "25608f76" 2023-03-10T12:25:00-05:00: Evaluation triggered by job "pytechco-redis" 2023-03-10T12:25:00-05:00: Evaluation within deployment: "0ea05651" 2023-03-10T12:25:00-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:25:00-05:00: Evaluation "25608f76" finished with status "complete" ==> 2023-03-10T12:25:00-05:00: Monitoring deployment "0ea05651" ✓ Deployment "0ea05651" successful 2023-03-10T12:25:00-05:00 ID = 0ea05651 Job ID = pytechco-redis Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline ptc-redis 1 1 1 0 2023-03-10T17:26:23Z Finally, stop and clean up the setup job. $ nomad job stop -purge pytechco-setup [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-stop-nomad#clean-up-the-cluster) Clean up the cluster ----------------------------------------------------------------------------------------------------------------------- Clean up the resources in use by stopping Nomad if you are running a local cluster or by destroying the cloud infrastructure with Terraform. LocalCloud Open the terminal tab running the Nomad dev agent and press `CTRL+C` to stop it. Navigate back to the directory from where you ran the Terraform `apply` command. AWSGCPAzure $ cd ../aws $ cd ../gcp $ cd ../azure Destroy the resources. Respond `yes` to the prompt to confirm the operation. $ terraform destroy [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-stop-nomad#next-steps) Next steps --------------------------------------------------------------------------------------------------- In this collection, you learned about Nomad, installed the Nomad CLI, created a cluster locally and in the cloud, and deployed and updated Nomad jobs. There are many directions you can go next to continue your learning so check out some of the tutorials and documentation below. * [Learn about Nomad jobs](https://developer.hashicorp.com/nomad/docs/job-declare) * [Learn more about the jobspec](https://developer.hashicorp.com/nomad/tutorials/job-specifications) * [Learn about deploying applications with Nomad Pack](https://developer.hashicorp.com/nomad/tools/nomad-pack) * [Learn about load balancing in Nomad](https://developer.hashicorp.com/nomad/tutorials/load-balancing) * [Learn how to integrate Consul for service discovery and more](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) * [Review the Nomad CLI commands](https://developer.hashicorp.com/nomad/docs/commands) **Was this tutorial helpful?** YesNo [Previous\ \ Deploy and update a job](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job) [Next Collection\ \ Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) --- # Agent - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Agent HTTP API ============== The `/agent` endpoints are used to interact with the local Nomad agent. [](https://developer.hashicorp.com/nomad/api-docs/agent#list-members) List Members ---------------------------------------------------------------------------------- This endpoint queries the agent for the known peers in the gossip pool. This endpoint is only applicable to servers. Due to the nature of gossip, this is eventually consistent. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/members` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request) Sample Request $ curl \ https://localhost:4646/v1/agent/members ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response) Sample Response { "ServerName": "bacon-mac", "ServerRegion": "global", "ServerDC": "dc1", "Members": [\ {\ "Name": "bacon-mac.global",\ "Addr": "127.0.0.1",\ "Port": 4648,\ "Tags": {\ "mvn": "1",\ "build": "0.5.5dev",\ "port": "4647",\ "bootstrap": "1",\ "role": "nomad",\ "region": "global",\ "dc": "dc1",\ "vsn": "1"\ },\ "Status": "alive",\ "ProtocolMin": 1,\ "ProtocolMax": 5,\ "ProtocolCur": 2,\ "DelegateMin": 2,\ "DelegateMax": 4,\ "DelegateCur": 4\ }\ ] } [](https://developer.hashicorp.com/nomad/api-docs/agent#list-servers) List Servers ---------------------------------------------------------------------------------- This endpoint lists the known server nodes. The `servers` endpoint is used to query an agent in client mode for its list of known servers. Client nodes register themselves with these server addresses so that they may dequeue work. The servers endpoint can be used to keep this configuration up to date if there are changes in the cluster. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/servers` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/agent/servers ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-1) Sample Response ["127.0.0.1:4647"] [](https://developer.hashicorp.com/nomad/api-docs/agent#update-servers) Update Servers -------------------------------------------------------------------------------------- This endpoint updates the list of known servers to the provided list. This **replaces** all previous server addresses with the new list. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/agent/servers` | `(empty body)` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`address`](https://developer.hashicorp.com/nomad/api-docs/agent#address) `(string: )` - Specifies the list of addresses in the format `ip:port`. This is specified as a query string. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-2) Sample Request $ curl \ --request POST \ "https://localhost:4646/v1/agent/servers?address=1.2.3.4:4647&address=5.6.7.8:4647" [](https://developer.hashicorp.com/nomad/api-docs/agent#query-self) Query Self ------------------------------------------------------------------------------ This endpoint queries the state of the target agent (self). | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/self` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-3) Sample Request $ curl \ https://localhost:4646/v1/agent/self ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-2) Sample Response { "config": { "Addresses": { "HTTP": "127.0.0.1", "RPC": "127.0.0.1", "Serf": "127.0.0.1" }, "AdvertiseAddrs": { "HTTP": "127.0.0.1:4646", "RPC": "127.0.0.1:4647", "Serf": "127.0.0.1:4648" }, "BindAddr": "127.0.0.1", "Client": { "AllocDir": "", "ChrootEnv": {}, "ClientMaxPort": 14512, "ClientMinPort": 14000, "DisableRemoteExec": false, "Enabled": true, "GCDiskUsageThreshold": 99, "GCInodeUsageThreshold": 99, "GCInterval": 600000000000, "MaxKillTimeout": "30s", "Meta": {}, "NetworkInterface": "lo0", "NetworkSpeed": 0, "NodeClass": "", "Options": { "driver.docker.volumes": "true" }, "Reserved": { "CPU": 0, "DiskMB": 0, "MemoryMB": 0, "ParsedReservedPorts": null, "ReservedPorts": "" }, "Servers": null, "StateDir": "" }, "Consul": { "Addr": "", "Auth": "", "AutoAdvertise": true, "CAFile": "", "CertFile": "", "ChecksUseAdvertise": false, "ClientAutoJoin": true, "ClientServiceName": "nomad-client", "EnableSSL": false, "KeyFile": "", "ServerAutoJoin": true, "ServerServiceName": "nomad", "Timeout": 5000000000, "Token": "", "VerifySSL": false }, "DataDir": "", "Datacenter": "dc1", "DevMode": true, "DisableAnonymousSignature": true, "DisableUpdateCheck": false, "EnableDebug": true, "EnableSyslog": false, "Files": null, "HTTPAPIResponseHeaders": {}, "LeaveOnInt": false, "LeaveOnTerm": false, "LogLevel": "DEBUG", "NodeName": "", "Ports": { "HTTP": 4646, "RPC": 4647, "Serf": 4648 }, "Region": "global", "Revision": "f551dcb83e3ac144c9dbb90583b6e82d234662e9", "Server": { "BootstrapExpect": 0, "DataDir": "", "Enabled": true, "EnabledSchedulers": null, "HeartbeatGrace": "", "NodeGCThreshold": "", "NumSchedulers": 0, "ProtocolVersion": 0, "RejoinAfterLeave": false, "RetryInterval": "30s", "RetryJoin": [], "RetryMaxAttempts": 0, "StartJoin": [] }, "SyslogFacility": "LOCAL0", "TLSConfig": { "CAFile": "", "CertFile": "", "EnableHTTP": false, "EnableRPC": false, "KeyFile": "", "VerifyServerHostname": false }, "Telemetry": { "CirconusAPIApp": "", "CirconusAPIToken": "", "CirconusAPIURL": "", "CirconusBrokerID": "", "CirconusBrokerSelectTag": "", "CirconusCheckDisplayName": "", "CirconusCheckForceMetricActivation": "", "CirconusCheckID": "", "CirconusCheckInstanceID": "", "CirconusCheckSearchTag": "", "CirconusCheckSubmissionURL": "", "CirconusCheckTags": "", "CirconusSubmissionInterval": "", "CollectionInterval": "1s", "DataDogAddr": "", "DataDogTags": [], "DisableHostname": false, "PublishAllocationMetrics": false, "PublishNodeMetrics": false, "StatsdAddr": "", "StatsiteAddr": "", "UseNodeName": false }, "Vault": { "Addr": "https://vault.service.consul:8200", "AllowUnauthenticated": true, "ConnectionRetryIntv": 30000000000, "Enabled": null, "Role": "", "TLSCaFile": "", "TLSCaPath": "", "TLSCertFile": "", "TLSKeyFile": "", "TLSServerName": "", "TLSSkipVerify": null, "TaskTokenTTL": "", "Token": "root" }, "Version": "0.5.5", "VersionPrerelease": "dev" }, "member": { "Addr": "127.0.0.1", "DelegateCur": 4, "DelegateMax": 4, "DelegateMin": 2, "Name": "bacon-mac.global", "Port": 4648, "ProtocolCur": 2, "ProtocolMax": 5, "ProtocolMin": 1, "Status": "alive", "Tags": { "role": "nomad", "region": "global", "dc": "dc1", "vsn": "1", "mvn": "1", "build": "0.5.5dev", "port": "4647", "bootstrap": "1" } }, "stats": { "runtime": { "cpu_count": "8", "kernel.name": "darwin", "arch": "amd64", "version": "go1.8", "max_procs": "7", "goroutines": "79" }, "nomad": { "server": "true", "leader": "true", "leader_addr": "127.0.0.1:4647", "bootstrap": "false", "known_regions": "1" }, "raft": { "num_peers": "0", "fsm_pending": "0", "last_snapshot_index": "0", "last_log_term": "2", "commit_index": "144", "term": "2", "last_log_index": "144", "snapshot_version_max": "1", "latest_configuration_index": "1", "latest_configuration": "[{Suffrage:Voter ID:127.0.0.1:4647 Address:127.0.0.1:4647}]", "last_contact": "never", "applied_index": "144", "state": "Leader", "last_snapshot_term": "0" }, "client": { "heartbeat_ttl": "17.79568937s", "node_id": "fb2170a8-257d-3c64-b14d-bc06cc94e34c", "known_servers": "127.0.0.1:4647", "num_allocations": "0", "last_heartbeat": "10.107423052s" }, "serf": { "event_time": "1", "event_queue": "0", "encrypted": "false", "member_time": "1", "query_time": "1", "intent_queue": "0", "query_queue": "0", "members": "1", "failed": "0", "left": "0", "health_score": "0" } } } [](https://developer.hashicorp.com/nomad/api-docs/agent#join-agent) Join Agent ------------------------------------------------------------------------------ This endpoint introduces a new member to the gossip pool. This endpoint is only eligible for servers. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/agent/join` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`address`](https://developer.hashicorp.com/nomad/api-docs/agent#address-1) `(string: )` - Specifies the address to join in the `ip:port` format. This is provided as a query parameter and may be specified multiple times to join multiple servers. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-4) Sample Request $ curl \ --request POST \ "https://localhost:4646/v1/agent/join?address=1.2.3.4&address=5.6.7.8" ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-3) Sample Response { "error": "", "num_joined": 2 } [](https://developer.hashicorp.com/nomad/api-docs/agent#force-leave-agent) Force Leave Agent -------------------------------------------------------------------------------------------- This endpoint forces a member of the gossip pool from the `"failed"` state to the `"left"` state. This allows the consensus protocol to remove the peer and stop attempting replication. This is only applicable for servers. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/agent/force-leave` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`node`](https://developer.hashicorp.com/nomad/api-docs/agent#node) `(string: )` - Specifies the name of the node to force leave. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`prune`](https://developer.hashicorp.com/nomad/api-docs/agent#prune) `(boolean: )` - Removes failed or left server from the Serf member list immediately. If member is actually still alive, it will eventually rejoin the cluster again. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-5) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/agent/force-leave?node=client-ab2e23dc&prune=true [](https://developer.hashicorp.com/nomad/api-docs/agent#health) Health ---------------------------------------------------------------------- This endpoint returns whether or not the agent is healthy. When using Consul it is the endpoint Nomad will register for its own health checks. When the agent is unhealthy 500 will be returned along with JSON response containing an error message. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/health` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-6) Sample Request $ curl \ https://localhost:4646/v1/agent/health ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-4) Sample Response { "client": { "message": "ok", "ok": true }, "server": { "message": "ok", "ok": true } } [](https://developer.hashicorp.com/nomad/api-docs/agent#host) Host ------------------------------------------------------------------ This endpoint returns data about the agent's host environment from the perspective of the agent. It is included in the archive produced by nomad operator debug. Known sensitive environment variables are shown as ``, but the response may still contain sensitive information. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/host` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`server_id`](https://developer.hashicorp.com/nomad/api-docs/agent#server_id) `(string: )` - Specify the server name for targeting. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`node_id`](https://developer.hashicorp.com/nomad/api-docs/agent#node_id) `(string: )` - Specify the client node id for targeting. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-7) Sample Request $ curl \ https://localhost:4646/v1/agent/host?node_id=4bb9aca7-d43b-43fc-d604-3a271ef0a6c0 ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-5) Sample Response { "AgentID": "4bb9aca7-d43b-43fc-d604-3a271ef0a6c0", "HostData": { "OS": "x86_64 ip-172-31-92-147 4.15.0-1007-aws Linux #7-Ubuntu SMP Tue Apr 24 10:56:17 UTC 2018", "Network": [\ {\ "DialPacket": "\"udp4\" \"\"",\ "DialStream": "\"tcp4\" \"\"",\ "ListenPacket": "\"udp4\" \"\"",\ "ListenStream": "\"tcp4\" \"\"",\ "address": "127.0.0.1",\ "binary": "01111111000000000000000000000001",\ "broadcast": "127.255.255.255",\ "first_usable": "127.0.0.1",\ "hex": "7f000001",\ "host": "127.0.0.1",\ "last_usable": "127.255.255.254",\ "mask_bits": "8",\ "netmask": "255.0.0.0",\ "network": "127.0.0.0",\ "octets": "127 0 0 1",\ "port": "0",\ "size": "16777216",\ "string": "127.0.0.1/8",\ "type": "IPv4",\ "uint32": "2130706433"\ }\ ], "ResolvConf": "nameserver 172.17.0.1\nnameserver 8.8.8.8\n", "Hosts": "127.0.0.1 localhost\n\n# The following lines are desirable for IPv6 capable hosts\n::1 ip6-localhost ip6-loopback\nfe00::0 ip6-localnet\nff00::0 ip6-mcastprefix\nff02::1 ip6-allnodes\nff02::2 ip6-allrouters\nff02::3 ip6-allhosts\n127.0.0.1 ip-172-31-71-163\n127.0.0.1 ip-172-31-92-147\n", "Environment": { "INVOCATION_ID": "b106b6ac67764c9b9f85c5cc5c3357e5", "JOURNAL_STREAM": "9:109527", "LANG": "C.UTF-8", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" }, "Disk": { "/": { "DiskMB": 7876, "UsedMB": 4287 } } } } [](https://developer.hashicorp.com/nomad/api-docs/agent#stream-logs) Stream Logs -------------------------------------------------------------------------------- This endpoint streams logs from the local agent until the connection is closed | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/monitor` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-4) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`log_level`](https://developer.hashicorp.com/nomad/api-docs/agent#log_level) `(string: "info")` - Specifies a text string containing a log level to filter on, such as `info`. Possible values include `trace`, `debug`, `info`, `warn`, `error` * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`log_json`](https://developer.hashicorp.com/nomad/api-docs/agent#log_json) `(bool: false)` - Specifies if the log format for streamed logs should be JSON. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`log_include_location`](https://developer.hashicorp.com/nomad/api-docs/agent#log_include_location) `(bool: false)` - Specifies if the logs streamed should include file and line information. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`node_id`](https://developer.hashicorp.com/nomad/api-docs/agent#node_id-1) `(string: "a57b2adb-1a30-2dda-8df0-25abb0881952")` - Specifies a text string containing a node-id to target for streaming. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`server_id`](https://developer.hashicorp.com/nomad/api-docs/agent#server_id-1) `(string: "server1.global")` - Specifies a text string containing a server name or "leader" to target a specific remote server or leader for streaming. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`plain`](https://developer.hashicorp.com/nomad/api-docs/agent#plain) `(bool: false)` - Specifies if the response should be JSON or plaintext ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-8) Sample Request $ curl \ "https://localhost:4646/v1/agent/monitor?log_level=debug&server_id=leader" $ curl \ "https://localhost:4646/v1/agent/monitor?log_level=debug&node_id=a57b2adb-1a30-2dda-8df0-25abb0881952" ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-6) Sample Response { "Offset": 0, "Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..." "FileEvent": "log" } #### [](https://developer.hashicorp.com/nomad/api-docs/agent#field-reference) Field Reference The return value is a stream of frames. These frames contain the following fields: * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`Data`](https://developer.hashicorp.com/nomad/api-docs/agent#data) - A base64 encoding of the bytes being streamed. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`FileEvent`](https://developer.hashicorp.com/nomad/api-docs/agent#fileevent) - An event that could cause a change in the streams position. The possible value for this endpoint is "log". * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`Offset`](https://developer.hashicorp.com/nomad/api-docs/agent#offset) - Offset is the offset into the stream. [](https://developer.hashicorp.com/nomad/api-docs/agent#agent-runtime-profiles) Agent Runtime Profiles ------------------------------------------------------------------------------------------------------ This endpoint is the equivalent of Go's /debug/pprof endpoint but is protected by ACLs and supports remote forwarding to a client node or server. See the [Golang documentation](https://golang.org/pkg/runtime/pprof/#Profile) for a list of available profiles. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/pprof/cmdline` | `text/plain` | | `GET` | `/v1/agent/pprof/profile` | `application/octet-stream` | | `GET` | `/v1/agent/pprof/trace` | `application/octet-stream` | | `GET` | `/v1/agent/pprof/` | `application/octet-stream` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#default-behavior) Default Behavior This endpoint is enabled whenever ACLs are enabled. Due to the potentially sensitive nature of data contained in profiles, as well as their significant performance impact, the agent/pprof endpoint is protected by a high level ACL: `agent:write`. For these reasons its recommended to leave [`enable_debug`](https://developer.hashicorp.com/nomad/docs/configuration#enable_debug) unset and only use the ACL-protected endpoints. The following table explains when each endpoint is available: | Endpoint | `enable_debug` | ACLs | **Available?** | | --- | --- | --- | --- | | `/v1/agent/pprof` | unset | n/a | no | | `/v1/agent/pprof` | `true` | n/a | yes | | `/v1/agent/pprof` | `false` | n/a | no | | `/v1/agent/pprof` | unset | off | no | | `/v1/agent/pprof` | unset | on | **yes** | | `/v1/agent/pprof` | `true` | off | yes | | `/v1/agent/pprof` | `false` | on | **yes** | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-5) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`node_id`](https://developer.hashicorp.com/nomad/api-docs/agent#node_id-2) `(string: "a57b2adb-1a30-2dda-8df0-25abb0881952")` - Specifies a text string containing a Node ID to target for profiling. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`server_id`](https://developer.hashicorp.com/nomad/api-docs/agent#server_id-2) `(string: "server1.global")` - Specifies a text string containing a Server ID, name, or `leader` to target a specific remote server or leader for profiling. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`seconds`](https://developer.hashicorp.com/nomad/api-docs/agent#seconds) `(int: 3)` - Specifies the amount of time to run a profile or trace request for. * [](https://developer.hashicorp.com/nomad/api-docs/agent#) [`debug`](https://developer.hashicorp.com/nomad/api-docs/agent#debug) `(int: 0)` - Specifies if a given pprof profile should be returned as human readable plain text instead of the pprof binary format. Defaults to 0, setting to 1 enables human readable plain text. For goroutine profiles, setting to 2 extends the plain text format with additional information helpful for debugging stalled goroutines. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-9) Sample Request $ curl -O -J \ --header "X-Nomad-Token: 8176afd3-772d-0b71-8f85-7fa5d903e9d4" \ "https://localhost:4646/v1/agent/pprof/goroutine?server_id=leader" $ go tool pprof goroutine $ curl -O -J \ --header "X-Nomad-Token: 8176afd3-772d-0b71-8f85-7fa5d903e9d4" \ "https://localhost:4646/v1/agent/pprof/profile?seconds=5&node_id=a57b2adb-1a30-2dda-8df0-25abb0881952" $ go tool pprof profile $ curl -O -J \ --header "X-Nomad-Token: 8176afd3-772d-0b71-8f85-7fa5d903e9d4" \ "https://localhost:4646/v1/agent/pprof/trace?&seconds=5&server_id=server1.global" go tool trace trace [](https://developer.hashicorp.com/nomad/api-docs/agent#fetch-all-scheduler-worker-s-status) Fetch all scheduler worker's status -------------------------------------------------------------------------------------------------------------------------------- The `/agent/schedulers` endpoint allow Nomad operators to inspect the state of a Nomad server agent's scheduler workers. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/schedulers` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-6) Parameters This endpoint accepts no additional parameters. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-10) Sample Request $ curl \ https://localhost:4646/v1/agent/schedulers ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-7) Sample Response { "schedulers": [\ {\ "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ],\ "id": "5669d6fa-0def-7369-6558-a47c35fdc675",\ "started": "2021-12-21T19:25:00.911883Z",\ "status": "Paused",\ "workload_status": "Paused"\ },\ {\ "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ],\ "id": "c919709d-6d14-66bf-b425-80b8167a267e",\ "started": "2021-12-21T19:25:00.91189Z",\ "status": "Paused",\ "workload_status": "Paused"\ },\ {\ "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ],\ "id": "f5edb69a-6122-be8f-b32a-23cd8511dba5",\ "started": "2021-12-21T19:25:00.911961Z",\ "status": "Paused",\ "workload_status": "Paused"\ },\ {\ "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ],\ "id": "458816ae-83cf-0710-d8d4-35d2ad2e42d7",\ "started": "2021-12-21T19:25:00.912119Z",\ "status": "Started",\ "workload_status": "WaitingToDequeue"\ }\ ], "server_id": "server1.global" } [](https://developer.hashicorp.com/nomad/api-docs/agent#read-scheduler-worker-configuration) Read scheduler worker configuration -------------------------------------------------------------------------------------------------------------------------------- This endpoint returns data about the agent's scheduler configuration from the perspective of the agent. This is only applicable for servers. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/agent/schedulers/config` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#parameters-7) Parameters This endpoint accepts no additional parameters. ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-11) Sample Request $ curl \ https://localhost:4646/v1/agent/schedulers/config ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-8) Sample Response { "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ], "num_schedulers": 8, "server_id": "server1.global" } [](https://developer.hashicorp.com/nomad/api-docs/agent#update-scheduler-worker-configuration) Update scheduler worker configuration ------------------------------------------------------------------------------------------------------------------------------------ This allows a Nomad operator to modify the server's running scheduler configuration, which will remain in effect until another update or until the node is restarted. For durable changes to this value, set the corresponding values—[`num_schedulers`](https://developer.hashicorp.com/nomad/docs/configuration/server#num_schedulers) and [`enabled_schedulers`](https://developer.hashicorp.com/nomad/docs/configuration/server#enabled_schedulers) —in the node's configuration file. The response contains the configuration after attempting to apply the provided values. This is only applicable for servers. | Method | Path | Produces | | --- | --- | --- | | `PUT` | `/v1/agent/schedulers/config` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `agent:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-payload) Sample Payload { "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ], "num_schedulers": 12 "server_id": "server1.global" } ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-request-12) Sample Request $ curl \ --request PUT \ --data @payload.json \ https://localhost:4646/v1/agent/schedulers/config ### [](https://developer.hashicorp.com/nomad/api-docs/agent#sample-response-9) Sample Response { "enabled_schedulers": [\ "service",\ "batch",\ "system",\ "sysbatch",\ "_core"\ ], "num_schedulers": 12, "server_id": "server1.global" } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/agent.mdx) --- # Scale your Nomad cluster with horizontal cluster autoscaling | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Autoscaling](https://developer.hashicorp.com/nomad/tutorials/autoscaler) As enterprises have accelerated their cloud adoption in the past 2-3 years, horizontal autoscaling has become a critical must-have capability for orchestrators. The dynamic nature of cloud environments allow for compute resources to be easily commissioned and be billed on a usage basis. Horizontal autoscaling is a feature that enables enterprises to: * Scale up infrastructure on an on-demand basis that is aligned with business SLAs. * Scale down infrastructure to minimize costs based on real demand. * Handle application load spikes or dips in real-time. * Handle cluster-wide excess capacity or shortages in real-time. * Reduce operator overhead and remove hard dependencies on manual intervention. This tutorial provides a basic demo for running full horizontal application and cluster autoscaling using the Nomad Autoscaler. During this tutorial you will: * Deploy sample infrastructure running a demonstration web application. * Review autoscaler policies to see their behaviors and thresholds. * While monitoring the included dashboard: 1. Generate traffic and observe the application scale up. 2. Generate additional traffic and observe the cluster scale out. 3. Finally, stop the traffic and observe the application scale down and the cluster scale in. Note The infrastructure built as part of the demo has billable costs and is not suitable for production use. Please consult the [reference architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul#high-availability) for production configuration. ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#requirements) Requirements In order to build and run the demo, you need the following applications with the listed version or greater locally. * HashiCorp Nomad [1.0.0+](https://releases.hashicorp.com/nomad/1.0.4/) * HashiCorp Packer [1.7.0+](https://releases.hashicorp.com/packer/1.7.0/) * HashiCorp Terraform [0.14.0+](https://releases.hashicorp.com/terraform/0.14.7/) * rakyll/hey [latest](https://github.com/rakyll/hey#installation) If you are running this demo in a Windows environment it is recommended to use the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/) . #### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#cloud-specific-dependencies) Cloud specific dependencies Amazon Web ServicesGoogle Cloud PlatformMicrosoft Azure There are not specific dependencies for Amazon Web Services. To deploy on Google Cloud Platform, you will also need to install the following. * [`gcloud` CLI](https://cloud.google.com/sdk/docs/install) * [`beta`](https://cloud.google.com/sdk/gcloud/reference/beta) commands enabled * GCP account with IAM roles `roles/resourcemanager.projectCreator` and `roles/billing.user` To deploy on Microsoft Azure, you will also need to install the following. * [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#fetch-the-nomad-autoscaler-demos) Fetch the Nomad Autoscaler demos ----------------------------------------------------------------------------------------------------------------------------------------------------------- Download the latest code for the Autoscaler demos from the [GitHub repository](https://github.com/hashicorp/nomad-autoscaler-demos) . You can use `git` to clone the repository or download the ZIP archive. Using GitDownload ZIP archive Clone the `hashicorp/nomad-autoscaler-demos` repository. $ git clone https://github.com/hashicorp/nomad-autoscaler-demos $ cd nomad-autoscaler-demos/cloud Check out the `learn` tag. Using this tag ensures that the instructions in this guide match your local copy of the code. $ git checkout learn $ wget https://github.com/hashicorp/nomad-autoscaler-demos/archive/learn.zip Unarchive the downloaded release. $ unzip learn.zip The unzipping process creates a directory named `nomad-autoscaler-demos-learn`. Change into it and into the `cloud` folder. $ cd nomad-autoscaler-demos-learn/cloud ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#change-into-the-cloud-specific-demonstration-directory) Change into the cloud specific demonstration directory Amazon Web ServicesGoogle Cloud PlatformMicrosoft Azure The AWS-specific demonstration code is located in the `aws` directory. Change there now. $ cd aws The GCP-specific demonstration code is located in the `gcp` directory. Change there now. $ cd gcp The Azure-specific demonstration code is located in the `azure` directory. Change there now. $ cd azure [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#create-the-demo-infrastructure) Create the demo infrastructure ------------------------------------------------------------------------------------------------------------------------------------------------------- There are specific steps to build the infrastructure depending on which provider you wish to use. Please navigate to the appropriate section below. Amazon Web ServicesGoogle Cloud PlatformMicrosoft Azure ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#configure-aws-credentials) Configure AWS credentials Configure AWS credentials for your environment so that Terraform can authenticate with AWS and create resources. To do this with IAM user authentication, set your AWS access key ID as an environment variable. $ export AWS_ACCESS_KEY_ID="" Now set your secret key. $ export AWS_SECRET_ACCESS_KEY="" Tip If you don't have access to IAM user credentials, use another authentication method described in the [AWS provider documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables) . ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#build-demo-environment-ami) Build demo environment AMI First, use Packer to build an AMI that is used for launching the Nomad server and client instances. Replace the placeholders where necessary. Packer will tag the AMI with the `created` values, so you can identify your new resources in the targeted environment. The region flag can be omitted if you are using the `us-east-1` region. $ cd packer $ packer build \ -var 'created_email=' \ -var 'created_name=' \ -var 'region=' \ aws-packer.pkr.hcl Now, navigate to the Terraform AWS environment you will be using to build the infrastructure components. $ cd ../terraform/control ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#build-terraform-variables-file) Build Terraform variables file In order for Terraform to run correctly you'll need to provide the appropriate variables within a file named `terraform.tfvars`. Create your own variables file by copying the provided `terraform.tfvars.sample` file. $ cp terraform.tfvars.sample terraform.tfvars #### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#update-the-variables-for-your-environment) Update the variables for your environment * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`region`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#region) - The region to deploy your infrastructure. This must match the region you deployed your AMI into. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`availability_zones`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#availability_zones) - A list of specific availability zones eligible to deploy your infrastructure into. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`ami`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#ami) - The AMI ID created by your Packer run. You will receive it in the output from `packer build` earlier. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`key_name`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#key_name) - The name of the AWS EC2 Key Pair that you want to associate to the instances. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`owner_name`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#owner_name) - Added to the created infrastructure as a tag. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`owner_email`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#owner_email) - Added to the created infrastructure as a tag. The most important are `ami`, `region`, and `key_name` For example, if your Packer run created an AMI `ami-03180edfa45c0fce2` in region `us-east-1`, and your AWS EC2 Key Pair is named `user-us-east-1`, your variables file would look similar to the following: region = "us-east-1" availability_zones = ["us-east-1a"] ami = "ami-0bd21458ecf89f85e" key_name = "user-us-east-1" owner_name = "alovelace" owner_email = "alovelace@example.com" ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#run-terraform) Run Terraform While in the `terraform/control` folder, provision your demo infrastructure by using the Terraform "init, plan, apply" cycle: $ terraform init $ terraform plan $ terraform apply --auto-approve Once the `terraform apply` finishes, a number of useful pieces of information should be output to your console. These include URLs to deployed resources as well as a Nomad Autoscaler job. ... Outputs: ip_addresses = < ZZZZZZZZZ $ gcloud beta billing accounts list ACCOUNT_ID NAME OPEN MASTER_ACCOUNT_ID Account 1 True AAAAAA-BBBBBB-CCCCCC ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#build-terraform-variables-file-1) Build Terraform variables file For Terraform to run correctly, you'll need to provide the values retrieved in the previous step into appropriate variables within a file named `terraform.tfvars`. Navigate to the Terraform control folder and create your own variables file by copying the provided `terraform.tfvars.sample` file. $ cd ./terraform/control $ cp terraform.tfvars.sample terraform.tfvars #### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#update-the-variables-for-your-environment-1) Update the variables for your environment * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`org_id`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#org_id) - The organization ID retrieved from the `gcloud organization list` command output. * [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#) [`billing_account`](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#billing_account) - The billing account ID retrieved from the `gcloud beta billing accounts list` command output. For example, if your organization ID is `123456789` and billing account ID is `824A25-7184CD-9217A0`, your variables file would look similar to the following. org_id = "123456789" billing_account = "824A25-7184CD-9217A0" ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#run-terraform-1) Run Terraform While in the `terraform/control` folder, provision your demo infrastructure by using the Terraform "init, plan, apply" cycle. $ terraform init $ terraform plan $ terraform apply --auto-approve Note Terraform may fail with an error message that says `Error 403: Compute Engine API has not been used in project ...`. If this happens, run `terraform apply --auto-approve` again. Once the `terraform apply` finishes, a number of useful pieces of information should be output to your console. These include URLs to deployed resources as well as a Nomad Autoscaler job. ... Outputs: stack_detail = <` command within your terminal or use the UI. You can test the integrity of the cluster by running: $ consul members $ nomad server members $ nomad node status The Nomad UI can be accessed at http://104.155.144.228:4646/ui The Consul UI can be accessed at http://104.155.144.228:8500/ui Grafana dashbaord can be accessed at http://34.72.51.47:3000/d/AQphTqmMk/demo?orgId=1&refresh=5s Traefik can be accessed at http://34.72.51.47:8081 Prometheus can be accessed at http://34.72.51.47:9090 Webapp can be accessed at http://34.72.51.47:80 CLI environment variables: export NOMAD_CLIENT_DNS=http://34.72.51.47 export NOMAD_ADDR=http://104.155.144.228:4646 EOT ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#configure-deployment-environment-variables) Configure deployment environment variables Login using the Azure command-line interface. $ az login [\ {\ "cloudName": "AzureCloud",\ "id": "",\ "isDefault": true,\ "name": "Free Trial",\ "state": "Enabled",\ "tenantId": "",\ "user": {\ "name": "user@example.com",\ "type": "user"\ }\ }\ \ \ Take a note of the values for `` and `` end export them as environment variables:\ \ $ export ARM_SUBSCRIPTION_ID=\ $ export ARM_TENANT_ID=\ \ \ Next, create an application ID and password that will be used to run Terraform.\ \ $ az ad sp create-for-rbac --role="Owner" --scopes="/subscriptions/$ARM_SUBSCRIPTION_ID"\ \ \ {\ "appId": "",\ "displayName": "azure-cli-...",\ "name": "http://azure-cli-...",\ "password": "",\ "tenant": ""\ }\ \ \ Export the values for `` and `` as environment variables as well:\ \ $ export ARM_CLIENT_ID=\ $ export ARM_CLIENT_SECRET=\ \ \ ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#run-terraform-2)\ Run Terraform\ \ Navigate to the Terraform control folder and execute the Terraform configuration to deploy the demo infrastructure:\ \ $ cd ./terraform/control\ \ \ $ terraform init\ \ \ $ terraform plan\ \ \ $ terraform apply --auto-approve\ \ \ Once the `terraform apply` finishes, a number of useful pieces of information should be output to your console. These include URLs to deployed resources as well as a Nomad Autoscaler job.\ \ ...\ Outputs:\ \ ip_addresses = <\ \ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#open-the-scenario-s-grafana-dashboard)\ Open the scenario's Grafana dashboard\ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------\ \ Retrieve the Grafana link from your Terraform output. Open it in a browser. It might take a minute to fully load if you didn't take some time earlier to look around.\ \ Once loaded, you will receive a dashboard similar to this.\ \ ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-start.png)\ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#generate-application-load)\ Generate application load\ ---------------------------------------------------------------------------------------------------------------------------------------------\ \ In order to generate some initial load, you will use the `hey` application. This will cause the application to scale up slightly.\ \ ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#run-a-load-generator)\ Run a load generator\ \ $ hey -z 10m -c 20 -q 40 $NOMAD_CLIENT_DNS:80 &\ \ \ Viewing the autoscaler logs or the Grafana dashboard should show the application count increase from `1` to `2`. Once this scaling has taken place, you can trigger additional load on the app that causes further scaling.\ \ DashboardLogs\ \ ![](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-app-scale-up.png)\ \ The application count is the graph in the top left.\ \ ![](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-app-scale-up-zoom.png)\ \ 2021-03-03T21:51:29.288Z [INFO] policy_eval.worker: scaling target: id=8e8be187-0ead-a0ef-5e41-a0db5f29f434 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target from=1 to=2 reason="scaling up because factor is 1.700000" meta=map[nomad_policy_id:56258c47-1a28-837c-568d-80dd9e9b3054]\ 2021-03-03T21:51:29.301Z [INFO] policy_eval.worker: successfully submitted scaling action to target: id=8e8be187-0ead-a0ef-5e41-a0db5f29f434 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target desired_count=2\ 2021-03-03T21:51:29.301Z [INFO] policy_eval.worker: policy evaluation complete: id=8e8be187-0ead-a0ef-5e41-a0db5f29f434 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target\ \ \ ### [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#run-a-second-load-generator)\ Run a second load generator\ \ $ hey -z 10m -c 20 -q 40 $NOMAD_CLIENT_DNS:80 &\ \ \ This again causes the application to scale, this time from `2` to `4`, which in-turn reduces the available resources on your cluster. The reduction is such that the Autoscaler will decide a cluster scaling action is required and trigger the appropriate action.\ \ DashboardLogs\ \ ![](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-client-scale-up-3.png)\ \ When watching the dashboard, you may see the cluster scale to three clients. This is because many cloud providers will add an additional instance to a scale-up request and then terminate the slowest instance to start.\ \ Once this instance is reaped, you will see the expected two clients.\ \ ![](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-client-settle-2.png)\ \ 2021-03-03T21:52:59.287Z [INFO] policy_eval.worker: scaling target: id=203a21b7-302f-ba72-bc1a-3d9ac6748a14 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target from=2 to=4 reason="scaling up because factor is 2.000000" meta=map[nomad_policy_id:56258c47-1a28-837c-568d-80dd9e9b3054]\ 2021-03-03T21:52:59.323Z [INFO] policy_eval.worker: successfully submitted scaling action to target: id=203a21b7-302f-ba72-bc1a-3d9ac6748a14 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target desired_count=4\ 2021-03-03T21:52:59.323Z [INFO] policy_eval.worker: policy evaluation complete: id=203a21b7-302f-ba72-bc1a-3d9ac6748a14 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target\ \ \ The additional allocations require more memory than is available on the current client. In response, the autoscaler starts another client to run some of the allocations.\ \ 2021-03-03T21:53:29.659Z [INFO] policy_eval.worker: scaling target: id=6cd5c72c-664d-0111-0e5b-8446638059ee policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg from=1 to=2 reason="scaling up because factor is 1.360405" meta=map[nomad_policy_id:484d733c-b28c-4a6e-364c-b36f10c6bb0b]\ 2021-03-03T21:53:50.319Z [INFO] internal_plugin.aws-asg: successfully performed and verified scaling out: action=scale_out asg_name=hashistack-nomad_client desired_count=2\ 2021-03-03T21:53:50.319Z [INFO] policy_eval.worker: successfully submitted scaling action to target: id=6cd5c72c-664d-0111-0e5b-8446638059ee policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg desired_count=2\ 2021-03-03T21:53:50.320Z [INFO] policy_eval.worker: policy evaluation complete: id=6cd5c72c-664d-0111-0e5b-8446638059ee policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg\ \ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#remove-load-on-the-application)\ Remove load on the application\ -------------------------------------------------------------------------------------------------------------------------------------------------------\ \ Now, simulate a reduction in load on the application by stopping the running `hey` processes using the `pkill` command.\ \ $ pkill hey\ [2] + 36851 terminated hey -z 10m -c 20 -q 40 $NOMAD_CLIENT_DNS:80\ [1] + 36827 terminated hey -z 10m -c 20 -q 40 $NOMAD_CLIENT_DNS:80\ \ \ The reduction in load causes the Autoscaler to firstly scale in the task group from `4` to `1`. Once the task group has scaled in a sufficient amount, the Autoscaler scales in the cluster from `2` to `1`. It performs this work by selecting a node to remove, draining the node of all work, and then terminating it within the provider.\ \ DashboardLogs\ \ ![](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs%2Fdash-client-scale-in-1.png)\ \ Once you stop the load generating processes, the autoscaler reduces the application's allocation count from 4 to 1\ \ 2021-03-03T21:57:29.287Z [INFO] policy_eval.worker: scaling target: id=5f7f0f2b-fe20-9fb6-5132-7554cd40c875 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target from=4 to=1 reason="capped count from 0 to 1 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:0 nomad_autoscaler.reason_history:[scaling down because factor is 0.000000] nomad_policy_id:56258c47-1a28-837c-568d-80dd9e9b3054]"\ 2021-03-03T21:57:29.308Z [INFO] policy_eval.worker: successfully submitted scaling action to target: id=5f7f0f2b-fe20-9fb6-5132-7554cd40c875 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target desired_count=1\ 2021-03-03T21:57:29.308Z [INFO] policy_eval.worker: policy evaluation complete: id=5f7f0f2b-fe20-9fb6-5132-7554cd40c875 policy_id=56258c47-1a28-837c-568d-80dd9e9b3054 queue=horizontal target=nomad-target\ \ \ Once the surplus allocations are stopped, the autoscaler then scales the cluster back to 1 client in response to excess available memory.\ \ 2021-03-03T21:58:29.585Z [INFO] policy_eval.worker: scaling target: id=9fb3b478-1a93-827b-4d5e-b9fa4924668b policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg from=2 to=1 reason="scaling down because factor is 0.178571" meta=map[nomad_policy_id:484d733c-b28c-4a6e-364c-b36f10c6bb0b]\ 2021-03-03T21:58:29.653Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=616d429e-74f7-c3a7-2052-445980e28987 deadline=5m0s\ 2021-03-03T21:58:29.668Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=616d429e-74f7-c3a7-2052-445980e28987 msg="Drain complete for node 616d429e-74f7-c3a7-2052-445980e28987"\ 2021-03-03T21:58:34.987Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=616d429e-74f7-c3a7-2052-445980e28987 msg="All allocations on node "616d429e-74f7-c3a7-2052-445980e28987" have stopped"\ 2021-03-03T21:58:34.987Z [INFO] internal_plugin.aws-asg: node drain complete: node_id=616d429e-74f7-c3a7-2052-445980e28987\ 2021-03-03T21:58:34.987Z [INFO] internal_plugin.aws-asg: pre scale-in tasks now complete\ 2021-03-03T21:58:45.522Z [INFO] internal_plugin.aws-asg: successfully detached instances from AutoScaling Group: action=scale_in asg_name=hashistack-nomad_client instances=[i-016ca3ede8b32bd49]\ 2021-03-03T21:59:56.219Z [INFO] internal_plugin.aws-asg: successfully terminated EC2 instances: action=scale_in asg_name=hashistack-nomad_client instances=[i-016ca3ede8b32bd49]\ 2021-03-03T21:59:56.431Z [INFO] policy_eval.worker: successfully submitted scaling action to target: id=9fb3b478-1a93-827b-4d5e-b9fa4924668b policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg desired_count=1\ 2021-03-03T22:00:43.194Z [INFO] policy_eval.worker: policy evaluation complete: id=9fb3b478-1a93-827b-4d5e-b9fa4924668b policy_id=484d733c-b28c-4a6e-364c-b36f10c6bb0b queue=cluster target=aws-asg\ \ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#destroy-the-demo-infrastructure)\ Destroy the demo infrastructure\ ---------------------------------------------------------------------------------------------------------------------------------------------------------\ \ Once you are done experimenting with the autoscaler, use the `terraform destroy` command to deprovision the demo infrastructure.\ \ It is important to destroy the created infrastructure as soon as you are finished with the demo to avoid unnecessary charges in your cloud provider account. To do this, issue the `terraform destroy` command.\ \ $ terraform destroy --auto-approve\ \ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#perform-cloud-specific-cleanup-activities)\ Perform cloud-specific cleanup activities\ -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ \ Amazon Web ServicesGoogle Cloud PlatformMicrosoft Azure\ \ Deregister the AMI that you created with Packer in the beginning of this demo. You can use the AWS console or the [AWS CLI](https://aws.amazon.com/cli/)\ if you have it installed.\ \ This set of commands will extract the AMI ID from your variables file, deregister the image, and delete the backing EBS snapshot.\ \ $ export IMAGE=$(awk '/ami/ {print $3}' terraform.tfvars | tr -d "\"")\ $ export REGION=$(awk '/region/ {print $3}' terraform.tfvars | tr -d "\"")\ $ export SNAP=$(aws ec2 describe-images --image-id $IMAGE --region $REGION --output json --query 'Images[0].BlockDeviceMappings[0].Ebs.SnapshotId' --no-paginate | tr -d "\"")\ $ aws ec2 deregister-image --image-id $IMAGE --region $REGION\ $ aws ec2 delete-snapshot --snapshot-id $SNAP --region $REGION\ \ \ There are no extra steps required for GCP.\ \ Remove the application ID you created for this demo by running the following command.\ \ $ az ad sp delete --id $ARM_CLIENT_ID\ \ \ [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling#next-steps)\ Next steps\ ---------------------------------------------------------------------------------------------------------------\ \ Now that you have explored horizontal cluster autoscaling with this demonstration, continue learning about the Nomad Autoscaler.\ \ * [Test Drive Horizontal Application Scaling](https://developer.hashicorp.com/nomad/tutorials/autoscaler/autoscaler-vagrant-demo)\ \ * [Test Drive Dynamic Application Sizing](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing)\ \ * [Dynamic Application Scaling Concepts](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts)\ \ * [Nomad Autoscaler Documentation](https://developer.hashicorp.com/nomad/tools/autoscaling)\ \ \ **Was this tutorial helpful?**\ \ YesNo\ \ [Previous\ \ Scale an application horizontally](https://developer.hashicorp.com/nomad/tutorials/autoscaler/autoscaler-vagrant-demo)\ [Next\ \ Scale cluster nodes with the Nomad Autoscaler](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling-on-demand-batch)\ \ This tutorial also appears in:\ ------------------------------\ \ *  [](https://developer.hashicorp.com/nomad/tutorials/nomad-1-0)\ \ 3 tutorials\ \ Nomad 1.0\ \ Explore Nomad v1.0 features like Nomad Autoscaler and the event-stream.\ \ * Nomad\ \ \ *  [](https://developer.hashicorp.com/nomad/tutorials/ecosystem)\ \ 5 tutorials\ \ Expand Nomad with Ecosystem Add-ins\ \ Explore applications that enhance how you use your Nomad cluster through their use of the Nomad HTTP API or plug-in interface.\ \ * Nomad --- # Autoscaling | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Autoscaler Overview ========================= This section details the Nomad Autoscaler, a horizontal application and cluster autoscaler for Nomad. The Nomad Autoscaler is built and released separately to Nomad. It runs as a daemon process, often referred to as the [Autoscaler Agent](https://developer.hashicorp.com/nomad/tools/autoscaling/agent) . An Autoscaler Agent is a separate and distinct process from a Nomad Agent. The source code can be viewed on [GitHub](https://github.com/hashicorp/nomad-autoscaler) and releases are available on the [HashiCorp releases page](https://releases.hashicorp.com/nomad-autoscaler/) or via [Docker Hub](https://hub.docker.com/r/hashicorp/nomad-autoscaler) . The Nomad Autoscaler repository includes a number of [demos](https://github.com/hashicorp/nomad-autoscaler-demos) which provide guided learning on running the Nomad AutoScaler. Since Autoscaler Agent runs as its own daemon, it requires its own configuration. Configuration can be passed as command line flags or parsed from a configuration file. The demos repository includes an example [config](https://github.com/hashicorp/nomad-autoscaler-demos/blob/1ecd9f32c749f1faaf4154b8a7e57fa68642fd33/cloud/demos/on-demand-batch/aws/jobs/autoscaler.nomad.tpl#L20) file that is useful to reference when configuring your Autoscaler Agent. [](https://developer.hashicorp.com/nomad/tools/autoscaling#horizontal-application-autoscaling) Horizontal Application Autoscaling --------------------------------------------------------------------------------------------------------------------------------- Horizontal application autoscaling is the process of automatically controlling the number of instances of an application to have sufficient work throughput to meet service-level agreements (SLA). In Nomad, horizontal application autoscaling can be achieved by modifying the number of allocations in a task group based on the value of a relevant metric, such as CPU and memory utilization or number of open connections. This is enabled by configuring [autoscaling policies](https://developer.hashicorp.com/nomad/tools/autoscaling/policy) on individual Nomad jobs using the [`scaling` block](https://developer.hashicorp.com/nomad/docs/job-specification/scaling#scaling-block) . [](https://developer.hashicorp.com/nomad/tools/autoscaling#horizontal-cluster-autoscaling) Horizontal Cluster Autoscaling ------------------------------------------------------------------------------------------------------------------------- Horizontal cluster autoscaling is the process of adding or removing Nomad clients from a cluster to ensure there is an appropriate amount of cluster resource for the scheduled applications. This is achieved by interacting with remote providers to start or terminate new Nomad clients based on metrics such as the remaining free schedulable CPU or memory. Cluster scaling is enabled by configuring the [autoscaler agent](https://developer.hashicorp.com/nomad/tools/autoscaling/agent#dir) with policies targeting the Nomad cluster. [](https://developer.hashicorp.com/nomad/tools/autoscaling#dynamic-application-sizing) Dynamic Application Sizing ----------------------------------------------------------------------------------------------------------------- Enterprise This functionality only exists in Nomad Autoscaler Enterprise. This is not present in the Community Edition of Nomad Autoscaler. Dynamic Application Sizing enables organizations to optimize the resource consumption of applications using sizing recommendations from Nomad. It evaluates, processes and stores historical task resource usage data, making recommendations for CPU and Memory resource parameters. The recommendations can be calculated using a number of different algorithms to ensure the recommendation best fits the application profile. Dynamic Application Sizing can be enabled on an individual task by configuring [autoscaling policies](https://developer.hashicorp.com/nomad/tools/autoscaling/policy) within the task block using the job specification [`scaling` block](https://developer.hashicorp.com/nomad/docs/job-specification/scaling#scaling-block) . [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/tools/autoscaling/index.mdx) --- # Generate mTLS certificates for Nomad using Vault | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Vault](https://developer.hashicorp.com/nomad/tutorials/integrate-vault) You can use Vault's [PKI Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/pki) to generate and renew dynamic X.509 certificates for your Nomad cluster nodes and [Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) to automatically create the appropriate certificate and key files on your nodes. With this method, each node has a unique certificate with a relatively short time-to-live (TTL). This feature, along with automatic certificate rotation, allows you to safely and securely scale your cluster while using mutual TLS (mTLS). In this tutorial, you will secure your existing Nomad cluster with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use Vault Agent to fetch, renew, and periodically rotate your mTLS certificates on your Nomad nodes. [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#prerequisites) Prerequisites --------------------------------------------------------------------------------------------------------------- Running this tutorial requires a Nomad environment with Vault installed. You can use the [Nomad sandbox](https://developer.hashicorp.com/nomad/sandbox) to follow along in your browser. Otherwise, you can use [this repository](https://github.com/hashicorp/nomad/tree/master/terraform) to provision a sandbox environment. If you use our hosted Nomad sandbox to follow this tutorial: * Load the Nomad, Consul, and Vault tokens, as described in the [Nomad sandbox documentation](https://developer.hashicorp.com/nomad/sandbox) . * Update the filepaths in this tutorial's configurations and commands to reference `/nomad-sandbox/` instead of `/opt/nomad/`. That way you can use the `Editor` table to create directories and update files. * Restart the Nomad agents as described in the [Nomad sandbox documentation](https://developer.hashicorp.com/nomad/sandbox) , instead of using `systemctl`. * Begin this tutorial by [unsealing Vault](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#unseal-vault) with the value stored in `/nomad-sandbox/configs/vault/unseal.key`. Note This tutorial is for demonstration purposes and is using a single Nomad server with a Vault server configured alongside it. In a production cluster, we recommend 3 or 5 Nomad server nodes along with a separate Vault cluster. Refer to the [Vault Reference Architecture](https://developer.hashicorp.com/vault/tutorials/raft/raft-reference-architecture) page to learn how to securely deploy a Vault cluster. [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#prepare-vault) Prepare Vault --------------------------------------------------------------------------------------------------------------- If you already have a Vault environment, skip ahead to [Log in to Vault](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#log-in-to-vault) . ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#initialize-vault-server) Initialize Vault server Run the following command to initialize the Vault server and receive an [unseal](https://developer.hashicorp.com/vault/docs/concepts/seal) key and initial root [token](https://developer.hashicorp.com/vault/docs/concepts/tokens) . If you are running the environment provided in this guide, the Vault server is co-located with the Nomad server. Be sure to note the unseal key and initial root token as you will need these two pieces of information. $ vault operator init -key-shares=1 -key-threshold=1 The [`vault operator init` command](https://developer.hashicorp.com/vault/docs/commands/operator/init) creates a single Vault unseal key for convenience. For a production environment, we recommended that you create at least five unseal key shares and securely distribute them to independent operators. The command defaults to five key shares and a key threshold of three. If you have provisioned more than one server, the others will become standby nodes and remain unsealed. ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#unseal-vault) Unseal Vault Run the command and provide your unseal key. $ vault operator unseal The output looks similar to the following. Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.0.3 Cluster Name vault-cluster-d1b6513f Cluster ID 87d6d13f-4b92-60ce-1f70-41a66412b0f1 HA Enabled true HA Cluster n/a HA Mode standby Active Node Address [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#log-in-to-vault) Log in to Vault ------------------------------------------------------------------------------------------------------------------- Use the [login command](https://developer.hashicorp.com/vault/docs/commands/login) to authenticate with Vault using the initial root token from earlier. $ vault login Token (will be hidden): Successful output looks similar to the following. Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. ... [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#prepare-the-pki-environment) Prepare the PKI environment ------------------------------------------------------------------------------------------------------------------------------------------- This tutorial uses a common and recommended pattern which is to have one mount act as the root CA and the other as the intermediate CA. The root CA signs only the intermediate CA CSRs from other PKI secrets engines. For a higher level of security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA. ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#generate-the-root-ca) Generate the root CA Enable the root [PKI secrets engine](https://developer.hashicorp.com/vault/docs/secrets/pki) at the `pki` path. $ vault secrets enable pki Success! Enabled the pki secrets engine at: pki/ Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours. $ vault secrets tune -max-lease-ttl=87600h pki Success! Tuned the secrets engine at: pki/ Generate the root certificate and save the certificate as `CA_cert.crt`. Note If you already have an existing root CA certificate in your organization, you can use that in place of generating a new one. $ vault write -field=certificate pki/root/generate/internal \ common_name="global.nomad" ttl=87600h > CA_cert.crt ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#generate-the-intermediate-ca-and-csr) Generate the intermediate CA and CSR Enable the intermediate PKI secrets engine at the `pki_int` path. $ vault secrets enable -path=pki_int pki Success! Enabled the pki secrets engine at: pki_int/ Tune the PKI secrets engine at the `pki_int` path to issue certificates with a maximum time-to-live (TTL) of 43800 hours. $ vault secrets tune -max-lease-ttl=43800h pki_int Success! Tuned the secrets engine at: pki_int/ Generate a CSR from your intermediate CA and save it as `pki_intermediate.csr`. $ vault write -format=json pki_int/intermediate/generate/internal \ common_name="global.nomad Intermediate Authority" \ ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#sign-and-deploy-the-intermediate-ca-certificate) Sign and deploy the intermediate CA certificate Sign the intermediate CA CSR with the root certificate and save it as `intermediate.cert.pem`. $ vault write -format=json pki/root/sign-intermediate \ csr=@pki_intermediate.csr format=pem_bundle \ ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem Import the signed intermediate certificate into Vault. $ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#set-up-the-approle-auth-method) Set up the AppRole auth method ------------------------------------------------------------------------------------------------------------------------------------------------- Vault offers many [authentication methods](https://developer.hashicorp.com/vault/docs/auth) including AWS, GCP, Azure, GitHub, LDAP, and Okta. In a production environment running in a cloud provider, we recommend using the machine's identity and the cloud provider's auth method. The [AppRole authentication method](https://developer.hashicorp.com/vault/docs/auth/approle) allows applications or services to authenticate with Vault-defined roles. For on-premises environments, we recommend AppRoles. In this tutorial, Nomad is using the AppRole auth method to retrieve certificates from Vault with the PKI engine. Enable the AppRole auth method. $ vault auth enable approle Success! Enabled approle auth method at: approle/ ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-the-pki-secrets-engine-role) Create the PKI secrets engine role Create a role to manage the intermediate PKI secrets engine authority information. $ vault write pki_int/roles/nomad-cluster \ allowed_domains="global.nomad" \ allow_subdomains=true \ generate_lease=true \ max_ttl="720h" The output includes a list of keys and values similar to the following. Key Value --- ----- allow_any_name false allow_bare_domains false allow_glob_domains false allow_ip_sans true allow_localhost true allow_subdomains true allow_token_displayname false allow_wildcard_certificates true allowed_domains [global.nomad] # ... ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-a-policy-to-access-the-role-endpoint) Create a policy to access the role endpoint Create a policy file named `tls-policy.hcl`, add the following contents to it, and save the file. This adds only the [`update` capability](https://developer.hashicorp.com/vault/docs/concepts/policies#capabilities) for the PKI secrets engine at the `pki_int/issue/nomad-cluster` path. Refer to the [Vault policies page](https://developer.hashicorp.com/vault/docs/concepts/policies#policies) for additional information. tls-policy.hcl path "pki_int/issue/nomad-cluster" { capabilities = ["update"] } Write the policy into Vault. $ vault policy write tls-policy tls-policy.hcl Success! Uploaded policy: tls-policy ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-an-approle) Create an approle Create an approle named `nomad-cluster`. Consider increasing or decreasing the TTL and SecretID values to match the your environment. $ vault write auth/approle/role/nomad-cluster \ token_type=batch \ token_policies="tls-policy" \ secret_id_ttl=10m \ token_ttl=10m \ token_max_ttl=15m \ secret_id_num_uses=10 The output looks similar to the following. Success! Data written to: auth/approle/role/nomad-cluster [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-and-populate-the-templates-directory) Create and populate the templates directory --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The templates directory contains the files used by Vault Agent to render the certificates and keys on the nodes in your cluster. Create a directory named `templates` in `/opt/nomad`. $ sudo mkdir -p /opt/nomad/templates Create a template file for each configuration below, add the contents to it, and save it with the filename given. We recommend creating only the _server_ and CLI files for server nodes and _client_ and CLI files for client nodes. Note The `common_name` value in the files below must match the [Nomad node names](https://developer.hashicorp.com/nomad/docs/configuration#name) in your cluster which include the node name and region. If you have a different configuration for the nodes in your cluster, be sure to update the values. Nomad server nodesNomad client nodesNomad CLI /opt/nomad/templates/server.agent.crt.tpl {{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}} {{ .Data.Cert }} {{ .Data.Key | writeToFile "/opt/nomad/rendered-cert-files/server.agent.key" "" "" "0644" }} {{ .Data.CA | writeToFile "/opt/nomad/rendered-cert-files/server.ca.crt" "" "" "0644" }} {{ end }} /opt/nomad/templates/client.agent.crt.tpl {{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}} {{ .Data.Cert }} {{ .Data.Key | writeToFile "/opt/nomad/rendered-cert-files/client.agent.key" "" "" "0644" }} {{ .Data.CA | writeToFile "/opt/nomad/rendered-cert-files/client.ca.crt" "" "" "0644" }} {{ end }} /opt/nomad/templates/cli.crt.tpl {{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=cli.global.nomad" "ttl=24h"}} {{ .Data.Cert }} {{ .Data.Key | writeToFile "/opt/nomad/rendered-cert-files/cli.key" "" "" "0644" }} {{ end }} ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-the-certificates-output-directory) Create the certificates output directory Create the `rendered-cert-files` directory to contain the generated certificate files. $ sudo mkdir -p /opt/nomad/rendered-cert-files [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#start-vault-agent) Start Vault Agent ----------------------------------------------------------------------------------------------------------------------- Create a directory for the configuration named `vault-agent` in `/opt/vault`. $ sudo mkdir -p /opt/vault/vault-agent Change to the directory. $ cd /opt/vault/vault-agent Get the role ID for the `nomad-cluster` approle and save it to a file. $ vault read -field="role_id" auth/approle/role/nomad-cluster/role-id > roleid Then, get the SecretID and save it to a file. The SecretID is similar to a password in that you should keep it secret and confidential. Refer to the [best practices page](https://developer.hashicorp.com/vault/docs/auth/approle/approle-pattern#secretid-delivery-best-practices) for additional information about working with the SecretID. $ vault write -field="secret_id" -force auth/approle/role/nomad-cluster/secret-id > secretid ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#create-the-vault-agent-configuration) Create the Vault Agent configuration Create the Vault Agent configuration file, name it `vault_agent.conf`, add the following contents to it, and save the file in the `/opt/vault/vault-agent` directory. Modify this configuration file to match the templates created for each specific node from the previous step. Include or remove the server and client template blocks as necessary. /opt/vault/vault-agent/vault\_agent.conf pid_file = "./pidfile" vault { address = "https://127.0.0.1:8200" # For demonstration purposes only. In a production environment, # tls_skip_verify value should NOT be set to true. tls_skip_verify = true } auto_auth { method { type = "approle" config = { role_id_file_path = "/opt/vault/vault-agent/roleid" secret_id_file_path = "/opt/vault/vault-agent/secretid" remove_secret_id_file_after_reading = false } } sink { type = "file" wrap_ttl = "30m" config = { path = "sink_file_wrapped_1.txt" } } sink { type = "file" config = { path = "sink_file_unwrapped_2.txt" } } } listener "tcp" { address = "127.0.0.1:8100" tls_disable = true } template_config { static_secret_render_interval = "10m" exit_on_retry_failure = true max_connections_per_host = 20 } # Server templates; remove if on client node template { source = "/opt/nomad/templates/server.agent.crt.tpl" destination = "/opt/nomad/rendered-cert-files/server.agent.crt" } # Client templates; remove if on server node template { source = "/opt/nomad/templates/client.agent.crt.tpl" destination = "/opt/nomad/rendered-cert-files/client.agent.crt" } # CLI templates template { source = "/opt/nomad/templates/cli.crt.tpl" destination = "/opt/nomad/rendered-cert-files/cli.crt" } Then, start Vault Agent on each node. $ sudo vault agent -config=/opt/vault/vault-agent/vault_agent.conf The output shows the agent rendering the local files with the certificate data. # ... 2024-11-26T10:09:54.347-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/cli.crt.tpl" => "/opt/nomad/rendered-cert-files/cli.crt" 2024-11-26T10:09:54.484-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/client.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/client.agent.crt" 2024-11-26T10:09:54.569-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/server.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/server.agent.crt" [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#configure-nomad-to-use-tls) Configure Nomad to use TLS ----------------------------------------------------------------------------------------------------------------------------------------- Add the following [tls stanza](https://developer.hashicorp.com/nomad/docs/configuration/tls) to the agent configuration file of each Nomad server and client node in the cluster. Server nodesClient nodes /etc/nomad.d/nomad.hcl tls { http = true rpc = true rpc_upgrade_mode = true ca_file = "/opt/nomad/rendered-cert-files/server.ca.crt" cert_file = "/opt/nomad/rendered-cert-files/server.agent.crt" key_file = "/opt/nomad/rendered-cert-files/server.agent.key" verify_server_hostname = true verify_https_client = true } The [`rpc_upgrade_mode`](https://developer.hashicorp.com/nomad/docs/configuration/tls#rpc_upgrade_mode) attribute ensures that the Nomad servers accept both TLS and non-TLS connections during the upgrade. /etc/nomad.d/nomad.hcl tls { http = true rpc = true ca_file = "/opt/nomad/rendered-cert-files/client.ca.crt" cert_file = "/opt/nomad/rendered-cert-files/client.agent.crt" key_file = "/opt/nomad/rendered-cert-files/client.agent.key" verify_server_hostname = true verify_https_client = true } Reload the Nomad configuration on each node. $ systemctl reload nomad After the reload, remove the `rpc_upgrade_mode = true` line from each server agent configuration file and save the file. This instructs the servers to only accept TLS connections. /etc/nomad.d/nomad.hcl tls { http = true rpc = true - rpc_upgrade_mode = true # ... Then, reload the Nomad configuration on each server node again. Refer to the [RPC Upgrade Mode page](https://developer.hashicorp.com/nomad/docs/secure/traffic/tls#rpc-upgrade-mode-for-nomad-servers) for additional information. $ systemctl reload nomad #### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#configure-the-nomad-cli) Configure the Nomad CLI The Nomad CLI defaults to HTTP instead of HTTPS. Configure the Nomad CLI to connect using TLS by providing paths to the certificate files through environment variables. export NOMAD_CACERT="/opt/nomad/rendered-cert-files/server.ca.crt" export NOMAD_CLIENT_CERT="/opt/nomad/rendered-cert-files/cli.crt" export NOMAD_CLIENT_KEY="/opt/nomad/rendered-cert-files/cli.key" Run the status command to confirm connectivity from the CLI. $ nomad status No running jobs ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#encrypt-server-gossip) Encrypt server gossip All communications over RPC and HTTP in your Nomad cluster are now secure with mTLS. Nomad servers also communicate with the Serf gossip protocol, which does not use TLS. To learn how to configure gossip encryption, follow the [Enable Gossip Encryption for Nomad](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption) tutorial. [](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad#next-steps) Next steps --------------------------------------------------------------------------------------------------------- In this tutorial, you configured the Vault PKI engine, installed and used Vault Agent to automatically create certificate files on cluster nodes, and enabled mTLS with those certificates to encrypt traffic between the nodes. Continue your learning with these resources: * [AppRole pull authentication tutorial](https://developer.hashicorp.com/vault/tutorials/auth-methods/approle) * [Vault Agent templates](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/template) * [Vault PKI Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/pki) * [Build Your Own Certificate Authority (CA)](https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine) * Learn about [Vault Proxy and how it compares to Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy#introduce-vault-agent-and-vault-proxy-to-the-workflow) **Was this tutorial helpful?** YesNo [Collection Overview\ \ Vault](https://developer.hashicorp.com/nomad/tutorials/integrate-vault) [Next Collection\ \ Workload identity](https://developer.hashicorp.com/nomad/tutorials/fed-workload-identity) This tutorial also appears in: ------------------------------ *  [](https://developer.hashicorp.com/vault/tutorials/cross-products) 9 tutorials HashiCorp product integrations Vault can manage secrets associated with other HashiCorp products. * Vault --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.10.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Nomad is a flexible workload orchestrator to deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run diverse workloads including Docker, non-containerized, microservice, and batch applications. If you are just getting started with Nomad, refer to the [Getting Started tutorial collection](https://developer.hashicorp.com/nomad/tutorials/get-started) . Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.10.x/content/docs/index.mdx) --- # Template abstract job specs with Levant | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) [Nomad Pack](https://developer.hashicorp.com/nomad/tools/nomad-pack) is a new package manager and templating tool that you may use instead of Levant. We recommend Nomad Pack for new projects. In this tutorial, you will iteratively modify the Zookeeper template created in the [DRY Nomad Job Specs with Levant](https://developer.hashicorp.com/nomad/tutorials/templates/dry-jobs-levant) tutorial to create an abstract job that you can deploy with Levant. When combined with sensible defaults and the ability to override these default values, you can create flexible deployments of Zookeeper clusters without modifying the base template. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs#challenge) Challenge --------------------------------------------------------------------------------------------------------------- Every Nomad job running in a cluster requires its own job configuration; typically, these are maintained as a file per job. While this does provide for archival quality job specifications suitable for source control, it precludes certain patterns that operators might desire with job specification: **composition**, **granularity**, and **abstraction** [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs#solution) Solution ------------------------------------------------------------------------------------------------------------- Levant, as a Nomad-aware template engine, allows you to create abstract job specifications, which when combined with user supplied values, can be used to render and deploy a composed job specification to your Nomad cluster. This tutorial uses the DRY Zookeeper job template created in the [DRY Nomad Job Specs with Levant](https://developer.hashicorp.com/nomad/tutorials/templates/dry-jobs-levant) tutorial to: * Export key configuration elements to a configuration file, so that they can be edited independently of the job specification * Allow for variable node counts, so that you can create highly available clusters * Allow for instance specific values, so that you can support more than one cluster You will enhance the previous Zookeeper template to create multiple Zookeeper clusters of different node count and configuration to support each of your internal use cases with one template. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs#prerequisites) Prerequisites ----------------------------------------------------------------------------------------------------------------------- You need a Nomad cluster with: * Consul integrated * Docker installed and available as a task driver on your clients * One or more distinct client instances. This tutorial describes a case using three clients. * A [Nomad static host volume](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume-stanza) configured on each client you would like to deploy ZK to named `zk«ID»`, where ID is the index of the ZK node that should run there. These static host volumes can be collocated on a client for the purposes of this tutorial. You should be: * Familiar with Go's text/template syntax. You can learn more about it in the [Learn Go Template Syntax](https://developer.hashicorp.com/nomad/docs/reference/go-template-syntax) tutorial. * Comfortable in your shell of choice, specifically adding executables to the path, editing text files, and managing directories. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs#download-levant) Download Levant --------------------------------------------------------------------------------------------------------------------------- If you haven't already, install Levant using the instructions found in the [README](https://github.com/hashicorp/levant/blob/master/README.md) of its GitHub repository. Use one of the methods that provides you with a binary, rather than the Docker image. Verify that you have installed it to your executable path by running `levant version`. $ levant version Levant v0.3.0-dev (d7d77077+CHANGES) [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/levant-abstract-jobs#get-started) Get started ------------------------------------------------------------------------------------------------------------------- If you just completed the [DRY Nomad Job Specs with Levant](https://developer.hashicorp.com/nomad/tutorials/templates/dry-jobs-levant) tutorial, you have the starting Zookeeper job specification files. If not, you need to create a working directory, change into it, and create these three files. Click each section to reveal each file's contents. zookeeper.nomad Create a text file named **zookeeper.nomad** with the following contents. job "zookeeper" { datacenters = ["dc1"] type = "service" update { max_parallel = 1 } group "zk1" { volume "zk" { type = "host" read_only = false source = "zk1" } count = 1 restart { attempts = 10 interval = "5m" delay = "25s" mode = "delay" } [[- $Protocols := list "client" "peer" "election" "admin" ]] network { [[- range $I, $Protocol := $Protocols -]] [[- $To := -1]] [[- if eq $Protocol "admin" -]] [[- $To = 8080 -]] [[- end ]] [[- if ne $I 0 -]][[- println "" -]][[- end ]] port "[[$Protocol]]" { to = [[$To]] } [[- end ]] } [[- range $I, $Protocol := $Protocols -]] [[- $Tags := list $Protocol -]] [[ if eq $Protocol "client" ]][[- $Tags = append $Tags "zk1" -]][[- end -]] [[ if eq $Protocol "admin" ]][[- $Tags = list "zk1-admin" -]][[- end -]] [[- println "" ]] service { tags = [[ $Tags | toJson ]] name = "zookeeper" port = "[[ $Protocol ]]" meta { ZK_ID = "1" } address_mode = "host" } [[- end ]] task "zookeeper" { driver = "docker" template { destination = "config/zoo.cfg" data = < Monitoring evaluation "fe77cd96" Evaluation triggered by job "batch/dispatch-1620256102-20467b5d" Evaluation status changed: "pending" -> "complete" ==> Evaluation "fe77cd96" finished with status "complete" but failed to place all allocations: Task Group "batch" (failed to place 1 allocation): * No nodes were eligible for evaluation * No nodes are available in datacenter "batch_workers" Evaluation "207358f1" waiting for additional capacity to place remainder The command output indicates that the job was not able to start, since there are no clients in the `batch_workers` datacenter to satisfy its constraint. The dashboard shows that a new instance is now enqueued. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-one-queued.png) The dashed vertical blue line indicates that the Autoscaler detected that the cluster needs to be scaled, and so, after a few minutes, a new client is added to the `batch_workers` datacenter and the queued job starts running. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-one-running.png) Dispatch two more instances of the batch job. $ nomad job dispatch batch $ nomad job dispatch batch Like the previous time, the command output indicates that the job can not be scheduled. However, this time the limitation is the amount of memory available in the `batch_workers` datacenter. Dispatched Job ID = batch/dispatch-1620256277-1240a1be Evaluation ID = 7b7f2a12 ==> Monitoring evaluation "7b7f2a12" Evaluation triggered by job "batch/dispatch-1620256277-1240a1be" Evaluation status changed: "pending" -> "complete" ==> Evaluation "7b7f2a12" finished with status "complete" but failed to place all allocations: Task Group "batch" (failed to place 1 allocation): * Resources exhausted on 1 nodes * Class "hashistack" exhausted on 1 nodes * Dimension "memory" exhausted on 1 nodes Evaluation "78de4e93" waiting for additional capacity to place remainder The dashboard now displays two instances in the queue and one running. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-two-queued.png) The total number of instances in progress is three, and so the Autoscaler scales out the cluster so that there are three clients running. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-three-running.png) After a few more minutes, the first job completes, and the Autoscaler removes one client since it is not used anymore. The client removed is the one with no allocations running, so the other batch jobs continue running without disruption. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-first-complete.png) Finally, the last two jobs complete, and the Autoscaler brings the `batch_workers` cluster back to zero, saving us the cost of having any clients running without work. ![Screenshot of Grafana dashboard](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fecosystem%2Fhcs-batch%2Fdash-done.png) Feel to dispatch more jobs and explore the scenario further. [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling-on-demand-batch#destroy-the-demo-infrastructure) Destroy the demo infrastructure ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you are done experimenting with the autoscaler, use the `terraform destroy` command to deprovision the demo infrastructure. It is important to destroy the created infrastructure as soon as you are finished with the demo to avoid unnecessary charges in your cloud provider account. $ terraform destroy --auto-approve [](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling-on-demand-batch#next-steps) Next steps ------------------------------------------------------------------------------------------------------------------------------- Now that you have explored horizontal cluster autoscaling for on-demand batch job with this demonstration, continue learning about the Nomad Autoscaler. * [Test Drive Horizontal Application Scaling](https://developer.hashicorp.com/nomad/tutorials/autoscaler/autoscaler-vagrant-demo) * [Test Drive Dynamic Application Sizing](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing) * [Dynamic Application Scaling Concepts](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts) * [Nomad Autoscaler Documentation](https://developer.hashicorp.com/nomad/tools/autoscaling) **Was this tutorial helpful?** YesNo [Previous\ \ Scale a cluster horizontally](https://developer.hashicorp.com/nomad/tutorials/autoscaler/horizontal-cluster-scaling) [Next\ \ Dynamic app sizing](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts) This tutorial also appears in: ------------------------------ *  [](https://developer.hashicorp.com/nomad/tutorials/ecosystem) 5 tutorials Expand Nomad with Ecosystem Add-ins Explore applications that enhance how you use your Nomad cluster through their use of the Nomad HTTP API or plug-in interface. * Nomad --- # Task HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Task API ======== Nomad's Task API provides every task managed by Nomad with a Unix Domain Socket (UDS) to access the local agent's HTTP API. Regardless of agent configuration the Task API does _not_ require [mTLS](https://developer.hashicorp.com/nomad/docs/secure/traffic/tls) , but _always_ requires authentication. See below for details. The Unix Domain Socket is located at `${NOMAD_SECRETS_DIR}/api.sock`. [](https://developer.hashicorp.com/nomad/api-docs/task-api#rationale) Rationale ------------------------------------------------------------------------------- Nomad's HTTP API is available on every agent at the configured [`bind_addr`](https://developer.hashicorp.com/nomad/docs/configuration) . While this is convenient for user access, it is not always accessible to workloads running on Nomad. These workloads may have a network configuration that makes it impossible to access the agent HTTP address, or the agent's HTTP address may be difficult for workloads to discover in a way that's portable between Nomad nodes and clusters. A Unix Domain Socket is a way to expose network services that works with most runtimes and operating systems and adds minimal complexity or runtime overhead to Nomad. [](https://developer.hashicorp.com/nomad/api-docs/task-api#security) Security ----------------------------------------------------------------------------- Unlike the agent's HTTP API, the Task API _always requires authentication_ even if [ACLs](https://developer.hashicorp.com/nomad/docs/secure/acl) are disabled. This allows Nomad to always make the Task API available even if the workload is untrusted. If ACLs are enabled, the [anonymous policy](https://developer.hashicorp.com/nomad/docs/secure/acl#policies) is not available via the Task API. Both [ACL Tokens](https://developer.hashicorp.com/nomad/docs/secure/acl#tokens) and [Workload Identities](https://developer.hashicorp.com/nomad/docs/concepts/workload-identity) are accepted. Once the Task API has authenticated the credentials, the normal endpoint-specific authorization is applied when ACLs are enabled. The Workload Identity should be used by tasks accessing the Task API. An ACL Token should be used when an operator is accessing the Task API via [`nomad alloc exec`](https://developer.hashicorp.com/nomad/commands/alloc/exec) or when a task is proxying Nomad HTTP requests on behalf of an authenticated user. The Task API could be used by a proxy presenting Nomad's UI with a standard TLS certificate for browsers. If [`task.user`](https://developer.hashicorp.com/nomad/docs/job-specification/task#user) is set in the jobspec, the Task API will only be usable by that user. Otherwise the Unix Domain Socket is accessible by any user. mTLS is never enabled for the Task API since traffic never leaves the node. [](https://developer.hashicorp.com/nomad/api-docs/task-api#using-the-task-api) Using the Task API ------------------------------------------------------------------------------------------------- The following jobspec will use the Task API to set [Dynamic Node Metadata](https://developer.hashicorp.com/nomad/api-docs/client#update-dynamic-node-metadata) and exit. job "taskapi-example" { type = "batch" group "taskapi-example" { task "taskapi" { driver = "docker" config { image = "curlimages/curl:7.87.0" args = [\ "--unix-socket", "${NOMAD_SECRETS_DIR}/api.sock",\ "-H", "Authorization: Bearer ${NOMAD_TOKEN}",\ "--data-binary", "{\"Meta\": {\"example\": \"Hello World!\"}}",\ "--fail-with-body",\ "--verbose",\ "localhost/v1/client/metadata",\ ] } identity { env = true } } } } If the job was able to run successfully after about 10 seconds you can observe the outcome by searching for the updated Node's metadata: $ nomad node status -filter 'Meta.example == "Hello World!"' [](https://developer.hashicorp.com/nomad/api-docs/task-api#limitations) Limitations ----------------------------------------------------------------------------------- * Using the Task API Unix Domain Socket on Windows [requires](https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/) Windows build 17063 or later. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/task-api.mdx) --- # Glossary | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Glossary ======== This glossary provides definitions and explanations for important terms and concepts used in Nomad. [](https://developer.hashicorp.com/nomad/docs/glossary#allocation) Allocation ----------------------------------------------------------------------------- An Allocation is a mapping between a task group in a job and a client node. A single job may have hundreds or thousands of task groups, meaning an equivalent number of allocations must exist to map the work to client machines. Allocations are created by the Nomad servers as part of scheduling decisions made during an evaluation. [](https://developer.hashicorp.com/nomad/docs/glossary#authoritative-and-non-authoritative-regions) Authoritative and Non-Authoritative Regions ----------------------------------------------------------------------------------------------------------------------------------------------- The authoritative region is the region in a federated multi-region cluster that holds the source of true for entities replicated across regions, such as ACL tokens, policies, and roles, namespaces, and node pools. All other regions are considered non-authoritative regions and replicate these entities by pulling them from the authoritative region. [](https://developer.hashicorp.com/nomad/docs/glossary#bin-packing) Bin Packing ------------------------------------------------------------------------------- Bin Packing is the process of filling bins with items in a way that maximizes the utilization of bins. This extends to Nomad, where the clients are "bins" and the items are task groups. Nomad optimizes resources by efficiently bin packing tasks onto client machines. [](https://developer.hashicorp.com/nomad/docs/glossary#client) Client --------------------------------------------------------------------- A Nomad client is an agent configured to run and manage tasks using available compute resources on a machine. The agent is responsible for registering with the servers, watching for any work to be assigned and executing tasks. The Nomad agent is a long lived process which interfaces with the servers. [](https://developer.hashicorp.com/nomad/docs/glossary#datacenters) Datacenters ------------------------------------------------------------------------------- Nomad models a datacenter as an abstract grouping of clients within a region. Nomad clients are not required to be in the same datacenter as the servers they are joined with, but do need to be in the same region. Datacenters provide a way to express fault tolerance among jobs as well as isolation of infrastructure. [](https://developer.hashicorp.com/nomad/docs/glossary#deployment) Deployment ----------------------------------------------------------------------------- Deployments are the mechanism by which Nomad rolls out changes to cluster state in a step-by-step fashion. Deployments are only available for Jobs with the type `service`. When an Evaluation is processed, the scheduler creates only the number of Allocations permitted by the [`update`](https://developer.hashicorp.com/nomad/docs/job-specification/update) block and the current state of the cluster. The Deployment is used to monitor the health of those Allocations and emit a new Evaluation for the next step of the update. [](https://developer.hashicorp.com/nomad/docs/glossary#driver) Driver --------------------------------------------------------------------- A Driver represents the basic means of executing your **Tasks**. Example Drivers include Docker, QEMU, Java, and static binaries. [](https://developer.hashicorp.com/nomad/docs/glossary#evaluation) Evaluation ----------------------------------------------------------------------------- Evaluations are the mechanism by which Nomad makes scheduling decisions. When either the _desired state_ (jobs) or _actual state_ (clients) changes, Nomad creates a new evaluation to determine if any actions must be taken. An evaluation may result in changes to allocations if necessary. [](https://developer.hashicorp.com/nomad/docs/glossary#job) Job --------------------------------------------------------------- A Job is a specification provided by users that declares a workload for Nomad. A Job is a form of _desired state_; the user is expressing that the job should be running, but not where it should be run. The responsibility of Nomad is to make sure the _actual state_ matches the user desired state. A Job is composed of one or more task groups. [](https://developer.hashicorp.com/nomad/docs/glossary#node) Node ----------------------------------------------------------------- A more generic term used to refer to machines running Nomad agents in client mode. Despite being different concepts, you may find `node` being used interchangeably with [`client`](https://developer.hashicorp.com/nomad/docs/glossary#client) in some materials and informal content. [](https://developer.hashicorp.com/nomad/docs/glossary#node-pool) Node Pool --------------------------------------------------------------------------- Node pools are used to group [nodes](https://developer.hashicorp.com/nomad/docs/glossary#node) and can be used to restrict which [jobs](https://developer.hashicorp.com/nomad/docs/glossary#job) are able to place [allocations](https://developer.hashicorp.com/nomad/docs/glossary#allocation) in a given set of nodes. Example use cases for node pools include segmenting nodes by environment (development, staging, production), by department (engineering, finance, support), or by functionality (databases, ingress proxy, applications). [](https://developer.hashicorp.com/nomad/docs/glossary#regions) Regions ----------------------------------------------------------------------- Nomad models infrastructure as regions and datacenters. A region will contain one or more datacenters. A set of servers joined together will represent a single region. Servers federate across regions to make Nomad globally aware. In federated clusters one of the regions must be defined as the [authoritative region](https://developer.hashicorp.com/nomad/docs/glossary#authoritative-and-non-authoritative-regions) . [](https://developer.hashicorp.com/nomad/docs/glossary#server) Server --------------------------------------------------------------------- Nomad servers are the brains of the cluster. There is a cluster of servers per region and they manage all jobs and clients, run evaluations, and create task allocations. The servers replicate data between each other and perform leader election to ensure high availability. More information about latency requirements for servers can be found in [Network Topology](https://developer.hashicorp.com/nomad/docs/deploy/production/requirements#network-topology) . [](https://developer.hashicorp.com/nomad/docs/glossary#task) Task ----------------------------------------------------------------- A Task is the smallest unit of work in Nomad. Tasks are executed by drivers, which allow Nomad to be flexible in the types of tasks it supports. Tasks specify their driver, configuration for the driver, constraints, and resources required. [](https://developer.hashicorp.com/nomad/docs/glossary#task-group) Task Group ----------------------------------------------------------------------------- A Task Group is a set of tasks that must be run together. For example, a web server may require that a log shipping co-process is always running as well. A task group is the unit of scheduling, meaning the entire group must run on the same client node and cannot be split. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/glossary.mdx) --- # Introduction to Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#what-is-nomad) What is Nomad? -------------------------------------------------------------------------------------------------------- Nomad is a flexible scheduler and workload orchestrator that enables you to deploy and manage any application across on-premise and cloud infrastructure at scale. Some of Nomad's main features include: * **Efficient resource usage** - Nomad optimizes the available cluster resources by efficiently placing the workloads onto the client nodes of the cluster through a process known as [bin packing](https://developer.hashicorp.com/nomad/docs/architecture#bin-packing) . * **Self-healing** - Nomad constantly monitors and detects if tasks stop responding and takes appropriate actions to reschedule them for high uptime. * **Zero downtime deployments** - Nomad supports several update strategies including rolling, blue/green, and canary deployments to make sure your applications are updated with zero downtime to your users. * **Different workload types** - Nomad's flexibility comes from its use of [task drivers](https://developer.hashicorp.com/nomad/docs/drivers) and allows orchestration of Docker and other containers, as well as Java Jar files, QEMU virtual machines, raw commands with the `exec` driver, and more. Additionally, users can create their own task driver plugins for customized workloads. * **Cross platform support** - Nomad runs as a single binary and allows you to orchestrate your application across macOS, Windows, and Linux clients running on-premises, in the cloud, or on the edge. * **Single unified and declarative workflow** - Regardless of the workload type, the workflow for deploying and maintaining applications on Nomad is unified within a declarative job specification that outlines important attributes like workload type and configuration, service definitions for communication between components, and location values such as region and datacenter. Read more about [Nomad's key features and how it compares to other tools](https://developer.hashicorp.com/nomad/intro) . [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#important-terms) Important terms ----------------------------------------------------------------------------------------------------------- ![Diagram illustrating the Nomad cluster terms](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_cluster_terms_diagram_light.png%26width%3D960%26height%3D540%23light-theme-only&w=1920&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Diagram illustrating the Nomad cluster terms](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_cluster_terms_diagram_dark.png%26width%3D960%26height%3D540%23dark-theme-only&w=1920&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) To create a solid foundational understanding of Nomad, review these terms before moving on to the other tutorials in this collection. ### [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#nomad-setup) Nomad setup In the process of getting Nomad deployed and functional, you will encounter the following terms. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`agent`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#agent) - An agent is a Nomad process running in server or client mode. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`client`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#client) - A Nomad client is responsible for running the tasks assigned to it. It also registers itself with the servers and watches for any work to be assigned. When running the agent, the client may be referred to as a _node_. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`server`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#server) - A Nomad server manages all jobs and clients, monitors tasks, and controls which tasks get placed on which client nodes. The servers replicate data between each other to ensure high availability. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`dev agent`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#dev-agent) - The development agent is an agent configuration that provides useful defaults for running a single node cluster of Nomad. It runs in server and client mode and does not persist its cluster state to disk, which allows the agent to start from a repeatable clean state without having to remove disk based state between runs. A Nomad cluster is typically made up of three to five _server agents_ and many _client agents_. ### [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#nomad-operation) Nomad operation In the process of Nomad scheduling and running workloads, you will encounter the following terms. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`task`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#task) - A task is the smallest unit of work in Nomad. Tasks are executed by `task drivers` like `docker` or `exec`, which allow Nomad to be flexible in the types of tasks it supports. Tasks specify their required task driver, configuration for the driver, constraints, and resources required. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`group`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#group) - A group is a series of tasks that run on the same Nomad client. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`job`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#job) - A job is the core unit of control for Nomad and defines the application and its configurations. It can contain one or many tasks. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`job specification`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#job-specification) - A job specification, also known as a _jobspec_ defines the schema for Nomad jobs. This describes the type of job, the tasks and resources necessary for the job to run, job information like which clients it can run on, and more. * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#) [`allocation`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#allocation) - An allocation is a mapping between a task group in a job and a client node. When a job is run, Nomad will choose a client capable of running it and allocate resources on the machine for the task(s) in the task group defined in the job. An application is defined in a _jobspec_ with _groups_ of _tasks_ and once submitted to Nomad, a _job_ is created along with _allocations_ for each group defined in that _jobspec_. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#application-workflow) Application workflow --------------------------------------------------------------------------------------------------------------------- A typical application workflow involves several steps and begins outside of Nomad. The prerequisite for any application running on Nomad is having a workload artifact. Nomad supports a variety of artifacts including [Docker images](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker) , [raw binaries](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/raw_exec) , [Java applications](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java) , and virtual machine images using [QEMU](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/qemu) . Nomad does not create these application artifacts but a CI tool like CircleCI, GitHub Actions, [HashiCorp Waypoint](https://developer.hashicorp.com/waypoint/plugins/nomad) , or a local build can be used to create and then push the artifacts to a repository from where Nomad can retrieve them when it schedules a job. Once the application has been created, the workflow continues with Nomad. ![Diagram illustrating a typical Nomad workflow](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_workflow_diagram.jpg%26width%3D1105%26height%3D244&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) * **Creating the job specification** - The jobspec contains the tasks required for the application including where the artifact resides, networking configurations like ports and service definitions, the number of instances desired, and more. * **Deploying the job** - The jobspec is submitted to Nomad and it [schedules](https://developer.hashicorp.com/nomad/docs/concepts/scheduling/how-scheduling-works) allocations for the job on one or more clients, depending on the job configuration. * **Updating and redeploying the job** - The application code or the jobspec are updated and then resubmitted to Nomad for scheduling. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#the-quick-start-collection) The Quick Start collection --------------------------------------------------------------------------------------------------------------------------------- This collection is your guide to becoming familiar with Nomad through hands-on tutorials. The following tutorials will show you how to install the Nomad CLI tool, create a Nomad cluster, and deploy an example application. By the end of the collection, you will have a Nomad cluster of your own and an application running on it that you can modify and redeploy. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-overview#next-steps) Next steps ------------------------------------------------------------------------------------------------- Continue on to the installation tutorial by clicking on the **Next** button below and learn how to install the Nomad CLI. **Was this tutorial helpful?** YesNo [Collection Overview\ \ Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) [Next\ \ Install Nomad](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-install) --- # Dynamic Application Sizing concepts | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#what-is-dynamic-application-sizing-in-nomad) What is Dynamic Application Sizing in Nomad? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dynamic Application Sizing (DAS) was designed with the following goals in mind: * **Reduce toil:** running a Nomad job requires knowledge of how much CPU and memory to allocate. This is often an unknown value which results in a frustrating loop of trial and error “guesstimations”. But these values are far from set-and-forget settings. As your user-base increases or the job is updated with new code, the resource usage profile of the job is likely to change as well. This requires even more work to monitor and track new limits. DAS monitors your jobs and provide you with recommendations for new limit values automatically. * **Maximize infrastructure usage:** overestimating job limits can result in resource waste as servers sit idle while Nomad is unable to schedule other jobs in them due to resource constraints. DAS detects overprovisioned jobs and recommends lower limits based on actual resource usage. * **Improve reliability:** underprovisioned jobs can suffer from problems like Out of Memory (OOM) errors and CPU throttling, causing reliability issues and requiring additional SRE attention. DAS recommendations are based on actual usage and can detect when an application is starting to require more resources. Prometheus Required Currently, Prometheus is the only APM supported for Dynamic Application Sizing. You can [test-drive DAS using Vagrant](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing) . Enterprise Only The functionality described here is available only in [Nomad Enterprise](https://www.hashicorp.com/products/nomad/pricing/) with the Multi-Cluster & Efficiency module. To explore Nomad Enterprise features, you can sign up for a free 30-day trial from [here](https://www.hashicorp.com/products/nomad/trial) . [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#how-dynamic-application-sizing-works) How Dynamic Application Sizing works ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nomad's Dynamic Application Sizing (DAS) feature is comprised of three new components: 1. Recommendations API 2. Nomad vertical autoscaling policies 3. DAS-specific plugins in nomad-autoscaler Using the new DAS plugins, the Nomad autoscaler pulls a list of vertical scaling policies from Nomad. These policies indicate which jobs should be subject to DAS and specify the strategy for making resource recommendations. The autoscaler pulls historical information about resource utilization from the configured APM and then proceeds to continuously collect resource utilization. The configured DAS strategy plugin consumes these metrics, determines a new resource value, and submits that value to the Recommendation API in Nomad. The UI displays these recommendations, along with statistics about the tasks that were computed by the autoscaler. After reviewing the recommendation in the Nomad UI, users can choose to dismiss the recommendation or apply it; applying the recommendation updates the job. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#configure-a-cluster-for-das) Configure a cluster for DAS ---------------------------------------------------------------------------------------------------------------------------------------------------------- Configuration for DAS occurs in 2 places: 1. Configuration of the autoscaler: 1. which plugins to load 2. where to find Nomad and the APM 3. number of workers 2. Job specification with scaling blocks under a task: 1. which jobs to analyze and which strategy to apply 2. policy-specific settings, such as cooldown interval and strategy configuration [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#configure-nomad-for-telemetry) Configure Nomad for telemetry -------------------------------------------------------------------------------------------------------------------------------------------------------------- Nomad needs to be configured to enable telemetry publishing. You need to enable allocation and node metrics. Since this tutorial also uses Prometheus as its APM, you need to set `prometheus_metrics` to true. Add this telemetry block to the configuration for every Nomad node in your cluster. telemetry { publish_allocation_metrics = true publish_node_metrics = true prometheus_metrics = true } [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#run-and-configure-an-apm) Run and configure an APM ---------------------------------------------------------------------------------------------------------------------------------------------------- The autoscaler uses an application performance monitor or metrics platform to retrieve historical metrics when starting to track a new target. In this beta, the APM is also used for ongoing monitoring metrics, but this is currently being shifted to using Nomad's metrics API. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#run-the-autoscaler) Run the autoscaler ---------------------------------------------------------------------------------------------------------------------------------------- The next step is to run the Nomad autoscaler. For the beta, an enterprise version of the Nomad Autoscaler is provided that includes the DAS plugins. The quickest approach is to run the autoscaler as a Nomad job; however, you can download the Nomad Autoscaler and run it as a standalone process. You need to configure the Nomad autoscaler with the information necessary to connect to your Nomad cluster and the information necessary to connect to your APM. Upon starting, the autoscaler loads the DAS-specific plugin and launches workers to evaluate vertical policies: [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=app-sizing-percentile [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=nomad-target [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=app-sizing-nomad [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=prometheus [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=app-sizing-avg [INFO] agent.plugin_manager: successfully launched and dispensed plugin: plugin_name=app-sizing-max [INFO] policy_eval.worker: starting worker: id=f6d205b3-9e48-ba9d-a230-9d3e8f2bdf81 queue=vertical_cpu [INFO] policy_eval.worker: starting worker: id=750bcea7-47af-94b3-820c-1770c757ed07 queue=vertical_mem If there are already jobs configured with vertical policies, the autoscaler begins dispatching policy evaluations from the broker to the workers; otherwise, this occurs when vertical policies are added to a job specification: [DEBUG] policy_eval.broker: dequeue eval: queue=vertical_mem Note The autoscaler does not immediately register recommendations. The `evaluate_after` field in the autoscaler configuration indicates the amount of historical metrics that must be available before a recommendation is made for a task. The purpose is to prevent recommendations with insufficient historical information; without representative data, appropriate recommendations cannot be made, which could result in under-provisioning a task. For the purpose of evaluating the feature, this can be reduced. For more production-like environments, this interval should be long enough to capture a representative sample of metrics. The default interval is 24 hours. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#add-das-to-a-job) Add DAS to a job ------------------------------------------------------------------------------------------------------------------------------------ In order to enable a Nomad job task for sizing recommendations, the following job specification contains a task scaling stanza for CPU and one for memory. These stanzas, when placed within a job specification's task stanza, configure the task for both CPU and memory recommendations. Once the job has been registered with its updated specification, the Nomad autoscaler automatically detects the new scaling policies and start the required internal processes. To enable application-sizing for multiple tasks with DAS, you need to add this scaling block to every new or additional task in the job spec. Below is a section on how you can further customize the application-sizing block to your needs (percentile, cooldown periods, sizing strategies). ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#vertical-autoscaling-policies) Vertical autoscaling policies In order to accommodate vertical application scaling, the Nomad job specification has been enhanced to allow scaling stanzas at the task level. The policy object follows the same syntax as horizontal policies with the exception that only a single check is allowed within a task scaling stanza. This validation is provided by the Nomad Autoscaler, and any validation error is displayed in the autoscaler's log output. An example of a Dynamic Application Sizing policy targeting the CPU resource looks as follows: task "app" { scaling "cpu" { enabled = true min = 50 max = 2000 policy { evaluation_interval = "30s" cooldown = "5m" check "95pct" { strategy "app-sizing-percentile" { percentile = "95" } } } } } The fields available are: * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`enabled`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#enabled) : indicates if the policy should be evaluated. It can be used to turn off recommendations for a task without having to remove the scaling stanza. * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`min`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#min) : defines the minimum value for a recommendation. In this example, recommendations for the CPU resource value for this task must be at least 50 MHz. If this field is omitted, this defaults to 10 MHz for CPU and 1 MB for memory. * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`max`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#max) : defines the maximum value for a recommendation. In this example, recommendations for the CPU resource value for this task are capped at 2000 MHz. * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`evaluation_interval`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#evaluation_interval) : specifies how often the autoscaler should consult the policy to see if a new recommendation is necessary. The autoscaler consults the policy above once every 30 seconds. * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`cooldown`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#cooldown) : indicates the Nomad autoscaler should not attempt a new recommendation within 5 minutes of a previous submission. * [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#) [`strategy`](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#strategy) : use the app-sizing-percentile strategy plugin and use a percentile value of 95. The autoscaler only provides recommendations for a given task resource if it includes a scaling stanza targeting that resource. The CPU stanza in the preceding example enables DAS for the task containing the block; enabling DAS for memory requires an explicit scaling block targeting memory, for example: scaling "mem" { # ... } [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#application-sizing-strategies) Application sizing strategies -------------------------------------------------------------------------------------------------------------------------------------------------------------- Nomad Autoscaler Enterprise delivers three new strategy plugins designed for Dynamic Application Sizing feature. The `app-sizing-max`, `app-sizing-avg`, and `app-sizing-percentile` plugins are all launched automatically when running Nomad Autoscaler Enterprise; further details of each plugin can be found below. Each of these strategies is used to compute a recommendation. Before submitting the recommendation to Nomad, the computed value is further post-processed to be: 1. increased by 15% as a safety margin 2. capped between the min and max values present in the policy 3. capped by any Nomad minimum/maximum resource values. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#app-sizing-max) App-Sizing-Max The `app-sizing-max` plugin calculates the maximum value seen for the target resource within the available dataset. This plugin is ideally suited for memory resources since workloads don’t release their memory too often and under-provisioning could cause OOM errors. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#app-sizing-percentile) App-Sizing-Percentile The `app-sizing-percentile` plugin calculates its result based on a desired percentile value from the dataset. The percentile value defaults to `99` but is configurable via the strategy config block and supports any value between 1 to 100: strategy "app-sizing-percentile" { percentile = "90" } The plugin applies an exponentially decaying weight to the data, in order to give more significance to recent values over older ones. It also adjusts its calculation based on the amount of resources used per unit of time. This load-adjusted calculation results in values that are more likely to actually meet the usage needs of the workload when compared to the traditional time-based percentile calculation. This plugin is the most versatile, since the percentile level can be fine-tuned as needed. If your workload can withstand occasional OOM errors gracefully, using a 98th percentile for memory instead of `app-sizing-max` could result in smaller recommendations and subsequently more resource availability for other tasks. A 95th to 90th percentile for CPU could have the same effect. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#app-sizing-avg) App-Sizing-Avg The `app-sizing-avg` plugin calculates the average value seen across the dataset. The plugin applies an exponentially decaying weight to the data, in order to give more significance to recent values over older ones. This plugin is only recommended for CPU values of workloads with very stable resource usage levels, such as batch jobs. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#maximize-your-das-outcomes) Maximize your DAS outcomes -------------------------------------------------------------------------------------------------------------------------------------------------------- Maximizing the benefit from DAS involves: * tuning the strategy to get good recommendations * effectively reviewing job recommendations * impact of DAS on automation ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#strategy-suggestions) Strategy suggestions * **CPU Batch:** usually will use the `app-sizing-avg` strategy as this calculates the most efficient limit. * **CPU Service/System:** most commonly will use the `app-sizing-percentile` strategy with a percentile value between 90-99. The more latency sensitive the application is, the higher the percentile value should be. * **Memory:** the strategy to use for memory largely depends on the OOM tolerance of the application. Tasks with low tolerance will commonly use the `app-sizing-percentile` strategy with a percentile value of 98 or 99. Tasks with minimal tolerance, or in situations where you are unsure, should use the `app-sizing-max` strategy. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts#review-das-recommendations) Review DAS recommendations -------------------------------------------------------------------------------------------------------------------------------------------------------- Once the autoscaler has generated recommendations, you can review them in the Nomad UI or using the Nomad API and accept or dismiss the recommendations. Because manual review is time-consuming, Nomad includes information in the recommendations API to support automatic notification of significant recommendations. The Nomad API has been enhanced to accommodate Dynamic Application Sizing with updates and new endpoints, which allow recommendations to be programmatically reviewed and applied/dismissed. While building confidence in DAS strategies and their tuning, operators may prefer to manually review the recommendations. Two sources of information are be useful in reviewing a given recommendation: the historical metrics in your APM and the statistics collected in the recommendation. The Nomad server emits metrics to the configured APM about the number of outstanding recommendations, as well as the effect of applying the recommendations. Monitoring these metrics can indicate when human review is appropriate. These are grouped on a per-namespace basis, and take the following form: [\ {\ "Labels": {\ "host": "server1.localdomain",\ "namespace": "default"\ },\ "Name": "nomad.nomad.recommendations.num_recommendations",\ "Value": 2.0\ },\ {\ "Labels": {\ "host": "server1.localdomain",\ "namespace": "default"\ },\ "Name": "nomad.nomad.recommendations.total_diff_cpu_ticks",\ "Value": -500.0\ },\ {\ "Labels": {\ "host": "server1.localdomain",\ "namespace": "default"\ },\ "Name": "nomad.nomad.recommendations.total_diff_memory_bytes",\ "Value": -1024.0\ }\ ] The recommendations API can be used to list outstanding recommendations, on a global or per-namespace basis. Dismissing the recommendation causes it to disappear. However, the autoscaler continues to monitor and eventually makes additional recommendations for the job until the vertical scaling policy is removed from the job specification. **Was this tutorial helpful?** YesNo [Previous\ \ Deploy an Enterprise cluster](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul) [Next\ \ Use dynamic app sizing](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing) This tutorial also appears in: ------------------------------ *  [](https://developer.hashicorp.com/nomad/tutorials/nomad-1-0) 3 tutorials Nomad 1.0 Explore Nomad v1.0 features like Nomad Autoscaler and the event-stream. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/ecosystem) 5 tutorials Expand Nomad with Ecosystem Add-ins Explore applications that enhance how you use your Nomad cluster through their use of the Nomad HTTP API or plug-in interface. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler) 5 tutorials Dynamically Resize with Nomad Autoscaler Automatically maintain your cluster and workload instance count to respond to demand while minimizing over-provisioning cost. * Nomad --- # nomad login command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad login command reference ============================= The `login` command is used to log in to an SSO provider and exchange the third party credentials for a newly minted Nomad ACL token. [](https://developer.hashicorp.com/nomad/commands/login#usage) Usage -------------------------------------------------------------------- nomad login [options] The login command will exchange the provided third party credentials with the requested auth method for a newly minted Nomad ACL token. [](https://developer.hashicorp.com/nomad/commands/login#options) Options ------------------------------------------------------------------------ * [](https://developer.hashicorp.com/nomad/commands/login#) [`-method`](https://developer.hashicorp.com/nomad/commands/login#method) : The name of the ACL auth method to log in via. If the cluster administrator has configured a default, this flag is optional. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-oidc-callback-addr`](https://developer.hashicorp.com/nomad/commands/login#oidc-callback-addr) : The address to use for the local OIDC callback server. This should be given in the form of `:` and defaults to `localhost:4649`. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-json`](https://developer.hashicorp.com/nomad/commands/login#json) : Output the ACL token in JSON format. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-t`](https://developer.hashicorp.com/nomad/commands/login#t) : Format and display the ACL token using a Go template. [](https://developer.hashicorp.com/nomad/commands/login#examples) Examples -------------------------------------------------------------------------- Login using an OIDC provider: $ nomad login -method=auth0 Successfully logged in via OIDC and auth0 Accessor ID = 68123fee-1e8b-7ecc-5b34-505ecd2dcb80 Secret ID = a47ed236-5a51-cadf-2ad0-4cd0fd5bc393 Name = OIDC-auth0 Type = client Global = false Create Time = 2023-01-12 14:13:04.863238 +0000 UTC Expiry Time = 2023-01-12 14:23:04.863238 +0000 UTC Create Index = 30 Modify Index = 30 Policies = [node-read] Roles ID Name ac9d4281-2079-aadb-6740-625f4ed156d8 engineering $ export NOMAD_TOKEN=a47ed236-5a51-cadf-2ad0-4cd0fd5bc393 $ nomad ... [](https://developer.hashicorp.com/nomad/commands/login#general-options) General options ---------------------------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/login#) [`-address=`](https://developer.hashicorp.com/nomad/commands/login#address) : The address of the Nomad server. Overrides the `NOMAD_ADDR` environment variable if set. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-region=`](https://developer.hashicorp.com/nomad/commands/login#region) : The region of the Nomad server to forward commands to. Overrides the `NOMAD_REGION` environment variable if set. Defaults to the Agent's local region. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-no-color`](https://developer.hashicorp.com/nomad/commands/login#no-color) : Disables colored command output. Alternatively, `NOMAD_CLI_NO_COLOR` may be set. This option takes precedence over `-force-color`. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-force-color`](https://developer.hashicorp.com/nomad/commands/login#force-color) : Forces colored command output. This can be used in cases where the usual terminal detection fails. Alternatively, `NOMAD_CLI_FORCE_COLOR` may be set. This option has no effect if `-no-color` is also used. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-ca-cert=`](https://developer.hashicorp.com/nomad/commands/login#ca-cert) : Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. Overrides the `NOMAD_CACERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-ca-path=`](https://developer.hashicorp.com/nomad/commands/login#ca-path) : Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `-ca-cert` and `-ca-path` are specified, `-ca-cert` is used. Overrides the `NOMAD_CAPATH` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-client-cert=`](https://developer.hashicorp.com/nomad/commands/login#client-cert) : Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `-client-key`. Overrides the `NOMAD_CLIENT_CERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-client-key=`](https://developer.hashicorp.com/nomad/commands/login#client-key) : Path to an unencrypted PEM encoded private key matching the client certificate from `-client-cert`. Overrides the `NOMAD_CLIENT_KEY` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-tls-server-name=`](https://developer.hashicorp.com/nomad/commands/login#tls-server-name) : The server name to use as the SNI host when connecting via TLS. Overrides the `NOMAD_TLS_SERVER_NAME` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-tls-skip-verify`](https://developer.hashicorp.com/nomad/commands/login#tls-skip-verify) : Do not verify TLS certificate. This is highly not recommended. Verification will also be skipped if `NOMAD_SKIP_VERIFY` is set. * [](https://developer.hashicorp.com/nomad/commands/login#) [`-token`](https://developer.hashicorp.com/nomad/commands/login#token) : The SecretID of an ACL token to use to authenticate API requests with. Overrides the `NOMAD_TOKEN` environment variable if set. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/login.mdx) --- # Deploy and update a Nomad job | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Quick Start](https://developer.hashicorp.com/nomad/tutorials/get-started) Now that you have deployed your Nomad cluster and configured the CLI to interact with it, the next step is deploying an application. In this tutorial, you will deploy and update an example application. In the process, you will learn about the Nomad job specification. The example application runs in Docker containers and consists of a database and a web frontend that reads from the database. You will set up the database with a [parameterized batch job](https://developer.hashicorp.com/nomad/docs/job-specification/parameterized) and then use a [periodic batch job](https://developer.hashicorp.com/nomad/docs/job-specification/periodic) to start additional short-lived jobs that write data to the database. ### [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#job-types) Job types Nomad supports several job types including service, parameterized and periodic batch jobs, and system. In this tutorial, you will be introduced to the service job and both parameterized and periodic batch jobs. [Service](https://developer.hashicorp.com/nomad/docs/concepts/scheduling/schedulers#service) jobs are for long lived services that run until explicitly stopped. [Batch](https://developer.hashicorp.com/nomad/docs/concepts/scheduling/schedulers#batch) jobs are for short lived jobs that run until they exit successfully. * The [parameterized](https://developer.hashicorp.com/nomad/docs/job-specification/parameterized) block lets you configure a batch job to accept required or optional inputs. You trigger the job with the `nomad job dispatch` command. * The [periodic](https://developer.hashicorp.com/nomad/docs/job-specification/periodic) block lets you schedule a Nomad job to run at set times. These are also known as a Nomad cron jobs. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#review-the-example-application) Review the example application ------------------------------------------------------------------------------------------------------------------------------------------- The example application simulates employees working at a technology company. They come online, complete their tasks, and then log off. Navigate to the `jobs` directory of the example repository on your local machine Note If you are following the getting started collection and are using a cloud-based cluster, navigate back to the root of the repository's directory first. $ cd jobs Each of the jobspec files below that make up the application sets the `driver` attribute to `docker` and specifies an image stored on the GitHub Container Registry in the `config` block with the `image` attribute. The Redis job is the exception as it uses an official Redis image hosted on Docker Hub. By default, Nomad looks for [images on Docker Hub](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker#image) so the full path including `https://` is not necessary for the Redis job. Snippet showing Docker configuration task "redis-task" { driver = "docker" config { image = "redis:7.0.7-alpine" } } The parameterized and periodic configurations of the `pytechco-setup` and `pytechco-employee` jobs below are more advanced use cases of the jobspec system and are here to provide an example of what you can do with batch jobs. Many users only use service type jobs because the applications they deploy are long-living. Not all applications require batch jobs and it is okay if yours does not use them. ![Diagram illustrating the relationship between the Pytechco application components](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fpytechco-diagram-light.png%26width%3D960%26height%3D540%23light-theme-only&w=1920&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) ![Diagram illustrating the relationship between the Pytechco application components](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fpytechco-diagram-dark.png%26width%3D960%26height%3D540%23dark-theme-only&w=1920&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#) [`pytechco-redis.nomad.hcl`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#pytechco-redis-nomad-hcl) - This service job runs and exposes a Redis database as a Nomad service for the other application components to connect to. The jobspec sets the type to `service`, configures a Nomad `service` block to use Nomad's [native service discovery](https://www.hashicorp.com/blog/nomad-1-3-adds-native-service-discovery-and-edge-workload-support) , and creates a service named `redis-svc`. Click here to view the contents of pytecho-redis.nomad.hcl job "pytechco-redis" { type = "service" group "ptc-redis" { count = 1 network { port "redis" { to = 6379 } } service { name = "redis-svc" port = "redis" provider = "nomad" } task "redis-task" { driver = "docker" config { image = "redis:7.0.7-alpine" ports = ["redis"] } } } } * [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#) [`pytechco-web.nomad.hcl`](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#pytechco-web-nomad-hcl) - This service jobs runs the web application frontend that displays the values stored in the database and the active employees. The jobspec sets the type to `service` and uses a static port of `5000` for the application. It also uses the `nomadService` built-in function to retrieve address and port information of the Redis database service. Click here to view the contents of pytecho-web.nomad.hcl job "pytechco-web" { type = "service" group "ptc-web" { count = 1 network { port "web" { static = 5000 } } service { name = "ptc-web-svc" port = "web" provider = "nomad" } task "ptc-web-task" { # Retrieves the .Address and .Port connection values for # redis-svc with nomadService and saves them to env vars # REFRESH_INTERVAL is how often the UI refreshes in milliseconds template { data = < 2023-03-10T12:16:09-05:00: Monitoring evaluation "d44af37b" 2023-03-10T12:16:09-05:00: Evaluation triggered by job "pytechco-redis" 2023-03-10T12:16:10-05:00: Evaluation within deployment: "0ea05651" 2023-03-10T12:16:10-05:00: Allocation "be0bda79" created: node "2c9f4b7e", group "ptc-redis" 2023-03-10T12:16:10-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:16:10-05:00: Evaluation "d44af37b" finished with status "complete" ==> 2023-03-10T12:16:10-05:00: Monitoring deployment "0ea05651" ✓ Deployment "0ea05651" successful 2023-03-10T12:16:24-05:00 ID = 0ea05651 Job ID = pytechco-redis Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline ptc-redis 1 1 1 0 2023-03-10T17:26:23Z Submit the webapp frontend job. $ nomad job run pytechco-web.nomad.hcl ==> 2023-03-10T12:16:42-05:00: Monitoring evaluation "62615d0e" 2023-03-10T12:16:42-05:00: Evaluation triggered by job "pytechco-web" 2023-03-10T12:16:43-05:00: Evaluation within deployment: "cdb7d282" 2023-03-10T12:16:43-05:00: Allocation "b5dc39d5" created: node "2c9f4b7e", group "ptc-web" 2023-03-10T12:16:43-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:16:43-05:00: Evaluation "62615d0e" finished with status "complete" ==> 2023-03-10T12:16:43-05:00: Monitoring deployment "cdb7d282" ✓ Deployment "cdb7d282" successful 2023-03-10T12:16:59-05:00 ID = cdb7d282 Job ID = pytechco-web Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline ptc-web 1 1 1 0 2023-03-10T17:26:57Z Get the IP address of the webapp. The following command gets the allocation ID of the web job and uses that ID to get the status of the allocation. It then searches for the IP address in the allocation status output and formats the IP address into a link with the webapp's port. Open the URL from the output in your browser to see the webapp frontend. LocalCloud $ nomad node status -verbose \ $(nomad job allocs pytechco-web | grep -i running | awk '{print $2}') | \ grep -i ip-address | awk -F "=" '{print $2}' | xargs | \ awk '{print "http://"$1":5000"}' AWSGCPAzure $ nomad node status -verbose \ $(nomad job allocs pytechco-web | grep -i running | awk '{print $2}') | \ grep -i public-ipv4 | awk -F "=" '{print $2}' | xargs | \ awk '{print "http://"$1":5000"}' $ nomad node status -verbose \ $(nomad job allocs pytechco-web | grep -i running | awk '{print $2}') | \ grep -i external-ip | awk -F "=" '{print $2}' | xargs | \ awk '{print "http://"$1":5000"}' $ nomad node status -verbose \ $(nomad job allocs pytechco-web | grep -i running | awk '{print $2}') | \ grep -i public-ipv4 | awk -F "=" '{print $2}' | xargs | \ awk '{print "http://"$1":5000"}' Submit the setup job. $ nomad job run pytechco-setup.nomad.hcl Job registration successful Dispatch the setup job by providing a value for budget. $ nomad job dispatch -meta budget="200" pytechco-setup Dispatched Job ID = pytechco-setup/dispatch-1678468734-396cfa83 Evaluation ID = 53a77034 ==> 2023-03-10T12:18:54-05:00: Monitoring evaluation "53a77034" 2023-03-10T12:18:54-05:00: Evaluation triggered by job "pytechco-setup/dispatch-1678468734-396cfa83" 2023-03-10T12:18:54-05:00: Allocation "d6c60ffd" created: node "2c9f4b7e", group "ptc-setup" 2023-03-10T12:18:54-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:18:54-05:00: Evaluation "53a77034" finished with status "complete" Submit the employee job. $ nomad job run pytechco-employee.nomad.hcl Job registration successful Approximate next launch time: 2023-03-10T17:19:18Z (3s from now) Navigate to the Nomad UI, click on the **Jobs** page, and then click on the `pytechco-employee` job. Since this is a cron batch job, you can see that it creates a new job every three seconds. ![Employee job generating new jobs](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fnomad_ui_job_allocs.gif%26width%3D1024%26height%3D768&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Navigate back to the webapp URL. Note how new employees come online under the **Online Employees** section as the `pytechco-employee` batch job creates new jobs. ![New employees coming online](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fapp_employees_coming_online.gif%26width%3D1024%26height%3D768&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#update-the-application) Update the application --------------------------------------------------------------------------------------------------------------------------- By default, the example application randomizes the type of employee it brings online. However, you can configure it to only bring on employees of a specific type. Update the employee jobspec to bring only sales engineers online and increase the budget by dispatching the setup job again. Stop the employee job. The [`-purge` flag](https://developer.hashicorp.com/nomad/commands/job/stop#purge) removes the job history and any children jobs from the Nomad database, and also removes them from the Nomad UI. If they still show up in the UI, you can force Nomad to do a garbage collection to remove them from the UI with [`nomad system gc`](https://developer.hashicorp.com/nomad/commands/system/gc) . $ nomad job stop -purge pytechco-employee Reset the database by dispatching the setup job again with a new budget of `500`. $ nomad job dispatch -meta budget="500" pytechco-setup Dispatched Job ID = pytechco-setup/dispatch-1678469008-6ffe1c0c Evaluation ID = e8e420f0 ==> 2023-03-10T12:23:28-05:00: Monitoring evaluation "e8e420f0" 2023-03-10T12:23:28-05:00: Evaluation triggered by job "pytechco-setup/dispatch-1678469008-6ffe1c0c" 2023-03-10T12:23:28-05:00: Allocation "74acf63e" created: node "2c9f4b7e", group "ptc-setup" 2023-03-10T12:23:28-05:00: Evaluation status changed: "pending" -> "complete" ==> 2023-03-10T12:23:28-05:00: Evaluation "e8e420f0" finished with status "complete" Open the `pytechco-employee.nomad.hcl` file and uncomment the `args` block of the task config to have the job create only employees of the type `sales_engineer`. Then update the cron to `0/1 * * * * * *` to have the job run every second. Save the file. pytechco-employee.nomad.hcl job "pytechco-employee" { datacenters = ["dc1"] type = "batch" periodic { cron = "0/1 * * * * * *" prohibit_overlap = false } group "ptc-employee" { count = 1 task "ptc-employee-task" { # ... config { image = "ghcr.io/hashicorp-education/learn-nomad-getting-started/ptc-employee:1.0" args = [\ "--employee-type", "sales_engineer"\ ] } } } } Submit the employee job again and navigate to the webapp. Note how all of the online employees are now only Sales Engineers and more of them are online at any given time. $ nomad job run pytechco-employee.nomad.hcl Job registration successful Approximate next launch time: 2023-03-10T17:24:18Z (1s from now) ![Only sales engineers coming online](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fgs%252Fapp_only_sales_engineers.gif%26width%3D1024%26height%3D768&w=2048&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The simulation will continue running until the amount of money reaches 0. [](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-deploy-job#next-steps) Next steps --------------------------------------------------------------------------------------------------- In this tutorial, you submitted jobs to Nomad, updated a jobspec, and resubmitted it to Nomad. Continue on to the next tutorial by clicking on the **Next** button below and learn how to clean up the jobs and your cluster. **Was this tutorial helpful?** YesNo [Previous\ \ Create a cluster](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster) [Next\ \ Stop the cluster](https://developer.hashicorp.com/nomad/tutorials/get-started/gs-stop-nomad) --- # Create a host volume plugin | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Create a host volume plugin =========================== This page describes the plugin specification for [dynamic host volumes](https://developer.hashicorp.com/nomad/docs/architecture/storage/host-volumes) , with examples, so you can write your own plugin to dynamically configure persistent storage on your Nomad client nodes. If you do not need to write your own plugin, consider Nomad's built-in [`mkdir` plugin](https://developer.hashicorp.com/nomad/docs/other-specifications/volume/host#mkdir-plugin) , which creates a directory on the host. If you have more complex needs, review the following examples to get a sense of the specification in action. A plugin can be as basic as a short shell script. The full [specification](https://developer.hashicorp.com/nomad/plugins/author/host-volume#specification) follows the examples, and is followed by a list of [general considerations](https://developer.hashicorp.com/nomad/plugins/author/host-volume#considerations) . [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#examples) Examples -------------------------------------------------------------------------------------- The specification is lean enough to be readily fulfilled in any language. The following examples are written in `bash`. The `custom-mkdir` plugin creates a directory, and `mkfs-ext4` creates a Linux filesystem. You may imagine others, such as an NFS mount or another storage infrastructure API. Each example includes a minimal [volume specification](https://developer.hashicorp.com/nomad/docs/other-specifications/volume/host) , which assumes that you have placed the plugin in the [host volume plugin directory](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume_plugin_dir) and made it executable. custom-mkdirmkfs-ext4 `custom-mkdir` only creates a directory. There is a plugin built into Nomad that does this called "mkdir", but this serves as a minimal example. Volume specification: type = "host" name = "mkdir-vol" plugin_id = "custom-mkdir" # plugin filename Plugin code: #!/usr/bin/env bash # set -u to error by name if an env var is not set set -eu # echo to stderr, because Nomad expects stdout to match the spec. # this (and stdout) show up in Nomad agent debug logs. stderr() { 1>&2 echo "$@" } get_path() { # since delete runs `rm -rf` (frightening), check to make sure # the path is something sensible. if [ -z "$DHV_VOLUMES_DIR" ]; then stderr "DHV_VOLUMES_DIR must not be empty" exit 1 fi if [ -z "$DHV_VOLUME_ID" ]; then stderr "DHV_VOLUME_ID must not be empty" exit 1 fi # create and delete assign this echo output to a variable echo "$DHV_VOLUMES_DIR/$DHV_VOLUME_ID" } case "$DHV_OPERATION" in "fingerprint") echo '{"version": "0.0.1"}' ;; "create") path="$(get_path)" stderr "creating directory: $path" # `mkdir -p` may be run repeatedly with the same result; # it is idempotent. mkdir -p "$path" # 0 bytes because plain directories are not any particular size. printf '{"path": "%s", "bytes": 0}' "$path" ;; "delete") path="$(get_path)" stderr "deleting directory: $path" rm -rf "$path" # `rm -f` is also idempotent. ;; *) echo "unknown operation: '$DHV_OPERATION'" exit 1 ;; esac `mkfs-ext4` creates a Linux ext4 filesystem with the `mkfs.ext4` command and mounts it as a loopback device. Unlike `mkdir`, `mkfs` can restrict the size of the volume. Volume specification: type = "host" name = "mkfs-vol" plugin_id = "mkfs-ext4" # plugin filename capacity_min = "50MB" # capacity values are required by this plugin capacity_max = "50MB" Plugin code: #!/usr/bin/env bash set -euo pipefail version='0.0.1' stderr() { 1>&2 echo "$@" } get_path() { if [ -z "$DHV_VOLUMES_DIR" ]; then stderr "DHV_VOLUMES_DIR must not be empty" exit 1 fi if [ -z "$DHV_VOLUME_ID" ]; then stderr "DHV_VOLUME_ID must not be empty" exit 1 fi echo "$DHV_VOLUMES_DIR/$DHV_VOLUME_ID" } is_mounted() { mount | grep -q " $1 " } create_volume() { local path="$1" local bytes="$2" # translate to mb for dd block size local megs=$((bytes / 1024 / 1024)) # lazy, approximate if [ $megs -le 0 ]; then stderr "minimum capacity must be greater than zero." exit 2 fi mkdir -p "$path" # the if statements ensure idempotency if [ ! -f "$path.ext4" ]; then # dd only writes to stderr, so safe to run without redirection dd if=/dev/zero of="$path.ext4" bs=1M count="$megs" # mkfs includes stdout, so we need to redirect that to stderr mkfs.ext4 "$path.ext4" 1>&2 fi if ! is_mounted "$path"; then mount "$path.ext4" "$path" fi } delete_volume() { local path="$1" is_mounted "$path" && umount "$path" rm -rf "$path" rm -f "$path.ext4" } case "$1" in "fingerprint") printf '{"version": "%s"}' "$version" ;; "create") path="$(get_path)" stderr "creating volume at $path" create_volume "$path" "$DHV_CAPACITY_MIN_BYTES" # output what Nomad expects bytes="$(stat --format='%s' "$path.ext4")" printf '{"path": "%s", "bytes": %s}' "$path" "$bytes" ;; "delete") path="$(get_path)" stderr "deleting volume at $path" delete_volume "$path" ;; *) echo "unknown operation: $1" exit 1 ;; esac [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#specification) Specification ------------------------------------------------------------------------------------------------ A host volume plugin is registered with Nomad if it: * Is an executable file, such as a script or binary * Is located in the [`client.host_volume_plugin_dir`](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume_plugin_dir) directory on Nomad client nodes * Responds appropriately to a `fingerprint` call ### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#operations) Operations To fully manage the lifecycle of a volume, plugins must fulfill all of the following operations: * [fingerprint](https://developer.hashicorp.com/nomad/plugins/author/host-volume#fingerprint) * [create](https://developer.hashicorp.com/nomad/plugins/author/host-volume#create) * [delete](https://developer.hashicorp.com/nomad/plugins/author/host-volume#delete) Nomad passes the operation as the first positional argument to the plugin. That and other information are passed as environment variables. Environment variables are prefixed with `"DHV_"`, which stands for "Dynamic Host Volume." Some variables may be required for certain plugins, such as `DHV_CAPACITY_*` for plugins that can restrict size. Most variables are for plugin author convenience. #### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#fingerprint) fingerprint > Nomad calls `fingerprint` to discover valid plugins when the client agent starts or is reloaded with a SIGHUP. The version it returns is used to register the plugin on the Nomad node for volume scheduling. > > **CLI arguments:** `$1=fingerprint` > > **Environment variables:** > > DHV_OPERATION=fingerprint > > > **Expected stdout:** > > {"version": "0.0.1"} > > > **Requirements:** > > * Must complete within 5 seconds, or Nomad will kill it. It should be much faster, as no actual work should be done. > * "version" value must be valid per the [hashicorp/go-version](https://pkg.go.dev/github.com/hashicorp/go-version#pkg-constants) > golang package. #### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#create) create > Nomad calls `create` when you run [`nomad volume create`](https://developer.hashicorp.com/nomad/commands/volume/create) > CLI or use the [create API](https://developer.hashicorp.com/nomad/api-docs/volumes#create-dynamic-host-volume) > ). > > You can run `create` again for the same volume if you include the volume `id` in the volume specification. > > When the agent starts, Nomad also calls `create` for each volume that was previously created on the node so that plugins can ensure the volumes are available after an agent restart or host reboot. > > **CLI Arguments:** `$1=create` > > **Environment variables:** > > DHV_OPERATION=create > DHV_VOLUMES_DIR={directory to put the volume in} > DHV_PLUGIN_DIR={path to directory containing plugins} > DHV_NAMESPACE={volume namespace} > DHV_VOLUME_NAME={name from the volume specification} > DHV_VOLUME_ID={volume ID generated by Nomad} > DHV_NODE_ID={Nomad node ID} > DHV_NODE_POOL={Nomad node pool} > DHV_CAPACITY_MIN_BYTES={capacity_min from the volume spec, expressed in bytes} > DHV_CAPACITY_MAX_BYTES={capacity_max from the volume spec, expressed in bytes} > DHV_PARAMETERS={stringified json of parameters from the volume spec} > > > **Expected stdout:** > > {"path": "/path/to/created/volume", "bytes": 50000000} > > > **Expected stdout on error:** > > {"error": "error message"} > > > Returning an error message is optional. Nomad returns the error message in any error returned to the user. > > **Requirements:** > > * Must complete within 60 seconds, or Nomad will kill it. > * Must create a path on disk, within `DHV_VOLUMES_DIR` or otherwise, and return it as `"path"`. Nomad will mount this path into workloads that request the volume, and it will also be sent to `delete` later as `DHV_CREATED_PATH`. > * Must be idempotent - running create with the same inputs should produce the same result. > * Must be safe to run concurrently, per volume name per node. > * If the plugin fails partway through create, it must clean up after itself and exit non-0. Nomad will not attempt to delete partial creates. > * However, if during an _initial_ create, Nomad fails to save the volume in its own state, it will issue `delete` automatically to avoid leaving any stray volumes on disk. #### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#delete) delete > Nomad calls `delete` when you run [`nomad volume delete`](https://developer.hashicorp.com/nomad/commands/volume/delete) > CLI or use the [delete API](https://developer.hashicorp.com/nomad/api-docs/volumes#delete-dynamic-host-volume) > . > > **CLI Arguments:** `$1=delete` > > **Environment variables:** > > DHV_OPERATION=delete > DHV_CREATED_PATH={path that `create` returned} > DHV_VOLUMES_DIR={directory that volumes should be put in} > DHV_PLUGIN_DIR={path to directory containing plugins} > DHV_NAMESPACE={volume namespace} > DHV_VOLUME_NAME={name from the volume specification} > DHV_VOLUME_ID={volume ID generated by Nomad} > DHV_NODE_ID={Nomad node ID} > DHV_NODE_POOL={Nomad node pool} > DHV_PARAMETERS={stringified json of parameters from the volume spec} > > > **Expected stdout:** none (stdout is discarded) > > **Expected stdout on error:** > > {"error": "error message"} > > > Returning an error message is optional. Nomad returns the error message in any error returned to the user. > > **Requirements:** > > * Must complete within 60 seconds, or Nomad will kill it. > * Must remove the `DHV_CREATED_PATH` that was returned by `create`, or derive the path from other variables the same way that `create` did. > * Must be idempotent - calling delete on an already deleted volume must not return an error. > * Must be safe to run concurrently, per volume name per node. > * Will be run after a failed `create` operation during initial volume creation. [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#considerations) Considerations -------------------------------------------------------------------------------------------------- Plugin authors should consider these details when writing plugins. ### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#execution) Execution * The plugin is executed as the same user as the `nomad` agent (likely root). * Plugin `stdout` and `stderr` are exposed as client agent debug logs, so plugins should not output sensitive information. * Nomad does not retry automatically on error. The caller of create/delete must retry manually. The plugin may do so internally with its own retry logic, provided it still completes within the deadline. * Errors from `create` while restoring a volume during Nomad agent start do not halt the client. The error will be in client logs, and the volume is not registered as available on the node. * Be aware that the `DHV_VOLUME_NAME` and `DHV_PARAMETERS` fields are controlled by the volume author. If the expected volume authors are not the Nomad administrators, you should ensure your plugin handles these fields safely or ignores them. ### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#uniqueness) Uniqueness * Volume `name` is unique per _node_, and volume `ID` is unique per _region_. * Only one create/delete operation at a time is executed per volume `name` per node, and similarly by `id` on Nomad servers, but many create/delete operations for different volume IDs may run concurrently, even on the same node. * We suggest placing volumes in `DHV_VOLUMES_DIR` for consistency, but it is not required. The built-in `mkdir` plugin uses `$DHV_VOLUMES_DIR/$DHV_VOLUME_ID` to ensure uniqueness across the cluster. We recommend against using `DHV_VOLUME_NAME` in the path unless the plugin guards against path traversal. * Plugins that write into network storage need to take care not to delete remote/shared state by `name`, unless they know that there are no other volumes with workloads using that name. ### [](https://developer.hashicorp.com/nomad/plugins/author/host-volume#configuration) Configuration * Per-_volume_ configuration should be set in the volume specification file's `parameters`. Per-_node_ configuration should be put in config file(s) as described next. * There is no mechanism built into Nomad for plugin configuration. As a convention, we suggest placing any necessary configuration file(s) next to the executable plugin in the plugin directory. You may use `DHV_PLUGIN_DIR` to refer to the directory. * If a plugin needs to retain state across operations (e.g. delete needs some value that was generated during create), then you may store that on the host filesystem, or some external data store of your choosing, perhaps even Nomad variables. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/author/host-volume.mdx) --- # nomad version command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad version command reference =============================== The `version` command displays build information about the running binary, including the release version, build date, and the exact revision. [](https://developer.hashicorp.com/nomad/commands/version#usage) Usage ---------------------------------------------------------------------- nomad version [](https://developer.hashicorp.com/nomad/commands/version#output) Output ------------------------------------------------------------------------ This command prints the version number and info about the git commit that was used to build the binary. `BuildDate` is when the commit was made, and `Revision` is the exact commit SHA. The SHA may also have the string `+CHANGES` appended to the end, indicating that local, uncommitted changes were detected at build time. [](https://developer.hashicorp.com/nomad/commands/version#examples) Examples ---------------------------------------------------------------------------- $ nomad version Nomad v1.5.0 BuildDate 2023-02-17T19:29:26Z Revision a536284ebcfb4ff26065955abae446d81cc92b87+CHANGES [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/version.mdx) --- # nomad ui command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad ui command reference ========================== The `ui` command is used to open the Nomad Web UI. [](https://developer.hashicorp.com/nomad/commands/ui#usage) Usage ----------------------------------------------------------------- nomad ui [options] The `ui` command can be called with no arguments, in which case the UI homepage will be opened in the default browser. An identifier may be provided, in which case the UI will be opened to view the details for that object. Supported identifiers are jobs, allocations and nodes. If ACLs are enabled, the web UI will start in an unauthenticated state and you may see a 403 Unauthorized page if anonymous read access is denied. The `nomad ui -authenticate` option will exchange your command line client's Nomad ACL token for a one-time token, which is passed to the web UI. That one-time token will be exchanged for your Nomad ACL token and stored in the browser's local storage for authentication. [](https://developer.hashicorp.com/nomad/commands/ui#options) Options --------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-authenticate`](https://developer.hashicorp.com/nomad/commands/ui#authenticate) : Exchange your Nomad ACL token for a one-time token in the web UI. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-show-url`](https://developer.hashicorp.com/nomad/commands/ui#show-url) : Show the Nomad UI URL instead of opening with the default browser. [](https://developer.hashicorp.com/nomad/commands/ui#examples) Examples ----------------------------------------------------------------------- Open the UI homepage: $ nomad ui Opening URL "http://127.0.0.1:4646" Open the UI directly to look at a job: $ nomad ui redis-job http://127.0.0.1:4646/ui/jobs/redis-job Open the UI directly to look at an allocation: $ nomad ui d4005969 Opening URL "http://127.0.0.1:4646/ui/allocations/d4005969-b16f-10eb-4fe1-a5374986083d" Open the UI and authenticate using your ACL token: $ NOMAD_ACL_TOKEN=e9674b26-763b-4637-a28f-0df95c53cdda nomad ui -authenticate Opening URL "http://127.0.0.1:4646" with token Show the UI URL without opening the browser: $ nomad ui -show-url URL for web UI: http://127.0.0.1:4646 [](https://developer.hashicorp.com/nomad/commands/ui#general-options) General options ------------------------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-address=`](https://developer.hashicorp.com/nomad/commands/ui#address) : The address of the Nomad server. Overrides the `NOMAD_ADDR` environment variable if set. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-region=`](https://developer.hashicorp.com/nomad/commands/ui#region) : The region of the Nomad server to forward commands to. Overrides the `NOMAD_REGION` environment variable if set. Defaults to the Agent's local region. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-namespace=`](https://developer.hashicorp.com/nomad/commands/ui#namespace) : The target namespace for queries and actions bound to a namespace. Overrides the `NOMAD_NAMESPACE` environment variable if set. If set to `'*'`, subcommands which support this functionality query all namespaces authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-no-color`](https://developer.hashicorp.com/nomad/commands/ui#no-color) : Disables colored command output. Alternatively, `NOMAD_CLI_NO_COLOR` may be set. This option takes precedence over `-force-color`. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-force-color`](https://developer.hashicorp.com/nomad/commands/ui#force-color) : Forces colored command output. This can be used in cases where the usual terminal detection fails. Alternatively, `NOMAD_CLI_FORCE_COLOR` may be set. This option has no effect if `-no-color` is also used. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-ca-cert=`](https://developer.hashicorp.com/nomad/commands/ui#ca-cert) : Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. Overrides the `NOMAD_CACERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-ca-path=`](https://developer.hashicorp.com/nomad/commands/ui#ca-path) : Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `-ca-cert` and `-ca-path` are specified, `-ca-cert` is used. Overrides the `NOMAD_CAPATH` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-client-cert=`](https://developer.hashicorp.com/nomad/commands/ui#client-cert) : Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `-client-key`. Overrides the `NOMAD_CLIENT_CERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-client-key=`](https://developer.hashicorp.com/nomad/commands/ui#client-key) : Path to an unencrypted PEM encoded private key matching the client certificate from `-client-cert`. Overrides the `NOMAD_CLIENT_KEY` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-tls-server-name=`](https://developer.hashicorp.com/nomad/commands/ui#tls-server-name) : The server name to use as the SNI host when connecting via TLS. Overrides the `NOMAD_TLS_SERVER_NAME` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-tls-skip-verify`](https://developer.hashicorp.com/nomad/commands/ui#tls-skip-verify) : Do not verify TLS certificate. This is highly not recommended. Verification will also be skipped if `NOMAD_SKIP_VERIFY` is set. * [](https://developer.hashicorp.com/nomad/commands/ui#) [`-token`](https://developer.hashicorp.com/nomad/commands/ui#token) : The SecretID of an ACL token to use to authenticate API requests with. Overrides the `NOMAD_TOKEN` environment variable if set. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/ui.mdx) --- # Migrate a Linux-based Java application to Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) Unlike many other application schedulers, Nomad can run non-containerized applications. If you have existing Java applications, you can run them directly on Nomad without converting them into containers. The Nomad Java task driver enables you to run Java applications in your Nomad cluster without the need to containerize them. The Nomad Java task driver also allows you to use Nomad's standardized, declarative job configuration with the familiar configuration elements of typical JVM applications. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#objective) Objective -------------------------------------------------------------------------------------------------------------- For this tutorial, you will use a sample Java application—Membrane Service Proxy—and: * Identify required values from your startup script and configuration files * Translate them to an equivalent Nomad job specification * Use fundamental job specification stanzas: `group`,`task`,`driver`,`config`, and `env` * Download files for a job with `artifact` stanzas * Include configuration elements as `template` stanzas ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#about-the-sample-application) About the sample application The Membrane Service Proxy application is an Open Source API Gateway & HTTP reverse proxy for REST & SOAP. The application configuration for this tutorial is based on the Membrane examples and: 1. Consumes a REST-style query 2. Converts it to a SOAP query 3. Proxies it to a remote SOAP server 4. Receives the response 5. Formats the response based on the `Accept` header 6. Returns the response to the caller Note This tutorial's sample application queries a remote SOAP service that is not maintained by HashiCorp. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#prerequisites) Prerequisites ---------------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#local-dependencies) Local dependencies * The `nomad` binary * Java * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) [`curl`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#curl) ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#target-dependencies) Target dependencies You can deploy the tutorial's job file to a local Nomad dev agent or a remote Nomad cluster. Local Nomad dev agentRemote Nomad cluster Start a shell session and run the following command. $ sudo nomad agent -dev -bind 0.0.0.0 When deploying to a remote Nomad cluster, ensure that: * The cluster has clients with Java installed. * [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/docs/commands#nomad_addr) and [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/docs/commands#nomad_token) are set as appropriate for [remote Nomad access](https://developer.hashicorp.com/nomad/docs/commands#remote-usage) . * Port 2000 and port 9000 are accessible on the client. This tutorial does not cover using dynamic ports. * You use the correct `class_path` separator for the Nomad clients. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) [`:`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) for Linux or macOS clients. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) [`;`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#-1) for Windows clients. * If your cluster has a mixture of Windows and Linux clients, you will need to add a [`constraint`](https://developer.hashicorp.com/nomad/docs/job-specification/constraint) to your job to target the operating system that matches your `class_path` separator. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#create-a-learning-environment) Create a learning environment ------------------------------------------------------------------------------------------------------------------------------------------------------ This tutorial uses the command line. Open a new shell session to get started. There is an [accompanying GitHub repository](https://github.com/hashicorp-education/learn-nomad-migrate-java) that contains the completed Nomad job file for Linux and Windows. It is not required to complete the tutorial but provided for a reference. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#make-a-working-directory) Make a working directory $ mkdir ~/java-guide $ cd ~/java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#run-the-workload-outside-of-nomad) Run the workload outside of Nomad -------------------------------------------------------------------------------------------------------------------------------------------------------------- To validate the sample Java workload runs as expected in your environment, try it locally or on a host outside of the Nomad cluster. This tutorial includes the test steps for you. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#download-and-decompress-the-application) Download and decompress the application Download Membrane-SOA to your local machine. $ curl --location \ --output membrane-service-proxy-4.7.3.zip \ https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip Once downloaded, unzip the `membrane-service-proxy-4.7.3.zip` file. $ unzip membrane-service-proxy-4.7.3.zip ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#change-into-the-workload-directory) Change into the workload directory $ cd membrane-service-proxy-4.7.3/examples/rest2soap-json You can consult the README in this folder for more information about the sample; however, you will find the vital information here. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#start-the-proxy) Start the proxy $ ./service-proxy.sh Logging will be returned in the terminal. A successful startup is indicated by a line containing `Membrane Service Proxy 4.7.3 up and running!` ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#observe-the-baseline-behavior) Observe the baseline behavior Open a new shell. Verify that the running proxy is behaving as expected by fetching data from `http://localhost:2000/bank/37050198`. Use the `curl` command to fetch the API endpoint. $ curl http://localhost:2000/bank/37050198 The proxy returns an XML response to your request.
Sparkasse KölnBonnCOLSDE33XXXKöln50667
 Now, add the `Accept: application/json` header to the request. $ curl --header "Accept: application/json" http://localhost:2000/bank/37050198 The proxy returns the response in JSON format. {"getBankResponse": {"details": {"bezeichnung": "Sparkasse KölnBonn","bic": "COLSDE33XXX","ort": "Köln","plz": 50667}}} Now that you have verified that the proxy is running as expected, close this shell session. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#stop-the-proxy) Stop the proxy Return to the terminal session running the proxy. Press `Ctrl-C` to stop it. The proxy process will stop and return you to a shell prompt. Now that you have validated the Java application runs as expected, you can migrate it to Nomad. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#read-the-example-startup-code) Read the example startup code ------------------------------------------------------------------------------------------------------------------------------------------------------ Open the `service-proxy.sh` file in a text editor to review its contents. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#key-observations) Key observations For now, observe that the `java` command does the following. This information is the foundation of your Nomad job specification. * Has a `-classpath` that depends on the value of `$CLASSPATH` * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) [`$CLASSPATH`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#classpath) is based on `$MEMBRANE_HOME`, which is the top-level directory created when decompressing the archive. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#) [`$CLASSPATH`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#classpath-1) expands to: $MEMBRANE_HOME/conf:$MEMBRANE_HOME/starter.jar * Runs a class named `com.predic8.membrane.core.Starter` * Passes the arguments `-c` and `proxies.xml` ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#change-into-your-work-directory) Change into your work directory $ cd ~/java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#build-your-nomad-job) Build your Nomad job ------------------------------------------------------------------------------------------------------------------------------------ Now that you have reviewed the startup script, you can use Nomad's declarative job specification to define the workload. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#required-job-specification) Required job specification Create a file named `rest2json.nomad.hcl` with the following template Nomad job specification. This job file is not runnable; however, every element here is required to create a parsable job. job "«job_name»" { datacenters = ["«datacenter»"] group "«group_name»" { task "«task_name»" { driver = "«driver_plugin_id»" } } } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#start-customizing-the-template-job) Start customizing the template job * Replace `«job_name»` with `rest2json` * Replace `«group_name»` with `proxy` * Replace `«task_name»` with `membrane` * Replace `«driver_plugin_id»` with `java` #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#fetch-valid-datacenter-values) Fetch valid `datacenter` values The `datacenters` value is a list of datacenter names as strings. Run the `nomad node status` command, which lists all of the clients in the target Nomad cluster or dev agent. $ nomad node status ID DC Name Class Drain Eligibility Status d715f8b4 dc1 nomad-client-1.node.consul false eligible ready 14ab9290 dc1 nomad-client-2.node.consul false eligible ready 0f357b26 dc1 nomad-client-3.node.consul false eligible ready In this output, all of the clients are in a datacenter named `dc1`. Replace `«datacenter»` with a valid value for your deployment target. The remaining tutorial content uses `dc1`; which is the default value for datacenter. Always use the correct value for your deployment target as the value for this field. Your job file should look like this now. job "rest2json" { datacenters = ["dc1"] group "proxy" { task "membrane" { driver = "java" } } } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#migrate-the-configuration-into-the-job) Migrate the configuration into the job The Nomad job specification provides job specific configuration to the task driver using the `config` stanza. This stanza tells Nomad how to start the Java application and contains attributes like `class`, `class_path`, and `args` Create a `config` stanza inside of the `task "membrane"` stanza with the information you discovered from reading the startup script. * Set the `class` argument to `com.predic8.membrane.core.Starter`. * Set the `class_path` argument to `${MEMBRANE_HOME}/conf:${MEMBRANE_HOME}/starter.jar` * Finally, set the `args` attribute to a list of quoted arguments: `["-c", "proxies.xml"]` This yields the following config stanza. config { class = "com.predic8.membrane.core.Starter" class_path = "${MEMBRANE_HOME}/conf:${MEMBRANE_HOME}/starter.jar" args = ["-c", "proxies.xml"] } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#download-the-app-with-an-artifact-stanza) Download the app with an artifact stanza Nomad jobs generally fetch the workload as a part of starting up. For Java workloads, operators typically use the `artifact` stanza to download files into the allocation's filesystem as the task starts up. The artifact stanza will also decompress archive files automatically by default. Inside of the `task "membrane"` stanza, add an `artifact` stanza to fetch the application from GitHub. artifact { source = "https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip" destination = "local" } This artifact stanza tells Nomad to download the application to the tasks's working directory named `local`. You can learn more about the allocation [Filesystem internals](https://developer.hashicorp.com/nomad/docs/concepts/filesystem) in the Nomad Documentation. Nomad will automatically unzip the application archive once it is downloaded. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#define-the-membrane_home-environment-variable) Define the MEMBRANE\_HOME environment variable Add an `env` stanza inside of the `task "membrane"` stanza. This creates the `MEMBRANE_HOME` environment variable with the appropriate path. [Nomad variable interpolation](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation) provides the correct value for the Nomad task directory (`$NOMAD_TASK_DIR`) in the `MEMBRANE_HOME` path. env { MEMBRANE_HOME = "${NOMAD_TASK_DIR}/membrane-service-proxy-4.7.3" } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#generate-the-configuration-files-with-template-stanzas) Generate the configuration files with template stanzas The `template` stanza provides the job specification creator the ability to generate dynamic content and save it into the Nomad job's working directories. This is useful for jobs that have runtime-environment specific configuration, especially when paired with variable interpolation like you used in the `env stanza`. This tutorial uses the template stanza to provide the configuration files necessary to run the example Java application: `proxies.xml`, `get2soap.xsl` and `strip-env.xsl` Inside of the `task "membrane"` stanza, add the following three template stanzas. These templates use the heredoc syntax to provide a long, formatted value to the `data` attribute. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#proxies-xml) `proxies.xml` template { destination = "local/proxy-conf/proxies.xml" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#get2soap-xsl) `get2soap.xsl` template { destination = "local/proxy-conf/get2soap.xsl" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#strip-env-xsl) `strip-env.xsl` template { destination = "local/proxy-conf/strip-env.xsl" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#fix-path-to-proxies-xml) Fix path to `proxies.xml` Update the `args` value in the `config` stanza from `"proxies.xml"` to `"local/proxy-conf/proxies.xml"` to reflect where Nomad will write the rendered templates. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#configure-the-required-network-resources) Configure the required network resources Nomad networking is generally configured at the `group` level. This job requires two network ports to be available. * **Port 2000** - proxies API requests. * **Port 9000** - administration interface for the proxy Create a `network` stanza inside the `group "proxy"` stanza. network { port "admin" { static = 9000 } port "proxy" { static = 2000 } } [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#run-the-job-file) Run the job file ---------------------------------------------------------------------------------------------------------------------------- At this point, you have a complete Nomad job specification for this Java application. If you think you might have missed a step, click the **Show the completed job specification** link below to reveal a complete version for comparison. command. Show the completed job specification job "rest2json" { datacenters = ["dc1"] group "proxy" { network { port "admin" { static = 9000 } port "proxy" { static = 2000 } } task "membrane" { template { destination = "local/proxy-conf/get2soap.xsl" data =< EOD } template { destination = "local/proxy-conf/strip-env.xsl" data =< EOD } template { destination = "local/proxy-conf/proxies.xml" data =< EOD } env { MEMBRANE_HOME="${NOMAD_TASK_DIR}/membrane-service-proxy-4.7.3" } artifact { source = "https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip" destination = "local" } driver = "java" config { class = "com.predic8.membrane.core.Starter" class_path = "${MEMBRANE_HOME}/conf:${MEMBRANE_HOME}/starter.jar" args = ["-c", "local/proxy-conf/proxies.xml"] } } } } $ nomad job run rest2json.nomad.hcl Look for the allocation ID in the output from the `nomad job run` command. ==> Monitoring evaluation "13b407ab" Evaluation triggered by job "rest2json" Allocation "38e715aa" created: node "d715f8b4", group "proxy" ==> Monitoring evaluation "13b407ab" Evaluation within deployment: "c1be19df" Evaluation status changed: "pending" -> "complete" ==> Evaluation "13b407ab" finished with status "complete" In this sample output, the allocation ID is `38e715aa`. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#check-the-allocation-health) Check the allocation health Use the `nomad alloc status` command to view the state of the allocation. Replace the sample allocation ID with your actual allocation ID. $ nomad alloc status 38e715aa ID = 38e715aa-7635-adee-634a-fe0026c2baad Eval ID = 13b407ab Name = soap-proxy.membrane[0] Node ID = d715f8b4 Node Name = nomad-client-1.node.consul Job ID = soap-proxy Job Version = 0 Client Status = running Client Description = Tasks are running Desired Status = run Desired Description = Created = 5m5s ago Modified = 4m22s ago Deployment ID = de2f0a4f Deployment Health = unset Allocation Addresses Label Dynamic Address *admin yes 10.0.2.51:9000 *proxy yes 10.0.2.51:2000 Task "membrane" is "running" Task Resources CPU Memory Disk Addresses 2/500 MHz 67 MiB/256 MiB 300 MiB Task Events: Started At = 2021-04-08T21:15:31Z Finished At = N/A Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 2021-04-08T17:15:31-04:00 Started Task started by client 2021-04-08T17:15:28-04:00 Downloading Artifacts Client is downloading artifacts 2021-04-08T17:14:49-04:00 Task Setup Building Task Directory 2021-04-08T17:14:49-04:00 Received Task received by client ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#troubleshooting) Troubleshooting If the application doesn't start, run the `nomad alloc logs -stderr` command passing in the allocation ID. nomad alloc logs -stderr 38e715aa #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#common-errors) Common errors Following are a few common error cases you might encounter while migrating the sample job to Nomad and troubleshooting tips. ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#could-not-find-or-load-main-class) Could not find or load main class Error: Could not find or load main class com.predic8.membrane.core.Starter Common causes for this error are making an error in the `class_path` value or using the incorrect separator for your operating system: Linux uses colons (`:`) where Windows uses semicolons (`;`). ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#class-not-found) Class not found ClassNotFoundException: com.predic8.membrane.core.RouterCLI This error indicates that MEMBRANE\_HOME is not set properly. Verify that your job contains the template stanza that produces it. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#make-requests-to-the-running-application) Make requests to the running application Verify that the running proxy application is behaving as expected by fetching data from it. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#running-the-job-on-a-remote-nomad-cluster) Running the job on a remote Nomad cluster When running the job on a remote Nomad cluster: * The Nomad client running the job must allow connections on port 2000 and port 9000. This guide does not cover using dynamic ports. * Use the `nomad alloc status` command to obtain the address of the proxy. It's displayed in the **Allocation Addresses** section of the output. Example output displaying the proxy application's address $ nomad alloc status 41e605ba [...] Allocation Addresses Label Dynamic Address *admin yes 67.122.55.51:9000 *proxy yes 67.122.55.51:2000 [...] Use the `curl` command to fetch the API endpoint. Note If you are running the job on a remote Nomad cluster, remember to replace `localhost` in the following commands with the deployed proxy's allocation address found in the output of the `nomad alloc status` command. $ curl http://localhost:2000/bank/37050198 The proxy returns an XML response to the request.
Sparkasse KölnBonnCOLSDE33XXXKöln50667
 Now, add the `Accept: application/json` header to the request. $ curl --header "Accept: application/json" \ http://localhost:2000/bank/37050198 The proxy returns the response in JSON format. {"getBankResponse": {"details": {"bezeichnung": "Sparkasse KölnBonn","bic": "COLSDE33XXX","ort": "Köln","plz": 50667}}} [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#clean-up) Clean up ------------------------------------------------------------------------------------------------------------ Now that you have run and tested a Java workload in Nomad, clean up your learning environment. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#stop-the-nomad-job) Stop the Nomad job Stop the tutorial job with the `nomad job stop` command. $ nomad job stop rest2json ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#stop-dev-agent-if-applicable) Stop dev agent (if applicable) If you are running a local Nomad dev agent, you can switch to the shell session containing it, stop Nomad with `Ctrl-C`, and then exit the session. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#remove-the-tutorial-s-working-directory) Remove the tutorial's working directory Ensure that you have closed any file editors that might have files in the working directory open. Change to your home directory and recursively remove the `java-guide` folder you created in the first step. $ cd ~; $ rm -rf java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#review-what-you-learned) Review what you learned ------------------------------------------------------------------------------------------------------------------------------------------ Question 1: What is the primary benefit of using the Nomad Java task driver? Nomad's Java task driver reduces time to deploy non-containerized Java applications into a cluster. Question 2: Which stanza enables Nomad to download your Java application? The `artifact` stanza is used to download items into a Nomad allocation as it is started. Read more in the [`artifact` stanza](https://developer.hashicorp.com/nomad/docs/job-specification/artifact) documentation. Question 3: Which operating system provides additional process isolation for Java workloads? Linux. The Nomad Java task driver uses a `chroot` to provides additional isolation. Read more in the [Resource Isolation](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java#resource-isolation) section of the Java task driver documentation. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux#next-steps) Next steps ---------------------------------------------------------------------------------------------------------------- You can read more about the specifics of the [Java task driver](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java) in the Nomad Documentation. **Was this tutorial helpful?** YesNo [Previous\ \ Create a parameterized job](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-parameterized) [Next\ \ Migrate a Windows Java app](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows) --- # Migrate a Windows-based Java application to Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Job Specifications](https://developer.hashicorp.com/nomad/tutorials/job-specifications) Unlike many other application schedulers, Nomad can run non-containerized applications. If you have existing Java applications, you can run them directly on Nomad without converting them into containers. The Nomad Java task driver enables you to run Java applications in your Nomad cluster without the need to containerize them. The Nomad Java task driver also allows you to use Nomad's standardized, declarative job configuration with the familiar configuration elements of typical JVM applications. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#objective) Objective ---------------------------------------------------------------------------------------------------------------- For this tutorial, you will use a sample Java application—Membrane Service Proxy—and: * Identify required values from your startup script and configuration files * Translate them to an equivalent Nomad job specification * Use fundamental job specification stanzas: `group`,`task`,`driver`,`config`, and `env` * Download files for a job with `artifact` stanzas * Include configuration elements as `template` stanzas ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#about-the-sample-application) About the sample application The Membrane Service Proxy application is an Open Source API Gateway & HTTP reverse proxy for REST & SOAP. The application configuration for this tutorial is based on the Membrane examples and: 1. Consumes a REST-style query 2. Converts it to a SOAP query 3. Proxies it to a remote SOAP server 4. Receives the response 5. Formats the response based on the `Accept` header 6. Returns the response to the caller Note This tutorial's sample application queries a remote SOAP service that is not maintained by HashiCorp. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#prerequisites) Prerequisites ------------------------------------------------------------------------------------------------------------------------ ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#local-dependencies) Local dependencies * The `nomad` binary * Java ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#target-dependencies) Target dependencies You can deploy the tutorial's job file to a local Nomad dev agent or a remote Nomad cluster. Local Nomad dev agentRemote Nomad cluster Start a PowerShell session and run the following command and then minimize the window: nomad agent -dev -bind 0.0.0.0 When deploying to a remote Nomad cluster, ensure that: * The cluster has clients with Java installed. * [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/docs/commands#nomad_addr) and [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/docs/commands#nomad_token) are set as appropriate for [remote Nomad access](https://developer.hashicorp.com/nomad/docs/commands#remote-usage) . * Port 2000 and port 9000 are accessible on the client. This tutorial does not cover using dynamic ports. * You use the correct `class_path` separator for the Nomad clients. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#) [`:`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#) for Linux or macOS clients. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#) [`;`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#-1) for Windows clients. * If your cluster has a mixture of Windows and Linux clients, you will need to add a [`constraint`](https://developer.hashicorp.com/nomad/docs/job-specification/constraint) to your job to target the operating system that matches your `class_path` separator. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#create-a-learning-environment) Create a learning environment -------------------------------------------------------------------------------------------------------------------------------------------------------- This tutorial uses the command line. Open a new PowerShell session to get started. There is an [accompanying GitHub repository](https://github.com/hashicorp-education/learn-nomad-migrate-java) that contains the completed Nomad job file for Linux and Windows. It is not required to complete the tutorial but provided for a reference. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#make-a-working-directory) Make a working directory mkdir ~\java-guide cd ~\java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#run-the-workload-outside-of-nomad) Run the workload outside of Nomad ---------------------------------------------------------------------------------------------------------------------------------------------------------------- To validate the sample Java workload runs as expected in your environment, try it locally or on a host outside of the Nomad cluster. This tutorial includes the test steps for you. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#download-and-decompress-the-application) Download and decompress the application Download Membrane-SOA to your local machine. $url="https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip" $progresspreference = 'silentlyContinue' Invoke-WebRequest $url -OutFile "membrane-service-proxy-4.7.3.zip" $progresspreference = 'Continue' Once downloaded, unzip the `membrane-service-proxy-4.7.3.zip` file. Note The archive contains a top-level folder named `membrane-service-proxy-4.7.3`. Prevent PowerShell from nesting it inside of another one by setting `-DestinationPath` to `.` Expand-Archive -LiteralPath .\membrane-service-proxy-4.7.3.zip -DestinationPath . ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#change-into-the-workload-directory) Change into the workload directory cd membrane-service-proxy-4.7.3\examples\rest2soap-json You can consult the README in this folder for more information about the sample; however, you will find the vital information here. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#start-the-proxy) Start the proxy .\service-proxy.bat Logging will be returned in the terminal. A successful startup is indicated by a line containing `Membrane Service Proxy 4.7.3 up and running!` ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#observe-the-baseline-behavior) Observe the baseline behavior Open a new shell. Verify that the running proxy is behaving as expected by fetching data from `http://localhost:2000/bank/37050198`. Use the `Invoke-WebRequest` cmdlet to fetch the API endpoint. Invoke-WebRequest http://localhost:2000/bank/37050198 | ` Select Content -ExpandProperty Content The proxy returns an XML response to your request.
Sparkasse KölnBonnCOLSDE33XXXKöln50667
 Now, add the `Accept: application/json` header to the request. Invoke-WebRequest -Headers @{'Accept'='application/json'} ` http://localhost:2000/bank/37050198 | ` Select Content -ExpandProperty Content The proxy returns the response in JSON format. {"getBankResponse": {"details": {"bezeichnung": "Sparkasse KölnBonn","bic": "COLSDE33XXX","ort": "Köln","plz": 50667}}} Now that you have verified that the proxy is running as expected, close this shell session. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#stop-the-proxy) Stop the proxy Return to the terminal session running the proxy. Press `Ctrl-C` to stop it. When prompted with `Terminate batch job (Y/N)?`, press `Y` and `Return`. The proxy process will stop and return you to a PowerShell prompt. Now that you have validated the Java application runs as expected, you can migrate it to Nomad. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#read-the-example-startup-code) Read the example startup code -------------------------------------------------------------------------------------------------------------------------------------------------------- Open the `service-proxy.bat` file in a text editor to review its contents. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#key-observations) Key observations For now, observe that the `java` command does the following. This information is the foundation of your Nomad job specification. * Has a `-classpath` that depends on the value of `%CLASSPATH%` * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#) [`%CLASSPATH%`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#classpath) is based on `%MEMBRANE_HOME%`, which is the top-level directory created when decompressing the archive. * [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#) [`%CLASSPATH%`](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#classpath-1) expands to: %MEMBRANE_HOME%/conf;%MEMBRANE_HOME%/starter.jar * Runs a class named `com.predic8.membrane.core.Starter` * Passes the arguments `-c` and `proxies.xml` ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#change-into-your-work-directory) Change into your work directory cd ~\java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#build-your-nomad-job) Build your Nomad job -------------------------------------------------------------------------------------------------------------------------------------- Now that you have reviewed the startup script, you can use Nomad's declarative job specification to define the workload. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#required-job-specification) Required job specification Create a file named `rest2json.nomad.hcl` with the following template Nomad job specification. This job file is not runnable; however, every element here is required to create a parsable job. job "«job_name»" { datacenters = ["«datacenter»"] group "«group_name»" { task "«task_name»" { driver = "«driver_plugin_id»" } } } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#start-customizing-the-template-job) Start customizing the template job * Replace `«job_name»` with `rest2json` * Replace `«group_name»` with `proxy` * Replace `«task_name»` with `membrane` * Replace `«driver_plugin_id»` with `java` #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#fetch-valid-datacenter-values) Fetch valid `datacenter` values The `datacenters` value is a list of datacenter names as strings. Run the `nomad node status` command, which lists all of the clients in the target Nomad cluster or dev agent. $ nomad node status ID DC Name Class Drain Eligibility Status d715f8b4 dc1 nomad-client-1.node.consul false eligible ready 14ab9290 dc1 nomad-client-2.node.consul false eligible ready 0f357b26 dc1 nomad-client-3.node.consul false eligible ready In this output, all of the clients are in a datacenter named `dc1`. Replace `«datacenter»` with a valid value for your deployment target. The remaining tutorial content uses `dc1`; which is the default value for datacenter. Always use the correct value for your deployment target as the value for this field. Your job file should look like this now. job "rest2json" { datacenters = ["dc1"] group "proxy" { task "membrane" { driver = "java" } } } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#migrate-the-configuration-into-the-job) Migrate the configuration into the job The Nomad job specification provides job specific configuration to the task driver using the `config` stanza. This stanza tells Nomad how to start the Java application and contains attributes like `class`, `class_path`, and `args` Create a `config` stanza inside of the `task "membrane"` stanza with the information you discovered from reading the startup script. * Set the `class` argument to `com.predic8.membrane.core.Starter`. * Set the `class_path` argument to `${MEMBRANE_HOME}/conf;${MEMBRANE_HOME}/starter.jar` * Finally, set the `args` attribute to a list of quoted arguments: `["-c", "proxies.xml"]` This yields the following config stanza. config { class = "com.predic8.membrane.core.Starter" class_path = "${MEMBRANE_HOME}/conf;${MEMBRANE_HOME}/starter.jar" args = ["-c", "proxies.xml"] } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#download-the-app-with-an-artifact-stanza) Download the app with an artifact stanza Nomad jobs generally fetch the workload as a part of starting up. For Java workloads, operators typically use the `artifact` stanza to download files into the allocation's filesystem as the task starts up. The artifact stanza will also decompress archive files automatically by default. Inside of the `task "membrane"` stanza, add an `artifact` stanza to fetch the application from GitHub. artifact { source = "https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip" destination = "local" } This artifact stanza tells Nomad to download the application to the tasks's working directory named `local`. You can learn more about the allocation [Filesystem internals](https://developer.hashicorp.com/nomad/docs/concepts/filesystem) in the Nomad Documentation. Nomad will automatically unzip the application archive once it is downloaded. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#define-the-membrane_home-environment-variable) Define the MEMBRANE\_HOME environment variable Add an `env` stanza inside of the `task "membrane"` stanza. This creates the `MEMBRANE_HOME` environment variable with the appropriate path. [Nomad variable interpolation](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation) provides the correct value for the Nomad task directory (`$NOMAD_TASK_DIR`) in the `MEMBRANE_HOME` path. env { MEMBRANE_HOME = "${NOMAD_TASK_DIR}/membrane-service-proxy-4.7.3" } ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#generate-the-configuration-files-with-template-stanzas) Generate the configuration files with template stanzas The `template` stanza provides the job specification creator the ability to generate dynamic content and save it into the Nomad job's working directories. This is useful for jobs that have runtime-environment specific configuration, especially when paired with variable interpolation like you used in the `env stanza`. This tutorial uses the template stanza to provide the configuration files necessary to run the example Java application: `proxies.xml`, `get2soap.xsl` and `strip-env.xsl` Inside of the `task "membrane"` stanza, add the following three template stanzas. These templates use the heredoc syntax to provide a long, formatted value to the `data` attribute. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#proxies-xml) `proxies.xml` template { destination = "local/proxy-conf/proxies.xml" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#get2soap-xsl) `get2soap.xsl` template { destination = "local/proxy-conf/get2soap.xsl" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#strip-env-xsl) `strip-env.xsl` template { destination = "local/proxy-conf/strip-env.xsl" data =< EOD } #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#fix-path-to-proxies-xml) Fix path to `proxies.xml` Update the `args` value in the `config` stanza from `"proxies.xml"` to `"local/proxy-conf/proxies.xml"` to reflect where Nomad will write the rendered templates. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#configure-the-required-network-resources) Configure the required network resources Nomad networking is generally configured at the `group` level. This job requires two network ports to be available. * **Port 2000** - proxies API requests. * **Port 9000** - administration interface for the proxy Create a `network` stanza inside the `group "proxy"` stanza. network { port "admin" { static = 9000 } port "proxy" { static = 2000 } } [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#run-the-job-file) Run the job file ------------------------------------------------------------------------------------------------------------------------------ At this point, you have a complete Nomad job specification for this Java application. If you think you might have missed a step, click the **Show the completed job specification** link below to reveal a complete version for comparison. command. Show the completed job specification job "rest2json" { datacenters = ["dc1"] group "proxy" { network { port "admin" { static = 9000 } port "proxy" { static = 2000 } } task "membrane" { template { destination = "local/proxy-conf/get2soap.xsl" data =< EOD } template { destination = "local/proxy-conf/strip-env.xsl" data =< EOD } template { destination = "local/proxy-conf/proxies.xml" data =< EOD } env { MEMBRANE_HOME="${NOMAD_TASK_DIR}/membrane-service-proxy-4.7.3" } artifact { source = "https://github.com/membrane/service-proxy/releases/download/v4.7.3/membrane-service-proxy-4.7.3.zip" destination = "local" } driver = "java" config { class = "com.predic8.membrane.core.Starter" class_path = "${MEMBRANE_HOME}/conf;${MEMBRANE_HOME}/starter.jar" args = ["-c", "local/proxy-conf/proxies.xml"] } } } } nomad job run rest2json.nomad.hcl Look for the allocation ID in the output from the `nomad job run` command. ==> Monitoring evaluation "13b407ab" Evaluation triggered by job "rest2json" Allocation "38e715aa" created: node "d715f8b4", group "proxy" ==> Monitoring evaluation "13b407ab" Evaluation within deployment: "c1be19df" Evaluation status changed: "pending" -> "complete" ==> Evaluation "13b407ab" finished with status "complete" In this sample output, the allocation ID is `38e715aa`. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#check-the-allocation-health) Check the allocation health Use the `nomad alloc status` command to view the state of the allocation. Replace the sample allocation ID with your actual allocation ID. nomad alloc status 38e715aa ID = 38e715aa-7635-adee-634a-fe0026c2baad Eval ID = 13b407ab Name = soap-proxy.membrane[0] Node ID = d715f8b4 Node Name = nomad-client-1.node.consul Job ID = soap-proxy Job Version = 0 Client Status = running Client Description = Tasks are running Desired Status = run Desired Description = Created = 5m5s ago Modified = 4m22s ago Deployment ID = de2f0a4f Deployment Health = unset Allocation Addresses Label Dynamic Address *admin yes 10.0.2.51:9000 *proxy yes 10.0.2.51:2000 Task "membrane" is "running" Task Resources CPU Memory Disk Addresses 2/500 MHz 67 MiB/256 MiB 300 MiB Task Events: Started At = 2021-04-08T21:15:31Z Finished At = N/A Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 2021-04-08T17:15:31-04:00 Started Task started by client 2021-04-08T17:15:28-04:00 Downloading Artifacts Client is downloading artifacts 2021-04-08T17:14:49-04:00 Task Setup Building Task Directory 2021-04-08T17:14:49-04:00 Received Task received by client ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#troubleshooting) Troubleshooting If the application doesn't start, run the `nomad alloc logs -stderr` command passing in the allocation ID. nomad alloc logs -stderr 38e715aa #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#common-errors) Common errors Following are a few common error cases you might encounter while migrating the sample job to Nomad and troubleshooting tips. ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#could-not-find-or-load-main-class) Could not find or load main class Error: Could not find or load main class com.predic8.membrane.core.Starter Common causes for this error are making an error in the `class_path` value or using the incorrect separator for your operating system: Linux uses colons (`:`) where Windows uses semicolons (`;`). ##### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#class-not-found) Class not found ClassNotFoundException: com.predic8.membrane.core.RouterCLI This error indicates that MEMBRANE\_HOME is not set properly. Verify that your job contains the template stanza that produces it. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#make-requests-to-the-running-application) Make requests to the running application Verify that the running proxy application is behaving as expected by fetching data from it. #### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#running-the-job-on-a-remote-nomad-cluster) Running the job on a remote Nomad cluster When running the job on a remote Nomad cluster: * The Nomad client running the job must allow connections on port 2000 and port 9000. This guide does not cover using dynamic ports. * Use the `nomad alloc status` command to obtain the address of the proxy. It's displayed in the **Allocation Addresses** section of the output. Example output displaying the proxy application's address $ nomad alloc status 41e605ba [...] Allocation Addresses Label Dynamic Address *admin yes 67.122.55.51:9000 *proxy yes 67.122.55.51:2000 [...] Use the `Invoke-WebRequest` cmdlet to fetch the API endpoint. Note If you are running the job on a remote Nomad cluster, remember to replace `localhost` in the following commands with the deployed proxy's allocation address found in the output of the `nomad alloc status` command. Invoke-WebRequest http://localhost:2000/bank/37050198 | ` Select Content -ExpandProperty Content The proxy returns an XML response to the request.
Sparkasse KölnBonnCOLSDE33XXXKöln50667
 Now, add the `Accept: application/json` header to the request. Invoke-WebRequest -Headers @{'Accept'='application/json'} ` http://localhost:2000/bank/37050198 | ` Select Content -ExpandProperty Content The proxy returns the response in JSON format. {"getBankResponse": {"details": {"bezeichnung": "Sparkasse KölnBonn","bic": "COLSDE33XXX","ort": "Köln","plz": 50667}}} [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#clean-up) Clean up -------------------------------------------------------------------------------------------------------------- Now that you have run and tested a Java workload in Nomad, clean up your learning environment. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#stop-the-nomad-job) Stop the Nomad job Stop the tutorial job with the `nomad job stop` command. nomad job stop rest2json If you are running a local Nomad dev agent, you can restore the window containing it, stop Nomad with `Ctrl-C`, and then exit the session. ### [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#remove-the-tutorial-s-working-directory) Remove the tutorial's working directory Ensure that you have closed any file editors that might have files in the working directory open. Change to your home directory and recursively remove the `java-guide` folder you created in the first step. cd ~; rm -Recurse java-guide [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#review-what-you-learned) Review what you learned -------------------------------------------------------------------------------------------------------------------------------------------- Question 1: What is the primary benefit of using the Nomad Java task driver? Nomad's Java task driver reduces time to deploy non-containerized Java applications into a cluster. Question 2: Which stanza enables Nomad to download your Java application? The `artifact` stanza is used to download items into a Nomad allocation as it is started. Read more in the [`artifact` stanza](https://developer.hashicorp.com/nomad/docs/job-specification/artifact) documentation. Question 3: Which operating system provides additional process isolation for Java workloads? Linux. The Nomad Java task driver uses a `chroot` to provides additional isolation. Read more in the [Resource Isolation](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java#resource-isolation) section of the Java task driver documentation. [](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-windows#next-steps) Next steps ------------------------------------------------------------------------------------------------------------------ You can read more about the specifics of the [Java task driver](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/java) in the Nomad Documentation. **Was this tutorial helpful?** YesNo [Previous\ \ Migrate a Linux Java app](https://developer.hashicorp.com/nomad/tutorials/job-specifications/job-spec-java-linux) [Next\ \ Create jobspecs with Levant](https://developer.hashicorp.com/nomad/tutorials/job-specifications/dry-jobs-levant) --- # nomad status command reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) nomad status command reference ============================== The `status` command displays the status output for any Nomad resource. [](https://developer.hashicorp.com/nomad/commands/status#usage) Usage --------------------------------------------------------------------- nomad status [options] The status command accepts any Nomad identifier or identifier prefix as its sole argument. The command detects the type of the identifier and routes to the appropriate status command to display more detailed output. If the ID is omitted, the command lists out all of the existing jobs. This is for backwards compatibility and should not be relied on. [](https://developer.hashicorp.com/nomad/commands/status#examples) Examples --------------------------------------------------------------------------- Display the status of a job: $ nomad status example ID = example Name = example Submit Date = 08/28/17 23:01:39 UTC Type = service Priority = 50 Datacenters = dc1 Status = running Periodic = false Parameterized = false Summary Task Group Queued Starting Running Failed Complete Lost cache 0 0 1 0 0 0 Latest Deployment ID = f5506391 Status = running Description = Deployment is running Deployed Task Group Desired Placed Healthy Unhealthy cache 1 1 0 0 Actions Action Name Task Group Task my-action cache my-task Allocations ID Node ID Task Group Version Desired Status Created At e1d14a39 f9dabe93 cache 0 run running 08/28/17 23:01:39 UTC Display the status of an allocation: $ nomad status e1d14a39 ID = e1d14a39 Eval ID = cc882755 Name = example.cache[0] Node ID = f9dabe93 Job ID = example Job Version = 0 Client Status = running Client Description = Desired Status = run Desired Description = Created At = 08/28/17 23:01:39 UTC Deployment ID = f5506391 Deployment Health = healthy Task "redis" is "running" Task Resources CPU Memory Disk Addresses 4/500 MHz 6.3 MiB/256 MiB 300 MiB db: 127.0.0.1:21752 Task Events: Started At = 08/28/17 23:01:39 UTC Finished At = N/A Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 08/28/17 23:01:39 UTC Started Task started by client 08/28/17 23:01:39 UTC Task Setup Building Task Directory 08/28/17 23:01:39 UTC Received Task received by client Display the status of a deployment: $ nomad status f5506391 ID = f5506391 Job ID = example Job Version = 0 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy cache 1 1 1 0 Display the status of a node: $ nomad status f9dabe93 ID = f9dabe93 Name = nomad-server01 Class = DC = dc1 Drain = false Status = ready Drivers = docker,exec,java,qemu,raw_exec,rkt Uptime = 4h17m24s Allocated Resources CPU Memory Disk 500/8709 MHz 256 MiB/2.0 GiB 300 MiB/24 GiB Allocation Resource Utilization CPU Memory 3/8709 MHz 6.3 MiB/2.0 GiB Host Resource Utilization CPU Memory Disk 116/8709 MHz 335 MiB/2.0 GiB 12 GiB/38 GiB Allocations ID Node ID Task Group Version Desired Status Created At e1d14a39 f9dabe93 cache 0 run running 08/28/17 23:01:39 UTC [](https://developer.hashicorp.com/nomad/commands/status#general-options) General options ----------------------------------------------------------------------------------------- * [](https://developer.hashicorp.com/nomad/commands/status#) [`-address=`](https://developer.hashicorp.com/nomad/commands/status#address) : The address of the Nomad server. Overrides the `NOMAD_ADDR` environment variable if set. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-region=`](https://developer.hashicorp.com/nomad/commands/status#region) : The region of the Nomad server to forward commands to. Overrides the `NOMAD_REGION` environment variable if set. Defaults to the Agent's local region. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-namespace=`](https://developer.hashicorp.com/nomad/commands/status#namespace) : The target namespace for queries and actions bound to a namespace. Overrides the `NOMAD_NAMESPACE` environment variable if set. If set to `'*'`, subcommands which support this functionality query all namespaces authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-no-color`](https://developer.hashicorp.com/nomad/commands/status#no-color) : Disables colored command output. Alternatively, `NOMAD_CLI_NO_COLOR` may be set. This option takes precedence over `-force-color`. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-force-color`](https://developer.hashicorp.com/nomad/commands/status#force-color) : Forces colored command output. This can be used in cases where the usual terminal detection fails. Alternatively, `NOMAD_CLI_FORCE_COLOR` may be set. This option has no effect if `-no-color` is also used. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-ca-cert=`](https://developer.hashicorp.com/nomad/commands/status#ca-cert) : Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. Overrides the `NOMAD_CACERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-ca-path=`](https://developer.hashicorp.com/nomad/commands/status#ca-path) : Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `-ca-cert` and `-ca-path` are specified, `-ca-cert` is used. Overrides the `NOMAD_CAPATH` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-client-cert=`](https://developer.hashicorp.com/nomad/commands/status#client-cert) : Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `-client-key`. Overrides the `NOMAD_CLIENT_CERT` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-client-key=`](https://developer.hashicorp.com/nomad/commands/status#client-key) : Path to an unencrypted PEM encoded private key matching the client certificate from `-client-cert`. Overrides the `NOMAD_CLIENT_KEY` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-tls-server-name=`](https://developer.hashicorp.com/nomad/commands/status#tls-server-name) : The server name to use as the SNI host when connecting via TLS. Overrides the `NOMAD_TLS_SERVER_NAME` environment variable if set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-tls-skip-verify`](https://developer.hashicorp.com/nomad/commands/status#tls-skip-verify) : Do not verify TLS certificate. This is highly not recommended. Verification will also be skipped if `NOMAD_SKIP_VERIFY` is set. * [](https://developer.hashicorp.com/nomad/commands/status#) [`-token`](https://developer.hashicorp.com/nomad/commands/status#token) : The SecretID of an ACL token to use to authenticate API requests with. Overrides the `NOMAD_TOKEN` environment variable if set. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/commands/status.mdx) --- # Nomad quickstart | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad quickstart ================ These installations are designed to get you started with Nomad and should be used only for experimentation purposes. If you are looking to install Nomad in production, refer to the [Production Installation guide](https://developer.hashicorp.com/nomad/docs/deploy/production) . [](https://developer.hashicorp.com/nomad/docs/quickstart#nomad-tutorials) Nomad tutorials ----------------------------------------------------------------------------------------- We recommend these tutorials, which provision a Nomad cluster for you. * [Get Started](https://developer.hashicorp.com/nomad/tutorials/get-started) : Get up and running with Nomad by learning about scheduling, setting up a cluster, and deploying an example job. Create a Nomad cluster on your local machine or in an AWS, Azure, or Google Cloud Platform (GCP) cloud environment. This tutorial series has an associated code repository so you can review the Terraform provisioning scripts and Nomad job specifications. * [Cluster Setup](https://developer.hashicorp.com/nomad/tutorials/cluster-setup) : Provision a Nomad cluster on AWS, Azure, or GCP. Enable Consul and access control lists (ACLs). This tutorial series has an associated code repository so you can review the Terraform provisioning scripts. [](https://developer.hashicorp.com/nomad/docs/quickstart#cloud-installation) Cloud installation ----------------------------------------------------------------------------------------------- The Nomad repository contains guides and Terraform scripts for provisioning a Nomad sandbox environment on AWS, Azure, or GCP with Consul and Vault. You can explore Nomad and its integrations with the HashiCorp stack. * [Provision a Nomad cluster on AWS](https://github.com/hashicorp/nomad/blob/main/terraform/aws/README.md) * [Provision a Nomad cluster on Azure](https://github.com/hashicorp/nomad/blob/main/terraform/azure/README.md) * [Provision a Nomad cluster on GCP](https://github.com/hashicorp/nomad/blob/main/terraform/gcp/README.md) [](https://developer.hashicorp.com/nomad/docs/quickstart#local-installation) Local installation ----------------------------------------------------------------------------------------------- Use one of the following methods to run a local Nomad sandbox environment: * Install a binary Refer to the [Install Nomad guide](https://developer.hashicorp.com/nomad/docs/deploy) to install the latest Nomad release binary. Run Nomad with the `sudo nomad agent -dev` [command](https://developer.hashicorp.com/nomad/commands/agent#dev) , which starts the Nomad agent in development mode. If you are using Docker Desktop for Windows or MacOS, refer to [How to connect to my host network when using Docker Desktop (Windows and MacOS)](https://developer.hashicorp.com/nomad/docs/faq#q-how-to-connect-to-my-host-network-when-using-docker-desktop-windows-and-macos) for instructions on binding Nomad to a non-loopback network interface. * Use Vagrant and VirtualBox This option requires AMD hardware. The Nomad repository contains a Vagrantfile. Refer to the [Developing with Vagrant section](https://github.com/hashicorp/nomad/blob/main/contributing/README.md#developing-with-vagrant) of the [Nomad Codebase Documentation](https://github.com/hashicorp/nomad/blob/main/contributing/README.md) file for instructions. * Compile and run from source This option requires software engineering experience and a workstation set up for Go, Node, and Docker development. Refer to the Nomad repository's [Nomad Codebase Documentation](https://github.com/hashicorp/nomad/blob/main/contributing/README.md) file for details. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/quickstart.mdx) --- # Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.10.x. [View latest version](https://developer.hashicorp.com/nomad/commands) . Nomad command-line interface (CLI) reference ============================================ The Nomad CLI is a well-behaved command-line application. In erroneous cases, the CLI returns a non-zero exit status. To view a list of available commands, run the `nomad` command with no arguments, `-h`, or `--help`. To access help for any specific subcommand, run the subcommand with the `-h` argument. [](https://developer.hashicorp.com/nomad/commands/v1.10.x#autocomplete) Autocomplete ------------------------------------------------------------------------------------ Nomad's CLI supports command autocomplete. To install autocomplete, run the `nomad` command with the `-autocomplete-install` option. $ nomad -autocomplete-install To uninstall autocomplete, run the `nomad` command with the `-autocomplete-uninstall` option. $ nomad -autocomplete-uninstall [](https://developer.hashicorp.com/nomad/commands/v1.10.x#command-contexts) Command contexts -------------------------------------------------------------------------------------------- Nomad's CLI commands have implied contexts in their naming convention. Because the CLI is most commonly used to manipulate or query jobs, you can assume that any given command is working in that context unless the command name implies otherwise. For example, the `nomad job run` command runs a new job and the `nomad status` command queries information about existing jobs. Conversely, commands with a prefix in their name likely operate in a different context. Examples include the `nomad agent-info` or `nomad node drain` commands, which operate in the agent or node contexts respectively. ### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#remote-usage) Remote usage You may use the Nomad CLI to interact with a remote Nomad cluster, even when the local machine does not have a running Nomad agent. To do so, set the `NOMAD_ADDR` environment variable. $ NOMAD_ADDR=https://remote-address:4646 $ nomad status Or use the `-address=` flag when running commands. $ nomad status -address=https://remote-address:4646 The Nomad address must be reachable from your local machine. If the Nomad port is exposed to the public internet, we recommend configuring TLS. Refer to the [`tls` block in agent configuration](https://developer.hashicorp.com/nomad/docs/v1.10.x/configuration/tls) for details. ### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#environment-variables) Environment variables Nomad can use environment variables to configure command-line tool options. You may override these environment variables with individual flags. Except where noted, these variables influence the behavior of the Nomad CLI and should not be set for Nomad agents. #### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#connection-environment-variables) Connection environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_addr) - The address of the Nomad server. Defaults to `http://127.0.0.1:4646`. For unix sockets, as with the [task API](https://developer.hashicorp.com/nomad/api-docs/v1.10.x/task-api) , the format is either `unix:/path/to/api.sock` or `unix:///path/to/api.sock`. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_REGION`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_region) - The region of the Nomad server to forward commands to. Defaults to the Agent's local region * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_NAMESPACE`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_namespace) - The target namespace for queries and actions bound to a namespace. If set to `*`, job and alloc subcommands query all namespacecs authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_HTTP_AUTH`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_http_auth) - (Optional) This allows users to supply "Basic" HTTP authentication scheme ([RFC 7617](https://tools.ietf.org/html/rfc7617) ) information in environments where the Nomad API is behind an authenticating proxy server. #### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#access-control-list-acl-environment-variables) Access control list (ACL) environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_token) - The SecretID of an ACL token to use to authenticate API requests with. #### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#cli-environment-variables) CLI environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CLI_NO_COLOR`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_cli_no_color) - Disables colored command output. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CLI_SHOW_HINTS`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_cli_show_hints) - Enables ui-hints in common CLI command output. #### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#mutual-tls-mtls-environment-variables) Mutual TLS (mTLS) environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CLIENT_CERT`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_client_cert) - Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `NOMAD_CLIENT_KEY`. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CLIENT_KEY`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_client_key) - Path to an unencrypted PEM encoded private key matching the client certificate from `NOMAD_CLIENT_CERT`. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CACERT`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_cacert) - Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_CAPATH`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_capath) - Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `NOMAD_CACERT` and `NOMAD_CAPATH` are specified, `NOMAD_CACERT` is used. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_SKIP_VERIFY`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_skip_verify) - Do not verify TLS certificate. **This is highly not recommended.** * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_TLS_SERVER_NAME`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_tls_server_name) - The server name to use as the SNI host when connecting via TLS. #### [](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad-enterprise-license-environment-variables) Nomad Enterprise license environment variables These environment variables influence the Nomad Enterprise license configuration. These values are only used for Nomad Enterprise agents, not the Nomad CLI. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_LICENSE_PATH`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_license_path) - An absolute path to a Nomad Enterprise license file, for example `/etc/nomad.d/license.hclic`. * [](https://developer.hashicorp.com/nomad/commands/v1.10.x#) [`NOMAD_LICENSE`](https://developer.hashicorp.com/nomad/commands/v1.10.x#nomad_license) - The Nomad Enterprise license file contents as a string. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.10.x/content/commands/index.mdx) --- # Use Cases | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Use Cases ========= This page features Nomad's core use cases. Note that the full range of potential use cases is broader than what is covered here. [](https://developer.hashicorp.com/nomad/docs/use-cases#docker-container-orchestration) Docker container orchestration ---------------------------------------------------------------------------------------------------------------------- Organizations are increasingly moving towards a Docker centric workflow for application deployment and management. This transition requires new tooling to automate placement, perform job updates, enable self-service for developers, and to handle failures automatically. Nomad supports a [first-class Docker workflow](https://developer.hashicorp.com/nomad/docs/job-declare/task-driver/docker) and integrates seamlessly with [Consul](https://developer.hashicorp.com/nomad/docs/networking/consul) and [Vault](https://developer.hashicorp.com/nomad/docs/secure/vault) to enable a complete solution while maximizing operational flexibility. Nomad is easy to use, can scale to thousands of nodes in a single cluster, and can easily deploy across private data centers and multiple clouds. [](https://developer.hashicorp.com/nomad/docs/use-cases#legacy-application-deployment) Legacy application deployment -------------------------------------------------------------------------------------------------------------------- A virtual machine based application deployment strategy can lead to low hardware utilization rates and high infrastructure costs. While a Docker-based deployment strategy can be impractical for some organizations or use cases, the potential for greater automation, increased resilience, and reduced cost is very attractive. Nomad natively supports running legacy applications, static binaries, JARs, and simple OS commands directly. Workloads are natively isolated at runtime and bin packed to maximize efficiency and utilization (reducing cost). Developers and operators benefit from API-driven automation and enhanced reliability for applications through automatic failure handling. [](https://developer.hashicorp.com/nomad/docs/use-cases#microservices) Microservices ------------------------------------------------------------------------------------ Microservices and Service Oriented Architectures (SOA) are a design paradigm in which many services with narrow scope, tight state encapsulation, and API driven communication interact together to form a larger solution. However, managing hundreds or thousands of services instead of a few large applications creates an operational challenge. Nomad elegantly integrates with [Consul](https://developer.hashicorp.com/nomad/docs/networking/consul) for automatic service registration and dynamic rendering of configuration files. Nomad and Consul together provide an ideal solution for managing microservices, making it easier to adopt the paradigm. [](https://developer.hashicorp.com/nomad/docs/use-cases#batch-processing-workloads) Batch processing workloads -------------------------------------------------------------------------------------------------------------- As data science and analytics teams grow in size and complexity, they increasingly benefit from highly performant and scalable tools that can run batch workloads with minimal operational overhead. Nomad can natively run batch jobs and [parameterized](https://www.hashicorp.com/blog/replacing-queues-with-nomad-dispatch) jobs. Nomad's architecture enables easy scalability and an optimistically concurrent scheduling strategy that can yield [thousands of container deployments per second](https://www.hashicorp.com/c1m) . Alternatives are overly complex and limited in terms of their scheduling throughput, scalability, and multi-cloud capabilities. [](https://developer.hashicorp.com/nomad/docs/use-cases#multi-region-and-multi-cloud-federated-deployments) Multi-Region and multi-cloud federated deployments -------------------------------------------------------------------------------------------------------------------------------------------------------------- Nomad is designed to natively handle multi-datacenter and multi-region deployments and is cloud agnostic. This allows Nomad to schedule in private datacenters running bare metal, OpenStack, or VMware alongside an AWS, Azure, or GCE cloud deployment. This makes it easier to migrate workloads incrementally and to utilize the cloud for bursting. Nomad now enables [Proof Key for Code Exchange (PKCE)](https://oauth.net/2/pkce/) by default for new or updated OIDC auth methods. [](https://developer.hashicorp.com/nomad/docs/use-cases#company-specific-use-cases) Company-specific use cases -------------------------------------------------------------------------------------------------------------- This section features talks from companies on how they use Nomad to solve critical, real-world business objectives. #### [](https://developer.hashicorp.com/nomad/docs/use-cases#cloudflare) Cloudflare * [How We Use HashiCorp Nomad (2020)](https://blog.cloudflare.com/how-we-use-hashicorp-nomad/) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#betterhelp) BetterHelp * [How the world's largest online therapy provider runs on Nomad (2020)](https://www.youtube.com/watch?v=eN2ghrGpiUo) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#navi-capital) Navi Capital * [How Nomad powers a \\$1B hedge fund in Brazil (2020)](https://www.hashicorp.com/blog/nomad-community-story-navi-capital/) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#trivago) Trivago * [Maybe You Don’t Need Kubernetes (2019)](https://endler.dev/2019/maybe-you-dont-need-kubernetes/) * [Nomad - Our Experiences and Best Practices (2019)](https://tech.trivago.com/2019/01/25/nomad-our-experiences-and-best-practices/) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#lob-com) Lob.com * [Hard pass Kubernetes, Hello Nomad! (2022)](https://www.lob.com/blog/alternative-to-kubernetes) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#behavox) Behavox * [Microservices Management in Behavox: Part 2 - Nomad (2023)](https://blog.behavox.engineering/microservices-management-in-behavox-part-2-nomad/) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#reaktor) Reaktor * [Nomad - Kubernetes, but without the complexity (2019)](https://youtu.be/GkmyNBUugg8) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#pandora) Pandora * [How Pandora Uses Nomad (2019)](https://www.youtube.com/watch?v=OsZeKTP2u98&t=2s) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#circleci) CircleCI * [How CircleCI Processes 4.5 Million Builds Per Month (2019)](https://stackshare.io/circleci/how-circleci-processes-4-5-million-builds-per-month) * [Security & Scheduling are Not Your Core Competencies (2018)](https://www.hashicorp.com/resources/nomad-vault-circleci-security-scheduling) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#pagerduty) PagerDuty * [PagerDuty’s Nomadic Journey (2017)](https://www.hashicorp.com/resources/pagerduty-nomad-journey) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#q2) Q2 * [Q2’s Nomad Use and Overview (2019)](https://www.youtube.com/watch?v=OsZeKTP2u98&feature=youtu.be&t=1499) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#sap) SAP * [HashiCorp Nomad @ SAP Ariba (2018)](https://www.hashicorp.com/resources/nomad-community-call-core-team-sap-ariba) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#deluxe-entertainment) Deluxe Entertainment * [How Deluxe Uses the Complete HashiStack for Video Production (2018)](https://www.hashicorp.com/resources/deluxe-hashistack-video-production) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#citadel) Citadel * [End-to-End Production Nomad at Citadel (2017)](https://www.hashicorp.com/resources/end-to-end-production-nomad-citadel) * [Extreme Scaling with HashiCorp Nomad & Consul (2016)](https://www.hashicorp.com/resources/citadel-scaling-hashicorp-nomad-consul) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#jet-com-walmart) Jet.com (Walmart) * [Driving down costs at Jet.com with HashiCorp Nomad (2017)](https://www.hashicorp.com/resources/jet-walmart-hashicorp-nomad-azure-run-apps) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#target) Target * [Nomad at Target - Scaling Microservices Across Public and Private Clouds (2018)](https://www.hashicorp.com/resources/nomad-scaling-target-microservices-across-cloud) * [Playing with Nomad from HashiCorp (2017)](https://danielparker.me/nomad/hashicorp/schedulers/nomad/) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#oscar-health) Oscar Health * [Scalable CI at Oscar Health with Nomad and Docker (2018)](https://www.hashicorp.com/resources/scalable-ci-oscar-health-insurance-nomad-docker) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#ebay) eBay * [HashiStack at eBay - A Fully Containerized Platform Based on Infrastructure as Code (2018)](https://www.hashicorp.com/resources/ebay-hashistack-fully-containerized-platform-iac) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#dutch-national-police) Dutch National Police * [Going Cloud-Native at the Dutch National Police (2018)](https://www.hashicorp.com/resources/going-cloud-native-at-the-dutch-national-police) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#n26) N26 * [Tech at N26 - The Bank in the Cloud (2018)](https://medium.com/insiden26/tech-at-n26-the-bank-in-the-cloud-e5ff818b528b) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#elsevier) Elsevier * [Elsevier’s Container Framework with Nomad, Terraform, and Consul (2017)](https://www.hashicorp.com/resources/elsevier-nomad-container-framework-demo) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#graymeta) Graymeta * [Backend Batch Processing At Scale with Nomad (2017)](https://www.hashicorp.com/resources/backend-batch-processing-nomad) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#nih-ncbi) NIH NCBI * [NCBI’s Legacy Migration to Hybrid Cloud with Consul & Nomad (2018)](https://www.hashicorp.com/resources/ncbi-legacy-migration-hybrid-cloud-consul-nomad) #### [](https://developer.hashicorp.com/nomad/docs/use-cases#imgix) imgix * [Cluster Schedulers & Why We Chose Nomad Over Kubernetes (2017)](https://medium.com/@copyconstruct/schedulers-kubernetes-and-nomad-b0f2e14a896) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/use-cases.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.9.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Nomad is a flexible workload orchestrator to deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run diverse workloads including Docker, non-containerized, microservice, and batch applications. If you are just getting started with Nomad, refer to the [Getting Started tutorial collection](https://developer.hashicorp.com/nomad/tutorials/get-started) . Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.9.x/content/docs/index.mdx) --- # Deploy a Nomad Enterprise cluster | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) This deployment tutorial covers the steps required to install and configure a single Nomad Enterprise cluster as defined in the [Nomad Enterprise Reference Architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) . This deployment tutorial is designed to work in combination with the Nomad [Reference Architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) and [Consul Deployment Guide](https://developer.hashicorp.com/consul/tutorials/production-deploy/deployment-guide) . Although it is not a strict requirement to follow the Nomad Reference Architecture, ensure you are familiar with the overall architecture design. For example, installing Nomad server agents on multiple physical or virtual (with correct anti-affinity) hosts for high-availability. To provide a highly available single cluster architecture, we recommend Nomad server agents be deployed to more than one host, as shown in the [Nomad Reference Architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) . [![Reference diagram](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fproduction%2Fnomad_reference_diagram.png)](https://content.hashicorp.com/api/assets?product=tutorials&version=main&asset=public%2Fimg%2Fnomad%2Fproduction%2Fnomad_reference_diagram.png) These instructions are for installing and configuring Nomad on Linux hosts running the systemd system and service manager. These setup steps should be completed on all Nomad hosts: 1. [Download Nomad](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#download-nomad) 2. [Install Nomad](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#install-nomad) 3. [Configure systemd](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#configure-systemd) 4. [Configure Nomad](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#configure-nomad) 5. [Start Nomad](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#start-nomad) [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#download-nomad) Download Nomad --------------------------------------------------------------------------------------------------------------------------------------- Nomad binaries for the current version are available at the [Nomad website](https://developer.hashicorp.com/nomad/install) , older versions can be found at [https://releases.hashicorp.com/nomad/](https://releases.hashicorp.com/nomad/) , and Nomad Enterprise binaries are available for download by following the instructions made available to HashiCorp Enterprise customers. You may perform checksum verification of the zip packages using the SHA256SUMS and SHA256SUMS.sig files available for the specific release version. HashiCorp provides [a tutorial on checksum verification](https://www.hashicorp.com/security) for precompiled binaries. v1.1v1.0v0.12v0.11v0.10v0.9 $ export NOMAD_VERSION="1.1.0" $ export NOMAD_VERSION="1.0.1" $ export NOMAD_VERSION="0.12.9" $ export NOMAD_VERSION="0.11.8" $ export NOMAD_VERSION="0.10.9" $ export NOMAD_VERSION="0.9.7" $ curl --silent --remote-name https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_linux_amd64.zip [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#install-nomad) Install Nomad ------------------------------------------------------------------------------------------------------------------------------------- Unzip the downloaded package and move the `nomad` binary to `/usr/local/bin/`. Check `nomad` is available on the system path. $ unzip nomad_${NOMAD_VERSION}_linux_amd64.zip $ sudo chown root:root nomad $ sudo mv nomad /usr/local/bin/ $ nomad version The `nomad` command features opt-in autocompletion for flags, subcommands, and arguments (where supported). Enable autocompletion. $ nomad -autocomplete-install $ complete -C /usr/local/bin/nomad nomad Create a data directory for Nomad. $ sudo mkdir --parents /opt/nomad Create a unique, non-privileged system user to run Nomad. $ sudo useradd --system --home /etc/nomad.d --shell /bin/false nomad [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#configure-systemd) Configure systemd --------------------------------------------------------------------------------------------------------------------------------------------- `systemd` uses [documented reasonable defaults](https://www.freedesktop.org/software/systemd/man/systemd.directives.html) so only non-default values must be set in the configuration file. Create a Nomad service file at `/etc/systemd/system/nomad.service`. $ sudo touch /etc/systemd/system/nomad.service Add this configuration to the Nomad service file: [Unit] Description=Nomad Documentation=https://www.nomadproject.io/docs/ Wants=network-online.target After=network-online.target # When using Nomad with Consul it is not necessary to start Consul first. These # lines start Consul before Nomad as an optimization to avoid Nomad logging # that Consul is unavailable at startup. #Wants=consul.service #After=consul.service [Service] # Nomad server should be run as the nomad user. Nomad clients # should be run as root User=nomad Group=nomad ExecReload=/bin/kill -HUP $MAINPID ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d KillMode=process KillSignal=SIGINT LimitNOFILE=65536 LimitNPROC=infinity Restart=on-failure RestartSec=2 ## Configure unit start rate limiting. Units which are started more than ## *burst* times within an *interval* time span are not permitted to start any ## more. Use `StartLimitIntervalSec` or `StartLimitInterval` (depending on ## systemd version) to configure the checking interval and `StartLimitBurst` ## to configure how many starts per interval are allowed. The values in the ## commented lines are defaults. # StartLimitBurst = 5 ## StartLimitIntervalSec is used for systemd versions >= 230 # StartLimitIntervalSec = 10s ## StartLimitInterval is used for systemd versions < 230 # StartLimitInterval = 10s TasksMax=infinity OOMScoreAdjust=-1000 [Install] WantedBy=multi-user.target The following parameters are set for the `[Unit]` stanza: * [`Description`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description=) - Free-form string describing the Nomad service * [`Documentation`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Documentation=) - Link to the Nomad documentation * [`Wants`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Wants=) - Configure a dependency on the network service * [`After`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#After=) - Configure an ordering dependency on the network service being started before the Nomad service The following parameters are set for the `[Service]` stanza: * [`User`, `Group`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#User=) - Nomad servers should run as the nomad user. Nomad clients should run as root. * [`ExecReload`](https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecReload=) - Send Nomad a `SIGHUP` signal to trigger a configuration reload * [`ExecStart`](https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStart=) - Start Nomad with the `agent` argument and path to a directory of configuration files * [`KillMode`](https://www.freedesktop.org/software/systemd/man/systemd.kill.html#KillMode=) - Treat Nomad as a single process * [`LimitNOFILE`, `LimitNPROC`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties) - Disable limits for file descriptors and processes * [`RestartSec`](https://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartSec=) - Restart Nomad after 2 seconds of it being considered 'failed' * [`Restart`](https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=) - Restart Nomad unless it returned a clean exit code * [`StartLimitBurst`, `StartLimitIntervalSec`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval) - Configure unit start rate limiting * [`TasksMax`](https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksMax=N) - Disable task limits (only available in systemd >= 226) The following parameters are set for the `[Install]` stanza: * [`WantedBy`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#WantedBy=) - Creates a weak dependency on Nomad being started by the multi-user run level [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#configure-nomad) Configure Nomad ----------------------------------------------------------------------------------------------------------------------------------------- Nomad uses [documented reasonable defaults](https://developer.hashicorp.com/nomad/docs/configuration) so only non-default values must be set in the configuration file. Configuration can be read from multiple files and is loaded in lexical order. See the [full description](https://developer.hashicorp.com/nomad/docs/configuration) for more information about configuration loading and merge semantics. Some configuration settings are common to both server and client Nomad agents, while some configuration settings must only exist on one or the other. Follow the [common configuration](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#common-configuration) guidance on all hosts and then the specific guidance depending on whether you are configuring a Nomad [server](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#server-configuration) or [client](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#client-configuration) . * [Common Nomad configuration](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#common-configuration) * [Configure a Nomad server](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#server-configuration) * [Configure a Nomad client](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#client-configuration) ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#common-configuration) Common configuration Create a configuration file at `/etc/nomad.d/nomad.hcl`: $ sudo mkdir --parents /etc/nomad.d $ sudo chmod 700 /etc/nomad.d $ sudo touch /etc/nomad.d/nomad.hcl Add this configuration to the `nomad.hcl` configuration file: Note Replace the `datacenter` parameter value with the identifier you are using for the datacenter this Nomad cluster is deployed in. datacenter = "dc1" data_dir = "/opt/nomad" * [`datacenter`](https://developer.hashicorp.com/nomad/docs/configuration#datacenter) - The datacenter in which the agent is running. * [`data_dir`](https://developer.hashicorp.com/nomad/docs/configuration#data_dir) - The data directory for the agent to store state. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#server-configuration) Server configuration Create a configuration file at `/etc/nomad.d/server.hcl`: $ sudo touch /etc/nomad.d/server.hcl Add this configuration to the `server.hcl` configuration file: Note Replace the `bootstrap_expect` value with the number of Nomad servers you are deploying; three or five [is recommended](https://developer.hashicorp.com/nomad/docs/architecture/cluster/consensus#deployment_table) . server { enabled = true bootstrap_expect = 3 } This [`server`](https://developer.hashicorp.com/nomad/docs/configuration/server) stanza contains the following parameters: * [`enabled`](https://developer.hashicorp.com/nomad/docs/configuration/server#enabled) - Specifies if this agent should run in server mode. All other server options depend on this value being set. * [`bootstrap_expect`](https://developer.hashicorp.com/nomad/docs/configuration/server#bootstrap_expect) \- The number of expected servers in the cluster. Either this value should not be provided or the value must agree with other servers in the cluster. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#client-configuration) Client configuration Create a configuration file at `/etc/nomad.d/client.hcl`: $ sudo touch /etc/nomad.d/client.hcl Add this configuration to the `client.hcl` configuration file: client { enabled = true } This [`client`](https://developer.hashicorp.com/nomad/docs/configuration/client) stanza contains the following parameters: * [`enabled`](https://developer.hashicorp.com/nomad/docs/configuration/client#enabled) - Specifies if this agent should run in client mode. All other client options depend on this value being set. Note The [`options`](https://developer.hashicorp.com/nomad/docs/configuration/client#options-parameters) parameter can be used to set specific configurations on Nomad clients unique to your use case requirements, like the list of enabled task drivers, permitted users for jobs, and node fingerprinters. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#acl-configuration) ACL configuration The [Access Control](https://developer.hashicorp.com/nomad/docs/secure/acl) collection of tutorials provides instructions on configuring and enabling ACLs. ### [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#tls-configuration) TLS configuration Securing Nomad's cluster communication with mutual TLS (mTLS) is recommended for production deployments and can even ease operations by preventing mistakes and misconfigurations. Nomad clients and servers should not be publicly accessible without mTLS enabled. The [Securing Nomad with TLS](https://developer.hashicorp.com/nomad/docs/secure/traffic/tls) tutorial provides instructions on configuring and enabling TLS. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#start-nomad) Start Nomad --------------------------------------------------------------------------------------------------------------------------------- Enable and start Nomad using the systemctl command responsible for controlling systemd managed services. Check the status of the nomad service using systemctl. $ sudo systemctl enable nomad $ sudo systemctl start nomad $ sudo systemctl status nomad [](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-deployment-guide-vm-with-consul#next-steps) Next steps ------------------------------------------------------------------------------------------------------------------------------- * Read [Outage Recovery](https://developer.hashicorp.com/nomad/docs/manage/outage-recovery) to learn the steps required to recover from a Nomad cluster outage. * Read [Autopilot](https://developer.hashicorp.com/nomad/docs/manage/autopilot) to learn about features in Nomad 0.8 to allow for automatic operator-friendly management of Nomad servers. **Was this tutorial helpful?** YesNo [Previous\ \ Discover reference architecture](https://developer.hashicorp.com/nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul) [Next\ \ Dynamic app sizing](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts) --- # Nomad Ecosystem | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Ecosystem =============== [](https://developer.hashicorp.com/nomad/docs/ecosystem#continuous-integration-delivery) Continuous Integration & Delivery -------------------------------------------------------------------------------------------------------------------------- * [GitLab](https://www.hashicorp.com/resources/nomad-ci-cd-developer-workflows-and-integrations) * [Codefresh](https://codefresh.io/docs/docs/yaml-examples/examples/nomad/) * [CircleCI](https://circleci.com/docs/server/operator/introduction-to-nomad-cluster-operation) * [Drone](https://docs.drone.io/runner/nomad/overview/) * [Jenkins](https://plugins.jenkins.io/nomad/) * [Buildkite](https://buildkite.com/works-with/hashicorp) [](https://developer.hashicorp.com/nomad/docs/ecosystem#task-drivers) Task Drivers ---------------------------------------------------------------------------------- * [Community task driver plugins](https://developer.hashicorp.com/nomad/plugins/drivers/community) * [HashiCorp task driver plugins](https://developer.hashicorp.com/nomad/plugins/drivers) [](https://developer.hashicorp.com/nomad/docs/ecosystem#application-definition-image-build) Application Definition & Image Build -------------------------------------------------------------------------------------------------------------------------------- * [Nomad Pack](https://github.com/hashicorp/nomad-pack) * [Levant](https://github.com/hashicorp/levant) * [Packer](https://www.packer.io/) * [Waypoint](https://developer.hashicorp.com/waypoint/tutorials/get-started-nomad) [](https://developer.hashicorp.com/nomad/docs/ecosystem#container-registry) Container Registry ---------------------------------------------------------------------------------------------- * [JFrog Artifactory](https://jfrog.com/blog/cluster-management-made-simple-with-jfrog-artifactory-and-hashicorp-nomad/) [](https://developer.hashicorp.com/nomad/docs/ecosystem#observability-and-analysis) Observability and Analysis -------------------------------------------------------------------------------------------------------------- * Prometheus * [Using Prometheus to Monitor Nomad Metrics](https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-metrics) * [Start Prometheus](https://developer.hashicorp.com/nomad/tutorials/nomad-1-0/dynamic-application-sizing#start-prometheus) * [Grafana](https://www.metricfire.com/blog/monitoring-hashicorp-nomad-with-prometheus-and-grafana/) * [DataDog](https://docs.datadoghq.com/integrations/nomad/) * [Turbonomics](https://www.youtube.com/watch?v=lwtIaPpdDsc) * [Circonus](https://developer.hashicorp.com/nomad/tools/autoscaling/agent/telemetry#circonus) * [Splunk](https://www.kmruddy.com/2020/deploying-splunk-enterprise-with-nomad/) [](https://developer.hashicorp.com/nomad/docs/ecosystem#secret-management) Secret Management -------------------------------------------------------------------------------------------- * [Vault](https://developer.hashicorp.com/nomad/docs/secure/vault) [](https://developer.hashicorp.com/nomad/docs/ecosystem#service-mesh) Service Mesh ---------------------------------------------------------------------------------- * [Consul](https://developer.hashicorp.com/nomad/docs/networking/consul/service-mesh) [](https://developer.hashicorp.com/nomad/docs/ecosystem#provisioning) Provisioning ---------------------------------------------------------------------------------- * [Terraform](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs) * [Chef](https://github.com/nathwill/chef-nomad) * [Ansible](https://github.com/ansible-community/ansible-nomad) [](https://developer.hashicorp.com/nomad/docs/ecosystem#cloud-native-network) Cloud Native Network -------------------------------------------------------------------------------------------------- * [CNI](https://www.cni.dev/) [](https://developer.hashicorp.com/nomad/docs/ecosystem#service-proxy) Service Proxy ------------------------------------------------------------------------------------ * [Envoy](https://developer.hashicorp.com/nomad/docs/networking/consul) * [NGINX](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-nginx) * [Traefik](https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-traefik) [](https://developer.hashicorp.com/nomad/docs/ecosystem#storage) Storage ------------------------------------------------------------------------ * [CSI](https://developer.hashicorp.com/nomad/docs/architecture/storage/csi) [](https://developer.hashicorp.com/nomad/docs/ecosystem#gpus) GPUs ------------------------------------------------------------------ * NVIDIA * [Using HashiCorp Nomad to Schedule GPU Workloads](https://developer.nvidia.com/blog/hashicorp-nomad-gpu-scheduling/) * [Running GPU-Accelerated Applications on Nomad](https://www.hashicorp.com/resources/running-gpu-accelerated-applications-on-nomad) [](https://developer.hashicorp.com/nomad/docs/ecosystem#autoscaling) Autoscaling -------------------------------------------------------------------------------- * AWS ASGs * [Dynamically autoscale your Nomad clusters](https://www.hashicorp.com/blog/cluster-scaling-with-the-hashicorp-nomad-autoscaler) * [Spot](https://docs.spot.io/container-management/nomad/nomad-integration-with-elastigroup/) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/ecosystem.mdx) --- # Allocations - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Allocations HTTP API ==================== The `/allocation` endpoints are used to query for and interact with allocations. [](https://developer.hashicorp.com/nomad/api-docs/allocations#list-allocations) List Allocations ------------------------------------------------------------------------------------------------ This endpoint lists all allocations. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/allocations` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`prefix`](https://developer.hashicorp.com/nomad/api-docs/allocations#prefix) `(string: "")`\- Specifies a string to filter allocations based on an ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters (0-9a-f). This is specified as a query string parameter. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`next_token`](https://developer.hashicorp.com/nomad/api-docs/allocations#next_token) `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which identifies the next expected allocation. This value can be obtained from the `X-Nomad-NextToken` header from the previous response. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`per_page`](https://developer.hashicorp.com/nomad/api-docs/allocations#per_page) `(int: 0)` - Specifies a maximum number of allocations to return for this request. If omitted, the response is not paginated. The value of the `X-Nomad-NextToken` header of the last response can be used as the `next_token` of the next request to fetch additional pages. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`filter`](https://developer.hashicorp.com/nomad/api-docs/allocations#filter) `(string: "")` - Specifies the [expression](https://developer.hashicorp.com/nomad/api-docs#filtering) used to filter the results. Consider using pagination or a query parameter to reduce resource used to serve the request. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/allocations#namespace) `(string: "default")` - Specifies the namespace to search. Specifying `*` would return all allocations across all the authorized namespaces. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`resources`](https://developer.hashicorp.com/nomad/api-docs/allocations#resources) `(bool: false)` - Specifies whether or not to include the `AllocatedResources` field in the response. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`task_states`](https://developer.hashicorp.com/nomad/api-docs/allocations#task_states) `(bool: true)` - Specifies whether or not to include the `TaskStates` field in the response. TaskStates are included by default but can represent a large percentage of the overall response size. Clusters with a large number of allocations may set `task_states=false` to significantly reduce the size of the response. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`reverse`](https://developer.hashicorp.com/nomad/api-docs/allocations#reverse) `(bool: false)` - Specifies the list of returned allocations should be sorted in the reverse order. By default allocations are returned sorted in chronological order (older evaluations first), or in lexicographical order by their ID if the `prefix` query parameter is used. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request) Sample Request $ curl \ https://localhost:4646/v1/allocations $ curl \ https://localhost:4646/v1/allocations?prefix=a8198d79 $ curl \ https://localhost:4646/v1/allocations?namespace=*&prefix=a8198d79 ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response) Sample Response [\ {\ "ClientDescription": "Tasks are running",\ "ClientStatus": "running",\ "CreateIndex": 10,\ "CreateTime": 1636017249798459000,\ "DeploymentStatus": {\ "Canary": false,\ "Healthy": true,\ "ModifyIndex": 15,\ "Timestamp": "2021-11-04T10:14:22.054814+01:00"\ },\ "DesiredDescription": "",\ "DesiredStatus": "run",\ "DesiredTransition": {\ "ForceReschedule": null,\ "Migrate": null,\ "Reschedule": null\ },\ "EvalID": "cb20d15d-861f-8d8d-8253-e93932beea2e",\ "FollowupEvalID": "",\ "ID": "5457f16d-0f87-8e6b-5e91-0c7da3a41eb7",\ "JobID": "example",\ "JobType": "service",\ "JobVersion": 0,\ "ModifyIndex": 15,\ "ModifyTime": 1636017262190928000,\ "Name": "example.cache[0]",\ "Namespace": "default",\ "NodeID": "f476d2b4-02dc-c216-d031-273396727347",\ "NodeName": "linux",\ "PreemptedAllocations": null,\ "PreemptedByAllocation": "",\ "RescheduleTracker": null,\ "TaskGroup": "cache",\ "TaskStates": {\ "redis": {\ "Events": [\ {\ "Details": {},\ "DiskLimit": 0,\ "DisplayMessage": "Task received by client",\ "DownloadError": "",\ "DriverError": "",\ "DriverMessage": "",\ "ExitCode": 0,\ "FailedSibling": "",\ "FailsTask": false,\ "GenericSource": "",\ "KillError": "",\ "KillReason": "",\ "KillTimeout": 0,\ "Message": "",\ "RestartReason": "",\ "SetupError": "",\ "Signal": 0,\ "StartDelay": 0,\ "TaskSignal": "",\ "TaskSignalReason": "",\ "Time": 1636017249803624000,\ "Type": "Received",\ "ValidationError": "",\ "VaultError": ""\ },\ {\ "Details": {\ "message": "Building Task Directory"\ },\ "DiskLimit": 0,\ "DisplayMessage": "Building Task Directory",\ "DownloadError": "",\ "DriverError": "",\ "DriverMessage": "",\ "ExitCode": 0,\ "FailedSibling": "",\ "FailsTask": false,\ "GenericSource": "",\ "KillError": "",\ "KillReason": "",\ "KillTimeout": 0,\ "Message": "Building Task Directory",\ "RestartReason": "",\ "SetupError": "",\ "Signal": 0,\ "StartDelay": 0,\ "TaskSignal": "",\ "TaskSignalReason": "",\ "Time": 1636017249805254000,\ "Type": "Task Setup",\ "ValidationError": "",\ "VaultError": ""\ },\ {\ "Details": {},\ "DiskLimit": 0,\ "DisplayMessage": "Task started by client",\ "DownloadError": "",\ "DriverError": "",\ "DriverMessage": "",\ "ExitCode": 0,\ "FailedSibling": "",\ "FailsTask": false,\ "GenericSource": "",\ "KillError": "",\ "KillReason": "",\ "KillTimeout": 0,\ "Message": "",\ "RestartReason": "",\ "SetupError": "",\ "Signal": 0,\ "StartDelay": 0,\ "TaskSignal": "",\ "TaskSignalReason": "",\ "Time": 1636017252049956000,\ "Type": "Started",\ "ValidationError": "",\ "VaultError": ""\ }\ ],\ "Failed": false,\ "FinishedAt": null,\ "LastRestart": null,\ "Restarts": 0,\ "StartedAt": "2021-11-04T09:14:12.04996Z",\ "State": "running",\ "TaskHandle": null\ }\ }\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/allocations#read-allocation) Read Allocation ---------------------------------------------------------------------------------------------- This endpoint reads information about a specific allocation. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/allocation/:alloc_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id) `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/allocation/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-1) Sample Response { "ID": "a8198d79-cfdb-6593-a999-1e9adabcba2e", "EvalID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "Name": "example.cache[0]", "NodeID": "fb2170a8-257d-3c64-b14d-bc06cc94e34c", "PreviousAllocation": "516d2753-0513-cfc7-57ac-2d6fac18b9dc", "NextAllocation": "cd13d9b9-4f97-7184-c88b-7b451981616b", "RescheduleTracker": { "Events": [\ {\ "PrevAllocID": "516d2753-0513-cfc7-57ac-2d6fac18b9dc",\ "PrevNodeID": "9230cd3b-3bda-9a3f-82f9-b2ea8dedb20e",\ "RescheduleTime": 1517434161192946200,\ "Delay": "5000000000"\ }\ ] }, "JobID": "example", "Job": { "Region": "global", "ID": "example", "ParentID": "", "Name": "example", "Type": "service", "Priority": 50, "AllAtOnce": false, "Datacenters": ["dc1"], "Constraints": null, "Affinities": null, "TaskGroups": [\ {\ "Name": "cache",\ "Count": 1,\ "Constraints": null,\ "Affinities": null,\ "RestartPolicy": {\ "Attempts": 10,\ "Interval": 300000000000,\ "Delay": 25000000000,\ "Mode": "delay"\ },\ "Spreads": null,\ "Tasks": [\ {\ "Name": "redis",\ "Driver": "docker",\ "User": "",\ "Config": {\ "port_map": [\ {\ "db": 6379\ }\ ],\ "image": "redis:7"\ },\ "Env": null,\ "Services": [\ {\ "Name": "redis-cache",\ "PortLabel": "db",\ "Tags": ["global", "cache"],\ "Checks": [\ {\ "Name": "alive",\ "Type": "tcp",\ "Command": "",\ "Args": null,\ "Path": "",\ "Protocol": "",\ "PortLabel": "",\ "Interval": 10000000000,\ "Timeout": 2000000000,\ "InitialStatus": ""\ }\ ]\ }\ ],\ "Vault": null,\ "Templates": null,\ "Constraints": null,\ "Affinities": null,\ "Resources": {\ "CPU": 500,\ "MemoryMB": 10,\ "DiskMB": 0,\ "Networks": [\ {\ "Device": "",\ "CIDR": "",\ "IP": "",\ "MBits": 10,\ "ReservedPorts": null,\ "DynamicPorts": [\ {\ "Label": "db",\ "Value": 0\ }\ ]\ }\ ]\ },\ "Spreads": null,\ "DispatchPayload": null,\ "Meta": null,\ "KillTimeout": 5000000000,\ "LogConfig": {\ "MaxFiles": 10,\ "MaxFileSizeMB": 10\ },\ "Artifacts": null,\ "Leader": false\ }\ ],\ "EphemeralDisk": {\ "Sticky": false,\ "SizeMB": 300,\ "Migrate": false\ },\ "Meta": null\ }\ ], "Update": { "Stagger": 10000000000, "MaxParallel": 0 }, "Periodic": null, "ParameterizedJob": null, "Payload": null, "Spreads": null, "Meta": null, "VaultToken": "", "Status": "pending", "StatusDescription": "", "CreateIndex": 52, "ModifyIndex": 52, "JobModifyIndex": 52 }, "TaskGroup": "cache", "Resources": { "CPU": 500, "MemoryMB": 10, "DiskMB": 300, "Networks": [\ {\ "Device": "lo0",\ "CIDR": "",\ "IP": "127.0.0.1",\ "MBits": 10,\ "ReservedPorts": null,\ "DynamicPorts": [\ {\ "Label": "db",\ "Value": 23116\ }\ ]\ }\ ] }, "SharedResources": { "CPU": 0, "MemoryMB": 0, "DiskMB": 300, "Networks": null }, "TaskResources": { "redis": { "CPU": 500, "MemoryMB": 10, "DiskMB": 0, "Networks": [\ {\ "Device": "lo0",\ "CIDR": "",\ "IP": "127.0.0.1",\ "MBits": 10,\ "ReservedPorts": null,\ "DynamicPorts": [\ {\ "Label": "db",\ "Value": 23116\ }\ ]\ }\ ] } }, "Metrics": { "NodesEvaluated": 1, "NodesFiltered": 0, "NodesAvailable": { "dc1": 1 }, "NodesInPool": 1, "ClassFiltered": null, "ConstraintFiltered": null, "NodesExhausted": 0, "ClassExhausted": null, "DimensionExhausted": null, "Scores": { "fb2170a8-257d-3c64-b14d-bc06cc94e34c.binpack": 0.6205732522109244 }, "AllocationTime": 31729, "CoalescedFailures": 0 }, "DesiredStatus": "run", "DesiredDescription": "", "ClientStatus": "running", "ClientDescription": "", "TaskStates": { "redis": { "State": "running", "Failed": false, "FinishedAt": "0001-01-01T00:00:00Z", "LastRestart": "0001-01-01T00:00:00Z", "Restarts": 0, "StartedAt": "2017-07-25T23:36:26.106431265Z", "Events": [\ {\ "Type": "Received",\ "Time": 1495747371795703800,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ },\ {\ "Type": "Driver",\ "Time": 1495747371798867200,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": "Downloading image redis:7"\ },\ {\ "Type": "Started",\ "Time": 1495747379525667800,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ }\ ] } }, "PreviousAllocation": "", "CreateIndex": 54, "ModifyIndex": 57, "AllocModifyIndex": 54, "CreateTime": 1495747371794276400, "ModifyTime": 1495747371794276400 } #### [](https://developer.hashicorp.com/nomad/api-docs/allocations#field-reference) Field Reference * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Job`](https://developer.hashicorp.com/nomad/api-docs/allocations#job) - A copy of the job at the time that the allocation is created or updated. This is primarily intended for use by the Nomad client when running the allocation, and it may include stale information on some job fields. Up-to-date information about the job status is available from the [jobs API](https://developer.hashicorp.com/nomad/api-docs/jobs) ; take care to fetch the version of the job associated with this allocation. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`TaskStates`](https://developer.hashicorp.com/nomad/api-docs/allocations#taskstates) - A map of tasks to their current state and the latest events that have effected the state. `TaskState` objects contain the following fields: * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`State`](https://developer.hashicorp.com/nomad/api-docs/allocations#state) : The task's current state. It can have one of the following values: * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`TaskStatePending`](https://developer.hashicorp.com/nomad/api-docs/allocations#taskstatepending) - The task is waiting to be run, either for the first time or due to a restart. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`TaskStateRunning`](https://developer.hashicorp.com/nomad/api-docs/allocations#taskstaterunning) - The task is currently running. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`TaskStateDead`](https://developer.hashicorp.com/nomad/api-docs/allocations#taskstatedead) - The task is dead and will not run again. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`StartedAt`](https://developer.hashicorp.com/nomad/api-docs/allocations#startedat) : The time the task was last started at. Can be updated through restarts. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`FinishedAt`](https://developer.hashicorp.com/nomad/api-docs/allocations#finishedat) : The time the task was finished at. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`LastRestart`](https://developer.hashicorp.com/nomad/api-docs/allocations#lastrestart) : The last time the task was restarted. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Restarts`](https://developer.hashicorp.com/nomad/api-docs/allocations#restarts) : The number of times the task has restarted. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Events`](https://developer.hashicorp.com/nomad/api-docs/allocations#events) - An event contains metadata about the event. The latest 10 events are stored per task. Each event is timestamped (Unix nanoseconds) and has one of the following types: * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Setup Failure`](https://developer.hashicorp.com/nomad/api-docs/allocations#setup-failure) - The task could not be started because there was a failure setting up the task prior to it running. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Driver Failure`](https://developer.hashicorp.com/nomad/api-docs/allocations#driver-failure) - The task could not be started due to a failure in the driver. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Started`](https://developer.hashicorp.com/nomad/api-docs/allocations#started) - The task was started; either for the first time or due to a restart. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Terminated`](https://developer.hashicorp.com/nomad/api-docs/allocations#terminated) - The task was started and exited. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Killing`](https://developer.hashicorp.com/nomad/api-docs/allocations#killing) - The task has been sent the kill signal. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Killed`](https://developer.hashicorp.com/nomad/api-docs/allocations#killed) - The task was killed by a user. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Received`](https://developer.hashicorp.com/nomad/api-docs/allocations#received) - The task has been pulled by the client at the given timestamp. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Failed Validation`](https://developer.hashicorp.com/nomad/api-docs/allocations#failed-validation) - The task was invalid and as such it didn't run. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Restarting`](https://developer.hashicorp.com/nomad/api-docs/allocations#restarting) - The task terminated and is being restarted. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Not Restarting`](https://developer.hashicorp.com/nomad/api-docs/allocations#not-restarting) - the task has failed and is not being restarted because it has exceeded its restart policy. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Downloading Artifacts`](https://developer.hashicorp.com/nomad/api-docs/allocations#downloading-artifacts) - The task is downloading the artifact(s) * specified in the task. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Failed Artifact Download`](https://developer.hashicorp.com/nomad/api-docs/allocations#failed-artifact-download) - Artifact(s) specified in the task failed to download. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Restart Signaled`](https://developer.hashicorp.com/nomad/api-docs/allocations#restart-signaled) - The task was signaled to be restarted. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Signaling`](https://developer.hashicorp.com/nomad/api-docs/allocations#signaling) - The task was is being sent a signal. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Sibling Task Failed`](https://developer.hashicorp.com/nomad/api-docs/allocations#sibling-task-failed) - A task in the same task group failed. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Leader Task Dead`](https://developer.hashicorp.com/nomad/api-docs/allocations#leader-task-dead) - The group's leader task is dead. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Driver`](https://developer.hashicorp.com/nomad/api-docs/allocations#driver) - A message from the driver. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Task Setup`](https://developer.hashicorp.com/nomad/api-docs/allocations#task-setup) - Task setup messages. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`Building Task Directory`](https://developer.hashicorp.com/nomad/api-docs/allocations#building-task-directory) - Task is building its file system. Depending on the type the event will have applicable annotations. [](https://developer.hashicorp.com/nomad/api-docs/allocations#stop-allocation) Stop Allocation ---------------------------------------------------------------------------------------------- This endpoint stops and reschedules a specific allocation. | Method | Path | Produces | | --- | --- | --- | | `POST` / `PUT` | `/v1/allocation/:alloc_id/stop` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:alloc-lifecycle` | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-1) `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`no_shutdown_delay`](https://developer.hashicorp.com/nomad/api-docs/allocations#no_shutdown_delay) `(bool: false)` - Ignore the group and task [`shutdown_delay`](https://developer.hashicorp.com/nomad/docs/job-specification/group#shutdown_delay) configuration so that there is no delay between service deregistration and task shutdown. Note that using this parameter will result in failed network connections to the allocation being stopped. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-2) Sample Request $ curl -X POST \ https://localhost:4646/v1/allocation/5456bd7a-9fc0-c0dd-6131-cbee77f57577/stop ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-2) Sample Response { "EvalID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "Index": 54 } [](https://developer.hashicorp.com/nomad/api-docs/allocations#signal-allocation) Signal Allocation -------------------------------------------------------------------------------------------------- This endpoint sends a signal to an allocation or task. | Method | Path | Produces | | --- | --- | --- | | `POST` / `PUT` | `/v1/client/allocation/:alloc_id/signal` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:alloc-lifecycle` | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-2) `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-payload) Sample Payload { "Signal": "SIGUSR1", "Task": "FOO" } If `Task` is omitted, the signal will be sent to all tasks in the allocation. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-3) Sample Request $ curl -X POST -d '{"Signal": "SIGUSR1" }' \ https://localhost:4646/v1/client/allocation/5456bd7a-9fc0-c0dd-6131-cbee77f57577/signal ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-3) Sample Response {} [](https://developer.hashicorp.com/nomad/api-docs/allocations#restart-allocation) Restart Allocation ---------------------------------------------------------------------------------------------------- This endpoint restarts an allocation or task in-place. | Method | Path | Produces | | --- | --- | --- | | `POST` / `PUT` | `/v1/client/allocation/:alloc_id/restart` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:alloc-lifecycle` | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-4) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-3) `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`TaskName`](https://developer.hashicorp.com/nomad/api-docs/allocations#taskname) `(string: "")` - Specifies the individual task to restart. Cannot be used with `AllTasks` set to `true`. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`AllTasks`](https://developer.hashicorp.com/nomad/api-docs/allocations#alltasks) `(bool: false)` - If set to `true` all tasks in the allocation will be restarted, even the ones that already ran. Cannot be set to `true` if `TaskName` is defined. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-payload-1) Sample Payload { "TaskName": "FOO" } ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-4) Sample Request $ curl -X POST -d '{"TaskName": "redis" }' \ https://localhost:4646/v1/client/allocation/5456bd7a-9fc0-c0dd-6131-cbee77f57577/restart ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-4) Sample Response {} [](https://developer.hashicorp.com/nomad/api-docs/allocations#exec-allocation) Exec Allocation ---------------------------------------------------------------------------------------------- This endpoint executes a command inside the isolation container where an allocation is running. It opens a WebSocket to transmit input to and output from the command. | Method | Path | Produces | | --- | --- | --- | | `WebSocket` | `/v1/client/allocation/:alloc_id/exec` | WebSocket JSON streams | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:alloc-exec` (and `namespace:alloc-node-exec` if target task uses raw\_exec driver) | ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-5) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-4) `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`command`](https://developer.hashicorp.com/nomad/api-docs/allocations#command) `(string: )` - Specifies the command to be executed. This must be a JSON-encoded array of the command to be executed, e.g. `["echo", "hi"]` or `["/bin/bash"]`. This is specified as a query parameter. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`task`](https://developer.hashicorp.com/nomad/api-docs/allocations#task) `(string: )` - Specifies the task name, as a query parameter. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`tty`](https://developer.hashicorp.com/nomad/api-docs/allocations#tty) `(bool: false)` - Specifies whether a TTY is allocated for this task, as a query parameter. * [](https://developer.hashicorp.com/nomad/api-docs/allocations#) [`ws_handshake`](https://developer.hashicorp.com/nomad/api-docs/allocations#ws_handshake) `(bool: false)` - Specifies whether to expect the authentication token in the first frame, as a query parameter. ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#request-frames) Request Frames Request frames represent the `stdin` stream from the command as well as TTY resize events. When `?ws_handshake=true`, the first request frame must contain the authentication token. The following are valid formats: # sending authentication token {"version":1,"auth_token":"fc3c1968-8d31-5c50-9617-3db2e19ef32e"}   # sending stdin data {"stdin": {"data": "...base64 encoded string of bytes ..."}}   # indicating stdin is closed {"stdin": {"close": true}}   # indicating that TTY was resized {"tty_size": {"height": , "width": }}   # basic application-level heartbeat {} ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#response-frames) Response Frames Response frames represent `stdout` and `stderr` output from the command as well as exit codes: # transferring stdout data {"stdout": {"data": "...base64 encoded string of bytes ..."}}   # signaling that host closed stdout {"stdout": {"close": true}}   # transferring stderr data {"stderr": {"data": "...base64 encoded string of bytes ..."}}   # signaling that host closed stderr {"stderr": {"close": true}}   # signaling process exited {"exited": true, "result": {"exit_code": }}   # basic application-level heartbeat {} ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-and-response) Sample Request and Response Request and response frames encompass the full range of terminal emulator inputs and outputs, including the control characters necessary to render interactive applications. The example response includes instances of the ANSI “control sequence introducer” (CSI), which is ASCII code 27 followed by `[`.\ \ # \x12: form feed, to clear terminal\ {"stdin":{"data":"DA=="}}\  \ # "\x1b[H\x1b[2J$ ":\ # CSI-H (move cursor to top left corner), CSI-2J (clear entire screen), print "$ "\ {"stdout":{"data":"G1tIG1sySiQg"}}\ \ \ [](https://developer.hashicorp.com/nomad/api-docs/allocations#allocation-services)\ Allocation Services\ ------------------------------------------------------------------------------------------------------\ \ The endpoint is used to read all services registered within Nomad belonging to the passed allocation ID.\ \ | Method | Path | Produces |\ | --- | --- | --- |\ | `GET` | `/v1/allocation/:alloc_id/services` | `application/json` |\ \ The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)\ , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes)\ and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls)\ .\ \ | Blocking Queries | Consistency Modes | ACL Required |\ | --- | --- | --- |\ | `YES` | `all` | `namespace:read-job` |\ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-6)\ Parameters\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-5)\ `(string: )` - Specifies the service name. This is specified as part of the path.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`namespace`](https://developer.hashicorp.com/nomad/api-docs/allocations#namespace-1)\ `(string: "default")` - Specifies the target namespace.\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-5)\ Sample Request\ \ $ curl \\ https://localhost:4646/v1/allocation/177160af-26f6-619f-9c9f-5e46d1104395/services\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-5)\ Sample Response\ \ [\ {\ "Address": "127.0.0.1",\ "AllocID": "177160af-26f6-619f-9c9f-5e46d1104395",\ "CreateIndex": 14,\ "Datacenter": "dc1",\ "ID": "_nomad-task-177160af-26f6-619f-9c9f-5e46d1104395-redis-example-cache-redis-db",\ "JobID": "example",\ "ModifyIndex": 24,\ "Namespace": "default",\ "NodeID": "7406e90b-de16-d118-80fe-60d0f2730cb3",\ "Port": 29702,\ "ServiceName": "example-cache-redis",\ "Tags": [\ "db",\ "cache"\ ]\ }\ ]\ \ \ [](https://developer.hashicorp.com/nomad/api-docs/allocations#allocation-checks)\ Allocation Checks\ --------------------------------------------------------------------------------------------------\ \ The endpoint is used to read all health checks registered within Nomad belonging to the passed allocation ID.\ \ | Method | Path | Produces |\ | --- | --- | --- |\ | `GET` | `/v1/allocation/:alloc_id/checks` | `application/json` |\ \ The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)\ , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes)\ and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls)\ .\ \ | Blocking Queries | ACL Required |\ | --- | --- |\ | `NO` | `namespace:read-job` |\ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-7)\ Parameters\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-6)\ `(string: )` - Specifies the allocation ID. This is specified as part of the path.\ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-6)\ Sample Request\ \ $ curl \\ https://localhost:4646/v1/allocation/177160af-26f6-619f-9c9f-5e46d1104395/checks\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-6)\ Sample Response\ \ {\ "a1ed96606694742bf201a640a607e306": {\ "Check": "redis_probe",\ "Group": "example.cache[0]",\ "ID": "a1ed96606694742bf201a640a607e306",\ "Mode": "healthiness",\ "Output": "nomad: tcp ok",\ "Service": "redis",\ "Status": "success",\ "Timestamp": 1690442203\ }\ }\ \ \ [](https://developer.hashicorp.com/nomad/api-docs/allocations#override-pause-schedule-state)\ Override Pause Schedule State\ --------------------------------------------------------------------------------------------------------------------------\ \ Enterprise\ \ This feature requires [Nomad Enterprise](https://www.hashicorp.com/products/nomad)\ (opens in new tab).\ \ The endpoint is used to override the [`schedule`](https://developer.hashicorp.com/nomad/docs/job-specification/schedule)\ for tasks using time based execution.\ \ | Method | Path | Produces |\ | --- | --- | --- |\ | `POST` / `PUT` | `/v1/client/allocation/:alloc_id/pause` | `application/json` |\ \ The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)\ and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls)\ .\ \ | Blocking Queries | ACL Required |\ | --- | --- |\ | `NO` | `namespace:job-submit` |\ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-8)\ Parameters\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-7)\ `(string: )`\- Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`Task`](https://developer.hashicorp.com/nomad/api-docs/allocations#task-1)\ `(string: )` - Specifies the name of the task whose schedule should be overridden.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`ScheduleState`](https://developer.hashicorp.com/nomad/api-docs/allocations#schedulestate)\ `(string: )` - Specifies the pause state to force the task into. One of:\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`"pause"`](https://developer.hashicorp.com/nomad/api-docs/allocations#pause)\ - Forces the task to pause.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`"run"`](https://developer.hashicorp.com/nomad/api-docs/allocations#run)\ - Forces the task to run.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`"scheduled"`](https://developer.hashicorp.com/nomad/api-docs/allocations#scheduled)\ - Removes any overrides and forces the task to adhere to its schedule.\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-7)\ Sample Request\ \ $ echo '{"Task": "schedtask", "ScheduleState": "run"}' | \\ nomad operator api /v1/client/allocation/23f520cc-629a-46ff-395f-0661e7aa939e/pause\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-7)\ Sample Response\ \ {\ "Index": 0\ }\ \ \ [](https://developer.hashicorp.com/nomad/api-docs/allocations#read-pause-schedule-state)\ Read Pause Schedule State\ ------------------------------------------------------------------------------------------------------------------\ \ Enterprise\ \ This feature requires [Nomad Enterprise](https://www.hashicorp.com/products/nomad)\ (opens in new tab).\ \ Retrieve the [`schedule`](https://developer.hashicorp.com/nomad/docs/job-specification/schedule)\ state for a task using time based execution.\ \ | Method | Path | Produces |\ | --- | --- | --- |\ | `GET` | `/v1/client/allocation/:alloc_id/pause` | `application/json` |\ \ The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)\ and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls)\ .\ \ | Blocking Queries | ACL Required |\ | --- | --- |\ | `NO` | `namespace:read-job` |\ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#parameters-9)\ Parameters\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/allocations#alloc_id-8)\ `(string: )` - Specifies the UUID of the allocation. This must be the full UUID, not the short 8-character one. This is specified as part of the path.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`task`](https://developer.hashicorp.com/nomad/api-docs/allocations#task-2)\ `(string: )` - Specifies the name of the task from which to retrieve the time-based task execution state.\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-request-8)\ Sample Request\ \ $ nomad operator api /v1/client/allocation/23f520cc-629a-46ff-395f-0661e7aa939e/pause?task=schedtask\ \ \ ### [](https://developer.hashicorp.com/nomad/api-docs/allocations#sample-response-8)\ Sample Response\ \ {\ "ScheduleState": "scheduled_pause"\ }\ \ \ #### [](https://developer.hashicorp.com/nomad/api-docs/allocations#field-reference-1)\ Field Reference\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`ScheduleState`](https://developer.hashicorp.com/nomad/api-docs/allocations#schedulestate-1)\ `(string)`: The task's current paused state. It can can have one of the following values:\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`""`](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ - The task is running. The only state returned for tasks with no schedule.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`force_run`](https://developer.hashicorp.com/nomad/api-docs/allocations#force_run)\ - The task's schedule has been overridden to run.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`force_pause`](https://developer.hashicorp.com/nomad/api-docs/allocations#force_pause)\ - The task's schedule has been overridden to pause.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`scheduled_pause`](https://developer.hashicorp.com/nomad/api-docs/allocations#scheduled_pause)\ - The task is paused according to its schedule.\ \ * [](https://developer.hashicorp.com/nomad/api-docs/allocations#)\ \ [`schedule_resume`](https://developer.hashicorp.com/nomad/api-docs/allocations#schedule_resume)\ - A schedule override is being removed. Subsequent calls should return running (`""`) or paused (`scheduled_pause`) states. This state is rarely possible to observe since it transitions immediately to another state.\ \ \ [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/allocations.mdx) --- # Deploy a Consul API Gateway on Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Consul](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) An [API Gateway](https://developer.hashicorp.com/consul/docs/connect/gateways/api-gateway) is used for controlling access at entry and traffic management. In this tutorial, you will: * Deploy Consul ACL roles, policies, and intentions for the API Gateway. * Deploy an API Gateway job to Nomad. * Deploy an example upstream job, and configure the gateway. This tutorial uses Nomad's [Workload Identity](https://developer.hashicorp.com/nomad/docs/concepts/workload-identity) to authorize a Consul task to bootstrap the API Gateway task and correctly register the API Gateway with Consul. The API Gateway is deployed in its own Nomad namespace. You will add a Consul ACL role that grants the appropriate permissions to the API Gateway and matches the Consul binding rule for that Nomad namespace. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#prerequisites) Prerequisites ---------------------------------------------------------------------------------------------------------------------------- Clone the [API Gateway on Nomad](https://github.com/hashicorp-guides/consul-api-gateway-on-nomad) repository. This repository contains all of the necessary Consul and Nomad configuration files. $ git clone https://github.com/hashicorp-guides/consul-api-gateway-on-nomad Navigate to the cloned repository directory. $ cd consul-gateway-on-nomad Follow the instructions in the [README file](https://github.com/hashicorp-guides/consul-api-gateway-on-nomad/blob/main/README.md) to create a Nomad and Consul cluster with the correct configuration. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#create-required-policies) Create required policies -------------------------------------------------------------------------------------------------------------------------------------------------- Create a Nomad namespace. $ nomad namespace apply \ -description "namespace for Consul API Gateways" \ ingress Create a Consul ACL binding rule for the API Gateway that assigns the `builtin/api-gateway` templated policy to Nomad workloads deployed into the Nomad namespace `ingress` that you just created. consul acl binding-rule create \ -method 'nomad-workloads' \ -description 'Nomad API gateway' \ -bind-type 'templated-policy' \ -bind-name 'builtin/api-gateway' \ -bind-vars 'Name=${value.nomad_job_id}' \ -selector '"nomad_service" not in value and value.nomad_namespace==ingress' [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#upload-certificates-for-api-gateway) Upload certificates for API Gateway ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The API Gateway job needs Consul mTLS certificates to communicate with Consul. This tutorial uses [Nomad Variables](https://developer.hashicorp.com/nomad/docs/concepts/variables) to store the certificates securely, but you can also use Vault secrets. Add the certificates to the `ingress` namespace. $ nomad var put -namespace ingress \ nomad/jobs/my-api-gateway/gateway/setup \ consul_cacert=@$CONSUL_CACERT \ consul_client_cert=@$CONSUL_CLIENT_CERT \ consul_client_key=@$CONSUL_CLIENT_KEY [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#deploy-api-gateway) Deploy API Gateway -------------------------------------------------------------------------------------------------------------------------------------- Run the Nomad job. You can pass additional values to the command with the [`-var` option](https://developer.hashicorp.com/nomad/commands/job/run#var) . $ nomad job run ./api-gateway.nomad.hcl Once the deployment is complete, check the Consul UI to see that the API Gateway service has been registered. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#run-an-example-upstream) Run an example upstream ------------------------------------------------------------------------------------------------------------------------------------------------ Add intentions to allow traffic from the API Gateway to the `hello` application. $ consul config write example/hello-app-intentions.hcl Register a listener for the API Gateway. $ consul config write example/gateway-listeners.hcl Register http routes for the API Gateway so that Envoy knows how and where to write the traffic. $ consul config write example/my-http-route.hcl Start the `hello` app. $ nomad run example/hello-app.nomad.hcl Once the deployment is complete, you can test the API Gateway. Find the allocation for the API gateway. $ nomad job status -namespace ingress my-api-gateway Find the address for the API Gateway allocation and provide the allocation ID from the `status` command above by replacing the placeholder `` in this command. $ nomad alloc -namespace ingress status Submit a request to the `hello` app and observe the response. Replace the placeholder values `` and `` with the address and port respectively from the `status` command. $ curl -v http://:/hello [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/deploy-api-gateway-on-nomad#next-steps) Next steps ---------------------------------------------------------------------------------------------------------------------- In this tutorial you deployed Consul ACL roles, policies, and intentions for an API Gateway, deployed an API Gateway job to Nomad, deployed an example upstream job, and configured an API gateway. Learn more by checking out these resources. * [`identity` block documentation](https://developer.hashicorp.com/nomad/docs/job-specification/identity) * [Vault ACLs with Nomad Workload Identities](https://developer.hashicorp.com/nomad/docs/secure/workload-identity/vault) **Was this tutorial helpful?** YesNo [Previous\ \ Consul in production](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist) [Next Collection\ \ Load Balancing](https://developer.hashicorp.com/nomad/tutorials/load-balancing) --- # Secure Nomad jobs with Consul service mesh | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Consul](https://developer.hashicorp.com/nomad/tutorials/integrate-consul) Nomad's first-class integration with Consul allows operators to design jobs that natively leverage Consul service mesh and transparent proxy. In this tutorial, you will learn how to configure Consul to allow access between workloads and run a sample Consul service mesh workload. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#prerequisites) Prerequisites -------------------------------------------------------------------------------------------------------------------- * Nomad v1.8.0 or greater. All recent versions of Nomad support Consul service mesh, but only Nomad 1.8.0 or greater supports the [transparent proxy block](https://developer.hashicorp.com/nomad/docs/job-specification/transparent_proxy) demonstrated here. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#create-the-consul-and-nomad-clusters) Create the Consul and Nomad clusters ------------------------------------------------------------------------------------------------------------------------------------------------------------------ LocalTerraform This section uses development agents for [Consul](https://developer.hashicorp.com/consul/commands/agent#_dev) and [Nomad](https://developer.hashicorp.com/nomad/commands/agent#dev) as a quick way to get started. Development agents have ephemeral state and should not be used in production environments. They run in the foreground of your terminal so be sure not to close the terminal window or the agent configuration steps will need to be rerun again. This setup uses hard-coded tokens in the Consul configuration. We do not recommend this method for production clusters. Follow the [Secure Consul with ACLs](https://developer.hashicorp.com/consul/tutorials/security/access-control-setup-production) tutorial to configure production Consul clusters. Create a directory for the tutorial on your local machine, change into that directory, and create a file named `consul.hcl` to store the Consul agent configuration. Add the following contents to it and replace the two instances of the placeholder `"<< IP ADDRESS >>"` with the IP address of your local machine. Save the file. consul.hcl datacenter = "dc1" bind_addr = "<< IP ADDRESS >>" client_addr = "0.0.0.0" recursors = ["1.1.1.1", "1.1.0.0"] addresses { dns = "<< IP ADDRESS >>" } acl { enabled = true tokens { initial_management = "consul-root-token" agent = "consul-root-token" } } ports { grpc = 8502 dns = 8600 } Start a Consul dev agent using `consul.hcl` as the configuration file. $ consul agent -dev -config-file 'consul.hcl' ==> Starting Consul agent... Version: '1.17.0' Build Date: '2023-11-03 14:56:56 +0000 UTC' Node ID: 'f9625116-3884-cd0b-01fa-2bc2e0d9a69d' ... Open another terminal window in the same directory and set the Consul management token as an environment variable. $ export CONSUL_HTTP_TOKEN=consul-root-token Create a file named `consul-policy-nomad-agents.hcl` to store the Consul ACL rules that grant the necessary permissions to Nomad agents. Add the following contents to it and save the file. consul-policy-nomad-agents.hcl agent_prefix "" { policy = "read" } node_prefix "" { policy = "read" } service_prefix "" { policy = "write" } Create a Consul ACL policy named `nomad-agents` with the rules defined in the `consul-policy-nomad-agents.hcl` file. $ consul acl policy create -name 'nomad-agents' -description 'Policy for Nomad agents' -rules '@consul-policy-nomad-agents.hcl' ID: 7a0fe00b-f7e6-809c-2227-bb0638b873bd Name: nomad-agents Description: Policy for Nomad agents Datacenters: Rules: agent_prefix "" { policy = "read" } node_prefix "" { policy = "read" } service_prefix "" { policy = "write" } Create a Consul ACL token for the Nomad agent using the `nomad-agents` ACL policy. $ consul acl token create -policy-name 'nomad-agents' AccessorID: 3f436657-823a-95e3-4755-79f3e1e43c8e SecretID: df179fd2-3211-3641-5901-a57331c14611 Description: Local: false Create Time: 2023-11-15 18:23:39.572365 -0500 EST Policies: a5ee20ed-7158-89be-9a19-be213d106d24 - nomad-agents Save the value of `SecretID` for the Consul ACL token. Download the [`consul-cni`](https://releases.hashicorp.com/consul-cni) CNI plugin. Unzip the archive and move the `consul-cni` binary to wherever you install the CNI reference plugins as described in Nomad's [Post Installation Steps](https://developer.hashicorp.com/nomad/docs/deploy#linux-post-installation-steps) . A commonly used path is `/opt/cni/bin`. Create a file named `nomad.hcl`. Add the following contents to it and replace the placeholder `` text with the value of `SecretID`. Save the file. nomad.hcl acl { enabled = true } consul { address = "127.0.0.1:8500" token = "" service_identity { aud = ["consul.io"] ttl = "1h" } task_identity { aud = ["consul.io"] ttl = "1h" } } Open another terminal window in the same directory and start the Nomad dev agent using `nomad.hcl` as the configuration file. $ sudo nomad agent -dev -dev-connect -config 'nomad.hcl' ==> Loaded configuration from nomad.hcl ==> Starting Nomad agent... ==> Nomad agent configuration: Advertise Addrs: HTTP: 192.168.1.170:4646; RPC: 192.168.1.170:4647; Serf: 192.168.1.170:4648 Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648 Client: true Log Level: DEBUG Node Id: a59f4059-4453-dd16-be81-6934962435f1 Region: global (DC: dc1) Server: true Version: 1.8.0 ==> Nomad agent started! Log data will stream in below: ... Bootstrap the Nomad ACL system. $ nomad acl bootstrap Accessor ID = d1de8625-8556-0932-a25c-3aa71bfc0134 Secret ID = 7f10099a-936c-3f3a-8783-f0980493e54b Name = Bootstrap Token Type = management Global = true Create Time = 2023-11-16 01:09:26.565422 +0000 UTC Expiry Time = Create Index = 23 Modify Index = 23 Policies = n/a Roles = n/a Copy the value of `Secret ID` and set it as the environment variable `NOMAD_TOKEN`. $ export NOMAD_TOKEN=... Use the `nomad setup` command to configure Consul to use Nomad workload identity. The `nomad.hcl` file contains the recommended configuration set by the command. $ nomad setup consul -y ... Follow the steps in the [Nomad repository](https://github.com/hashicorp/nomad/tree/main/terraform) to provision a cluster on your cloud provider of choice. Then complete the [Consul ACL with Nomad Workload Identities](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-acl) tutorial to configure the Nomad cluster and the Consul auth methods and ACL policies. Refer to the [Consul ACL documentation](https://developer.hashicorp.com/consul/docs/secure/acl) for more information on Consul ACL set up. Download the [`consul-cni`](https://releases.hashicorp.com/consul-cni) CNI plugin. Unzip the archive and move the `consul-cni` binary to wherever you install the CNI reference plugins as described in Nomad's [Post Installation Steps](https://developer.hashicorp.com/nomad/docs/deploy#linux-post-installation-steps) . A commonly used path `/opt/cni/bin`. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#update-consul-agent-configuration) Update Consul agent configuration ------------------------------------------------------------------------------------------------------------------------------------------------------------ Open the Consul agent configuration file on each of your Nomad client nodes and ensure that Consul's DNS listener is exposed to a private IP address by updating the `bind_addr`. For example, if the client hosts are on a network `192.168.1.0/24`, use the following value: bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"192.168.1.0/24\" | limit 1 | attr \"address\" }}" Workloads in network namespace will not be able to reach Consul DNS if the `bind_addr` is set to localhost. Use `iptables` rules or cloud firewall rules to prevent access to the Consul agent from other hosts on the local network. Consul DNS allows your workload to connect to other applications in the service mesh via a [virtual IP](https://developer.hashicorp.com/consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups) . Configure the [`recursors`](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#_recursors) block to allow workloads to make DNS queries for applications outside the service mesh. The `recursors` block is a list of DNS nameservers that Consul will recursively query if it gets a request for a record it does not have. Configure Consul to use CloudFlare DNS. recursors = ["1.1.1.1", "1.1.0.0"] Configure the [`ports`](https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#ports) for DNS and gRPC. ports { dns = 8600 grpc = 8502 } Ensure that the configuration has Consul service mesh enabled. This is the default behavior. connect { enabled = true } Save the configuration file and restart the Consul agent to pick up the new configuration values. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#verify-nomad-client-consul-configuration) Verify Nomad client Consul configuration -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Verify that the Nomad client nodes detect Consul and the correct configuration for Consul service mesh using [`nomad node status -verbose`](https://developer.hashicorp.com/nomad/commands/node/status) . Specifically, verify that `consul.connect` is `true`, `consul.dns.addr` and `consul.dns.port` are enabled (not set to `-1`), and `plugins.cni.version.consul-cni` is showing version 1.4.2 or above. $ nomad node status -verbose fdb32e59 | grep consul consul.connect = true consul.datacenter = dc1 consul.dns.addr = 192.168.1.170 consul.dns.port = 8600 consul.ft.namespaces = true consul.grpc = 8502 consul.partition = default consul.revision = d6969061 consul.sku = ent consul.version = 1.16.0+ent plugins.cni.version.consul-cni = v1.4.2 The Nomad client agents will periodically fingerprint the Consul agent every few minutes. Restarting the Nomad agent will force Nomad to immediately detect any changes. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#) ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#alternative-architectures-non-x86-amd64) Alternative architectures (non-x86/amd64) If you are using an ARM or another non-x86/amd64 architecture, go to the [Alternative Architectures](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#alternative-architectures) section of this tutorial for additional setup details. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#run-a-connect-enabled-job) Run a `connect`\-enabled job ----------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#create-the-job-specification) Create the job specification Create the "countdash" job by copying this job specification into a file named `countdash.nomad.hcl`. Save the file. countdash.nomad.hcl job "countdash" { group "api" { network { mode = "bridge" } service { name = "count-api" port = "9001" check { type = "http" path = "/health" expose = true interval = "3s" timeout = "1s" check_restart { limit = 0 } } connect { sidecar_service { proxy { transparent_proxy {} } } } } task "web" { driver = "docker" config { image = "hashicorpdev/counter-api:v3" auth_soft_fail = true } } } group "dashboard" { network { mode = "bridge" port "http" { static = 9002 to = 9002 } } service { name = "count-dashboard" port = "9002" check { type = "http" path = "/health" expose = true interval = "3s" timeout = "1s" check_restart { limit = 0 } } connect { sidecar_service { proxy { transparent_proxy {} } } } } task "dashboard" { driver = "docker" env { COUNTING_SERVICE_URL = "http://count-api.virtual.consul" } config { image = "hashicorpdev/counter-dashboard:v3" auth_soft_fail = true } } } } ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#create-a-service-intention) Create a service intention In Consul, the default intention behavior is defined by the default ACL policy. If the default ACL policy is "allow all", then all service mesh connections are allowed by default. If the default ACL policy is "deny all", then all service mesh connections are denied by default. To avoid unexpected behavior around this, it is better to create an explicit intention. Create an intention to allow traffic from the count-dashboard service to the count-api service. First, create a file for a config entry definition named `intention-config.hcl`. intention-config.hcl Kind = "service-intentions" Name = "count-api" Sources = [\ {\ Name = "count-dashboard"\ Action = "allow"\ }\ ] Initialize the intention rules. For more information, check out the [Service Intentions documentation](https://developer.hashicorp.com/consul/docs/connect/config-entries/service-intentions) . $ consul config write intention-config.hcl Config entry written: service-intentions/count-api ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#run-the-job) Run the job Run the job by calling `nomad run countdash.nomad.hcl`. The command will output the result of running the job and show the allocation IDs of the two new allocations that are created. $ nomad run countdash.nomad.hcl ==> Monitoring evaluation "3e7ebb57" Evaluation triggered by job "countdash" Evaluation within deployment: "9eaf6878" Allocation "012eb94f" created: node "c0e8c600", group "api" Allocation "02c3a696" created: node "c0e8c600", group "dashboard" Evaluation status changed: "pending" -> "complete" ==> Evaluation "3e7ebb57" finished with status "complete" Get the address of the dashboard application from the allocation status output. $ nomad alloc status 02c3a696 | grep 9002 *http yes 10.37.105.17:9002 -> 9002 Navigating to the dashboard on port `9002` of the Nomad client host shows a green "Connected" badge. Stop the job. The `stop` command will stop the allocations in the background and output evaluation information about the stop request. $ nomad stop countdash ==> Monitoring evaluation "d4796df1" Evaluation triggered by job "countdash" Evaluation within deployment: "18b25bb6" Evaluation status changed: "pending" -> "complete" ==> Evaluation "d4796df1" finished with status "complete" [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#advanced-considerations) Advanced considerations ---------------------------------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#alternative-architectures) Alternative architectures Nomad provides a default link to a pause image. This image, however, is architecture specific and is only provided for the amd64 architecture. In order to use Consul service mesh on non-x86/amd64 hardware, you will need to configure Nomad to use a different pause container. If Nomad is trying to use a version of Envoy earlier than 1.16, you will need to specify a different version it as well. Read through the section on airgapped networks below. It explains the same configuration elements that you will need to set to use alternative containers for service mesh. Special thanks to `@GusPS`, who reported this working configuration. Envoy 1.16 now has ARM64 support. Configure it as your sidecar image by setting the `connect.sidecar_image` [`meta`](https://developer.hashicorp.com/nomad/docs/configuration/client#meta) variable on each of your ARM64 clients. meta { "connect.sidecar_image" = "envoyproxy/envoy:v1.16.0" } The `rancher/pause` container has versions for several different architectures as well. Override the default pause container and use it instead. In your client configuration, add an `infra_image` to your docker plugin configuration overriding the default with the rancher version. plugin "docker" { config { infra_image = "rancher/pause:3.2" } } If you came here from "Alternative Architectures" note above, [return there now](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#alt-arch) . ### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#airgapped-networks-or-proxied-environments) Airgapped networks or proxied environments If you are in an airgapped network or need to access Docker Hub via a proxy, you will have to perform some additional configuration on your Nomad clients to enable Nomad's Consul service mesh integration. #### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#set-the-infra_image-path) Set the "infra\_image" path Set the [`infra_image`](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker#infra_image) configuration option for the Docker driver plugin on your Nomad clients to a path that is accessible in your environment. For example, plugin "docker" { config { infra_image = "dockerhub.myproxy.com:8080/google_containers/pause-amd64:3.0" } } Changing this value will require a restart of the Nomad client agent. #### [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#set-the-sidecar_image-path) Set the "sidecar\_image" path You will also need the Envoy proxy image used for Consul service mesh networking. Configure the sidecar image on your Nomad clients to override the default container path by adding a `"connect.sidecar_image"` value to the [`client.meta`](https://developer.hashicorp.com/nomad/docs/configuration/client#meta) block of your Nomad client configuration. If you do not have a `meta` block inside of your top-level `client` block, add one as follows. client { # ... meta { # Set this value to a proxy or internal registry that can provide an # appropriate envoy image. "connect.sidecar_image" = "dockerhub.myproxy.com:8080/envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09" } # ... } Changing this value will require a restart of the Nomad client agent. Alternately, you can set the value with the [`nomad node meta apply`](https://developer.hashicorp.com/nomad/commands/node/meta/apply) command. [](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#next-steps) Next steps -------------------------------------------------------------------------------------------------------------- Learn more about Nomad's Consul service mesh integration by checking out these resources. * [Nomad Consul Service Mesh documentation](https://developer.hashicorp.com/nomad/docs/networking/consul/service-mesh) * [`consul` block in job specification](https://developer.hashicorp.com/nomad/docs/job-specification/consul) * [`connect` block in job specification](https://developer.hashicorp.com/nomad/docs/job-specification/connect) * [`consul` block in agent configuration](https://developer.hashicorp.com/nomad/docs/configuration/consul) Learn more about Consul service mesh with these tutorials. * [Secure Service-to-Service Communication](https://developer.hashicorp.com/consul/tutorials/developer-mesh/service-mesh-with-envoy-proxy) * [Consul Service Mesh in Production](https://developer.hashicorp.com/consul/tutorials/developer-mesh/service-mesh-production-checklist) Learn more about Consul ACLs. * [Manage Consul ACL Policies](https://developer.hashicorp.com/consul/tutorials/security/access-control-manage-policies) * [Consul Reference Architecture](https://developer.hashicorp.com/consul/tutorials/production-deploy/reference-architecture) **Was this tutorial helpful?** YesNo [Previous\ \ Consul ACL with Nomad WI](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-acl) [Next\ \ Consul in production](https://developer.hashicorp.com/nomad/tutorials/integrate-consul/service-mesh-production-checklist) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.1.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [HashiCorp Learn "Getting Started" collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.1.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.4.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [Getting Started collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.4.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.0.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [HashiCorp Learn "Getting Started" collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.0.x/content/docs/index.mdx) --- # Frequently Asked Questions | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Frequently Asked Questions ========================== [](https://developer.hashicorp.com/nomad/docs/faq#q-what-is-checkpoint-does-nomad-call-home) Q: What is Checkpoint? / Does Nomad call home? ------------------------------------------------------------------------------------------------------------------------------------------- Nomad makes use of a HashiCorp service called [Checkpoint](https://checkpoint.hashicorp.com/) which is used to check for updates and critical security bulletins. Only anonymous information, which cannot be used to identify the user or host, is sent to Checkpoint. An anonymous ID is sent which helps de-duplicate warning messages. This anonymous ID can be disabled. Using the Checkpoint service is optional and can be disabled. See [`disable_anonymous_signature`](https://developer.hashicorp.com/nomad/docs/configuration#disable_anonymous_signature) and [`disable_update_check`](https://developer.hashicorp.com/nomad/docs/configuration#disable_update_check) . [](https://developer.hashicorp.com/nomad/docs/faq#q-is-nomad-eventually-or-strongly-consistent) Q: Is Nomad eventually or strongly consistent? ---------------------------------------------------------------------------------------------------------------------------------------------- Nomad makes use of both a [consensus protocol](https://developer.hashicorp.com/nomad/docs/architecture/cluster/consensus) and a [gossip protocol](https://developer.hashicorp.com/nomad/docs/architecture/security/gossip) . The consensus protocol is strongly consistent, and is used for all state replication and scheduling. The gossip protocol is used to manage the addresses of servers for automatic clustering and multi-region federation. This means all data that is managed by Nomad is strongly consistent. [](https://developer.hashicorp.com/nomad/docs/faq#q-is-nomad-s-datacenter-parameter-the-same-as-consul-s) Q: Is Nomad's `datacenter` parameter the same as Consul's? -------------------------------------------------------------------------------------------------------------------------------------------------------------------- No. For those familiar with Consul, [Consul's notion of a datacenter](https://developer.hashicorp.com/consul/docs/agent/config/config-files#datacenter) is more equivalent to a [Nomad region](https://developer.hashicorp.com/nomad/docs/configuration#datacenter) . Nomad supports grouping nodes into multiple datacenters, which should reflect nodes being colocated, while being managed by a single set of Nomad servers. Consul on the other hand does not have this two-tier approach to servers and agents and instead [relies on federation to create larger logical clusters](https://developer.hashicorp.com/consul/tutorials/networking/federation-gossip-wan) . [](https://developer.hashicorp.com/nomad/docs/faq#q-what-is-bootstrapping-a-nomad-cluster) [](https://developer.hashicorp.com/nomad/docs/faq#) Q: What is "bootstrapping" a Nomad cluster? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Bootstrapping is the process when a Nomad cluster elects its first leader and writes the initial cluster state to that leader's state store. Bootstrapping will not occur until at least a given number of servers, defined by [`bootstrap_expect`](https://developer.hashicorp.com/nomad/docs/configuration/server#bootstrap_expect) , have connected to each other. Once this process has completed, the cluster is said to be bootstrapped and is ready to use. Certain configuration options are only used to influence the creation of the initial cluster state during bootstrapping and are not consulted again so long as the state data remains intact. These typically are values that must be consistent across server members. For example, the [`default_scheduler_config`](https://developer.hashicorp.com/nomad/docs/configuration/server#default_scheduler_config) option allows an operator to set the SchedulerConfig to non-default values during this bootstrap process rather than requiring an immediate call to the API once the cluster is up and running. If the state is completely destroyed, whether intentionally or accidentally, on all of the Nomad servers in the same outage, the cluster will re-bootstrap based on the Nomad defaults and any configuration present that impacts the bootstrap process. [](https://developer.hashicorp.com/nomad/docs/faq#q-how-to-connect-to-my-host-network-when-using-docker-desktop-windows-and-macos) Q: How to connect to my host network when using Docker Desktop (Windows and MacOS)? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Since Docker is based on Linux-native technologies, Docker Desktop for Windows and MacOS uses a small Linux virtual machine to run containers. This extra step adds a layer of indirection between the network of your host (the computer you are currently using) and the network of the VM running your containers. This means that, by default, your Docker tasks will not be able to access endpoints that are available in your host network, such as a local Consul agent. In order to properly setup this connection you will need to explicitly bind the Nomad client to a non-loopback network interface, and anything else you would like to access must also be in the same interface. On Windows, we recommend you to start with the [WSL2 backend for Docker Desktop](https://docs.docker.com/docker-for-windows/wsl/) . Once you are more familiarized with Nomad you can start running it natively. To use the network named `en0` that has the IP address `192.168.0.10`, you can start Nomad with this command. $ sudo nomad agent -dev -bind=0.0.0.0 -network-interface=en0 To start Consul in the same network, you can run this command. $ consul agent -dev -client=0.0.0.0 -bind=192.168.0.10 Now your services will be registered in Consul using the right IP and your tasks will be able to reach each other. To access your tasks from your host machine you will need to use the network interface IP address. $ curl http://192.168.0.10:8080 [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/faq.mdx) --- # Client - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Client HTTP API =============== The `/client` endpoints are used to interact with the Nomad clients. Both clients and servers can handle client endpoints. This is particularly useful for when a direct connection to a client is not possible due to the network configuration. For high volume access to the client endpoints, particularly endpoints streaming file contents, direct access to the node should be preferred as it avoids adding additional load to the servers. When accessing the endpoints via the server, if the desired node is ambiguous based on the URL, an additional `?node_id` query parameter must be provided to disambiguate. The [`/node`](https://developer.hashicorp.com/nomad/api-docs/nodes) endpoints provide node information that are retried from Nomad servers. [](https://developer.hashicorp.com/nomad/api-docs/client#read-dynamic-node-metadata) Read Dynamic Node Metadata --------------------------------------------------------------------------------------------------------------- This endpoint queries Node metadata on a specific Client agent and responds with the following fields: * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/client#meta) `(object)` - The Node metadata that will be registered with the Nomad servers and used by the scheduler (after up to 10 seconds of delay for batching). This is the merged version of the `Static` and `Dynamic` fields. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Static`](https://developer.hashicorp.com/nomad/api-docs/client#static) `(object)` - The Node metadata set in the Client agent's configuration file. Only loaded when an agent starts. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Dynamic`](https://developer.hashicorp.com/nomad/api-docs/client#dynamic) `(object)` - The Node metadata set via the API (see below). Unlike `Meta` and `Static`, this object may contain `null` values to differentiate "unset" keys from keys with an empty string value (`""`). Note that [`/v1/node/:node_id`](https://developer.hashicorp.com/nomad/api-docs/nodes) only contains the `Meta` object. It may take up to 10 seconds for dynamic Node metadata to be sent to Servers and visible through the Node API. Use the Node API to see the version of Node metadata the scheduler uses. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/metadata` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:node_id`](https://developer.hashicorp.com/nomad/api-docs/client#node_id) `(string: )` - Specifies the node to query. This is required when the endpoint is being accessed via a server. Defaults to the node receiving the request otherwise. This is specified as part of the URL. Note, this must be the _full_ node ID, not the short 8-character one. This must be specified as part of the path (`?node_id=...`). ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request) Sample Request $ nomad operator api /v1/client/metadata ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response) Sample Response Formatted by appending `?pretty` above. { "Meta": { "connect.proxy_concurrency": "1", "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.log_level": "debug", "foo": "bar" }, "Dynamic": { "key_to_unset": null, "foo": "bar", "connect.log_level": "debug" }, "Static": { "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.log_level": "info", "connect.proxy_concurrency": "1" } } ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-1) Sample Request [](https://developer.hashicorp.com/nomad/api-docs/client#update-dynamic-node-metadata) Update Dynamic Node Metadata ------------------------------------------------------------------------------------------------------------------- This endpoint updates dynamic Node metadata on a specific Client agent. Since dynamic Node metadata is only periodically synchronized to Nomad Servers, the `Meta` returned in this API may not be reflected in the [`/v1/node/:node_id`](https://developer.hashicorp.com/nomad/api-docs/nodes) API for up to 10 seconds. Scheduling uses the Node API version of `Meta`. For convenience this endpoint returns the same response as a GET. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/client/metadata` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`NodeID`](https://developer.hashicorp.com/nomad/api-docs/client#nodeid) or `:node_id` `(string: )` - Specifies the node to query. This is required when the endpoint is being accessed via a server. Defaults to the node receiving the request otherwise. This is specified as part of the URL. Note, this must be the _full_ node ID, not the short 8-character one. This may be specified as part of the path (`?node_id=...`) or request (`NodeID: "..."`). * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/client#meta-1) `(object: )` - Specifies the Node metadata keys to update. Only specified keys are updated. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [``](https://developer.hashicorp.com/nomad/api-docs/client#) `(string: )` - Specifies a metadata key to update to a particular value. Since `""` is a valid value and distinct from unset, the `null` value is used to mark a key as unset. Keys must be valid dotted HCL identifiers. For example `connect.log_level` is a valid key while `some/path` is not. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-payload) Sample Payload { "Meta": { "connect.log_level": "debug", "key_to_unset": null, "foo": "bar" } } ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-2) Sample Request Assuming the above payload is in a file called `meta.json`. $ nomad operator api /v1/client/metadata < meta.json ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-1) Sample Response Formatted by appending `?pretty` above. { "Meta": { "connect.proxy_concurrency": "1", "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.log_level": "debug", "foo": "bar" }, "Dynamic": { "key_to_unset": null, "foo": "bar", "connect.log_level": "debug" }, "Static": { "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}", "connect.log_level": "info", "connect.proxy_concurrency": "1" } } [](https://developer.hashicorp.com/nomad/api-docs/client#read-stats) Read Stats ------------------------------------------------------------------------------- This endpoint queries the actual resources consumed on a node. The API endpoint is hosted by the Nomad client and requests have to be made to the nomad client whose resource usage metrics are of interest. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/stats` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`node_id`](https://developer.hashicorp.com/nomad/api-docs/client#node_id-1) `(string: )` - Specifies the node to query. This is required when the endpoint is being accessed via a server. This is specified as part of the URL. Note, this must be the _full_ node ID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-3) Sample Request $ nomad operator api /v1/client/stats ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-2) Sample Response { "AllocDirStats": { "Available": 142943150080, "Device": "", "InodesUsedPercent": 0.05312946180421879, "Mountpoint": "", "Size": 249783500800, "Used": 106578206720, "UsedPercent": 42.668233241448746 }, "CPU": [\ {\ "CPU": "cpu0",\ "Idle": 80,\ "System": 11,\ "Total": 20,\ "User": 9\ },\ {\ "CPU": "cpu1",\ "Idle": 99,\ "System": 0,\ "Total": 1,\ "User": 1\ },\ {\ "CPU": "cpu2",\ "Idle": 89,\ "System": 7.000000000000001,\ "Total": 11,\ "User": 4\ },\ {\ "CPU": "cpu3",\ "Idle": 100,\ "System": 0,\ "Total": 0,\ "User": 0\ },\ {\ "CPU": "cpu4",\ "Idle": 92.92929292929293,\ "System": 4.040404040404041,\ "Total": 7.07070707070707,\ "User": 3.0303030303030303\ },\ {\ "CPU": "cpu5",\ "Idle": 99,\ "System": 1,\ "Total": 1,\ "User": 0\ },\ {\ "CPU": "cpu6",\ "Idle": 92.07920792079209,\ "System": 4.9504950495049505,\ "Total": 7.920792079207921,\ "User": 2.9702970297029703\ },\ {\ "CPU": "cpu7",\ "Idle": 99,\ "System": 0,\ "Total": 1,\ "User": 1\ }\ ], "CPUTicksConsumed": 1126.8044804480448, "DeviceStats": [\ {\ "InstanceStats": {\ "6a61929e-d572-092d-5921-156a913f8e56": {\ "Stats": {\ "Attributes": {\ "Used Memory": {\ "Desc": "Memory in use by the device",\ "IntNumeratorVal": 128,\ "Unit": "MiB"\ }\ },\ "Nested": {}\ },\ "Summary": {\ "Desc": "Memory in use by the device",\ "IntNumeratorVal": 128,\ "Unit": "MiB"\ },\ "Timestamp": "2020-12-18T17:15:08.949806Z"\ }\ },\ "Name": "modelA",\ "Type": "skeleton",\ "Vendor": "hashicorp"\ },\ {\ "InstanceStats": {\ "73af5d3e-00f9-0786-9bc1-8f5ffa953f15": {\ "Stats": {\ "Attributes": {\ "Used Memory": {\ "Desc": "Memory in use by the device",\ "IntNumeratorVal": 128,\ "Unit": "MiB"\ }\ },\ "Nested": {}\ },\ "Summary": {\ "Desc": "Memory in use by the device",\ "IntNumeratorVal": 128,\ "Unit": "MiB"\ },\ "Timestamp": "2020-12-18T17:15:08.949806Z"\ }\ },\ "Name": "modelB",\ "Type": "skeleton",\ "Vendor": "hashicorp"\ }\ ], "DiskStats": [\ {\ "Available": 142943150080,\ "Device": "/dev/disk1",\ "InodesUsedPercent": 0.05312946180421879,\ "Mountpoint": "/",\ "Size": 249783500800,\ "Used": 106578206720,\ "UsedPercent": 42.668233241448746\ }\ ], "Memory": { "Available": 6232244224, "Free": 470618112, "Total": 17179869184, "Used": 10947624960 }, "Timestamp": 1495743032992498200, "Uptime": 193520 } [](https://developer.hashicorp.com/nomad/api-docs/client#read-allocation-statistics) Read Allocation Statistics --------------------------------------------------------------------------------------------------------------- The client `allocation` endpoint is used to query the actual resources consumed by an allocation. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/allocation/:alloc_id/stats` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-4) Sample Request $ nomad operator api \ /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/stats ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-3) Sample Response { "ResourceUsage": { "CpuStats": { "Measured": ["Throttled Periods", "Throttled Time", "Percent"], "Percent": 0.14159538847117795, "SystemMode": 0, "ThrottledPeriods": 0, "ThrottledTime": 0, "TotalTicks": 3.256693934837093, "UserMode": 0 }, "MemoryStats": { "Cache": 1744896, "KernelMaxUsage": 0, "KernelUsage": 0, "MaxUsage": 4710400, "Measured": ["RSS", "Cache", "Swap", "Max Usage"], "RSS": 1486848, "Swap": 0 } }, "Tasks": { "redis": { "Pids": null, "ResourceUsage": { "CpuStats": { "Measured": ["Throttled Periods", "Throttled Time", "Percent"], "Percent": 0.14159538847117795, "SystemMode": 0, "ThrottledPeriods": 0, "ThrottledTime": 0, "TotalTicks": 3.256693934837093, "UserMode": 0 }, "MemoryStats": { "Cache": 1744896, "KernelMaxUsage": 0, "KernelUsage": 0, "MaxUsage": 4710400, "Measured": ["RSS", "Cache", "Swap", "Max Usage"], "RSS": 1486848, "Swap": 0 } }, "Timestamp": 1495743243970720000 } }, "Timestamp": 1495743243970720000 } [](https://developer.hashicorp.com/nomad/api-docs/client#read-file) Read File ----------------------------------------------------------------------------- This endpoint reads the contents of a file in an allocation directory. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/cat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-4) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-1) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`path`](https://developer.hashicorp.com/nomad/api-docs/client#path) `(string: "/")` - Specifies the path of the file to read, relative to the root of the allocation directory. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-5) Sample Request $ nomad operator api \ /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99 $ nomad operator api \ /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=alloc/file.json ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-4) Sample Response (whatever was in the file...) [](https://developer.hashicorp.com/nomad/api-docs/client#read-file-at-offset) Read File at Offset ------------------------------------------------------------------------------------------------- This endpoint reads the contents of a file in an allocation directory at a particular offset and limit. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/readat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-5) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-2) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`path`](https://developer.hashicorp.com/nomad/api-docs/client#path-1) `(string: "/")` - Specifies the path of the file to read, relative to the root of the allocation directory. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`offset`](https://developer.hashicorp.com/nomad/api-docs/client#offset) `(int: )` - Specifies the byte offset from where content will be read. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`limit`](https://developer.hashicorp.com/nomad/api-docs/client#limit) `(int: )` - Specifies the number of bytes to read from the offset. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-6) Sample Request $ nomad operator api \ /v1/client/fs/readat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/foo&offset=1323&limit=19303 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-5) Sample Response (whatever was in the file, starting from offset, up to limit bytes...) [](https://developer.hashicorp.com/nomad/api-docs/client#stream-file) Stream File --------------------------------------------------------------------------------- This endpoint streams the contents of a file in an allocation directory. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/stream/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-6) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-3) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`path`](https://developer.hashicorp.com/nomad/api-docs/client#path-2) `(string: "/")` - Specifies the path of the file to read, relative to the root of the allocation directory. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`follow`](https://developer.hashicorp.com/nomad/api-docs/client#follow) `(bool: true)`\- Specifies whether to tail the file. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`offset`](https://developer.hashicorp.com/nomad/api-docs/client#offset-1) `(int: )` - Specifies the byte offset from where content will be read. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`origin`](https://developer.hashicorp.com/nomad/api-docs/client#origin) `(string: "start|end")` - Applies the relative offset to either the start or end of the file. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-7) Sample Request $ nomad operator api \ /v1/client/fs/stream/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/logs/redis.log ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-6) Sample Response ({ "File": "alloc/logs/redis.log", "Offset": 3604480, "Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..." }, { "File": "alloc/logs/redis.log", "FileEvent": "file deleted" }) #### [](https://developer.hashicorp.com/nomad/api-docs/client#field-reference) Field Reference The return value is a stream of frames. These frames contain the following fields: * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Data`](https://developer.hashicorp.com/nomad/api-docs/client#data) - A base64 encoding of the bytes being streamed. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`FileEvent`](https://developer.hashicorp.com/nomad/api-docs/client#fileevent) - An event that could cause a change in the streams position. The possible values are "file deleted" and "file truncated". * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Offset`](https://developer.hashicorp.com/nomad/api-docs/client#offset-2) - Offset is the offset into the stream. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`File`](https://developer.hashicorp.com/nomad/api-docs/client#file) - The name of the file being streamed. [](https://developer.hashicorp.com/nomad/api-docs/client#stream-logs) Stream Logs --------------------------------------------------------------------------------- This endpoint streams a task's stderr/stdout logs. Note that if logging is set to [disabled=true](https://developer.hashicorp.com/nomad/docs/job-specification/logs#disabled) for the task, this endpoint will return a 404 error. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/logs/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-logs` or `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-7) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-4) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`task`](https://developer.hashicorp.com/nomad/api-docs/client#task) `(string: )` - Specifies the name of the task inside the allocation to stream logs from. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`follow`](https://developer.hashicorp.com/nomad/api-docs/client#follow-1) `(bool: false)`\- Specifies whether to tail the logs. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`type`](https://developer.hashicorp.com/nomad/api-docs/client#type) `(string: "stderr|stdout")` - Specifies the stream to stream. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`offset`](https://developer.hashicorp.com/nomad/api-docs/client#offset-3) `(int: 0)` - Specifies the offset to start streaming from. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`origin`](https://developer.hashicorp.com/nomad/api-docs/client#origin-1) `(string: "start|end")` - Specifies either "start" or "end" and applies the offset relative to either the start or end of the logs respectively. Defaults to "start". * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`plain`](https://developer.hashicorp.com/nomad/api-docs/client#plain) `(bool: false)` - Return just the plain text without framing. This can be useful when viewing logs in a browser. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-8) Sample Request $ nomad operator api \ /v1/client/fs/logs/5fc98185-17ff-26bc-a802-0c74fa471c99 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-7) Sample Response ({ "File": "alloc/logs/redis.stdout.0", "Offset": 3604480, "Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..." }, { "File": "alloc/logs/redis.stdout.0", "FileEvent": "file deleted" }) #### [](https://developer.hashicorp.com/nomad/api-docs/client#field-reference-1) Field Reference The return value is a stream of frames. These frames contain the following fields: * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Data`](https://developer.hashicorp.com/nomad/api-docs/client#data-1) - A base64 encoding of the bytes being streamed. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`FileEvent`](https://developer.hashicorp.com/nomad/api-docs/client#fileevent-1) - An event that could cause a change in the streams position. The possible values are "file deleted" and "file truncated". * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`Offset`](https://developer.hashicorp.com/nomad/api-docs/client#offset-4) - Offset is the offset into the stream. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`File`](https://developer.hashicorp.com/nomad/api-docs/client#file-1) - The name of the file being streamed. [](https://developer.hashicorp.com/nomad/api-docs/client#list-files) List Files ------------------------------------------------------------------------------- This endpoint lists files in an allocation directory. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/ls/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-8) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-5) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`path`](https://developer.hashicorp.com/nomad/api-docs/client#path-3) `(string: "/")` - Specifies the path of the file to read, relative to the root of the allocation directory. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-9) Sample Request $ nomad operator api \ /v1/client/fs/ls/5fc98185-17ff-26bc-a802-0c74fa471c99 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-8) Sample Response [\ {\ "Name": "alloc",\ "IsDir": true,\ "Size": 4096,\ "FileMode": "drwxrwxr-x",\ "ModTime": "2016-03-15T15:40:00.414236712-07:00"\ },\ {\ "Name": "redis",\ "IsDir": true,\ "Size": 4096,\ "FileMode": "drwxrwxr-x",\ "ModTime": "2016-03-15T15:40:56.810238153-07:00"\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/client#stat-file) Stat File ----------------------------------------------------------------------------- This endpoint stats a file in an allocation. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/fs/stat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-fs` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-9) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-6) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`path`](https://developer.hashicorp.com/nomad/api-docs/client#path-4) `(string: "/")` - Specifies the path of the file to read, relative to the root of the allocation directory. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-10) Sample Request $ nomad operator api \ /v1/client/fs/stat/5fc98185-17ff-26bc-a802-0c74fa471c99 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-9) Sample Response { "Name": "redis-syslog-collector.out", "IsDir": false, "Size": 96, "FileMode": "-rw-rw-r--", "ModTime": "2016-03-15T15:40:56.822238153-07:00" } [](https://developer.hashicorp.com/nomad/api-docs/client#gc-allocation) GC Allocation ------------------------------------------------------------------------------------- This endpoint forces a garbage collection of a particular, stopped allocation on a node. Note that the allocation will still exist on the server and appear in server responses. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/allocation/:alloc_id/gc` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-10) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:alloc_id`](https://developer.hashicorp.com/nomad/api-docs/client#alloc_id-7) `(string: )` - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the _full_ allocation ID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-11) Sample Request $ nomad operator api \ /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/gc [](https://developer.hashicorp.com/nomad/api-docs/client#gc-all-allocation) GC All Allocation --------------------------------------------------------------------------------------------- This endpoint forces a garbage collection of all stopped allocations on a node. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/gc` | `text/plain` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-11) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`node_id`](https://developer.hashicorp.com/nomad/api-docs/client#node_id-2) `(string: )` - Specifies the node to target. This is required when the endpoint is being accessed via a server. This is specified as part of the URL. Note, this must be the _full_ node ID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-12) Sample Request $ nomad operator api /v1/client/gc [](https://developer.hashicorp.com/nomad/api-docs/client#read-a-node-s-identity-claims) Read a Node's Identity Claims --------------------------------------------------------------------------------------------------------------------- This endpoint reads the identity claims for a node. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/identity` | `application/json` | This table shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-12) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`:node_id`](https://developer.hashicorp.com/nomad/api-docs/client#node_id-3) `(string: )` - Specifies the node to target. This is required when the endpoint is being accessed via a server that is specified as part of the path (`?node_id=...`). Note, this must be the full node ID, not the short 8-character one. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-13) Sample Request $ nomad operator api \ /v1/client/identity?node_id=c172799d-1592-06c9-ffc8-308ed12f8080 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-10) Sample Response { "Claims": { "aud": "nomadproject.io", "exp": 1758032770, "jti": "a881c528-64f5-4fdd-70cb-c759c3427bb8", "nomad_node_datacenter": "dc1", "nomad_node_id": "c172799d-1592-06c9-ffc8-308ed12f8080", "nomad_node_pool": "default", "iat": 1757946370, "nbf": 1757946370, "sub": "node:global:default:c172799d-1592-06c9-ffc8-308ed12f8080:default" } } [](https://developer.hashicorp.com/nomad/api-docs/client#renew-a-node-s-identity) Renew a Node's Identity --------------------------------------------------------------------------------------------------------- This endpoint instructs a node to renew its identity at the next heartbeat. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/client/identity/renew` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `node:write` | ### [](https://developer.hashicorp.com/nomad/api-docs/client#parameters-13) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/client#) [`NodeID`](https://developer.hashicorp.com/nomad/api-docs/client#nodeid-1) or `:node_id` `(string: )` - Specifies the node to target. This is required when the endpoint is being accessed via a server. This may be specified as part of the path (`?node_id=...`) or request body (`NodeID: "..."`), with the query parameter taking precedence when both are provided. Note, this must be the full node ID, not the short 8-character one. ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-request-14) Sample Request $ nomad operator api \ -X POST \ /v1/client/identity/renew?node_id=c172799d-1592-06c9-ffc8-308ed12f8080 ### [](https://developer.hashicorp.com/nomad/api-docs/client#sample-response-11) Sample Response {} [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/client.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.2.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [HashiCorp Learn "Getting Started" collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.2.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.8.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [Getting Started collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.8.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.3.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [HashiCorp Learn "Getting Started" collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.3.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.5.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [Getting Started collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.5.x/content/docs/index.mdx) --- # Deployments - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Deployments HTTP API ==================== The `/deployment` endpoints are used to query for and interact with deployments. [](https://developer.hashicorp.com/nomad/api-docs/deployments#list-deployments) List Deployments ------------------------------------------------------------------------------------------------ This endpoint lists all deployments. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/deployments` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`prefix`](https://developer.hashicorp.com/nomad/api-docs/deployments#prefix) `(string: "")`\- Specifies a string to filter deployments based on an ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters (0-9a-f) .This is specified as a query string parameter and is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/deployments#namespace) `(string: "default")` - Specifies the target namespace. Specifying `*` will return all evaluations across all authorized namespaces. This parameter is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`next_token`](https://developer.hashicorp.com/nomad/api-docs/deployments#next_token) `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which is the `ID` field of the next expected deployment. This value can be obtained from the `X-Nomad-NextToken` header from the previous response. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`per_page`](https://developer.hashicorp.com/nomad/api-docs/deployments#per_page) `(int: 0)` - Specifies a maximum number of deployments to return for this request. If omitted, the response is not paginated. The `ID` of the last deployment in the response can be used as the `last_token` of the next request to fetch additional pages. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`filter`](https://developer.hashicorp.com/nomad/api-docs/deployments#filter) `(string: "")` - Specifies the [expression](https://developer.hashicorp.com/nomad/api-docs#filtering) used to filter the results. Consider using pagination or a query parameter to reduce resource used to serve the request. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`reverse`](https://developer.hashicorp.com/nomad/api-docs/deployments#reverse) `(bool: false)` - Specifies the list of returned deployments should be sorted in the reverse order. By default deployments are returned sorted in chronological order (older deployments first), or in lexicographical order by their ID if the `prefix` query parameter is used. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request) Sample Request $ curl \ https://localhost:4646/v1/deployments $ curl \ https://localhost:4646/v1/deployments?prefix=25ba81c ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response) Sample Response [\ {\ "ID": "70638f62-5c19-193e-30d6-f9d6e689ab8e",\ "JobID": "example",\ "JobVersion": 1,\ "JobModifyIndex": 17,\ "JobSpecModifyIndex": 17,\ "JobCreateIndex": 7,\ "TaskGroups": {\ "cache": {\ "Promoted": false,\ "DesiredCanaries": 1,\ "DesiredTotal": 3,\ "PlacedAllocs": 1,\ "HealthyAllocs": 0,\ "UnhealthyAllocs": 0\ }\ },\ "Status": "running",\ "StatusDescription": "",\ "CreateIndex": 19,\ "ModifyIndex": 19\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/deployments#read-deployment) Read Deployment ---------------------------------------------------------------------------------------------- This endpoint reads information about a specific deployment by ID. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/deployment/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/deployment/70638f62-5c19-193e-30d6-f9d6e689ab8e ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-1) Sample Response { "ID": "70638f62-5c19-193e-30d6-f9d6e689ab8e", "JobID": "example", "JobVersion": 1, "JobModifyIndex": 17, "JobSpecModifyIndex": 17, "JobCreateIndex": 7, "TaskGroups": { "cache": { "Promoted": false, "DesiredCanaries": 1, "DesiredTotal": 3, "PlacedAllocs": 1, "HealthyAllocs": 0, "UnhealthyAllocs": 0 } }, "Status": "running", "StatusDescription": "", "CreateIndex": 19, "ModifyIndex": 19 } [](https://developer.hashicorp.com/nomad/api-docs/deployments#list-allocations-for-deployment) List Allocations for Deployment ------------------------------------------------------------------------------------------------------------------------------ This endpoint lists the allocations created or modified for the given deployment. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/deployment/allocations/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-1) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-2) Sample Request $ curl \ https://localhost:4646/v1/deployment/allocations/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-2) Sample Response [\ {\ "ID": "287b65cc-6c25-cea9-0332-e4a75ca2af98",\ "EvalID": "9751cb74-1a0d-190e-d026-ad2bc666ad2c",\ "Name": "example.cache[0]",\ "NodeID": "cb1f6030-a220-4f92-57dc-7baaabdc3823",\ "JobID": "example",\ "TaskGroup": "cache",\ "DesiredStatus": "run",\ "DesiredDescription": "",\ "ClientStatus": "running",\ "ClientDescription": "",\ "TaskStates": {\ "redis": {\ "State": "running",\ "Failed": false,\ "StartedAt": "2017-06-29T22:29:41.52000268Z",\ "FinishedAt": "0001-01-01T00:00:00Z",\ "Events": [\ {\ "Type": "Received",\ "Time": 1498775380693307400,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ },\ {\ "Type": "Task Setup",\ "Time": 1498775380693659000,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "Building Task Directory",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ },\ {\ "Type": "Started",\ "Time": 1498775381508493800,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ }\ ]\ }\ },\ "DeploymentStatus": null,\ "CreateIndex": 19,\ "ModifyIndex": 22,\ "CreateTime": 1498775380678486300,\ "ModifyTime": 1498775380678486300\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/deployments#fail-deployment) Fail Deployment ---------------------------------------------------------------------------------------------- This endpoint is used to mark a deployment as failed. This should be done to force the scheduler to stop creating allocations as part of the deployment or to cause a rollback to a previous job version. This endpoint only triggers a rollback if the most recent stable version of the job has a different specification than the job being reverted. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/deployment/fail/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-2) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-3) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/deployment/fail/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-3) Sample Response { "EvalID": "0d834913-58a0-81ac-6e33-e452d83a0c66", "EvalCreateIndex": 20, "DeploymentModifyIndex": 20, "RevertedJobVersion": 1, "Index": 20 } [](https://developer.hashicorp.com/nomad/api-docs/deployments#pause-deployment) Pause Deployment ------------------------------------------------------------------------------------------------ This endpoint is used to pause or unpause a deployment. This is done to pause a rolling upgrade or resume it. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/deployment/pause/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-4) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-3) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path and in the JSON payload. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`Pause`](https://developer.hashicorp.com/nomad/api-docs/deployments#pause) `(bool: false)` - Specifies whether to pause or resume the deployment. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-payload) Sample Payload { "DeploymentID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "Pause": true } ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-4) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/deployment/pause/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-4) Sample Response { "EvalID": "0d834913-58a0-81ac-6e33-e452d83a0c66", "EvalCreateIndex": 20, "DeploymentModifyIndex": 20, "Index": 20 } [](https://developer.hashicorp.com/nomad/api-docs/deployments#promote-deployment) Promote Deployment ---------------------------------------------------------------------------------------------------- This endpoint is used to promote task groups that have canaries for a deployment. This should be done when the placed canaries are healthy and the rolling upgrade of the remaining allocations should begin. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/deployment/promote/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-5) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-4) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path and JSON payload. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`All`](https://developer.hashicorp.com/nomad/api-docs/deployments#all) `(bool: false)` - Specifies whether all task groups should be promoted. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`Groups`](https://developer.hashicorp.com/nomad/api-docs/deployments#groups) `(array: nil)` - Specifies a particular set of task groups that should be promoted. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-payload-1) Sample Payload { "DeploymentID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "All": true } { "DeploymentID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "Groups": ["web", "api-server"] } ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-5) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/deployment/promote/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-5) Sample Response { "EvalID": "0d834913-58a0-81ac-6e33-e452d83a0c66", "EvalCreateIndex": 20, "DeploymentModifyIndex": 20, "Index": 20 } [](https://developer.hashicorp.com/nomad/api-docs/deployments#set-allocation-health-in-deployment) Set Allocation Health in Deployment -------------------------------------------------------------------------------------------------------------------------------------- This endpoint is used to set the health of an allocation that is in the deployment manually. In some use cases, automatic detection of allocation health may not be desired. As such those task groups can be marked with an upgrade policy that uses `health_check = "manual"`. Those allocations must have their health marked manually using this endpoint. Marking an allocation as healthy will allow the rolling upgrade to proceed. Marking it as failed will cause the deployment to fail. This endpoint only triggers a rollback if the most recent stable version of the job has a different specification than the job being reverted. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/deployment/allocation-health/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-6) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-5) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path and the JSON payload. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`HealthyAllocationIDs`](https://developer.hashicorp.com/nomad/api-docs/deployments#healthyallocationids) `(array: nil)` - Specifies the set of allocation that should be marked as healthy. * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`UnhealthyAllocationIDs`](https://developer.hashicorp.com/nomad/api-docs/deployments#unhealthyallocationids) `(array: nil)` - Specifies the set of allocation that should be marked as unhealthy. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-payload-2) Sample Payload { "DeploymentID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577", "HealthyAllocationIDs": [\ "eb13bc8a-7300-56f3-14c0-d4ad115ec3f5",\ "6584dad8-7ae3-360f-3069-0b4309711cc1"\ ] } ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-6) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/deployment/allocation-health/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-6) Sample Response { "EvalID": "0d834913-58a0-81ac-6e33-e452d83a0c66", "EvalCreateIndex": 20, "DeploymentModifyIndex": 20, "RevertedJobVersion": 1, "Index": 20 } [](https://developer.hashicorp.com/nomad/api-docs/deployments#unblock-deployment) Unblock Deployment ---------------------------------------------------------------------------------------------------- This endpoint is used to manually mark a blocked multiregion deployment as successful. A blocked deployment is a multiregion deployment within a region that has completed within a region but is waiting on the other federated regions. The endpoint can be used in cases where a failed peer region is unable to communicate its failed deployment status to other regions to force a deployment to complete. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/deployment/unblock/:deployment_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#parameters-7) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/deployments#) [`:deployment_id`](https://developer.hashicorp.com/nomad/api-docs/deployments#deployment_id-6) `(string: )`\- Specifies the UUID of the deployment. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-request-7) Sample Request $ curl \ --request POST \ https://localhost:4646/v1/deployment/unblock/5456bd7a-9fc0-c0dd-6131-cbee77f57577 ### [](https://developer.hashicorp.com/nomad/api-docs/deployments#sample-response-7) Sample Response { "EvalID": "0d834913-58a0-81ac-6e33-e452d83a0c66", "EvalCreateIndex": 20, "DeploymentModifyIndex": 20, "Index": 20 } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/deployments.mdx) --- # Install Nomad | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Install Nomad ============= Nomad is available as a pre-compiled binary or as a package for several operating systems. You can also [build Nomad from source](https://developer.hashicorp.com/nomad/docs/deploy#from-source) . If you are interested in trialing Nomad without installing it locally, see the [Quickstart](https://developer.hashicorp.com/nomad/docs/quickstart) for options to get started with Nomad. LinuxMacWindows Ubuntu/DebianRHEL/CentOSFedoraAmazon LinuxManual Install the required packages. $ sudo apt-get update && \ sudo apt-get install wget gpg coreutils Add the HashiCorp [GPG key](https://apt.releases.hashicorp.com/gpg "HashiCorp GPG key") . $ wget -O- https://apt.releases.hashicorp.com/gpg | \ sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg Add the official HashiCorp Linux repository. $ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \ | sudo tee /etc/apt/sources.list.d/hashicorp.list Update and install. $ sudo apt-get update && sudo apt-get install nomad Install `yum-config-manager` to manage your repositories. $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. $ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo Install. $ sudo yum -y install nomad Install `dnf config-manager` to manage your repositories. $ sudo dnf install -y dnf-plugins-core Use `dnf config-manager` to add the official HashiCorp Linux repository. $ sudo dnf config-manager addrepo --from-repofile=https://rpm.releases.hashicorp.com/fedora/hashicorp.repo Install. $ sudo dnf -y install nomad Install `yum-config-manager` to manage your repositories. $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. $ sudo yum-config-manager \ --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo Install. $ sudo yum -y install nomad Download a [precompiled binary](https://releases.hashicorp.com/nomad/) , verify the binary using the available SHA-256 sums, and unzip the package to a location on your machine. Make sure that the location of the `nomad` binary is available on your `PATH` before continuing with the other guides. HomebrewManual [Homebrew](https://brew.sh/) is a free and open source package management system for Mac OS X. Install the official [Nomad formula](https://github.com/hashicorp/homebrew-tap) from the terminal. First, install the HashiCorp tap, a repository of all of the HashiCorp Homebrew packages. $ brew tap hashicorp/tap Now, install Nomad with `hashicorp/tap/nomad`. $ brew install hashicorp/tap/nomad This installs a signed binary and is automatically updated with every new official release. To update to the latest, run $ brew upgrade hashicorp/tap/nomad Download a [precompiled binary](https://releases.hashicorp.com/nomad/) , verify the binary using the available SHA-256 sums, and unzip the package to a location on your machine. Make sure that the location of the `nomad` binary is available on your `PATH` before continuing with the other guides. ChocolateyManual [Chocolatey](https://chocolatey.org/) is a free and open-source package management system for Windows. Install the [Nomad package](https://chocolatey.org/packages/nomad) from the command-line. $ choco install nomad Chocolatey and the Nomad package are **NOT** directly maintained by HashiCorp. The latest version of Nomad is always available by manual installation. Download a [precompiled binary](https://releases.hashicorp.com/nomad/) , verify the binary using the available SHA-256 sums, and unzip the package to a location on your machine. Make sure that the location of the `nomad` binary is available on your `PATH` before continuing with the other guides. [](https://developer.hashicorp.com/nomad/docs/deploy#linux-post-installation-steps) Linux post-installation steps ----------------------------------------------------------------------------------------------------------------- These steps are optional but can be helpful for running Nomad and to take advantage of additional Nomad functionalities. You need to run client agents as root (or with `sudo`) so that cpuset accounting and network namespaces work correctly. ### [](https://developer.hashicorp.com/nomad/docs/deploy#install-cni-reference-plugins) Install CNI reference plugins Nomad uses CNI plugins to configure network namespaces when using the `bridge` network mode. You must install the CNI plugins on all Linux Nomad client nodes that use network namespaces. Refer to the [CNI Plugins external guide](https://www.cni.dev/plugins/current/) for details on individual plugins. The following series of commands determines your operating system architecture, downloads the [CNI 1.6.2 release](https://github.com/containernetworking/plugins/releases/tag/v1.6.2) , and then extracts the CNI plugin binaries into the `/opt/cni/bin` directory. Update the `CNI_PLUGIN_VERSION` value to use a different release version. $ export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64) $ export CNI_PLUGIN_VERSION=v1.6.2 $ curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-linux-${ARCH_CNI}-${CNI_PLUGIN_VERSION}".tgz && \ sudo mkdir -p /opt/cni/bin && \ sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz Your Linux distribution's package manager may provide the CNI reference plugins but we recommend installing the most recent stable version to ensure you have fixes for known bugs shipping in those versions. Nomad looks for CNI plugin binaries by default in the `/opt/cni/bin` directory. However, you may install in the binaries in a different directory and then configure using the [`cni_path`](https://developer.hashicorp.com/nomad/docs/configuration/client#cni_path) attribute. ### [](https://developer.hashicorp.com/nomad/docs/deploy#install-consul-cni-plugin) Install consul-cni plugin When you use the [`transparent_proxy` block](https://developer.hashicorp.com/nomad/docs/job-specification/transparent_proxy) for Consul service mesh, you must also install the [`consul-cni` plugin](https://releases.hashicorp.com/consul-cni) on each client node for Consul to properly redirect inbound and outbound traffic for services to the Envoy proxy. For more information, refer to [Enable the Consul CNI plugin](https://developer.hashicorp.com/consul/docs/k8s/connect/transparent-proxy/enable-transparent-proxy#enable-the-consul-cni-plugin) in the Consul documentation. You must install the CNI plugins before you install the Consul CNI plugin. The following commands assume that you already installed the CNI plugins. Install the `consul-cni` plugin on each client node. Ubuntu/DebianRHEL/CentOSFedoraAmazon LinuxManual $ sudo apt-get install -y consul-cni $ sudo yum -y install consul-cni $ sudo dnf -y install consul-cni $ sudo yum -y install consul-cni $ export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64) $ curl -L -o consul-cni.zip "https://releases.hashicorp.com/consul-cni/1.5.1/consul-cni_1.5.1_linux_${ARCH_CNI}".zip && \ sudo unzip consul-cni.zip -d /opt/cni/bin -x LICENSE.txt ### [](https://developer.hashicorp.com/nomad/docs/deploy#install-dmidecode) Install dmidecode When running on a virtualized host such as Amazon EC2, Nomad makes use of the `dmidecode` tool to detect CPU performance data. Some Linux distributions require installing the dmidecode package manually. ### [](https://developer.hashicorp.com/nomad/docs/deploy#configure-bridge-network-to-route-traffic-through-iptables) Configure bridge network to route traffic through iptables Nomad's task group networks integrate with Consul's service mesh using bridge networking and iptables to send traffic between containers. **Warning:** New Linux versions, such as Ubuntu 24.04, may not enable bridge networking by default. Use `sudo modprobe bridge` to load the bridge module if it is missing. The Linux kernel bridge module has three tunable parameters that control whether iptables processes traffic crossing the bridge. Some operating systems, including RedHat, CentOS, and Fedora, might have iptables rules that are not correctly configured for guest traffic because these tunable parameters are optimized for VM workloads. Ensure your Linux operating system distribution is configured to allow iptables to route container traffic through the bridge network. Run the following commands to set the tunable parameters to allow iptables processing for the bridge network. $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables To preserve these settings on startup of a client node, add a file to `/etc/sysctl.d/` or remove the file your Linux distribution puts in that directory. The following example configures the tunable parameters for a client node. /etc/sysctl.d/bridge.conf net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 ### [](https://developer.hashicorp.com/nomad/docs/deploy#verify-cgroup-controllers) Verify cgroup controllers On Linux, Nomad uses cgroups to control access to resources like CPU and memory. Nomad supports both [cgroups v2](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) and the legacy [cgroups v1](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cgroups.html) . When Nomad clients start, they determine the available cgroup controllers and include the attribute `os.cgroups.version` in their fingerprint. On cgroups v2, you can run the following command to verify that you have all required controllers. $ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory pids On legacy cgroups v1, this same list of required controllers appears as a series of sub-directories under the directory `/sys/fs/cgroup`. Refer to the [cgroup controller requirements](https://developer.hashicorp.com/nomad/docs/deploy/production/requirements#cgroup-controllers) for more details and to enable missing cgroups. [](https://developer.hashicorp.com/nomad/docs/deploy#verify-the-installation) Verify the installation ----------------------------------------------------------------------------------------------------- To verify Nomad was installed correctly, try the `nomad` command. $ nomad You should see help output, similar to the following. Usage: nomad [-version] [-help] [-autocomplete-(un)install] [args] Common commands: run Run a new job or update an existing job stop Stop a running job status Display the status output for a resource alloc Interact with allocations job Interact with jobs node Interact with nodes agent Runs a Nomad agent Other commands: acl Interact with ACL policies and tokens agent-info Display status information about the local agent config Interact with configurations deployment Interact with deployments eval Interact with evaluations exec Execute commands in task fmt Rewrites Nomad config and job files to canonical format license Interact with Nomad Enterprise License login Login to Nomad using an auth method monitor Stream logs from a Nomad agent namespace Interact with namespaces operator Provides cluster-level tools for Nomad operators plugin Inspect plugins quota Interact with quotas recommendation Interact with the Nomad recommendation endpoint scaling Interact with the Nomad scaling endpoint sentinel Interact with Sentinel policies server Interact with servers service Interact with registered services system Interact with the system API tls Generate Self Signed TLS Certificates for Nomad ui Open the Nomad Web UI var Interact with variables version Prints the Nomad version volume Interact with volumes * * * [](https://developer.hashicorp.com/nomad/docs/deploy#compiling-from-source) [](https://developer.hashicorp.com/nomad/docs/deploy#) Compiling from source -------------------------------------------------------------------------------------------------------------------------------------------------------- To compile from source, you will need [Go](https://golang.org/) installed at the version described by the [.go-version](https://github.com/hashicorp/nomad/blob/main/.go-version) file. You should properly configure your Go environment, including setting a `GOPATH` environment variable and ensuring `GOPATH/bin` is within your `PATH`. A copy of [`git`](https://www.git-scm.com/) is also needed in your `PATH`. 1. Clone the Nomad repository from GitHub into your `GOPATH`: $ mkdir -p $GOPATH/src/github.com/hashicorp && cd $_ $ git clone https://github.com/hashicorp/nomad.git $ cd nomad 2. Bootstrap the project. This will download and compile libraries and tools needed to compile Nomad: $ make bootstrap 3. Build Nomad for your current system and put the binary in `./bin/` (relative to the git checkout). The `make dev` target is just a shortcut that builds `nomad` for only your local build environment (no cross-compiled targets). $ make dev [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/deploy/index.mdx) --- # Use Dynamic Application Sizing | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Enterprise](https://developer.hashicorp.com/nomad/tutorials/enterprise) Enterprise Only The functionality described here is available only in [Nomad Enterprise](https://www.hashicorp.com/products/nomad/pricing/) with the Multi-Cluster & Efficiency module. To explore Nomad Enterprise features, you can sign up for a free 30-day trial from [here](https://www.hashicorp.com/products/nomad/trial) . Prometheus Required Currently, Prometheus is the only APM supported for Dynamic Application Sizing Using a Vagrant virtual machine, you will deploy a simple environment containing: * An APM, specifically Prometheus, to collect metric data. * Nomad Autoscaler Enterprise. * A sample job, which will be configured to enable DAS recommendations with: * one NGINX instance used as a TCP load balancer. * three Redis instances to service requests. * A sample dispatch job to create load on the Redis nodes. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#prerequisites) Prerequisites --------------------------------------------------------------------------------------------------------------------- Familiarity with the [Dynamic application scaling concepts](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts) tutorial. This [Vagrantfile](https://raw.githubusercontent.com/hashicorp/nomad-education-content/main/nomad-das-vagrant/Vagrantfile) to create a suitable environment to run the demonstration. This Vagrantfile provisions: * one Ubuntu 20.04 VM preinstalled with: * Nomad Enterprise v1.0.0 beta 2 * The current version of Consul installable via package * The current version of Docker installable via package [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#start-and-connect-to-the-vagrant-environment) Start and connect to the Vagrant environment ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Download the [Vagrantfile](https://raw.githubusercontent.com/hashicorp/nomad-education-content/main/nomad-das-vagrant/Vagrantfile) . Start the test-drive environment by running `vagrant up`. $ vagrant up Once the environment is provisioned and you are returned to your command prompt, connect to the Vagrant instance. $ vagrant ssh Once you are at the `vagrant@ubuntu-focal:~$` prompt, you are ready to continue. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#verify-nomad-telemetry-configuration) Verify Nomad telemetry configuration ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nomad needs to be configured to enable telemetry publishing. You need to enable allocation and node metrics. Since this tutorial also uses Prometheus as its APM, you need to set `prometheus_metrics` to true. The configuration for the Nomad inside the test-drive already has the appropriate telemetry configuration. View the configuration using `cat /etc/nomad.d/nomad.hcl` file and note the following stanza is included. telemetry { publish_allocation_metrics = true publish_node_metrics = true prometheus_metrics = true } Given this configuration, Nomad generates node and allocation metrics and make them available in a format that Prometheus can consume. If you are using this test-drive with your own Nomad cluster, add this telemetry block to the configuration for every Nomad node in your cluster and restart them to load the new configuration. Return to the vagrant user's home directory if you changed away from it. $ cd /home/vagrant [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#start-prometheus) Start Prometheus --------------------------------------------------------------------------------------------------------------------------- The autoscaler configuration in this test-drive uses Prometheus to retrieve historical metrics when starting to track a new target. In this beta, Prometheus is also used for ongoing monitoring metrics, but this is currently being shifted to using Nomad's metrics API. The first step is to run an instance of Prometheus for the Nomad Autoscaler to use. The simplest way to do this is to run Prometheus as a Nomad job. The environment contains a complete Prometheus job file to get started with. You can create a file called `prometheus.nomad` with the following content, or you can copy `prometheus.nomad` from the `~/nomad-autoscaler/jobs` folder when logged into a vagrant user's shell inside the VM. job "prometheus" { datacenters = ["dc1"] group "prometheus" { count = 1 network { port "prometheus_ui" { static = 9090 } } task "prometheus" { driver = "docker" config { image = "prom/prometheus:v2.18.1" args = [\ "--config.file=/etc/prometheus/config/prometheus.yml",\ "--storage.tsdb.path=/prometheus",\ "--web.console.libraries=/usr/share/prometheus/console_libraries",\ "--web.console.templates=/usr/share/prometheus/consoles",\ ] volumes = [\ "local/config:/etc/prometheus/config",\ ] ports = ["prometheus_ui"] } template { data = < Created = 6m55s ago Modified = 6m36s ago Deployment ID = c3ee5e5d Deployment Health = healthy Allocation Addresses Label Dynamic Address *db yes 10.0.2.15:25465 -> 6379 Task "redis" is "running" Task Resources CPU Memory Disk Addresses 3/57 MHz 992 KiB/10 MiB 300 MiB Task Events: Started At = 2020-11-02T16:29:11Z Finished At = N/A Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 2020-11-02T16:29:11Z Started Task started by client 2020-11-02T16:29:11Z Task Setup Building Task Directory 2020-11-02T16:29:11Z Received Task received by client Note that the **Task Resources** section shows the updated values for memory and CPU given by the autoscaler. From the earlier job status output, a `cache-lb` allocation has allocation ID **8ceec492**. Run the `nomad alloc status 8ceec492` command to get the Task Resources information about this allocation. $ nomad alloc status 8ceec492 ID = 8ceec492-9549-e563-40d9-bf76a47940f2 Eval ID = f0c24365 Name = example.cache-lb[0] Node ID = c442fcaa Node Name = ubuntu-focal Job ID = example Job Version = 2 Client Status = running Client Description = Tasks are running Desired Status = run Desired Description = Created = 7m44s ago Modified = 7m12s ago Deployment ID = c3ee5e5d Deployment Health = healthy Allocation Addresses Label Dynamic Address *lb yes 10.0.2.15:29363 -> 6379 Task "nginx" is "running" Task Resources CPU Memory Disk Addresses 0/57 MHz 1.5 MiB/10 MiB 300 MiB Task Events: Started At = 2020-11-02T16:28:54Z Finished At = N/A Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 2020-11-02T16:29:24Z Signaling Template re-rendered 2020-11-02T16:29:16Z Signaling Template re-rendered 2020-11-02T16:29:13Z Signaling Template re-rendered 2020-11-02T16:29:04Z Signaling Template re-rendered 2020-11-02T16:28:57Z Signaling Template re-rendered 2020-11-02T16:28:55Z Signaling Template re-rendered 2020-11-02T16:28:54Z Started Task started by client 2020-11-02T16:28:53Z Driver Downloading image 2020-11-02T16:28:53Z Task Setup Building Task Directory 2020-11-02T16:28:52Z Received Task received by client Here, also, the **Task Resources** section shows the updated values for memory and CPU given by the autoscaler. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#generate-load-to-create-new-recommendations) Generate load to create new recommendations --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Create a parameterized dispatch job to generate load in your cluster. Create a file named `das-load-test.nomad` with the following content. You can also copy this file from the `~/nomad-autoscaler/jobs` folder in the Vagrant instance. job "das-load-test" { datacenters = ["dc1"] type = "batch" parameterized { payload = "optional" meta_optional = ["requests", "clients"] } group "redis-benchmark" { task "redis-benchmark" { driver = "docker" config { image = "redis:3.2" command = "redis-benchmark" args = [\ "-h","${HOST}",\ "-p","${PORT}",\ "-n","${REQUESTS}",\ "-c","${CLIENTS}",\ ] } template { destination = "secrets/env.txt" env = true data = < Monitoring evaluation "1793fe23" Evaluation triggered by job "das-load-test/dispatch-1604336299-70a3923e" Allocation "589a1825" created: node "c442fcaa", group "redis-benchmark" Evaluation status changed: "pending" -> "complete" ==> Evaluation "1793fe23" finished with status "complete Each run of this job creates 100,000 requests against your Redis cluster using 50 Redis clients. Once you have run the job, watch the Optimize view for new suggestions based on the latest activity. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#exit-and-clean-up) Exit and clean up ----------------------------------------------------------------------------------------------------------------------------- Exit the shell session on the Vagrant VM by typing `exit`. Run the `vagrant destroy` command to stop and remove the virtual box instance. Delete the Vagrantfile once you no longer want to use the test-drive environment. [](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing#learn-more) Learn more --------------------------------------------------------------------------------------------------------------- If you have not already, review the [Dynamic Application Sizing Concepts](https://developer.hashicorp.com/nomad/tutorials/autoscaler/dynamic-application-sizing-concepts) tutorial for more information about the individual parameters and available strategies. You can also find more information in the [Nomad Autoscaler Scaling Policies](https://developer.hashicorp.com/nomad/tools/autoscaling/policy) documentation, including how you can further customize the application-sizing block to your needs (percentile, cooldown periods, and sizing strategies). **Was this tutorial helpful?** YesNo [Previous\ \ Dynamic app sizing](https://developer.hashicorp.com/nomad/tutorials/enterprise/dynamic-application-sizing-concepts) [Next Collection\ \ Manage Clusters](https://developer.hashicorp.com/nomad/tutorials/manage-clusters) This tutorial also appears in: ------------------------------ *  [](https://developer.hashicorp.com/nomad/tutorials/nomad-1-0) 3 tutorials Nomad 1.0 Explore Nomad v1.0 features like Nomad Autoscaler and the event-stream. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/ecosystem) 5 tutorials Expand Nomad with Ecosystem Add-ins Explore applications that enhance how you use your Nomad cluster through their use of the Nomad HTTP API or plug-in interface. * Nomad *  [](https://developer.hashicorp.com/nomad/tutorials/autoscaler) 5 tutorials Dynamically Resize with Nomad Autoscaler Automatically maintain your cluster and workload instance count to respond to demand while minimizing over-provisioning cost. * Nomad --- # Federate multi-region clusters | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Federate multi-region clusters ============================== Nomad operates at a regional level and provides first-class support for federation. Federation enables users to submit jobs or interact with the HTTP API targeting any region, from any server, even if that server resides in a different region. Federating multiple Nomad clusters requires network connectivity between the clusters. Servers in each cluster must be able to communicate over [RPC and Serf](https://developer.hashicorp.com/nomad/docs/deploy/production/requirements#ports-used) . Federated clusters are expected to communicate over WANs, so they do not need the same low latency as servers within a region. Once Nomad servers are able to connect over the network, you can issue the [nomad server join](https://developer.hashicorp.com/nomad/commands/server/join) command from any server in one region to a server in a remote region to federate the clusters. [![Multi-Region](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/clusters/nomad-multi-region.png)](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/clusters/nomad-multi-region.png) [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#prerequisites) Prerequisites ----------------------------------------------------------------------------------------------------------- To perform the tasks described in this guide, you need to have two Nomad environments with ports 4646, 4647, and 4648 exposed. You can use this [Terraform environment](https://github.com/hashicorp/nomad/tree/master/terraform#provision-a-nomad-cluster-in-the-cloud) to provision the sandbox environments. This guide assumes two clusters with one server node and two client nodes in each cluster. While the Terraform code already opens port 4646, you will also need to expose ports 4647 and 4648 on the server you wish to run [nomad server join](https://developer.hashicorp.com/nomad/commands/server/join) against (consult the [Nomad Port Requirements](https://developer.hashicorp.com/nomad/docs/deploy/production/requirements#ports-used) documentation for more information). Note This tutorial is for demo purposes and only assumes a single server node in each cluster. Consult the [reference architecture](https://developer.hashicorp.com/nomad/docs/deploy/production/reference-architecture) for production configuration. [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#verify-current-regions) Verify current regions ----------------------------------------------------------------------------------------------------------------------------- Currently, each of your clusters is in the default `global` [region](https://developer.hashicorp.com/nomad/docs/configuration#region) . You can verify this by running [nomad server members](https://developer.hashicorp.com/nomad/commands/server/members) on any node in each of your clusters: $ nomad server members Name Address Port Status Leader Protocol Build Datacenter Region ip-172-31-29-34.global 172.31.29.34 4648 alive true 2 0.10.1 dc1 global [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#change-the-regions) Change the regions --------------------------------------------------------------------------------------------------------------------- Respectively change the region of your individual clusters into `west` and `east` by adding the [region](https://developer.hashicorp.com/nomad/docs/configuration#region) parameter into the agent configuration on the servers and clients (if you are using the provided sandbox environment, this configuration is located at `/etc/nomad.d/nomad.hcl`). Below is a snippet of the configuration file showing the required change on a node for one of the clusters (remember to change this value to `east` on the servers and clients in your other cluster): data_dir = "/opt/nomad/data" bind_addr = "0.0.0.0" region = "west" # ... Once you have made the necessary changes for each cluster, restart the nomad service on each node: $ sudo systemctl restart nomad Re-run the `nomad server members` command on any node in the cluster to verify that your server is configured to be the in the correct region. The output below is from running the command in the `west` region (make sure to run this command in your other cluster to make sure it is in the `east` region): $ nomad server members Name Address Port Status Leader Protocol Build Datacenter Region ip-172-31-29-34.west 172.31.29.34 4648 alive true 2 0.10.1 dc1 west [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#federate-the-regions) Federate the regions ------------------------------------------------------------------------------------------------------------------------- Run the [`nomad server join`](https://developer.hashicorp.com/nomad/commands/server/join) command from a server in one cluster and supply it the IP address of the server in your other cluster while specifying port 4648. Below is an example of running the `nomad server join` command from the server in the `west` region while targeting the server in the `east` region: $ nomad server join 172.31.26.138:4648 Joined 1 servers successfully [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#verify-the-clusters-have-been-federated) Verify the clusters have been federated --------------------------------------------------------------------------------------------------------------------------------------------------------------- After you have federated your clusters, the output from the `nomad server members` command will show the servers from both regions: $ nomad server members Name Address Port Status Leader Protocol Build Datacenter Region ip-172-31-26-138.east 172.31.26.138 4648 alive true 2 0.10.1 dc1 east ip-172-31-29-34.west 172.31.29.34 4648 alive true 2 0.10.1 dc1 west [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#check-job-status-in-remote-cluster) Check job status in remote cluster ----------------------------------------------------------------------------------------------------------------------------------------------------- From the Nomad cluster in the `west` region, try to run the [`nomad status`](https://developer.hashicorp.com/nomad/commands/status) command to check the status of jobs in the `east` region: $ nomad status -region="east" No running jobs If your regions were not federated properly, you will receive the following output: $ nomad status -region="east" Error querying jobs: Unexpected response code: 500 (No path to region) [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/federate-regions#learn-more-about-federation) Learn more about federation --------------------------------------------------------------------------------------------------------------------------------------- * [Deployment Topology across Multiple Regions](https://developer.hashicorp.com/nomad/docs/deploy/production/reference-architecture#deployment-topology-across-multiple-regions) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/deploy/clusters/federate-regions.mdx) --- # Evaluations - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Evaluations HTTP API ==================== The `/evaluation` endpoints are used to query for and interact with evaluations. [](https://developer.hashicorp.com/nomad/api-docs/evaluations#list-evaluations) List Evaluations ------------------------------------------------------------------------------------------------ This endpoint lists all evaluations. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/evaluations` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`prefix`](https://developer.hashicorp.com/nomad/api-docs/evaluations#prefix) `(string: "")`\- Specifies a string to filter evaluations based on an ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters (0-9a-f). This is specified as a query string parameter and is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`next_token`](https://developer.hashicorp.com/nomad/api-docs/evaluations#next_token) `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which is the `ID` field of the next expected evaluation. This value can be obtained from the `X-Nomad-NextToken` header from the previous response. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`per_page`](https://developer.hashicorp.com/nomad/api-docs/evaluations#per_page) `(int: 0)` - Specifies a maximum number of evaluations to return for this request. If omitted, the response is not paginated. The `ID` of the last evaluation in the response can be used as the `last_token` of the next request to fetch additional pages. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`filter`](https://developer.hashicorp.com/nomad/api-docs/evaluations#filter) `(string: "")` - Specifies the [expression](https://developer.hashicorp.com/nomad/api-docs#filtering) used to filter the results. Consider using pagination or a query parameter to reduce resource used to serve the request. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`job`](https://developer.hashicorp.com/nomad/api-docs/evaluations#job) `(string: "")` - Filter the list of evaluations to a specific job ID. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`status`](https://developer.hashicorp.com/nomad/api-docs/evaluations#status) `(string: "")` - Filter the list of evaluations to a specific evaluation status (one of `blocked`, `pending`, `complete`, `failed`, or `canceled`). * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/evaluations#namespace) `(string: "default")` - Specifies the target namespace. Specifying `*` will return all evaluations across all authorized namespaces. This parameter is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`reverse`](https://developer.hashicorp.com/nomad/api-docs/evaluations#reverse) `(bool: false)` - Specifies the list of returned evaluations should be sorted in the reverse order. By default evaluations are returned sorted in chronological order (older evaluations first), or in lexicographical order by their ID if the `prefix` query parameter is used. ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-request) Sample Request $ curl \ https://localhost:4646/v1/evaluations $ curl \ https://localhost:4646/v1/evaluations?prefix=25ba81 ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-response) Sample Response [\ {\ "ID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577",\ "Priority": 50,\ "Type": "service",\ "TriggeredBy": "job-register",\ "JobID": "example",\ "JobModifyIndex": 52,\ "NodeID": "",\ "NodeModifyIndex": 0,\ "Status": "complete",\ "StatusDescription": "",\ "Wait": 0,\ "NextEval": "",\ "PreviousEval": "",\ "BlockedEval": "",\ "FailedTGAllocs": null,\ "ClassEligibility": null,\ "EscapedComputedClass": false,\ "AnnotatePlan": false,\ "SnapshotIndex": 53,\ "QueuedAllocations": {\ "cache": 0\ },\ "CreateIndex": 53,\ "ModifyIndex": 55\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/evaluations#read-evaluation) Read Evaluation ---------------------------------------------------------------------------------------------- This endpoint reads information about a specific evaluation by ID. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/evaluation/:eval_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`:eval_id`](https://developer.hashicorp.com/nomad/api-docs/evaluations#eval_id) `(string: )`\- Specifies the UUID of the evaluation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`related`](https://developer.hashicorp.com/nomad/api-docs/evaluations#related) `(bool: false)` - Specifies if related evaluations should be returned. Related evaluations are the ones that can be reached by following the trail of IDs for `NextEval`, `PreviousEval`, and `BlockedEval`. This is specified as a query parameter. ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/evaluation/2deb5f06-a100-f01a-3316-5e501a4965e7?related=true ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-response-1) Sample Response { "CreateIndex": 28, "CreateTime": 1647394818583344000, "FailedTGAllocs": { "cache": { "AllocationTime": 4111, "ClassExhausted": null, "ClassFiltered": null, "CoalescedFailures": 0, "ConstraintFiltered": null, "DimensionExhausted": null, "NodesAvailable": { "dc1": 0 }, "NodesEvaluated": 0, "NodesExhausted": 0, "NodesFiltered": 0, "NodesInPool": 0, "QuotaExhausted": null, "ResourcesExhausted": null, "ScoreMetaData": null, "Scores": null } }, "ID": "2deb5f06-a100-f01a-3316-5e501a4965e7", "JobID": "example", "ModifyIndex": 28, "ModifyTime": 1647394818583344000, "Namespace": "default", "PreviousEval": "0f98f7ea-59ae-4d90-d9bd-b8ce80b9e100", "Priority": 50, "RelatedEvals": [\ {\ "BlockedEval": "2deb5f06-a100-f01a-3316-5e501a4965e7",\ "CreateIndex": 27,\ "CreateTime": 1647394818582736000,\ "DeploymentID": "79ae0a49-acf6-0fcf-183f-8646f3167b88",\ "ID": "0f98f7ea-59ae-4d90-d9bd-b8ce80b9e100",\ "JobID": "example",\ "ModifyIndex": 30,\ "ModifyTime": 1647394818583565000,\ "Namespace": "default",\ "NextEval": "",\ "NodeID": "",\ "PreviousEval": "",\ "Priority": 50,\ "Status": "complete",\ "StatusDescription": "",\ "TriggeredBy": "node-drain",\ "Type": "service",\ "WaitUntil": null\ }\ ], "SnapshotIndex": 27, "Status": "blocked", "StatusDescription": "created to place remaining allocations", "TriggeredBy": "queued-allocs", "Type": "service" } [](https://developer.hashicorp.com/nomad/api-docs/evaluations#delete-evaluations) Delete Evaluations ---------------------------------------------------------------------------------------------------- This endpoint deletes evaluations. In order to utilise this endpoint the eval broker should be paused via the [operator scheduler update configuration](https://developer.hashicorp.com/nomad/api-docs/operator/scheduler#update-scheduler-configuration) API endpoint. This API endpoint should be used cautiously and only in outage situations where there is a large backlog of evaluations not being processed. During most normal and outage scenarios, Nomad's reconciliation and state management will handle evaluations as needed. | Method | Path | Produces | | --- | --- | --- | | `DELETE` | `/v1/evaluations` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`EvalIDs`](https://developer.hashicorp.com/nomad/api-docs/evaluations#evalids) `(array: )`\- An array of evaluation UUIDs to delete. This must be a full length UUID and not a prefix. ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-payload) Sample Payload { "EvalIDs": [\ "167ec27d-2e36-979a-280a-a6b920d382db",\ "6c193955-ac66-42e2-f4c7-f1fc707f1f5e"\ ] } ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-request-2) Sample Request $ curl \ --request DELETE \ --data @payload.json \ https://localhost:4646/v1/evaluations [](https://developer.hashicorp.com/nomad/api-docs/evaluations#list-allocations-for-evaluation) List Allocations for Evaluation ------------------------------------------------------------------------------------------------------------------------------ This endpoint lists the allocations created or modified for the given evaluation. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/evaluation/:eval_id/allocations` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`:eval_id`](https://developer.hashicorp.com/nomad/api-docs/evaluations#eval_id-1) `(string: )`\- Specifies the UUID of the evaluation. This must be the full UUID, not the short 8-character one. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-request-3) Sample Request $ curl \ https://localhost:4646/v1/evaluation/5456bd7a-9fc0-c0dd-6131-cbee77f57577/allocations ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-response-2) Sample Response [\ {\ "ID": "a8198d79-cfdb-6593-a999-1e9adabcba2e",\ "EvalID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577",\ "Name": "example.cache[0]",\ "NodeID": "fb2170a8-257d-3c64-b14d-bc06cc94e34c",\ "JobID": "example",\ "TaskGroup": "cache",\ "DesiredStatus": "run",\ "DesiredDescription": "",\ "ClientStatus": "running",\ "ClientDescription": "",\ "TaskStates": {\ "redis": {\ "State": "running",\ "Failed": false,\ "Events": [\ {\ "Type": "Received",\ "Time": 1495747371795703800,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ },\ {\ "Type": "Driver",\ "Time": 1495747371798867200,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": "Downloading image redis:7"\ },\ {\ "Type": "Started",\ "Time": 1495747379525667800,\ "FailsTask": false,\ "RestartReason": "",\ "SetupError": "",\ "DriverError": "",\ "ExitCode": 0,\ "Signal": 0,\ "Message": "",\ "KillTimeout": 0,\ "KillError": "",\ "KillReason": "",\ "StartDelay": 0,\ "DownloadError": "",\ "ValidationError": "",\ "DiskLimit": 0,\ "FailedSibling": "",\ "VaultError": "",\ "TaskSignalReason": "",\ "TaskSignal": "",\ "DriverMessage": ""\ }\ ]\ }\ },\ "CreateIndex": 54,\ "ModifyIndex": 57,\ "CreateTime": 1495747371794276400\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/evaluations#count-evaluations) Count Evaluations -------------------------------------------------------------------------------------------------- This endpoint counts evaluations. Note that Nomad's state store architecture makes calculating this count unexpectedly expensive (similar in cost to the List API), and this API was designed for use during recovery operations with the `nomad eval delete` command. It is not recommended to use this API for monitoring. The `nomad.nomad.broker.*` metrics are better for that use case. See the [metrics reference](https://developer.hashicorp.com/nomad/docs/reference/metrics) for details. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/evaluations/count` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#parameters-4) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`prefix`](https://developer.hashicorp.com/nomad/api-docs/evaluations#prefix-1) `(string: "")`\- Specifies a string to filter evaluations based on an ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters (0-9a-f). This is specified as a query string parameter and is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`filter`](https://developer.hashicorp.com/nomad/api-docs/evaluations#filter-1) `(string: "")` - Specifies the [expression](https://developer.hashicorp.com/nomad/api-docs#filtering) used to filter the results. * [](https://developer.hashicorp.com/nomad/api-docs/evaluations#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/evaluations#namespace-1) `(string: "default")` - Specifies the target namespace. Specifying `*` will return all evaluations across all authorized namespaces. This parameter is used before any `filter` expression is applied. ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-request-4) Sample Request $ curl \ https://localhost:4646/v1/evaluations/count $ curl \ https://localhost:4646/v1/evaluations/count?prefix=25ba81 ### [](https://developer.hashicorp.com/nomad/api-docs/evaluations#sample-response-3) Sample Response { "Count": 36 "Index": 133, "KnownLeader": true, "LastContact": 0, "NextToken": "" } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/evaluations.mdx) --- # JSON Job Specification - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) JSON Job Specification ====================== Nomad's HTTP API uses JSON formatted job specifications except for the [`/job/parse` API](https://developer.hashicorp.com/nomad/api-docs/jobs#parse-job) which exists to convert HCL to JSON. The Nomad CLI includes a number of useful commands for working with JSON jobs. The [`nomad job run -output`](https://developer.hashicorp.com/nomad/commands/job/run#output) flag converts HCL jobs to JSON without submitting the job: $ nomad job run -output my-job.nomad The [`nomad job inspect`](https://developer.hashicorp.com/nomad/commands/job/inspect) command retrieves the JSON specification for an existing job: $ nomad job inspect example The [`nomad job run -json`](https://developer.hashicorp.com/nomad/commands/job/run#json) flag submits a JSON formatted job: $ nomad job run -json example.json [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#syntax) Syntax -------------------------------------------------------------------------- Below is the JSON representation of the example job as well as the commands to reproduce it: $ nomad init Example job file written to example.nomad.hcl $ nomad job run -output example.nomad.hcl { "Job": { "Region": null, "Namespace": null, "ID": "example", "Name": "example", "Type": "service", "Priority": null, "AllAtOnce": null, "Datacenters": [\ "dc1"\ ], "NodePool": "prod", "Constraints": null, "Affinities": null, "TaskGroups": [\ {\ "Name": "cache",\ "Count": 1,\ "Constraints": null,\ "Disconnect": null,\ "Affinities": null,\ "Tasks": [\ {\ "Name": "redis",\ "Driver": "docker",\ "User": "",\ "Lifecycle": null,\ "Config": {\ "auth_soft_fail": true,\ "image": "redis:7",\ "ports": [\ "db"\ ]\ },\ "Constraints": null,\ "Affinities": null,\ "Env": null,\ "Services": null,\ "Resources": {\ "CPU": 500,\ "Cores": null,\ "MemoryMB": 256,\ "MemoryMaxMB": null,\ "DiskMB": null,\ "Networks": null,\ "Devices": null,\ "IOPS": null\ },\ "RestartPolicy": null,\ "Meta": null,\ "KillTimeout": null,\ "LogConfig": null,\ "Artifacts": null,\ "Vault": null,\ "Templates": null,\ "DispatchPayload": null,\ "VolumeMounts": null,\ "Leader": false,\ "ShutdownDelay": 0,\ "KillSignal": "",\ "Kind": "",\ "ScalingPolicies": null,\ "Identity": {\ "Env": true,\ "File": true\ }\ }\ ],\ "Spreads": null,\ "Volumes": null,\ "RestartPolicy": {\ "Interval": 1800000000000,\ "Attempts": 2,\ "Delay": 15000000000,\ "Mode": "fail"\ },\ "ReschedulePolicy": null,\ "EphemeralDisk": {\ "Sticky": null,\ "Migrate": null,\ "SizeMB": 300\ },\ "Update": null,\ "Migrate": null,\ "Networks": [\ {\ "Mode": "",\ "Device": "",\ "CIDR": "",\ "IP": "",\ "DNS": null,\ "ReservedPorts": null,\ "DynamicPorts": [\ {\ "Label": "db",\ "Value": 0,\ "To": 6379,\ "HostNetwork": ""\ }\ ],\ "Hostname": "",\ "MBits": null\ }\ ],\ "Meta": null,\ "Services": [\ {\ "Name": "redis-cache",\ "Tags": [\ "global",\ "cache"\ ],\ "CanaryTags": null,\ "EnableTagOverride": false,\ "PortLabel": "db",\ "AddressMode": "",\ "Address": "",\ "Checks": null,\ "CheckRestart": null,\ "Connect": null,\ "Meta": null,\ "CanaryMeta": null,\ "TaggedAddresses": null,\ "TaskName": "",\ "OnUpdate": "",\ "Provider": "nomad"\ }\ ],\ "ShutdownDelay": null,\ "StopAfterClientDisconnect": null,\ "MaxClientDisconnect": null,\ "Scaling": null,\ "Consul": null\ }\ ], "Update": { "Stagger": null, "MaxParallel": 1, "HealthCheck": null, "MinHealthyTime": 10000000000, "HealthyDeadline": 180000000000, "ProgressDeadline": 600000000000, "Canary": 0, "AutoRevert": false, "AutoPromote": null }, "Multiregion": null, "Spreads": null, "Periodic": null, "ParameterizedJob": null, "Reschedule": null, "Migrate": { "MaxParallel": 1, "HealthCheck": "checks", "MinHealthyTime": 10000000000, "HealthyDeadline": 300000000000 }, "Meta": null, "ConsulToken": null, "VaultToken": null, "Stop": null, "ParentID": null, "Dispatched": false, "DispatchIdempotencyToken": null, "Payload": null, "ConsulNamespace": null, "VaultNamespace": null, "NomadTokenID": null, "Status": null, "StatusDescription": null, "Stable": null, "Version": null, "SubmitTime": null, "CreateIndex": null, "ModifyIndex": null, "JobModifyIndex": null } } [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#syntax-reference) Syntax Reference ---------------------------------------------------------------------------------------------- Following is a syntax reference for the possible keys that are supported and their default values if any for each type of object. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#job) Job The `Job` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`AllAtOnce`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#allatonce) - Controls whether the scheduler can make partial placements if optimistic scheduling resulted in an oversubscribed node. This does not control whether all allocations for the job, where all would be the desired count for each task group, must be placed atomically. This should only be used for special circumstances. Defaults to `false`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Constraints`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constraints) - A list to define additional constraints where a job can be run. See the constraint reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Affinities`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#affinities) - A list to define placement preferences on nodes where a job can be run. See the affinity reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Spreads`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spreads) - A list to define allocation spread across attributes. See the spread reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Datacenters`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#datacenters) - A list of datacenters in the region which are eligible for task placement. This must be provided, and does not have a default. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`NodePool`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#nodepool) - The node pool in which the job can be placed. Defaults to `"default"`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`TaskGroups`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#taskgroups) - A list to define additional task groups. See the task group reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#meta) - Annotates the job with opaque metadata. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ConsulToken`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#consultoken) - Specifies the Consul token that proves the submitter of the job has access to the Service Identity policies associated with the job's Consul service mesh enabled services. This field is only used to transfer the token and is not stored after job submission. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`VaultToken`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#vaulttoken) - Specifies the Vault token that proves the submitter of the job has access to the specified policies in the `vault` block. This field is only used to transfer the token and is not stored after job submission. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Namespace`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#namespace) - The namespace to execute the job in, defaults to "default". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ParameterizedJob`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#parameterizedjob) - Specifies the job as a parameterized job such that it can be dispatched against. The `ParameterizedJob` object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MetaOptional`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#metaoptional) - Specifies the set of metadata keys that may be provided when dispatching against the job as a string array. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MetaRequired`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#metarequired) - Specifies the set of metadata keys that must be provided when dispatching against the job as a string array. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Payload`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#payload) - Specifies the requirement of providing a payload when dispatching against the parameterized job. The options for this field are "optional", "required" and "forbidden". The default value is "optional". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`DispatchIdempotencyToken`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#dispatchidempotencytoken) - Optional identifier used to prevent more than one instance of the job from being dispatched. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Payload`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#payload-1) - The payload may not be set when submitting a job but may appear in a dispatched job. The `Payload` will be a base64 encoded string containing the payload that the job was dispatched with. The `payload` has a **maximum size of 16 KiB**. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Priority`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#priority) - Specifies the job priority which is used to prioritize scheduling and access to resources. Must be between 1 and 100 inclusively, and defaults to 50. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Region`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#region) - The region to run the job in, defaults to "global". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Type`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#type) - Specifies the job type and switches which scheduler is used. Nomad provides the `service`, `system` and `batch` schedulers, and defaults to `service`. To learn more about each scheduler type visit [here](https://developer.hashicorp.com/nomad/docs/concepts/scheduling/schedulers) * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Update`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#update) - Specifies an update strategy to be applied to all task groups within the job. When specified both at the job level and the task group level, the update blocks are merged with the task group's taking precedence. For more details on the update block, please see below. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Periodic`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#periodic) - `Periodic` allows the job to be scheduled at fixed times, dates or intervals. The periodic expression is always evaluated in the UTC timezone to ensure consistent evaluation when Nomad Servers span multiple time zones. The `Periodic` object is optional and supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Enabled`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#enabled) - `Enabled` determines whether the periodic job will spawn child jobs. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`TimeZone`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#timezone) - Specifies the time zone to evaluate the next launch interval against. This is useful when wanting to account for day light savings in various time zones. The time zone must be parsable by Golang's [LoadLocation](https://golang.org/pkg/time/#LoadLocation) . The default is UTC. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`SpecType`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spectype) - `SpecType` determines how Nomad is going to interpret the periodic expression. `cron` is the only supported `SpecType` currently. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Spec`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spec) - A cron expression configuring the interval the job is launched at. Supports predefined expressions such as "@daily" and "@weekly" See [here](https://github.com/gorhill/cronexpr#implementation) for full documentation of supported cron specs and the predefined expressions. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Specs`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#specs) - A list of cron expressions configuring the intervals the job is launched at. The job runs at the next earliest time that matches any of the expressions. Supports predefined expressions such as `@daily` and `@weekly`. Refer to [the documentation](https://github.com/gorhill/cronexpr#implementation) for full details about the supported cron specs and the predefined expressions. Conflicts with `Spec`. * [\`ProhibitOverlap\`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) - \`ProhibitOverlap\` can be set to true to enforce that the periodic job doesn't spawn a new instance of the job if any of the previous jobs are still running. It is defaulted to false. An example `periodic` block: { "Periodic": { "Spec": "*/15 - *", "TimeZone": "Europe/Berlin", "SpecType": "cron", "Enabled": true, "ProhibitOverlap": true } } * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ReschedulePolicy`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#reschedulepolicy) - Specifies a reschedule policy to be applied to all task groups within the job. When specified both at the job level and the task group level, the reschedule blocks are merged, with the task group's taking precedence. For more details on `ReschedulePolicy`, please see below. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Ui`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ui) - Options to modify the presentation of the Job index page in the Web UI. When specified, a description and any number of links will be added to the top of the job page in question. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Description`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#description) - a markdown-enabled string description of the job jobs. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Links`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#links) - Anchor link that will appear at the top of the job page. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Label`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#label) - The shown name of the link * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Url`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#url) - The address that the link will point to An example `ui` block: "Ui": { "Description": "A job that uses **Nomad Variables** to modify its output", "Links": [\ {\ "Label": "Learn more about Nomad Variables",\ "Url": "https://developer.hashicorp.com/nomad/docs/concepts/variables"\ },\ {\ "Label": "See this job on Github",\ "Url": "https://github.com/hashicorp/nomad/blob/main/ui/app/utils/default_jobs/variables.js"\ }\ ] } ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#task-group) Task Group `TaskGroups` is a list of `TaskGroup` objects, each supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Constraints`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constraints-1) - This is a list of `Constraint` objects. See the constraint reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Affinities`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#affinities-1) - This is a list of `Affinity` objects. See the affinity reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Spreads`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spreads-1) - This is a list of `Spread` objects. See the spread reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Count`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#count) - Specifies the number of the task groups that should be running. Must be non-negative, defaults to one. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#meta-1) - A key-value map that annotates the task group with opaque metadata. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Migrate`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#migrate) - Specifies a migration strategy to be applied during [node drains](https://developer.hashicorp.com/nomad/commands/node/drain) . * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`HealthCheck`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#healthcheck) - One of `checks` or `task_states`. Indicates how task health should be determined: either via Consul health checks or whether the task was able to run successfully. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`HealthyDeadline`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#healthydeadline) - Specifies duration a task must become healthy within before it is considered unhealthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MaxParallel`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#maxparallel) - Specifies how many allocations may be migrated at once. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MinHealthyTime`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#minhealthytime) - Specifies duration a task must be considered healthy before the migration is considered healthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#name) - The name of the task group. Must be specified. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RestartPolicy`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restartpolicy) - Specifies the restart policy to be applied to tasks in this group. If omitted, a default policy for batch and non-batch jobs is used based on the job type. See the [restart policy reference](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restart_policy) for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ReschedulePolicy`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#reschedulepolicy-1) - Specifies the reschedule policy to be applied to tasks in this group. If omitted, a default policy is used for batch and service jobs. System jobs are not eligible for rescheduling. See the [reschedule policy reference](https://developer.hashicorp.com/nomad/api-docs/json-jobs#reschedule_policy) for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Scaling`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#scaling) - Specifies the autoscaling policy for the task group. This is primarily for supporting external autoscalers. See the [scaling policy reference](https://developer.hashicorp.com/nomad/api-docs/json-jobs#scaling_policy) for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`EphemeralDisk`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ephemeraldisk) - Specifies the group's ephemeral disk requirements. See the [ephemeral disk reference](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ephemeral_disk) for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Update`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#update-1) - Specifies an update strategy to be applied to all task groups within the job. When specified both at the job level and the task group level, the update blocks are merged with the task group's taking precedence. For more details on the update block, please see below. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Tasks`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#tasks) - A list of `Task` object that are part of the task group. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#task) Task The `Task` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Artifacts`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#artifacts) - `Artifacts` is a list of `Artifact` objects which define artifacts to be downloaded before the task is run. See the artifacts reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Config`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#config) - A map of key-value configuration passed into the driver to start the task. The details of configurations are specific to each driver. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Constraints`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constraints-2) - This is a list of `Constraint` objects. See the constraint reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Affinities`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#affinities-2) - This is a list of `Affinity` objects. See the affinity reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Spreads`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spreads-2) - This is a list of `Spread` objects. See the spread reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`DispatchPayload`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#dispatchpayload) - Configures the task to have access to dispatch payloads. The `DispatchPayload` object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`File`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#file) - Specifies the file name to write the content of dispatch payload to. The file is written relative to the task's local directory. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Driver`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#driver) - Specifies the task driver that should be used to run the task. See the [driver documentation](https://developer.hashicorp.com/nomad/docs/job-declare/task-driver) for what is available. Examples include `docker`, `qemu`, `java`, and `exec`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Env`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#env) - A map of key-value representing environment variables that will be passed along to the running process. Nomad variables are interpreted when set in the environment variable values. See the table of interpreted variables [here](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation) . For example the below environment map will be reinterpreted: { "Env": { "NODE_CLASS": "${nomad.class}" } } * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Identity`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#identity) - Specifies whether to expose Nomad's [Workload Identity](https://developer.hashicorp.com/nomad/docs/concepts/workload-identity "Nomad Workload Identity") to the task. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Env`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#env-1) - If `true` the `NOMAD_TOKEN` environment variable will be set. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`File`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#file-1) - If `true` the `secrets/nomad_token` file will be created. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`KillSignal`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#killsignal) - Specifies a configurable kill signal for a task, where the default is SIGINT. Note that this is only supported for drivers which accept sending signals (currently `docker`, `exec`, `raw_exec`, and `java` drivers). * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`KillTimeout`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#killtimeout) - `KillTimeout` is a time duration in nanoseconds. It can be used to configure the time between signaling a task it will be killed and actually killing it. Drivers first sends a task the `SIGINT` signal and then sends `SIGTERM` if the task doesn't die after the `KillTimeout` duration has elapsed. The default `KillTimeout` is 5 seconds. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Leader`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#leader) - Specifies whether the task is the leader task of the task group. If set to true, when the leader task completes, all other tasks within the task group will be gracefully shutdown. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`LogConfig`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#logconfig) - This allows configuring log rotation for the `stdout` and `stderr` buffers of a Task. See the log rotation reference below for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#meta-2) - Annotates the task group with opaque metadata. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#name-1) - The name of the task. This field is required. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Resources`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#resources) - Provides the resource requirements of the task. See the resources reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RestartPolicy`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restartpolicy-1) - Specifies the task-specific restart policy. If omitted, the restart policy from the encapsulating task group is used. If both are present, they are merged. See the [restart policy reference](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restart_policy) for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Services`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#services) - `Services` is a list of `Service` objects. Nomad integrates with Consul for service discovery. A `Service` object represents a routable and discoverable service on the network. Nomad automatically registers when a task is started and de-registers it when the task transitions to the dead state. [Click here](https://developer.hashicorp.com/nomad/docs/networking/service-discovery) to learn more about services. Below is the fields in the `Service` object: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#name-2) : An explicit name for the Service. Nomad will replace `${JOB}`, `${TASKGROUP}` and `${TASK}` by the name of the job, task group or task, respectively. `${BASE}` expands to the equivalent of `${JOB}-${TASKGROUP}-${TASK}`, and is the default name for a Service. Each service defined for a given task must have a distinct name, so if a task has multiple services only one of them can use the default name and the others must be explicitly named. Names must adhere to [RFC-1123 §2.1](https://tools.ietf.org/html/rfc1123#section-2) and are limited to alphanumeric and hyphen characters (i.e. `[a-z0-9\-]`), and be less than 64 characters in length. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Tags`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#tags) : A list of string tags associated with this Service. String interpolation is supported in tags. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#meta-3) : A key-value map that annotates the Consul service with user-defined metadata. String interpolation is supported in meta. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`CanaryTags`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#canarytags) : A list of string tags associated with this Service while it is a canary. Once the canary is promoted, the registered tags will be updated to the set defined in the `Tags` field. String interpolation is supported in tags. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`CanaryMeta`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#canarymeta) : A key-value map that annotates this Service while it is a canary. Once the canary is promoted, the registered meta will be updated to the set defined in the `Meta` field or removed if the `Meta` field is not set. String interpolation is supported in meta keys and values. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`PortLabel`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#portlabel) : `PortLabel` is an optional string and is used to associate a port with the service. If specified, the port label must match one defined in the resources block. This could be a label of either a dynamic or a static port. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Provider`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#provider) : Specifies the service registration provider to use for service registrations. Valid options are either `consul` or `nomad`. All services within a single task group must utilise the same provider value. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Address`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#address) : Specifies a custom address to advertise in Consul or Nomad service registration. If set, `AddressMode` must be in `auto` mode. Useful with interpolation - for example to advertise the public IP address of an AWS EC2 instance set this to `${attr.unique.platform.aws.public-ipv4}`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`AddressMode`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#addressmode) : Specifies what address (host or driver-specific) this service should advertise. Valid options are: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`auto`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#auto) - Allows the driver to determine whether the host or driver address should be used. Defaults to `host` and only implemented by Docker. If you use a Docker network plugin such as weave, Docker will automatically use its address. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`driver`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#driver-1) - Use the IP specified by the driver, and the port specified in a port map. A numeric port may be specified since port maps aren't required by all network plugins. Useful for advertising SDN and overlay network addresses. Task will fail if driver network cannot be determined. Only implemented for Docker. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`host`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#host) - Use the host IP and port. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Checks`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#checks) : `Checks` is an array of check objects. A check object defines a health check associated with the service. Nomad supports the `script`, `http` and `tcp` Consul Checks. Script checks are not supported for the qemu driver since the Nomad client doesn't have access to the file system of a task using the QEMU driver. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Type`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#type-1) : This indicates the check types supported by Nomad. Valid options are currently `script`, `http` and `tcp`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#name-3) : The name of the health check. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`AddressMode`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#addressmode-1) : Same as `AddressMode` on `Service`. Unlike services, checks do not have an `auto` address mode as there's no way for Nomad to know which is the best address to use for checks. Consul needs access to the address for any HTTP or TCP checks. Unlike `PortLabel`, this setting is _not_ inherited from the `Service`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`PortLabel`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#portlabel-1) : Specifies the label of the port on which the check will be performed. Note this is the _label_ of the port and not the port number unless `AddressMode: "driver"`. The port label must match one defined in the Network block. If a port value was declared on the `Service`, this will inherit from that value if not supplied. If supplied, this value takes precedence over the `Service.PortLabel` value. This is useful for services which operate on multiple ports. `http` and `tcp` checks require a port while `script` checks do not. Checks will use the host IP and ports by default. Numeric ports may be used if `AddressMode: "driver"` is set on the check. * [`Header`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#header) : Headers for HTTP checks. Should be an object where the values are an array of values. Headers will be written once for each value. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Interval`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#interval) : This indicates the frequency of the health checks that Consul will perform. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Timeout`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#timeout) : This indicates how long Consul will wait for a health check query to succeed. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Method`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#method) : The HTTP method to use for HTTP checks. Defaults to GET. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Body`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#body) : The HTTP body to use for HTTP checks. Defaults to an empty string. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Path`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#path) : The path of the HTTP endpoint which Consul will query to query the health of a service if the type of the check is `http`. Nomad will add the IP of the service and the port, users are only required to add the relative URL of the health check endpoint. Absolute paths are not allowed. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Protocol`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#protocol) : This indicates the protocol for the HTTP checks. Valid options are `http` and `https`. We default it to `http`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Command`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#command) : This is the command that the Nomad client runs for doing script based health check. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Args`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#args) : Additional arguments to the `command` for script based health checks. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`TLSSkipVerify`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#tlsskipverify) : If true, Consul will not attempt to verify the certificate when performing HTTPS checks. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`CheckRestart`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#checkrestart) : `CheckRestart` is an object which enables restarting of tasks based upon Consul health checks. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Limit`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#limit) : The number of unhealthy checks allowed before the service is restarted. Defaults to `0` which disables health-based restarts. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Grace`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#grace) : The duration to wait after a task starts or restarts before counting unhealthy checks count against the limit. Defaults to "1s". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`IgnoreWarnings`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ignorewarnings) : Treat checks that are warning as passing. Defaults to false which means warnings are considered unhealthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ShutdownDelay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#shutdowndelay) - Specifies the duration to wait when killing a task between removing it from Consul and sending it a shutdown signal. Ideally services would fail healthchecks once they receive a shutdown signal. Alternatively `ShutdownDelay` may be set to give in flight requests time to complete before shutting down. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Templates`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#templates) - Specifies the set of [`Template`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#template) objects to render for the task. Templates can be used to inject both static and dynamic configuration with data populated from environment variables, Consul and Vault. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`User`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#user) - Set the user that will run the task. It defaults to the same user the Nomad client is being run as. This can only be set on Linux platforms. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#resources-1) Resources The `Resources` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`CPU`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#cpu) - The CPU required in MHz. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MemoryMB`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#memorymb) - The memory required in MB. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Networks`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#networks) - A list of network objects. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Devices`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#devices) - A list of device objects. The Network object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MBits`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#mbits) - The number of MBits in bandwidth required. Nomad can allocate two types of ports to a task - Dynamic and Static/Reserved ports. A network object allows the user to specify a list of `DynamicPorts` and `ReservedPorts`. Each object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Value`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#value) - The port number for static ports. If the port is dynamic, then this attribute is ignored. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Label`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#label-1) - The label to annotate a port so that it can be referred in the service discovery block or environment variables. The Device object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#name-4) - Specifies the device required. The following inputs are valid: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [``](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) : If a single value is given, it is assumed to be the device type, such as "gpu", or "fpga". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`/`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#-1) : If two values are given separated by a `/`, the given device type will be selected, constraining on the provided vendor. Examples include "nvidia/gpu" or "amd/gpu". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`//`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#-2) : If three values are given separated by a `/`, the given device type will be selected, constraining on the provided vendor, and model name. Examples include "nvidia/gpu/1080ti" or "nvidia/gpu/2080ti". * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Count`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#count-1) - The count of devices being requested per task. Defaults to 1. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Constraints`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constraints-3) - A list to define constraints on which device can satisfy the request. See the constraint reference for more details. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Affinities`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#affinities-3) - A list to define preferences for which device should be chosen. See the affinity reference for more details. [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ephemeral-disk) Ephemeral Disk The `EphemeralDisk` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Migrate`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#migrate-1) - Specifies that the Nomad client should make a best-effort attempt to migrate the data from a remote machine if placement cannot be made on the original node. During data migration, the task will block starting until the data migration has completed. Value is a boolean and the default is false. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`SizeMB`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#sizemb) - Specifies the size of the ephemeral disk in MB. Default is 300. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Sticky`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#sticky) - Specifies that Nomad should make a best-effort attempt to place the updated allocation on the same machine. This will move the `local/` and `alloc/data` directories to the new allocation. Value is a boolean and the default is false. [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#reschedule-policy) Reschedule Policy The `ReschedulePolicy` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Attempts`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#attempts) - `Attempts` is the number of reschedule attempts allowed in an `Interval`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Interval`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#interval-1) - `Interval` is a time duration that is specified in nanoseconds. The `Interval` is a sliding window within which at most `Attempts` number of reschedule attempts are permitted. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Delay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#delay) - A duration to wait before attempting rescheduling. It is specified in nanoseconds. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`DelayFunction`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#delayfunction) - Specifies the function that is used to calculate subsequent reschedule delays. The initial delay is specified by the `Delay` parameter. Allowed values for `DelayFunction` are listed below: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`constant`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constant) - The delay between reschedule attempts stays at the `Delay` value. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`exponential`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#exponential) - The delay between reschedule attempts doubles. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`fibonacci`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#fibonacci) - The delay between reschedule attempts is calculated by adding the two most recent delays applied. For example if `Delay` is set to 5 seconds, the next five reschedule attempts will be delayed by 5 seconds, 5 seconds, 10 seconds, 15 seconds, and 25 seconds respectively. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MaxDelay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#maxdelay) - `MaxDelay` is an upper bound on the delay beyond which it will not increase. This parameter is used when `DelayFunction` is `exponential` or `fibonacci`, and is ignored when `constant` delay is used. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Unlimited`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#unlimited) - `Unlimited` enables unlimited reschedule attempts. If this is set to true the `Attempts` and `Interval` fields are not used. [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restart-policy) Restart Policy The `RestartPolicy` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Attempts`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#attempts-1) - `Attempts` is the number of restarts allowed in an `Interval`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Interval`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#interval-2) - `Interval` is a time duration that is specified in nanoseconds. The `Interval` begins when the first task starts and ensures that only `Attempts` number of restarts happens within it. If more than `Attempts` number of failures happen, behavior is controlled by `Mode`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Delay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#delay-1) - A duration to wait before restarting a task. It is specified in nanoseconds. A random jitter of up to 25% is added to the delay. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Mode`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#mode) - `Mode` is given as a string and controls the behavior when the task fails more than `Attempts` times in an `Interval`. Possible values are listed below: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`delay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#delay-2) - `delay` will delay the next restart until the next `Interval` is reached. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`fail`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#fail) - `fail` will not restart the task again. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#update-2) Update Specifies the task group update strategy. When omitted, rolling updates are disabled. The update block can be specified at the job or task group level. When specified at the job, the update block is inherited by all task groups. When specified in both the job and in a task group, the blocks are merged with the task group's taking precedence. The `Update` object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MaxParallel`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#maxparallel-1) - `MaxParallel` is given as an integer value and specifies the number of tasks that can be updated at the same time. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`HealthCheck`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#healthcheck-1) - Specifies the mechanism in which allocations health is determined. The potential values are: * "checks" - Specifies that the allocation should be considered healthy when all of its tasks are running and their associated checks are healthy, and unhealthy if any of the tasks fail or not all checks become healthy. This is a superset of "task\_states" mode. * "task\_states" - Specifies that the allocation should be considered healthy when all its tasks are running and unhealthy if tasks fail. * "manual" - Specifies that Nomad should not automatically determine health and that the operator will specify allocation health using the [HTTP API](https://developer.hashicorp.com/nomad/api-docs/deployments#set-allocation-health-in-deployment) . * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MinHealthyTime`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#minhealthytime-1) - Specifies the minimum time the allocation must be in the healthy state before it is marked as healthy and unblocks further allocations from being updated. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`HealthyDeadline`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#healthydeadline-1) - Specifies the deadline in which the allocation must be marked as healthy after which the allocation is automatically transitioned to unhealthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ProgressDeadline`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#progressdeadline) - Specifies the deadline in which an allocation must be marked as healthy. The deadline begins when the first allocation for the deployment is created and is reset whenever an allocation as part of the deployment transitions to a healthy state. If no allocation transitions to the healthy state before the progress deadline, the deployment is marked as failed. If the `progress_deadline` is set to `0`, the first allocation to be marked as unhealthy causes the deployment to fail. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`AutoRevert`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#autorevert) - Specifies if the job should auto-revert to the last stable job on deployment failure. A job is marked as stable if all the allocations as part of its deployment were marked healthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Canary`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#canary) - Specifies that changes to the job that would result in destructive updates should create the specified number of canaries without stopping any previous allocations. Once the operator determines the canaries are healthy, they can be promoted which unblocks a rolling update of the remaining allocations at a rate of `max_parallel`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`AutoPromote`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#autopromote) - Specifies if the job should automatically promote to the new deployment if all canaries become healthy. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Stagger`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#stagger) - Specifies the delay between migrating allocations off nodes marked for draining. An example `Update` block: { "Update": { "MaxParallel": 3, "HealthCheck": "checks", "MinHealthyTime": 15000000000, "HealthyDeadline": 180000000000, "AutoRevert": false, "AutoPromote": false, "Canary": 1 } } ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#constraint) Constraint The `Constraint` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`LTarget`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ltarget) - Specifies the attribute to examine for the constraint. See the table of attributes [here](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation#interpreted_node_vars) . * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RTarget`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#rtarget) - Specifies the value to compare the attribute against. This can be a literal value, another attribute or a regular expression if the `Operator` is in "regexp" mode. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Operand`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#operand) - Specifies the test to be performed on the two targets. It takes on the following values: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`regexp`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#regexp) - Allows the `RTarget` to be a regular expression to be matched. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`set_contains`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#set_contains) - Allows the `RTarget` to be a comma separated list of values that should be contained in the LTarget's value. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`distinct_hosts`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#distinct_hosts) - If set, the scheduler will not co-locate any task groups on the same machine. This can be specified as a job constraint which applies the constraint to all task groups in the job, or as a task group constraint which scopes the effect to just that group. The constraint may not be specified at the task level. Placing the constraint at both the job level and at the task group level is redundant since when placed at the job level, the constraint will be applied to all task groups. When specified, `LTarget` and `RTarget` should be omitted. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`distinct_property`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#distinct_property) - If set, the scheduler selects nodes that have a distinct value of the specified property. The `RTarget` specifies how many allocations are allowed to share the value of a property. The `RTarget` must be 1 or greater and if omitted, defaults to 1. This can be specified as a job constraint which applies the constraint to all task groups in the job, or as a task group constraint which scopes the effect to just that group. The constraint may not be specified at the task level. Placing the constraint at both the job level and at the task group level is redundant since when placed at the job level, the constraint will be applied to all task groups. When specified, `LTarget` should be the property that should be distinct and `RTarget` should be omitted. * Comparison Operators - `=`, `==`, `is`, `!=`, `not`, `>`, `>=`, `<`, `<=`. The ordering is compared lexically. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#affinity) Affinity Affinities allow operators to express placement preferences. More details on how they work are described in [affinities](https://developer.hashicorp.com/nomad/docs/job-specification/affinity) The `Affinity` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`LTarget`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#ltarget-1) - Specifies the attribute to examine for the affinity. See the table of attributes [here](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation#interpreted_node_vars) . * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RTarget`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#rtarget-1) - Specifies the value to compare the attribute against. This can be a literal value, another attribute or a regular expression if the `Operator` is in "regexp" mode. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Operand`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#operand-1) - Specifies the test to be performed on the two targets. It takes on the following values: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`regexp`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#regexp-1) - Allows the `RTarget` to be a regular expression to be matched. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`set_contains_all`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#set_contains_all) - Allows the `RTarget` to be a comma separated list of values that should be contained in the LTarget's value. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`set_contains_any`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#set_contains_any) - Allows the `RTarget` to be a comma separated list of values any of which should be contained in the LTarget's value. * Comparison Operators - `=`, `==`, `is`, `!=`, `not`, `>`, `>=`, `<`, `<=`. The ordering is compared lexically. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Weight`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#weight) - A non zero weight, valid values are from -100 to 100. Used to express relative preference when there is more than one affinity. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#log-rotation) Log Rotation The `LogConfig` object configures the log rotation policy for a task's `stdout` and `stderr`. The `LogConfig` object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Disabled`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#disabled) - Disables log collection. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MaxFiles`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#maxfiles) - The maximum number of rotated files Nomad will retain for `stdout` and `stderr`, each tracked individually. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`MaxFileSizeMB`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#maxfilesizemb) - The size of each rotated file. The size is specified in `MB`. If the amount of disk resource requested for the task is less than the total amount of disk space needed to retain the rotated set of files, Nomad will return a validation error when a job is submitted. { "LogConfig": { "Disabled": false, "MaxFiles": 3, "MaxFileSizeMB": 10 } } In the above example we have asked Nomad to retain 3 rotated files for both `stderr` and `stdout` and size of each file is 10 MB. The minimum disk space that would be required for the task would be 60 MB. ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#artifact) Artifact Nomad downloads artifacts using [`go-getter`](https://github.com/hashicorp/go-getter) . The `go-getter` library allows downloading of artifacts from various sources using a URL as the input source. The key-value pairs given in the `options` block map directly to parameters appended to the supplied `source` URL. These are then used by `go-getter` to appropriately download the artifact. `go-getter` also has a CLI tool to validate its URL and can be used to check if the Nomad `artifact` is valid. Nomad allows downloading `http`, `https`, and `S3` artifacts. If these artifacts are archives (zip, tar.gz, bz2, etc.), these will be unarchived before the task is started. The `Artifact` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`GetterSource`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#gettersource) - The path to the artifact to download. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RelativeDest`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#relativedest) - An optional path to download the artifact into relative to the root of the task's directory. If omitted, it will default to `local/`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`GetterOptions`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#getteroptions) - A `map[string]string` block of options for `go-getter`. Full documentation of supported options are available [here](https://github.com/hashicorp/go-getter/tree/ef5edd3d8f6f482b775199be2f3734fd20e04d4a#protocol-specific-options-1) . An example is given below: { "GetterOptions": { "checksum": "md5:c4aa853ad2215426eb7d70a21922e794", "aws_access_key_id": "", "aws_access_key_secret": "", "aws_access_token": "" } } An example of downloading and unzipping an archive is as simple as: { "Artifacts": [\ {\ "GetterSource": "https://example.com/my.zip",\ "GetterOptions": {\ "checksum": "md5:7f4b3e3b4dd5150d4e5aaaa5efada4c3"\ }\ }\ ] } #### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#s3-examples) S3 examples S3 has several different types of addressing and more detail can be found [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro) S3 region specific endpoints can be found [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) Path based style: { "Artifacts": [\ {\ "GetterSource": "https://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"\ }\ ] } or to override automatic detection in the URL, use the S3-specific syntax { "Artifacts": [\ {\ "GetterSource": "s3::https://my-bucket-example.s3-eu-west-1.amazonaws.com/my_app.tar.gz"\ }\ ] } Virtual hosted based style { "Artifacts": [\ {\ "GetterSource": "my-bucket-example.s3-eu-west-1.amazonaws.com/my_app.tar.gz"\ }\ ] } ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#template) Template The `Template` block instantiates an instance of a template renderer. This creates a convenient way to ship configuration files that are populated from environment variables, Consul data, Vault secrets, or just general configurations within a Nomad task. Nomad utilizes a tool called [Consul Template](https://github.com/hashicorp/consul-template "Consul Template by HashiCorp") . The template can reference [Nomad's runtime environment variables](https://developer.hashicorp.com/nomad/docs/reference/runtime-environment-settings "Nomad Runtime Environment") . For a full list of the API template functions, please refer to the [Consul Template README](https://github.com/hashicorp/consul-template "Consul Template by HashiCorp") . `Template` object supports following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ChangeMode`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#changemode) - Specifies the behavior Nomad should take if the rendered template changes. The default value is `"restart"`. The possible values are: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`"noop"`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#noop) - take no action (continue running the task) * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`"restart"`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#restart) - restart the task * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`"signal"`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#signal) - send a configurable signal to the task * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`"script"`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#script) - run a script * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ChangeSignal`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#changesignal) - Specifies the signal to send to the task as a string like "SIGUSR1" or "SIGINT". This option is required if the `ChangeMode` is `signal`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`ChangeScript`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#changescript) - Configures the script triggered on template change. This option is required if the `ChangeMode` is `script`. The `ChangeScript` object supports the following attributes: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Command`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#command-1) - Specifies the full path to a script or executable that is to be executed on template change. Path is relative to the driver, e.g., if running with a container driver the path must be existing in the container. This option is required is the `change_mode` is `script`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Args`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#args-1) - List of arguments that are passed to the script that is to be executed on template change. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Timeout`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#timeout-1) - Timeout for script execution specified using a label suffix like "30s" or "1h". Default value is `"5s"`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`FailOnError`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#failonerror) - If `true`, Nomad will kill the task if the script execution fails. If `false`, script failure will be logged but the task will continue uninterrupted. Default value is `false`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`DestPath`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#destpath) - Specifies the location where the resulting template should be rendered, relative to the task directory. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`EmbeddedTmpl`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#embeddedtmpl) - Specifies the raw template to execute. One of `SourcePath` or `EmbeddedTmpl` must be specified, but not both. This is useful for smaller templates, but we recommend using `SourcePath` for larger templates. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Envvars`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#envvars) - Specifies the template should be read back as environment variables for the task. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`LeftDelim`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#leftdelim) - Specifies the left delimiter to use in the template. The default is "{{" for some templates, it may be easier to use a different delimiter that does not conflict with the output file itself. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Perms`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#perms) - Specifies the rendered template's permissions. File permissions are given as octal of the Unix file permissions `rwxrwxrwx`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Uid`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#uid) - Specifies the rendered template owner's user ID. **Caveat:** Works only on Unix-based systems. Be careful when using containerized drivers, such as `docker` or `podman`, as groups and users inside the container may have different IDs than on the host system. This feature will also **not** work with Docker Desktop. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Gid`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#gid) - Specifies the rendered template owner's group ID. **Caveat:** Works only on Unix-based systems. Be careful when using containerized drivers, such as `docker` or `podman`, as groups and users inside the container may have different IDs than on the host system. This feature will also **not** work with Docker Desktop. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`RightDelim`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#rightdelim) - Specifies the right delimiter to use in the template. The default is "}}" for some templates, it may be easier to use a different delimiter that does not conflict with the output file itself. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`SourcePath`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#sourcepath) - Specifies the path to the template to be rendered. `SourcePath` is mutually exclusive with `EmbeddedTmpl` attribute. The source can be fetched using an [`Artifact`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#artifact) resource. The template must exist on the machine prior to starting the task; it is not possible to reference a template inside of a Docker container, for example. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Splay`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#splay) - Specifies a random amount of time to wait between 0 ms and the given splay value before invoking the change mode. Should be specified in nanoseconds. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`VaultGrace`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#vaultgrace) - [Deprecated](https://github.com/hashicorp/consul-template/issues/1268) { "Templates": [\ {\ "SourcePath": "local/config.conf.tpl",\ "DestPath": "local/config.conf",\ "EmbeddedTmpl": "",\ "ChangeMode": "signal",\ "ChangeSignal": "SIGUSR1",\ "Splay": 5000000000\ }\ ] } ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spread) Spread Spread allow operators to target specific percentages of allocations based on any node attribute or metadata. More details on how they work are described in [spread](https://developer.hashicorp.com/nomad/docs/job-specification/spread) . The `Spread` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Attribute`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#attribute) - Specifies the attribute to examine for the spread. See the [table of attributes](https://developer.hashicorp.com/nomad/docs/reference/runtime-variable-interpolation#interpreted_node_vars) for examples. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`SpreadTarget`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#spreadtarget) - Specifies a list of attribute values and percentages. This is an optional field, when left empty Nomad will evenly spread allocations across values of the attribute. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Value`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#value-1) - The value of a specific target attribute, like "dc1" for `${node.datacenter}`. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Percent`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#percent) - Desired percentage of allocations for this attribute value. The sum of all spread target percentages must add up to 100. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Weight`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#weight-1) - A non zero weight, valid values are from -100 to 100. Used to express relative preference when there is more than one spread or affinity. [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) ### [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#scaling-1) Scaling Scaling policies allow operators to attach autoscaling configuration to a task group. This information can be queried by [external autoscalers](https://github.com/hashicorp/nomad-autoscaler) . The `Scaling` object supports the following keys: * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Min`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#min) - The minimum allowable count for the task group. This is optional; if absent, the default is the `Count` specified in the task group. Attempts to set the task group `Count` below `Min` will result in a 400 error during job registration. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Max`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#max) - The maximum allowable count for the task group. This is required if a scaling policy is provided. This must be larger than `Min`. Attempts to set the task group `Count` above `Max` wil result in a 400 error during job registration. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Enabled`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#enabled-1) - An optional boolean (default: `true`). This indicates to the autoscaler that this scaling policy should be ignored. It is intended to allow autoscaling to be temporarily disabled for a task group. * [](https://developer.hashicorp.com/nomad/api-docs/json-jobs#) [`Policy`](https://developer.hashicorp.com/nomad/api-docs/json-jobs#policy) - An optional JSON block. This is opaque to Nomad; see the documentation for the external autoscaler (e.g., [nomad-autoscaler](https://github.com/hashicorp/nomad-autoscaler) ). [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/json-jobs.mdx) --- # What is Nomad? | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.10.x. [View latest version](https://developer.hashicorp.com/nomad/docs/what-is-nomad) . Introduction to Nomad ===================== Welcome to the intro guide to Nomad. This guide is the best place to start with Nomad. We cover what Nomad is, what problems it can solve, how it compares to existing software, and how you can get started using it. If you are familiar with the basics of Nomad, the [documentation](https://developer.hashicorp.com/nomad/docs/v1.10.x) and [tutorials](https://developer.hashicorp.com/nomad/tutorials) provide a more detailed reference of available features. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#what-is-nomad) What is Nomad? ------------------------------------------------------------------------------------------------- Nomad is a flexible workload orchestrator that enables an organization to easily deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run a diverse workload of Docker, non-containerized, microservice, and batch applications. Nomad enables developers to use declarative infrastructure-as-code for deploying applications. Nomad uses bin packing to efficiently schedule jobs and optimize for resource utilization. Nomad is supported on macOS, Windows, and Linux. Nomad is widely adopted and used in production by PagerDuty, Target, Citadel, Trivago, SAP, Pandora, Roblox, eBay, Deluxe Entertainment, and more. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#key-features) Key features ---------------------------------------------------------------------------------------------- * **Deploy Containers and Legacy Applications**: Nomad’s flexibility as an orchestrator enables an organization to run containers, legacy, and batch applications together on the same infrastructure. Nomad brings core orchestration benefits to legacy applications without needing to containerize via pluggable [task drivers](https://developer.hashicorp.com/nomad/docs/v1.10.x/job-declare/task-driver) . * **Simple & Reliable**: Nomad runs as a single binary and is entirely self contained - combining resource management and scheduling into a single system. Nomad does not require any external services for storage or coordination. Nomad automatically handles application, node, and driver failures. Nomad is distributed and resilient, using leader election and state replication to provide high availability in the event of failures. * **Device Plugins & GPU Support**: Nomad offers built-in support for GPU workloads such as machine learning (ML) and artificial intelligence (AI). Nomad uses [device plugins](https://developer.hashicorp.com/nomad/plugins/v1.10.x/devices) to automatically detect and utilize resources from hardware devices such as GPU, FPGAs, and TPUs. * **Federation for Multi-Region**: Nomad has native support for multi-region federation. This built-in capability allows multiple clusters to be linked together, which in turn enables developers to deploy jobs to any cluster in any region. Federation also enables automatic replication of ACL policies, namespaces, resource quotas and Sentinel policies across all clusters. * **Proven Scalability**: Nomad is optimistically concurrent, which increases throughput and reduces latency for workloads. Nomad has been proven to scale to clusters of 10K+ nodes in real-world production environments. * **HashiCorp Ecosystem**: Nomad integrates seamlessly with Terraform, Consul, Vault for provisioning, service discovery, and secrets management. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#how-nomad-compares-to-other-tools) How Nomad compares to other tools ---------------------------------------------------------------------------------------------------------------------------------------- The following characteristics generally differentiate Nomad from related products: * **Simplicity**: Nomad runs as a single process with zero external dependencies. Operators can easily provision, manage, and scale Nomad. Developers can easily define and run applications. * **Flexibility**: Nomad can run a diverse workload of containerized, legacy, microservice, and batch applications. Nomad can schedule service, batch processing and system jobs, and can run on both Linux and Windows. * **Scalability and High Performance**: Nomad can schedule thousands of containers per second, scale to thousands of nodes in a single cluster, and easily federate across regions and cloud providers. * **HashiCorp Interoperability**: Nomad elegantly integrates with Vault for secrets management and Consul for service discovery and dynamic configuration. Nomad's Consul-like architecture and Terraform-like job specification lower the barrier to entry for existing users of the HashiCorp stack. There are many relevant categories for comparison including cluster managers, resource managers, workload managers, and schedulers. There are many existing tools in each category, and the comparisons are not exhaustive of the entire space. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#nomad-versus-kubernetes) Nomad versus Kubernetes -------------------------------------------------------------------------------------------------------------------- Kubernetes and Nomad support similar core use cases for application deployment and management, but they differ in a few key ways. Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, and secrets management. Nomad only aims to focus on cluster management and scheduling, and Nomad is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#simplicity) Simplicity Kubernetes is designed as a collection of more than a half-dozen interoperating services which together provide the full functionality. Coordination and storage is provided by etcd at the core. The state is wrapped by API controllers which are consumed by other services that provide higher level APIs for features like scheduling. Kubernetes supports running in a highly available configuration but is operationally complex to setup. Nomad is architecturally much simpler. Nomad is a single binary, both for clients and servers, and requires no external services for coordination or storage. Nomad combines a lightweight resource manager and a sophisticated scheduler into a single system. By default, Nomad is distributed, highly available, and operationally simple. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#flexible-workload-support) Flexible Workload Support While Kubernetes is specifically focused on Linux containers, Nomad is more general purpose. Nomad supports virtualized, containerized and standalone applications, including Docker, Java, IIS on Windows, Qemu, etc. Nomad is designed with extensible drivers and support will be extended to all common drivers. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#consistent-deployment) Consistent Deployment A full Kubernetes installation for a production environment is time consuming, operationally complex, and resource intensive. An increasing number of implementations are created by the Kubernetes community to mitigate these challenges, such as minikube, kubeadm, k3s, and more. These trimmed versions of Kubernetes offer easier adoption for development and testing, but lead to inconsistency in capabilities, configuration, and management when moving into production. In contrast to Kubernetes' fragmented distributions, Nomad as a single lightweight binary can be deployed in local dev, production, on-prem, at the edge, and in the cloud in a consistent manner, and provides the same operational ease-of-use across all environments. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#scalability) Scalability [Kubernetes documentation](https://kubernetes.io/docs/setup/best-practices/cluster-large/) states that they support clusters up to 5,000 nodes and 300,000 total containers. As the environment grows, the interoperating components with different constraints compound the operational complexity. [Even operators at Google revealed the significant challenges of managing the system at scale](https://blog.dave.tf/post/new-kubernetes/) . The lack of maturity in the Federation project and the additional overhead of managing a centralized management plane also make it a hard experience to deploy a distributed system that spans multiple clusters. Nomad has been proven to scale to cluster sizes that exceed 10,000 nodes in real-world production environments. It can be deployed across multiple availability zones, regions, and data centers with a single cluster or multiple clusters. Nomad is designed to natively handle multi-cluster deployments without the overhead of running clusters on clusters. This makes it easier to scale the application deployment across multiple datacenters, regions, and clouds with no additional complexity. Nomad has performed strenuous benchmark on scalability with [1 million container challenge](https://www.hashicorp.com/c1m) in 2016 and [2 million container challenge](https://www.hashicorp.com/c2m) in 2020. These tests are aimed to validate Nomad's architectural design and ensure that Nomad performs under the most extreme requirements. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#supplement-to-kubernetes) Supplement to Kubernetes Enterprises are comprised of multiple groups of people (business units) with different projects, infrastructure environments, technical competencies, team sizes, budgets, and SLAs. Each group has different requirements and leverages technologies based on their particular needs and constraints. Medium to large scale enterprises run into challenges when trying to standardize hundreds to thousands of software developers and administrators onto one single orchestrator (Kubernetes, Nomad, Mesos) as no scheduler today fits all applications, environments, projects, and teams. Companies in the Global 2000 today such as Intel, Autodesk and GitHub with multiple products and business units organically run Nomad and Kubernetes to supplement each other. They leverage each scheduler to its strengths with Kubernetes for its cutting edge ecosystem and Nomad for simple maintenance and flexibility in core scheduling. ![How organizations leverage Nomad and Kubernetes](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/v1.10.x/img/nomad-kubernetes.png) These are the characteristics we see in teams that typically adopt self-hosted Kubernetes: * Greenfield use cases such as machine learning (ML), serverless, and big data that require the Kubernetes ecosystem and Helm chart * High budget and full-time staffing to maintain Kubernetes * High-profile projects with significant investment and long-term timeline (multi-year) * Deploying and managing new, cloud-native applications * Public cloud environment such as AWS, GCP, Azure Characteristics of teams that typically adopt Nomad: * Run a mix of containerized and non-containerized workloads (Windows, Java) * Small/medium-sized teams with limited capacity to maintain an orchestrator * Deploying and managing core, existing applications * On-premises environment, or hybrid environments * Require simplicity to move fast and fulfill business needs with hard deadlines We continue to see small enterprises continue to standardize on a single orchestrator given the natural staffing and organizational constraints. There are not enough DevOps members to maintain more than one orchestrator, not enough developers to warrant diverging workflows, or simply not enough workload diversity to require more than one orchestrator. ### [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#resources) Resources Review the following resources for in-depth comparisons between Nomad and Kubernetes: * [A Kubernetes User's Guide to HashiCorp Nomad](https://www.hashicorp.com/en/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad) * [The Kubernetes to Nomad Cheat Sheet](https://www.hashicorp.com/en/blog/the-kubernetes-to-nomad-cheat-sheet) * [A Kubernetes User's Guide to HashiCorp Nomad Secret Management](https://www.hashicorp.com/en/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad-secret-management) [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#nomad-versus-aws-ecs) Nomad versus AWS ECS -------------------------------------------------------------------------------------------------------------- Amazon Web Services provides the Elastic Container Service (ECS), which is a cluster manager. The ECS service is only available within AWS and can only be used for Docker workloads. Amazon provides customers with the agent that is installed on EC2 instances, but does not provide the servers which are a hosted service of AWS. There are a number of fundamental differences between Nomad and ECS. Nomad is completely open source, including both the client and server components. By contrast, only the agent code for ECS is open and the servers are closed sourced and managed by Amazon. As a side effect of the ECS servers being managed by AWS, it is not possible to use ECS outside of AWS. Nomad is agnostic to the environment in which it is run, supporting public and private clouds, as well as bare metal datacenters. Clusters in Nomad can span multiple datacenters and regions, meaning a single cluster could be managing machines on AWS, Azure, and GCE simultaneously. The ECS service is specifically focused on containers and the Docker engine, while Nomad is more general purpose. Nomad supports virtualized, containerized, and standalone applications, including Docker. Nomad is designed with extensible drivers and support will be extended to all common drivers. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#nomad-versus-terraform) Nomad versus Terraform ------------------------------------------------------------------------------------------------------------------ [Terraform](https://www.terraform.io/) is a tool for building, changing, and versioning infrastructure safely and efficiently. Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied. Nomad differs from Terraform in a number of key ways. Terraform is designed to support any type of resource including low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc. Terraform knows how to create, provision, and manage the lifecycle of these resources. Nomad runs on existing infrastructure and manages the lifecycle of applications running on that infrastructure. Another major distinction is that Terraform is an offline tool that runs to completion, while Nomad is an online system with long lived servers. Nomad allows new jobs to be submitted, existing jobs updated or deleted, and can handle node failures. This requires operating continuously instead of in a single shot like Terraform. For small infrastructures with only a handful of servers or applications, the complexity of Nomad may not outweigh simply using Terraform to statically assign applications to machines. At larger scales, Terraform should be used to provision capacity for Nomad, and Nomad used to manage scheduling applications to machines dynamically. [](https://developer.hashicorp.com/nomad/docs/v1.10.x/what-is-nomad#next-steps) Next steps ------------------------------------------------------------------------------------------ Review [use cases](https://developer.hashicorp.com/nomad/docs/v1.10.x/use-cases) to understand the many ways Nomad is used in production today across many industries to solve critical, real-world business objectives. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.10.x/content/docs/what-is-nomad.mdx) --- # Connect nodes into a cluster | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Connect nodes into a cluster ============================ In order to create a Nomad cluster out of individual nodes, you need to introduce them to one another. There are several ways to perform this: * Manual bootstrap * Cloud Auto-Join * Consul This guide describes each method and provides configuration snippets, which you can use as starting points for your own configuration. You may also use client node introduction tokens to restrict which clients join your cluster. [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#manual-clustering) Manual clustering ---------------------------------------------------------------------------------------------------------------- Manually bootstrapping a Nomad cluster does not rely on additional tooling, but does require operator participation in the cluster formation process. When bootstrapping, Nomad servers and clients must be started and informed with the address of at least one Nomad server. As you can tell, this creates a chicken-and-egg problem where one server must first be fully bootstrapped and configured before the remaining servers and clients can join the cluster. This requirement can add additional provisioning time as well as ordered dependencies during provisioning. First, you need to bootstrap a single Nomad server and capture its IP address. Place this address in the configuration once you have that nodes IP address. For Nomad servers, this configuration may look something like this: server { enabled = true bootstrap_expect = 3 # This is the IP address of the first server provisioned server_join { retry_join = [":4648"] } } Alternatively, you can supply a server's address after the servers have all been started by running the [`server join` command](https://developer.hashicorp.com/nomad/commands/server/join) on the servers individually to cluster the servers. All servers can join one other server, and then rely on the gossip protocol to discover the rest. $ nomad server join For Nomad clients, the configuration may look something like: client { enabled = true server_join { retry_join = [":4647"] } } The client node's server list can be updated at run time using the [`node config` command](https://developer.hashicorp.com/nomad/commands/node/config) . $ nomad node config -update-servers :4647 The port corresponds to the RPC port. If no port is specified with the IP address, the default RPC port of `4647` is assumed. As servers are added or removed from the cluster, this information is pushed to the client. This means only one server must be specified because, after initial contact, the full set of servers in the client's region are shared with the client. [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#use-cloud-auto-join) Use cloud auto-join -------------------------------------------------------------------------------------------------------------------- The [`retry_join`](https://developer.hashicorp.com/nomad/docs/configuration/server_join#retry_join) parameter accepts a unified interface using the [go-discover](https://github.com/hashicorp/go-discover) library for doing automatic cluster joining using cloud metadata. To use retry-join with a supported cloud provider, specify the configuration on the command line or configuration file as a `key=value key=value ...` string. Values are taken literally and must not be URL encoded. If the values contain spaces, backslashes or double quotes they need to be double quoted and the usual escaping rules apply. { "retry_join": ["provider=my-cloud config=val config2=\"some other val\" ..."] } Consult the [cloud provider-specific configurations](https://developer.hashicorp.com/nomad/docs/configuration/server_join#cloud-auto-join) in the cloud-autojoin documentation. This can be combined with static IP or DNS addresses or even multiple configurations for different providers. In order to use discovery behind a proxy, you will need to set `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables per [Golang `net/http` library](https://golang.org/pkg/net/http/#ProxyFromEnvironment) . [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#use-consul-to-automatically-cluster-nodes) Use Consul to automatically cluster nodes ---------------------------------------------------------------------------------------------------------------------------------------------------------------- To automatically bootstrap a Nomad cluster, Nomad can leverage another HashiCorp open source tool, [Consul](https://developer.hashicorp.com/consul) . Bootstrapping Nomad is easiest against an existing Consul cluster. The Nomad servers and clients will become informed of each other's existence when the Consul agent is installed and configured on each host. As an added benefit, integrating Consul with Nomad provides service and health check registration for applications which later run under Nomad. Consul models infrastructures as datacenters and multiple Consul datacenters can be connected over the WAN so that clients can discover nodes in other datacenters. Since Nomad regions can encapsulate many datacenters, you should be running a Consul cluster in every Nomad region and connecting them over the WAN. Refer to the Consul tutorial for both [bootstrapping](https://developer.hashicorp.com/consul/docs/deploy/server/vm/bootstrap) a single datacenter and [connecting multiple Consul clusters over the WAN](https://developer.hashicorp.com/consul/docs/east-west/wan-federation) . If a Consul agent is installed on the host prior to Nomad starting, the Nomad agent will register with Consul and discover other nodes. For servers, you must inform the cluster how many servers you expect to have. This is required to form the initial quorum, since Nomad is unaware of how many peers to expect. For example, to form a region with three Nomad servers, you would use the following Nomad configuration file: # /etc/nomad.d/server.hcl # data_dir tends to be environment specific. data_dir = "/opt/nomad/data" server { enabled = true bootstrap_expect = 3 } This configuration would be saved to disk and then run: $ nomad agent -config=/etc/nomad.d/server.hcl A similar configuration is available for Nomad clients: # /etc/nomad.d/client.hcl datacenter = "dc1" # data_dir tends to be environment specific. data_dir = "/opt/nomad/data" client { enabled = true } The agent is started in a similar manner: $ sudo nomad agent -config=/etc/nomad.d/client.hcl Nomad clients should always run as root (or with `sudo`). The above configurations include no IP or DNS addresses between the clients and servers. This is because Nomad detected the existence of Consul and utilized service discovery to form the cluster. ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#consul-auto-join-internals) Consul auto-join internals This section discusses the internals of the Consul and Nomad integration at a very high level. Reading is only recommended for those curious to the implementation. As discussed in the previous section, Nomad merges multiple configuration files together, so the `-config` may be specified more than once: $ nomad agent -config=base.hcl -config=server.hcl In addition to merging configuration on the command line, Nomad also maintains its own internal configurations (called "default configs") which include reasonable base defaults. One of those default configurations includes a "consul" block, which specifies reasonable defaults for connecting to and integrating with Consul. In essence, this configuration file resembles the following: # You do not need to add this to your configuration file. This is an example # that is part of Nomad's internal default configuration for Consul integration. consul { # The address to the Consul agent. address = "127.0.0.1:8500" # The service name to register the server and client with Consul. server_service_name = "nomad" client_service_name = "nomad-client" # Enables automatically registering the services. auto_advertise = true # Enabling the server and client to bootstrap using Consul. server_auto_join = true client_auto_join = true } Refer to the [`consul` stanza](https://developer.hashicorp.com/nomad/docs/configuration/consul) documentation for the complete set of configuration options. [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#use-client-node-introduction-tokens) Use client node introduction tokens ---------------------------------------------------------------------------------------------------------------------------------------------------- Use client introduction tokens to restrict which clients join your cluster. The client node introduction feature is like multi-factor authentication for your Nomad clusters. **It does not replace mTLS**, but instead adds an additional layer of security that prevents an unauthorized or misconfigured client from joining a Nomad cluster. When you generate a client introduction token, you may specify the following optional parameters to further secure cluster access: * Node pool: The node pool that clients with this token may join. This token is not valid with any other node pool. * Node name: The token is scoped to the node with this name. No other node may use this token to join the cluster. * TTL: Token expiration. The token is not valid after expiration. ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#create-a-client-introduction-token) Create a client introduction token Follow these steps to use client node introduction tokens: 1. [Create an ACL node policy](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#create-an-acl-policy) . 2. [Create an ACL role from the policy](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#create-an-acl-role) . 3. [Generate a client introduction token](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#generate-a-client-introduction-token) . 4. [Update your Nomad server configuration to use client introduction](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#update-your-nomad-server-configuration) . 5. [Start the Nomad server](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#start-the-nomad-server) . 6. [Monitor client join failures](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#monitor-client-join-failures) . ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#prerequisites) Prerequisites You bootstrapped the ACL system and configured the CLI to use your management token. Refer to the [Bootstrap the ACL system guide](https://developer.hashicorp.com/nomad/docs/secure/acl/bootstrap) for instructions. ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#create-an-acl-policy) Create an ACL policy This example creates the `node:write` policy required to generate tokens. 1. Create a policy file called `client-introduction.hcl`. node { policy = "write" } 2. Add the policy to the cluster. nomad acl policy apply client-introduction client-introduction.hcl ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#create-an-acl-role) Create an ACL role Create an ACL role with the `client-introduction` policy. nomad acl role create --tls-skip-verify -name="client-introduction" \ -policy="client-introduction" The output describes the ACL role, including its attached policies and the Raft index at its creation. ID = cf0b4a43-b00f-cc30-b656-b34d66151b04 Name = client-introduction Description = Policies = client-introduction Create Index = 117 Modify Index = 117 ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#generate-a-client-introduction-token) Generate a client introduction token Use the [`nomad node intro create` command](https://developer.hashicorp.com/nomad/commands/node/intro/create) to generate the client introduction token, which is a JSON Web Token (JWT). This example scopes the token to a node pool named `staging` and writes the result to a file called `intro_token.jwt`. nomad node intro create -node-pool=staging > intro_token.jwt The `intro-token.jwt` file contains the JWT. "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..." ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#update-your-nomad-server-configuration) Update your Nomad server configuration Configure your Nomad server's `server.client_introduction` block. This example sets strict enforcement, which means the server rejects any client without a valid token. Refer to the [`server.client_introduction` block documentation](https://developer.hashicorp.com/nomad/docs/configuration/server#client_introduction-parameters) for additional enforcement options. data_dir = "/opt/nomad/" acl { enabled = true } server { enabled = true bootstrap_expect = 1 client_introduction { enforcement = "strict" # Default = "warn" default_identity_ttl = "5m" # Default = "5m" max_identity_ttl = "30m" # Default = "30m" } } tls { http = true rpc = true ca_file = "/opt/nomad/tls/nomad-agent-ca.pem" cert_file = "/opt/nomad/tls/global-server-nomad.pem" key_file = "/opt/nomad/tls/global-server-nomad-key.pem" } No additional configuration is required on the client node. ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#start-the-nomad-server) Start the Nomad server You have the following options for passing the client introduction token to the Nomad server: * Set the token as the value of a `NOMAD_CLIENT_INTRO_TOKEN` environment variable. * Use the [`-client-intro-token` parameter](https://developer.hashicorp.com/nomad/commands/agent#client-intro-token) of the `nomad agent` command. * Place the `intro_token.jwt` file in the client's state directory, which is by default [the `/client_state_dir>` directory](https://developer.hashicorp.com/nomad/docs/configuration/client#state_dir) . This example starts the server with the client introduction token passed in the `-client-intro-token` parameter. nomad agent -config /etc/nomad.d/nomad.hcl \ -client-intro-token "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..." ### [](https://developer.hashicorp.com/nomad/docs/deploy/clusters/connect-nodes#monitor-client-join-failures) Monitor client join failures You have the following options to determine when client registration fails: * Check the server logs for `[ERROR] nomad.client: node registration without introduction token` messages. * Monitor the [`nomad.client.introduction.enforcement` counter](https://developer.hashicorp.com/nomad/docs/monitor#client-introduction) , which increments when a client tries to join without a valid client introduction token. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/deploy/clusters/connect-nodes.mdx) --- # Nomad command-line interface (CLI) reference | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.9.x. [View latest version](https://developer.hashicorp.com/nomad/commands) . Nomad command-line interface (CLI) reference ============================================ The Nomad CLI is a well-behaved command-line application. In erroneous cases, the CLI returns a non-zero exit status. To view a list of available commands, run the `nomad` command with no arguments, `-h`, or `--help`. To access help for any specific subcommand, run the subcommand with the `-h` argument. [](https://developer.hashicorp.com/nomad/commands/v1.9.x#autocomplete) Autocomplete ----------------------------------------------------------------------------------- Nomad's CLI supports command autocomplete. To install autocomplete, run the `nomad` command with the `-autocomplete-install` option. $ nomad -autocomplete-install To uninstall autocomplete, run the `nomad` command with the `-autocomplete-uninstall` option. $ nomad -autocomplete-uninstall [](https://developer.hashicorp.com/nomad/commands/v1.9.x#command-contexts) Command contexts ------------------------------------------------------------------------------------------- Nomad's CLI commands have implied contexts in their naming convention. Because the CLI is most commonly used to manipulate or query jobs, you can assume that any given command is working in that context unless the command name implies otherwise. For example, the `nomad job run` command runs a new job and the `nomad status` command queries information about existing jobs. Conversely, commands with a prefix in their name likely operate in a different context. Examples include the `nomad agent-info` or `nomad node drain` commands, which operate in the agent or node contexts respectively. ### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#remote-usage) Remote usage You may use the Nomad CLI to interact with a remote Nomad cluster, even when the local machine does not have a running Nomad agent. To do so, set the `NOMAD_ADDR` environment variable. $ NOMAD_ADDR=https://remote-address:4646 $ nomad status Or use the `-address=` flag when running commands. $ nomad status -address=https://remote-address:4646 The Nomad address must be reachable from your local machine. If the Nomad port is exposed to the public internet, we recommend configuring TLS. Refer to the [`tls` block in agent configuration](https://developer.hashicorp.com/nomad/docs/v1.9.x/configuration/tls) for details. ### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#environment-variables) Environment variables Nomad can use environment variables to configure command-line tool options. You may override these environment variables with individual flags. Except where noted, these variables influence the behavior of the Nomad CLI and should not be set for Nomad agents. #### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#connection-environment-variables) Connection environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_addr) - The address of the Nomad server. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_REGION`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_region) - The region of the Nomad server to forward commands to. Defaults to the Agent's local region * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_NAMESPACE`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_namespace) - The target namespace for queries and actions bound to a namespace. If set to `*`, job and alloc subcommands query all namespacecs authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_HTTP_AUTH`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_http_auth) - (Optional) This allows users to supply "Basic" HTTP authentication scheme ([RFC 7617](https://tools.ietf.org/html/rfc7617) ) information in environments where the Nomad API is behind an authenticating proxy server. #### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#access-control-list-acl-environment-variables) Access control list (ACL) environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_token) - The SecretID of an ACL token to use to authenticate API requests with. #### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#cli-environment-variables) CLI environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CLI_NO_COLOR`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_cli_no_color) - Disables colored command output. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CLI_SHOW_HINTS`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_cli_show_hints) - Enables ui-hints in common CLI command output. #### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#mutual-tls-mtls-environment-variables) Mutual TLS (mTLS) environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CLIENT_CERT`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_client_cert) - Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `NOMAD_CLIENT_KEY`. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CLIENT_KEY`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_client_key) - Path to an unencrypted PEM encoded private key matching the client certificate from `NOMAD_CLIENT_CERT`. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CACERT`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_cacert) - Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_CAPATH`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_capath) - Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `NOMAD_CACERT` and `NOMAD_CAPATH` are specified, `NOMAD_CACERT` is used. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_SKIP_VERIFY`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_skip_verify) - Do not verify TLS certificate. **This is highly not recommended.** * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_TLS_SERVER_NAME`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_tls_server_name) - The server name to use as the SNI host when connecting via TLS. #### [](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad-enterprise-license-environment-variables) Nomad Enterprise license environment variables These environment variables influence the Nomad Enterprise license configuration. These values are only used for Nomad Enterprise agents, not the Nomad CLI. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_LICENSE_PATH`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_license_path) - An absolute path to a Nomad Enterprise license file, for example `/etc/nomad.d/license.hclic`. * [](https://developer.hashicorp.com/nomad/commands/v1.9.x#) [`NOMAD_LICENSE`](https://developer.hashicorp.com/nomad/commands/v1.9.x#nomad_license) - The Nomad Enterprise license file contents as a string. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.9.x/content/commands/index.mdx) --- # Regions - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Regions HTTP API ================ The `/regions` endpoints list all known regions. [](https://developer.hashicorp.com/nomad/api-docs/regions#list-regions) List Regions ------------------------------------------------------------------------------------ | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/regions` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/regions#sample-request) Sample Request $ curl \ https://localhost:4646/v1/regions ### [](https://developer.hashicorp.com/nomad/api-docs/regions#sample-response) Sample Response ["region1", "region2"] [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/regions.mdx) --- # Status - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Status HTTP API =============== The `/status` endpoints query the Nomad system status. [](https://developer.hashicorp.com/nomad/api-docs/status#read-leader) Read Leader --------------------------------------------------------------------------------- This endpoint returns the address of the current leader in the region. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/status/leader` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/status#sample-request) Sample Request $ curl \ https://localhost:4646/v1/status/leader ### [](https://developer.hashicorp.com/nomad/api-docs/status#sample-response) Sample Response "127.0.0.1:4647" [](https://developer.hashicorp.com/nomad/api-docs/status#list-peers) List Peers ------------------------------------------------------------------------------- This endpoint returns the set of raft peers in the region. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/status/peers` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/status#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/status/peers ### [](https://developer.hashicorp.com/nomad/api-docs/status#sample-response-1) Sample Response ["127.0.0.1:4647"] [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/status.mdx) --- # Exec2 task driver plugin | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Exec2 task driver plugin ======================== Name: `exec2` The `exec2` driver is used to execute a command for a task. It offers a security model optimized for running 'ordinary' processes with very low startup times and minimal overhead in terms of CPU, disk, and memory utilization. The `exec2` driver leverages kernel features such as the [Landlock LSM](https://github.com/shoenig/go-landlock) , cgroups v2, and the [`unshare`](https://www.man7.org/linux/man-pages/man1/unshare.1.html) system utility. Review the source on [GitHub](https://github.com/hashicorp/nomad-driver-exec2) . [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#installation) Installation ----------------------------------------------------------------------------------------- Manual installationUbuntu/DebianCentOS/RHEL You can download a [precompiled binary](https://releases.hashicorp.com/nomad-driver-exec2/) and verify the binary using the available SHA-256 sums. After downloading nomad-driver-exec2 driver, unzip the package. Make sure that the `nomad-driver-exec2` binary is available on your [plugin\_dir](https://developer.hashicorp.com/nomad/docs/configuration#plugin_dir) path, specified by the client's config file, before continuing with the other guides. Install the required packages. $ sudo apt-get update && \ sudo apt-get install wget gpg coreutils Add the HashiCorp \[GPG key\]\[gpg-key\]. $ wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg Add the official HashiCorp Linux test repository. $ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) test" | sudo tee /etc/apt/sources.list.d/hashicorp.list Update and install. $ sudo apt-get update && sudo apt-get install nomad-driver-exec2 Install `yum-config-manager` to manage your repositories. $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. $ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo Edit the repo file at `/etc/yum.repos.d/hashicorp.repo` and set `enabled=1` for `[hashicorp-test]`. Install. $ sudo yum -y install nomad-driver-exec2 [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#usage) Usage --------------------------------------------------------------------------- The example below is a short demonstration of what a task making use of the exec2 task driver looks like: job "http" { group "web" { task "python" { driver = "exec2" config { command = "python3" args = ["-m", "http.server", "8080", "--directory", "${NOMAD_TASK_DIR}"] unveil = ["r:/etc/mime.types"] } } } } [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#client-requirements) Client Requirements ------------------------------------------------------------------------------------------------------- The `exec2` task driver is not built into Nomad. It must be [downloaded](https://releases.hashicorp.com/nomad-driver-exec2) onto the client host in the configured plugin directory. * Linux 5.15+ * Cgroups v2 enabled * Landlock LSM enabled * Commands `nsenter` and `unshare` * Nomad client running as root [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#capabilities) Capabilities ----------------------------------------------------------------------------------------- The `exec2` task driver implements the following driver [capabilities](https://developer.hashicorp.com/nomad/plugins/author/task-driver#task-driver-plugin-api) | Feature | Implementation | | --- | --- | | `nomad alloc signal` | true | | `nomad alloc exec` | false | | filesystem isolation | unveil | | network isolation | host, group, none | | volume mounting | false | [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#concepts) Concepts --------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#filesystem-isolation) Filesystem Isolation ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#landlock) landlock The `exec2` task driver makes use of [`go-landlock`](https://github.com/shoenig/go-landlock) for providing filesystem isolation, making the host filesystem unreachable except where explicitly allowed. By default a task is enabled to access its task directory and its shared alloc directory. The paths to these directories are accessible by reading the environment variables `$NOMAD_TASK_DIR` and `$NOMAD_ALLOC_DIR` respectively. A file access mode must also be specified when granting additional filesystem access to a path. This is done by prefix the path with `'r'`, `'w'`, `'x'`, and/or `'c'` indicating read, write, executable, and create permissions, respectively. e.g., * `r:/srv/www` - read-only access to \`/srv/www\` * `rwc:/tmp` - read, write, and create files in \`/tmp\` * `rx:/opt/bin/application` - read and execute a specific application * `wc:/var/log` - write and create files in \`/var/log\` This style of permission control is modeled after the [`unveil`](https://man.openbsd.org/unveil) system call pioneered by the OpenBSD project. In configuration parameters we refer to "unveil"-ing of filesystem paths as `exec2` is leveraging landlock to emulate the semantics of `unveil`. ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#dynamic-workload-users) dynamic workload users While landlock prevents tasks from accessing the host filesystem, Nomad 1.8 introduces `dynamic workload users` which enable tasks to be run as a UID/GID that is not assigned to any user. This provides protection from non-root users getting access inside the task and allocation directories created for the task. To make use of a dynamic workload user, simply leave the `user` field blank in the task definition of an `exec2` task. ### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#resource-isolation) Resource Isolation Similar to `exec` and other container runtimes, `exec2` makes use of cgroups for limiting the amount of CPU and RAM a task may consume. [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#configuration) Configuration ------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#plugin-configuration) Plugin Configuration The default plugin configuration is shown below. System default paths are enabled, but nothing else. These default paths enable basic functionality like reading system TLS certificates, executing programs in `/bin`, `/usr/bin`, and accessing system shared object files. The exact set of paths is system dependent, and can be disabled or customized in plugin config. The default set of paths are listed below. These paths are enabled only if they are found to exist at the time of the task launching. ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#shared-objects) shared objects rx:/lib rx:/lib64 rx:/usr/lib rx:/usr/libexec rx:/usr/local/lib rx:/usr/local/lib64 r:/etc/ld.so.conf r:/etc/ld.so.cache r:/etc/ld.so.conf.d ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#io-common) io, common rwc:/tmp rw:/dev/null r:/dev/zero r:/dev/fd rw:/dev/stdin rw:/dev/stdout r:/dev/urandom w:/dev/log r:/usr/share/locale r:/proc/self/cmdline r:/usr/share/zoneinfo r:/usr/share/common-licenses r:/proc/sys/kernel/ngroups_max r:/proc/sys/kernel/cap_last_cap r:/proc/sys/vm/overcommit_memory ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#dns) dns r:/etc/hosts r:/hostname r:/etc/services r:/etc/protocols r:/etc/resolv.conf ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#certificates) certificates r:/etc/ssl/certs r:/etc/pki/tls/certs r:/etc/ssl/ca-bundle.pem r:/etc/pki/tls/cacert.pem r:/etc/pki/ca-trust-extracted/pem/tls-ca-bundle.pem r:/etc/ssl/cert.pem Additional allowable paths can be specified at the plugin level, which applies to all tasks making use of the `exec2` task driver, or at the task level, which will apply specifically to each task. plugin "nomad-driver-exec2" { config { unveil_defaults = true unveil_paths = [] unveil_by_task = false } } * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`unveil_defaults`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#unveil_defaults) - (default: `true`) - enable or disable default system paths useful for running basic commands * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`unveil_paths`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#unveil_paths) - (default: `[]`) - a list of filesystem paths with permissions to grant all tasks unveil_paths = ["rx:/opt/bin", "r:/srv/certs"] * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`unveil_by_task`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#unveil_by_task) - (default: `false`) - enable or disable job submitters to specify additional filesystem path access within task config ### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#task-configuration) Task Configuration ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#config) config Task configuration for an exec2 task includes setting a `command`, `args` for the command, and additional `unveil` filesystem paths if `unveil_by_task` is enabled in plugin configuration. config { command = "/usr/bin/cat" args = ["/etc/os-release"] unveil = ["r:/etc/os-release"] oom_score_adj = 500 } * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`command`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#command) - (string: required) - command to run * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`args`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#args) - (list(string): optional) - list of arguments to provide to `command` * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`unveil`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#unveil) - (list(string): optional) - list of additional filesystem paths to provide access to the task (requires `unveil_by_task` in plugin config). * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`oom_score_adj`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#oom_score_adj) - (optional) - The likelihood of the task being OOM killed, must be a positive integer. Defaults to `0`. ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#cpu) cpu Tasks are limited in CPU resources by setting the `cpu` or `cores` values in the task `resources` block. * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`cpu`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#cpu-1) - (default: `100`) - limits the CPU bandwidth allowable for the task to make use of in MHz, may not be used with `cores`. * [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#) [`cores`](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#cores) - (int: optional) - specifies the number of CPU cores to reserve specifically for the task, may not be used with `cpu` ##### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#memory) memory Tasks are limited in memory resources by setting `memory` and optionally the `memory_max` values in the task `resources` block. ### [](https://developer.hashicorp.com/nomad/plugins/drivers/exec2#node-attributes) Node Attributes When installed, the `exec2` plugin provides the following node attributes which can be used as constraints when authoring jobs. driver.exec2.unveil.defaults = true driver.exec2.unveil.tasks = true [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/plugins/drivers/exec2.mdx) --- # Oversubscribe memory | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Advanced Scheduling](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling) Job authors must set a memory limit for each task. If the memory limit is too low, then the task may exceed it and stop running. If the memory limit is too high, the cluster is left underutilized and resources are wasted. Job authors usually set limits based on the task's typical memory usage—plus an extra safety margin to handle unexpected load spikes or uncommon scenarios. Cumulatively, this can lead to a significant amount of the cluster memory being reserved but unused in clusters. To help prevent this, Nomad 1.1 now provides job authors with two separate memory limits: * A reserve limit to represent the task’s typical memory usage. This value is used by the Nomad scheduler to reserve and place the task. * A max limit, which is the largest amount of memory the task may burst up to. If another process competes for the client’s memory or the client's available memory becomes too low, Nomad uses the operating system primitives to recover. In Linux via cgroups, Nomad reclaims memory by pushing the tasks back to their reserved memory limits. It may also reschedule tasks to other clients. Memory oversubscription is not enabled by default. You enable it by sending a [payload](https://developer.hashicorp.com/nomad/api-docs/operator/scheduler#sample-payload) with the appropriate options specified to the [scheduler configuration](https://developer.hashicorp.com/nomad/api-docs/operator#update-scheduler-configuration) API endpoint. In this tutorial, you will enable the oversubscription feature and observe the memory utilization of a service job. You will change the memory parameters and observe the behaviors of the job as the memory parameters are adjusted. Launch Terminal This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure. Start interactive lab ![]() ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#requirements) Requirements * Linux or macOS host * [Nomad 1.1.0+](https://developer.hashicorp.com/nomad/install) * Docker * 2GB+ RAM * [jq](https://stedolan.github.io/jq/) —This tutorial uses the `jq` command to filter and rewrite JSON. [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#configure-your-learning-environment) Configure your learning environment ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#fetch-the-tutorial-content) Fetch the tutorial content This tutorial uses content provided in the [`hashicorp-education/learn-nomad-features` repository](https://github.com/hashicorp-education/learn-nomad-features/tree/memory-oversubscription) on GitHub. You can download a ZIP archive directly or use `git` to clone the repository. Download ZIP archiveClone repository $ wget https://github.com/hashicorp-education/learn-nomad-features/archive/memory-oversubscription.zip Unarchive the downloaded release. $ unzip memory-oversubscription.zip The unzipping process creates the `learn-nomad-features-memory-oversubscription` directory, which contains the `memory-oversubscription` directory you will use in this tutorial. Change to the tutorial directory. $ cd learn-nomad-features-memory-oversubscription/memory-oversubscription Clone the `hashicorp-education/learn-nomad-features` repository. $ git clone https://github.com/hashicorp-education/learn-nomad-features Change into the project directory. $ cd learn-nomad-features Check out the release tag. $ git checkout memory-oversubscription Change to the `memory-oversubscription` directory, which contains the tutorial files. $ cd memory-oversubscription [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#start-the-tutorial-environment) Start the tutorial environment ------------------------------------------------------------------------------------------------------------------------------------------------------------- This tutorial includes a Nomad job specification that starts a monitoring application and a job that allocates memory in a bursty fashion that makes it difficult to determine its memory resource needs. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#start-a-nomad-agent) Start a Nomad agent Open another terminal session in the same folder, and run a Nomad dev agent with the following command. $ sudo nomad agent -dev -config=config/nomad.hcl Switch back to the first terminal session so that you can run the commands in the rest of the tutorial. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#manage-environment-variables) Manage environment variables Because this tutorial uses a local Nomad dev agent, you need to unset the `NOMAD_ADDR` and `NOMAD_TOKEN` variables if they are set in your current shell environment. $ unset NOMAD_ADDR NOMAD_TOKEN The shell does not provide any feedback when you run this command. To simplify the `curl` commands you will be running, create a `NOMAD_ADDR` environment variable that points to your running Nomad dev agent. $ export NOMAD_ADDR=http://127.0.0.1:4646 As before, the shell does not provide any feedback when you run this command. [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#view-the-current-scheduler-configuration) View the current scheduler configuration --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration | jq . The response is a SchedulerConfig JSON object and information about its last modified index. { "SchedulerConfig": { "SchedulerAlgorithm": "binpack", "PreemptionConfig": { "SystemSchedulerEnabled": true, "BatchSchedulerEnabled": false, "ServiceSchedulerEnabled": false }, "MemoryOversubscriptionEnabled": false, "CreateIndex": 5, "ModifyIndex": 5 }, "Index": 5, "LastContact": 0, "KnownLeader": true } If you don't receive a JSON response from Nomad, make sure that the `NOMAD_ADDR` environment variable is set correctly and that your Nomad dev agent is running. Using the `jq` command, you can filter the response down to the SchedulerConfig object itself. $ curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration | jq '.SchedulerConfig' { "CreateIndex": 5, "MemoryOversubscriptionEnabled": false, "ModifyIndex": 5, "PreemptionConfig": { "BatchSchedulerEnabled": false, "ServiceSchedulerEnabled": false, "SystemSchedulerEnabled": true }, "SchedulerAlgorithm": "binpack" } ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#run-the-monitoring-job) Run the monitoring job The `monitoring.nomad` job includes an ephemeral monitoring environment for use with the tutorial. Use the [`nomad job run` command](https://developer.hashicorp.com/nomad/commands/job/run) to run the `monitoring.nomad` job. $ nomad job run monitoring.nomad ==> 2021-07-01T14:34:00-04:00: Monitoring evaluation "b5186f1d" 2021-07-01T14:34:00-04:00: Evaluation triggered by job "monitoring" ==> 2021-07-01T14:34:01-04:00: Monitoring evaluation "b5186f1d" 2021-07-01T14:34:01-04:00: Evaluation within deployment: "f7ec09be" 2021-07-01T14:34:01-04:00: Allocation "6c125a03" created: node "54308685", group "metrics" 2021-07-01T14:34:01-04:00: Evaluation status changed: "pending" -> "complete" ==> 2021-07-01T14:34:01-04:00: Evaluation "b5186f1d" finished with status "complete" ==> 2021-07-01T14:34:01-04:00: Monitoring deployment "f7ec09be" ⠧ Deployment "f7ec09be" in progress... 2021-07-01T14:34:09-04:00 ID = f7ec09be Job ID = monitoring Job Version = 0 Status = running Description = Deployment is running Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline metrics 1 1 0 0 2021-07-01T14:44:00-04:00 [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#observe-the-sample-application) Observe the sample application ------------------------------------------------------------------------------------------------------------------------------------------------------------- The tutorial repository includes a sample application that uses memory in a predictable way. The code for the sample application and a Dockerfile to build it for yourself is included in the `memory-wave` folder at the root of the [`learn-nomad-features`](https://github.com/hashicorp-education/learn-nomad-features) repository. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#run-the-sample-job) Run the sample job The `wave.nomad` job runs a prebuilt instance of the container from Docker Hub. You will need to update the image in the job specification if you would like to use your own image. Use the [`nomad job run` command](https://developer.hashicorp.com/nomad/commands/job/run) to run the `wave.nomad` job. $ nomad job run wave.nomad ==> 2021-07-01T14:35:00-04:00: Monitoring evaluation "450db87b" 2021-07-01T14:35:00-04:00: Evaluation triggered by job "wave" 2021-07-01T14:35:00-04:00: Evaluation within deployment: "404b29b2" 2021-07-01T14:35:00-04:00: Allocation "97fcb825" created: node "54308685", group "wave" 2021-07-01T14:35:00-04:00: Evaluation status changed: "pending" -> "complete" ==> 2021-07-01T14:35:00-04:00: Evaluation "450db87b" finished with status "complete" ==> 2021-07-01T14:35:00-04:00: Monitoring deployment "404b29b2" ⠏ Deployment "404b29b2" in progress... 2021-07-01T14:35:02-04:00 ID = 404b29b2 Job ID = wave Job Version = 0 Status = running Description = Deployment is running Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline wave 1 1 0 0 2021-07-01T14:45:00-04:00 ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#open-the-influx-ui) Open the Influx UI Use the Nomad UI to determine the IP address and port of the Influx instance. In your browser, navigate to [http://localhost:4646/ui/jobs/monitoring/metrics](http://localhost:4646/ui/jobs/monitoring/metrics) ![Nomad UI open to "metrics" task group detail page](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Fui_jobs_monitoring_metrics_1.png%26width%3D2732%26height%3D2048&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Next, select the running allocation by its ID. Nomad will show the allocation detail page. ![Nomad allocation detail page for running "metrics" allocation](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Fui_jobs_monitoring_metrics_2.png%26width%3D2732%26height%3D2048&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) If not visible, scroll down to the **Ports** section of the page. ![Earlier page scrolled down to show "Ports" section with hyperlinked address in the "Host Address" column.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Fui_jobs_monitoring_metrics_3.png%26width%3D2732%26height%3D2048&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Click on the hyperlinked **Host Address** to open the Influx UI in a new browser tab. ![Influx Web UI login page](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-01.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Sign in to the Influx UI using the username `admin` and the password `password`. The **Getting Started** page opens. ![Influx "Getting Started" page, Le](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-02.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Click on the **Dashboards** option in the side bar. ![Influx "Dashboards" page showing "Wave Dashboard" card.](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-03.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Select the **Wave Dashboard** to open the tutorial's dashboard. ![Influx UI with "Wave Dashboard" open](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-04.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Change the timeframe to **Past 5m**. !["Wave Dashboard" page with timeframe dropdown open](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-05.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) Change the refresh rate to **5s**. !["Wave Dashboard" page with refresh rate dropdown open](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-06.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) As the application runs, the **Memory Stats** graph should show regular peaks and valleys. !["Wave Dashboard" page showing sinusoidal memory usage in the "Memory Stats" cell](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-08.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) The **Max Memory Usage for Period** statistic shows that the wave task is using 499 MiB in the 5 minute period captured in the metrics view. The **Average Memory Usage for Period** shows that the job uses around 400 MiB. Without memory oversubscription, the job needs to reserve more than the memory required for the entire life of the task. If the task uses more memory than the job indicates, Docker forcibly stops the task because it's out of memory ("OOM kill"). Once you have enabled memory oversubscription, your job can reserve an amount closer to the actual average usage of the application. [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#enable-memory-oversubscription) Enable memory oversubscription ------------------------------------------------------------------------------------------------------------------------------------------------------------- To enable memory oversubscription, you must set `MemoryOversubscriptionEnabled` to `true`. The general process is: * Fetch the current SchedulerConfig. * Update `MemoryOversubscriptionEnabled` to `true`. * POST the updated value back to Nomad. You can save the SchedulerConfig contents to your filesystem and edit them, or you can use the `jq` command in a shell-command pipeline to update the value without writing it to disk. This tutorial demonstrates the pipeline method. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#update-the-memoryoversubscriptionenabled-value) Update the MemoryOversubscriptionEnabled value Run this single-line command to get the current scheduler configuration with `curl`, update the value with `jq`, and send it back to Nomad using a `curl` PUT request. $ curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration | \ jq '.SchedulerConfig | .MemoryOversubscriptionEnabled=true' | \ curl -X PUT $NOMAD_ADDR/v1/operator/scheduler/configuration -d @- Nomad's response shows that the value was updated and provides you with the change index. { "Updated": true, "Index": 40 } ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#verify-the-value) Verify the value Verify the cluster's `MemoryOversubscriptionEnabled` value by running the curl command to query the `/v1/operator/scheduler/configuration` endpoint again. $ curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration | jq . { "SchedulerConfig": { "SchedulerAlgorithm": "binpack", "PreemptionConfig": { "SystemSchedulerEnabled": true, "BatchSchedulerEnabled": false, "ServiceSchedulerEnabled": false }, "MemoryOversubscriptionEnabled": true, "CreateIndex": 5, "ModifyIndex": 40 }, "Index": 40, "LastContact": 0, "KnownLeader": true } [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#update-the-job-to-use-oversubscription) Update the job to use oversubscription ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open `wave.nomad` in a text editor and scroll down to the `resources` stanza. Reduce the memory value from `520` to the observed average value of `400`. Add a `memory_max` value to inform Nomad of how much extra memory the job can use; set it to `520`. Once complete, your `resources` stanza should look like the following. wave.nomad job "wave" { datacenters = ["dc1"] group "wave" { task "wave" { driver = "docker" config { image = "voiselle/wave:v5" args = [ "300", "200", "15", "64", "4" ] } resources { memory = 400 memory_max = 520 } } } } ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#re-run-the-job) Re-run the job Run the job to update the configuration. $ nomad job run wave.nomad ==> 2021-07-01T14:43:30-04:00: Monitoring evaluation "d6f822a1" 2021-07-01T14:43:30-04:00: Evaluation triggered by job "wave" ==> 2021-07-01T14:43:31-04:00: Monitoring evaluation "d6f822a1" 2021-07-01T14:43:31-04:00: Evaluation within deployment: "692890d2" 2021-07-01T14:43:31-04:00: Allocation "c78818af" created: node "54308685", group "wave" 2021-07-01T14:43:31-04:00: Evaluation status changed: "pending" -> "complete" ==> 2021-07-01T14:43:31-04:00: Evaluation "d6f822a1" finished with status "complete" ==> 2021-07-01T14:43:31-04:00: Monitoring deployment "692890d2" ⠙ Deployment "692890d2" in progress... 2021-07-01T14:43:33-04:00 ID = 692890d2 Job ID = wave Job Version = 1 Status = running Description = Deployment is running Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline wave 1 1 0 0 2021-07-01T14:53:30-04:00 ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#examine-the-dashboard) Examine the dashboard Switch back to the Influx browser tab and watch the **Wave Dashboard**. Observe that the running allocation uses more than the allocated value of 400 MiB without being OOM-killed. ![](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-10.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#validate-memory_max-setting) Validate memory\_max setting -------------------------------------------------------------------------------------------------------------------------------------------------------- Open `wave.nomad` in a text editor and scroll down to the `config` stanza's `args` value. Update the second value in the list to `300`. Once complete, your `args` stanza should look like the following. wave.nomad job "wave" { datacenters = ["dc1"] group "wave" { task "wave" { driver = "docker" config { image = "voiselle/wave:v5" args = [ "300", "300", "15", "64", "4" ] } resources { memory = 400 memory_max = 520 } } } } This change causes the wave application to use approximately 600 MiB of RAM at its peak usage. Because the job's `max_memory` value of 550 MiB, Docker will OOM-kill the job once it passes that value. Run the job to update the configuration. $ nomad job run wave.nomad ==> 2021-07-01T14:46:33-04:00: Monitoring evaluation "f54c7610" 2021-07-01T14:46:33-04:00: Evaluation triggered by job "wave" 2021-07-01T14:46:33-04:00: Allocation "f51cd2c2" created: node "54308685", group "wave" ==> 2021-07-01T14:46:34-04:00: Monitoring evaluation "f54c7610" 2021-07-01T14:46:34-04:00: Evaluation within deployment: "c76d1227" 2021-07-01T14:46:34-04:00: Evaluation status changed: "pending" -> "complete" ==> 2021-07-01T14:46:34-04:00: Evaluation "f54c7610" finished with status "complete" ==> 2021-07-01T14:46:34-04:00: Monitoring deployment "c76d1227" ⠸ Deployment "c76d1227" in progress... 2021-07-01T14:46:39-04:00 ID = c76d1227 Job ID = wave Job Version = 2 Status = running Description = Deployment is running Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline wave 1 1 0 0 2021-07-01T14:56:33-04:00 ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#examine-the-dashboard-1) Examine the dashboard Switch back to the Influx browser tab and watch the **Wave Dashboard**. Observe that once the application uses more than the specified memory\_max value— 550 MiB—that the container is OOM-killed. !["Wave Dashboard" showing instances of OOM Killer behavior](https://developer.hashicorp.com/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fnomad%252Fmemory-oversubscription%252Finflux_ui-11.png%26width%3D2048%26height%3D1536&w=3840&q=75&dpl=dpl_8efKEmne3ssm1jMbymC8YpsYWpqn) [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#clean-up) Clean up ----------------------------------------------------------------------------------------------------------------- Now that you have configured memory oversubscription in your local Nomad dev instance, you can clean up the running containers and Docker images. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#stop-nomad-jobs) Stop Nomad jobs Use the [`nomad job stop` command](https://developer.hashicorp.com/nomad/commands/job/stop) to stop the wave job. $ nomad job stop wave ==> Monitoring evaluation "85b1c158" Evaluation triggered by job "wave" Evaluation status changed: "pending" -> "complete" ==> Evaluation "85b1c158" finished with status "complete" Use the [`nomad job stop` command](https://developer.hashicorp.com/nomad/commands/job/stop) to stop the monitoring job. $ nomad job stop monitoring ==> Monitoring evaluation "dd37cc88" Evaluation triggered by job "monitoring" ==> Monitoring evaluation "dd37cc88" Evaluation within deployment: "aeff4677" Evaluation status changed: "pending" -> "complete" ==> Evaluation "dd37cc88" finished with status "complete" ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#stop-the-nomad-dev-agent) Stop the Nomad dev agent Switch to the terminal running your Nomad dev agent and stop it by pressing `Ctrl-C`. You can now close this terminal session. ### [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#remove-tutorial-docker-images-optional) Remove tutorial Docker images (optional) The tutorial pulls three Docker containers that will be cached by your local Docker daemon. Once you are completely done with the tutorial, run the following command to remove them if you wish. $ docker image rm voiselle/wave:v5 influxdb:2.0.7 telegraf:1.19.0 [](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/memory-oversubscription#next-steps) Next steps --------------------------------------------------------------------------------------------------------------------- In this tutorial, you learned how to update the `SchedulerConfig` value to enable memory oversubscription in your Nomad cluster, how to configure the `memory` attribute for oversubscription, and how to use the `memory_max` attribute to prevent a misbehaving workload from depleting the available memory on your Nomad client nodes. Read more about memory oversubscription in the [Nomad documentation](https://developer.hashicorp.com/nomad/docs/job-specification/resources#memory-oversubscription) . To learn more about advanced scheduling configuration, visit the [Define Application Placement Preferences](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling) collection. **Was this tutorial helpful?** YesNo [Collection Overview\ \ Advanced Scheduling](https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling) [Next Collection\ \ Edge Computing](https://developer.hashicorp.com/nomad/tutorials/edge) --- # Commands (CLI) | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.8.x. [View latest version](https://developer.hashicorp.com/nomad/commands) . Nomad Commands (CLI) ==================== Nomad is controlled via a very easy to use command-line interface (CLI). Nomad is only a single command-line application: `nomad`, which takes a subcommand such as "agent" or "status". The complete list of subcommands is in the navigation to the left. The Nomad CLI is a well-behaved command line application. In erroneous cases, a non-zero exit status will be returned. It also responds to `-h` and `--help` as you would most likely expect. To view a list of the available commands at any time, just run Nomad with no arguments. To get help for any specific subcommand, run the subcommand with the `-h` argument. Each command has been conveniently documented on this website. Links to each command can be found on the left. [](https://developer.hashicorp.com/nomad/commands/v1.8.x#autocomplete) Autocomplete ----------------------------------------------------------------------------------- Nomad's CLI supports command autocomplete. Autocomplete can be installed or uninstalled by running the following on bash, zsh or fish shells: $ nomad -autocomplete-install $ nomad -autocomplete-uninstall [](https://developer.hashicorp.com/nomad/commands/v1.8.x#command-contexts) Command Contexts ------------------------------------------------------------------------------------------- Nomad's CLI commands have implied contexts in their naming convention. Because the CLI is most commonly used to manipulate or query jobs, you can assume that any given command is working in that context unless the command name implies otherwise. For example, the `nomad job run` command is used to run a new job, the `nomad status` command queries information about existing jobs, etc. Conversely, commands with a prefix in their name likely operate in a different context. Examples include the `nomad agent-info` or `nomad node drain` commands, which operate in the agent or node contexts respectively. ### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#remote-usage) Remote Usage The Nomad CLI may be used to interact with a remote Nomad cluster, even when the local machine does not have a running Nomad agent. To do so, set the `NOMAD_ADDR` environment variable. $ NOMAD_ADDR=https://remote-address:4646 $ nomad status Or use the `-address=` flag when running commands. $ nomad status -address=https://remote-address:4646 The provided address must be reachable from your local machine. There are a variety of ways to accomplish this (VPN, SSH Tunnel, etc). If the port is exposed to the public internet it is highly recommended to configure TLS. ### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#environment-variables) Environment Variables Nomad can use environment variables to configure command-line tool options. You may override these environment variables with individual flags. Except where noted, these variables influence the behavior of the Nomad CLI and should not be set for Nomad agents. #### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#connection-environment-variables) Connection Environment Variables * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_ADDR`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_addr) - The address of the Nomad server. Defaults to `http://127.0.0.1:4646`. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_REGION`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_region) - The region of the Nomad server to forward commands to. Defaults to the Agent's local region * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_NAMESPACE`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_namespace) - The target namespace for queries and actions bound to a namespace. If set to `*`, job and alloc subcommands query all namespacecs authorized to user. Defaults to the "default" namespace. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_HTTP_AUTH`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_http_auth) - (Optional) This allows users to supply "Basic" HTTP authentication scheme ([RFC 7617](https://tools.ietf.org/html/rfc7617) ) information in environments where the Nomad API is behind an authenticating proxy server. #### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#acl-environment-variables) ACL Environment Variables * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_TOKEN`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_token) - The SecretID of an ACL token to use to authenticate API requests with. #### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#cli-environment-variables) CLI Environment Variables * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_CLI_NO_COLOR`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_cli_no_color) - Disables colored command output. #### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#mutual-tls-mtls-environment-variables) Mutual TLS (mTLS) environment variables * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_CLIENT_CERT`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_client_cert) - Path to a PEM encoded client certificate for TLS authentication to the Nomad server. Must also specify `NOMAD_CLIENT_KEY`. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_CLIENT_KEY`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_client_key) - Path to an unencrypted PEM encoded private key matching the client certificate from `NOMAD_CLIENT_CERT`. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_CACERT`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_cacert) - Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_CAPATH`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_capath) - Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate. If both `NOMAD_CACERT` and `NOMAD_CAPATH` are specified, `NOMAD_CACERT` is used. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_SKIP_VERIFY`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_skip_verify) - Do not verify TLS certificate. **This is highly not recommended.** * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_TLS_SERVER_NAME`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_tls_server_name) - The server name to use as the SNI host when connecting via TLS. #### [](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad-enterprise-licensing-environment-variables) Nomad Enterprise Licensing Environment Variables These environment variables influence the Nomad Enterprise license configuration. These values are only used for Nomad Enterprise agents, not the Nomad CLI. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_LICENSE_PATH`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_license_path) - An absolute path to a Nomad Enterprise license file, for example `/etc/nomad.d/license.hclic`. * [](https://developer.hashicorp.com/nomad/commands/v1.8.x#) [`NOMAD_LICENSE`](https://developer.hashicorp.com/nomad/commands/v1.8.x#nomad_license) - The Nomad Enterprise license file contents as a string. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.8.x/content/commands/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.6.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [Getting Started collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.6.x/content/docs/index.mdx) --- # Documentation | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.7.x. [View latest version](https://developer.hashicorp.com/nomad/docs) . Nomad Documentation =================== Welcome to the Nomad documentation. Nomad is a scheduler and workload orchestrator. This documentation is a reference for all available features and options of Nomad. If you are just getting started with Nomad, please start with the [Getting Started collection](https://developer.hashicorp.com/nomad/tutorials/get-started) instead. Interested in talking with HashiCorp about your experience building, deploying, or managing your applications? [Set up a time to chat!](https://forms.gle/2tAmxJbyPbcL2nRW9) [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.7.x/content/docs/index.mdx) --- # Namespace - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Namespace HTTP API ================== The `/namespace` endpoints are used to query for and interact with namespaces. [](https://developer.hashicorp.com/nomad/api-docs/namespaces#list-namespaces) List Namespaces --------------------------------------------------------------------------------------------- This endpoint lists all namespaces. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/namespaces` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:*`
Any capability on the namespace authorizes the endpoint | ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`prefix`](https://developer.hashicorp.com/nomad/api-docs/namespaces#prefix) `(string: "")`\- Specifies a string to filter namespaces on based on an index prefix. This is specified as a query string parameter. ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-request) Sample Request $ curl \ https://localhost:4646/v1/namespaces $ curl \ https://localhost:4646/v1/namespaces?prefix=prod ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-response) Sample Response [\ {\ "Capabilities": null,\ "CreateIndex": 1,\ "Description": "Default shared namespace",\ "Meta": null,\ "ModifyIndex": 1,\ "Name": "default",\ "Quota": ""\ },\ {\ "Capabilities": null,\ "CreateIndex": 17,\ "Description": "Development Staging Namespace",\ "Meta": {\ "type": "dev",\ "contact": "helpdesk@example.com"\ },\ "ModifyIndex": 17,\ "Name": "staging",\ "Quota": ""\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/namespaces#read-namespace) Read Namespace ------------------------------------------------------------------------------------------- This endpoint reads information about a specific namespace. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/namespace/:namespace` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:*`
Any capability on the namespace authorizes the endpoint | ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`:namespace`](https://developer.hashicorp.com/nomad/api-docs/namespaces#namespace) `(string: )`\- Specifies the namespace to query. ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/namespace/staging ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-response-1) Sample Response { "Capabilities": null, "CreateIndex": 17, "Description": "Development Staging Namespace", "Meta": { "type": "dev", "contact": "helpdesk@example.com" }, "ModifyIndex": 17, "Name": "staging", "Quota": "" } [](https://developer.hashicorp.com/nomad/api-docs/namespaces#create-or-update-namespace) Create or Update Namespace ------------------------------------------------------------------------------------------------------------------- This endpoint is used to create or update a namespace. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/namespace/:namespace`
`/v1/namespace` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#parameters-2) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/namespaces#name) `(string: )`\- Specifies the namespace to create or update. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Description`](https://developer.hashicorp.com/nomad/api-docs/namespaces#description) `(string: "")` - Specifies an optional human-readable description of the namespace. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Meta`](https://developer.hashicorp.com/nomad/api-docs/namespaces#meta) `(object: null)` - Optional object with string keys and values of metadata to attach to the namespace. Namespace metadata is not used by Nomad and is intended for use by operators and third party tools. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Quota`](https://developer.hashicorp.com/nomad/api-docs/namespaces#quota) `(string: "")` Enterprise \- Specifies an quota to attach to the namespace. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Capabilities`](https://developer.hashicorp.com/nomad/api-docs/namespaces#capabilities) `(Capabilities: )` - Specifies capabilities allowed in the namespace. These values are checked at job submission. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`EnabledTaskDrivers`](https://developer.hashicorp.com/nomad/api-docs/namespaces#enabledtaskdrivers) `(array: [])` - List of task drivers allowed in the namespace. If empty all task drivers are allowed. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`DisabledTaskDrivers`](https://developer.hashicorp.com/nomad/api-docs/namespaces#disabledtaskdrivers) `(array: [])` - List of task drivers disabled in the namespace. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`NodePoolConfiguration`](https://developer.hashicorp.com/nomad/api-docs/namespaces#nodepoolconfiguration) `(NodePoolConfiguration: )` Enterprise \- Specifies node pool configurations. These values are checked at job submission. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Default`](https://developer.hashicorp.com/nomad/api-docs/namespaces#default) `(string: "default")` - Specifies the node pool to use for jobs in this namespace that don't define a node pool in their specification. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`Allowed`](https://developer.hashicorp.com/nomad/api-docs/namespaces#allowed) `(array: [])` - Specifies the node pools that are allowed to be used by jobs in this namespace. This field supports wildcard globbing through the use of `*` for multi-character matching. If specified, only the node pools that match these patterns are allowed. This field cannot be used with `Disabled`. * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`disabled`](https://developer.hashicorp.com/nomad/api-docs/namespaces#disabled) `(array: [])` - Specifies the node pools that are not allowed to be used by jobs in this namespace. This field supports wildcard globbing through the use of `*` for multi-character matching. If specified, any node pool is allowed except for those that match any of these patterns. This field cannot be used with `Enabled`. ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-payload) Sample Payload { "Name": "api-prod", "Description": "Production API Servers", "Meta": { "contact": "platform-eng@example.com" }, "Quota": "prod-quota", "Capabilities": { "DisabledTaskDrivers": ["raw_exec"] }, "NodePoolConfiguration": { "Default": "prod-pool", "Allowed": ["default"] } } ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-request-2) Sample Request $ curl \ --request POST \ --data @namespace.json \ https://localhost:4646/v1/namespace/api-prod $ curl \ --request POST \ --data @namespace.json \ https://localhost:4646/v1/namespace [](https://developer.hashicorp.com/nomad/api-docs/namespaces#delete-namespace) Delete Namespace ----------------------------------------------------------------------------------------------- This endpoint is used to delete a namespace. | Method | Path | Produces | | --- | --- | --- | | `DELETE` | `/v1/namespace/:namespace` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#parameters-3) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/namespaces#) [`:namespace`](https://developer.hashicorp.com/nomad/api-docs/namespaces#namespace-1) `(string: )`\- Specifies the namespace to delete. ### [](https://developer.hashicorp.com/nomad/api-docs/namespaces#sample-request-3) Sample Request $ curl \ --request DELETE \ https://localhost:4646/v1/namespace/api-prod [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/namespaces.mdx) --- # System - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) System HTTP API =============== The `/system` endpoints are used to for system maintenance and should not be necessary for most users. [](https://developer.hashicorp.com/nomad/api-docs/system#force-gc) Force GC --------------------------------------------------------------------------- This endpoint initializes a garbage collection of jobs, evaluations, allocations, and nodes. This is an asynchronous operation. | Method | Path | Produces | | --- | --- | --- | | `PUT` | `/v1/system/gc` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/system#sample-request) Sample Request $ curl \ --request PUT \ https://localhost:4646/v1/system/gc [](https://developer.hashicorp.com/nomad/api-docs/system#reconcile-summaries) Reconcile Summaries ------------------------------------------------------------------------------------------------- This endpoint reconciles the summaries of all registered jobs. | Method | Path | Produces | | --- | --- | --- | | `PUT` | `/v1/system/reconcile/summaries` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/system#sample-request-1) Sample Request $ curl \ --request PUT \ https://localhost:4646/v1/system/reconcile/summaries [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/system.mdx) --- # Validate - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Validate HTTP API ================= The `/validate` endpoints are used to validate object structs, fields, and types. [](https://developer.hashicorp.com/nomad/api-docs/validate#validate-job) Validate Job ------------------------------------------------------------------------------------- This endpoint validates a Nomad job file. The local Nomad agent forwards the request to the leader. In the event the leader cannot be reached the agent verifies the job file locally but skips validating driver configurations. This endpoint accepts a **JSON job file**, not an HCL job file. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/validate/job` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/validate#parameters) Parameters There are no parameters, but the request _body_ contains the entire job file. ### [](https://developer.hashicorp.com/nomad/api-docs/validate#sample-payload) Sample Payload (any valid nomad job IN JSON FORMAT) ### [](https://developer.hashicorp.com/nomad/api-docs/validate#sample-request) Sample Request $ curl \ --request POST \ --data @my-job.nomad.json \ https://localhost:4646/v1/validate/job ### [](https://developer.hashicorp.com/nomad/api-docs/validate#sample-response) Sample Response { "DriverConfigValidated": true, "ValidationErrors": [\ "Task group cache validation failed: 1 error(s) occurred:\n\n* Task redis validation failed: 1 error(s) occurred:\n\n* 1 error(s) occurred:\n\n* minimum CPU value is 20; got 1"\ ], "Warnings": "1 warning(s):\n\n* Group \"cache\" has warnings: 1 error(s) occurred:\n\n* Update max parallel count is greater than task group count (13 > 1). A destructive change would result in the simultaneous replacement of all allocations.", "Error": "1 error(s) occurred:\n\n* Task group cache validation failed: 1 error(s) occurred:\n\n* Task redis validation failed: 1 error(s) occurred:\n\n* 1 error(s) occurred:\n\n* minimum CPU value is 20; got 1" } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/validate.mdx) --- # Metrics - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Metrics HTTP API ================ The `/metrics` endpoint returns metrics for the current Nomad process. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/metrics` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `none` | ### [](https://developer.hashicorp.com/nomad/api-docs/metrics#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/metrics#) [`format`](https://developer.hashicorp.com/nomad/api-docs/metrics#format) `(string: "")` - Specifies the metrics format to be other than the JSON default. Currently, only `prometheus` is supported as an alternative format. This is specified as a query string parameter. ### [](https://developer.hashicorp.com/nomad/api-docs/metrics#sample-request) Sample Request $ curl https://localhost:4646/v1/metrics $ curl https://localhost:4646/v1/metrics?format=prometheus ### [](https://developer.hashicorp.com/nomad/api-docs/metrics#sample-response) Sample Response { "Counters": [\ {\ "Count": 11,\ "Labels": {},\ "Max": 1.0,\ "Mean": 1.0,\ "Min": 1.0,\ "Name": "nomad.nomad.rpc.query",\ "Stddev": 0.0,\ "Sum": 11.0\ }\ ], "Gauges": [\ {\ "Labels": {\ "node_id": "cd7c3e0c-0174-29dd-17ba-ea4609e0fd1f",\ "datacenter": "dc1"\ },\ "Name": "nomad.client.allocations.blocked",\ "Value": 0.0\ },\ {\ "Labels": {\ "datacenter": "dc1",\ "node_id": "cd7c3e0c-0174-29dd-17ba-ea4609e0fd1f"\ },\ "Name": "nomad.client.allocations.migrating",\ "Value": 0.0\ }\ ], "Samples": [\ {\ "Count": 20,\ "Labels": {},\ "Max": 0.03544100001454353,\ "Mean": 0.023678050097078084,\ "Min": 0.00956599973142147,\ "Name": "nomad.memberlist.gossip",\ "Stddev": 0.005445327799243976,\ "Sum": 0.4735610019415617\ },\ {\ "Count": 1,\ "Labels": {},\ "Max": 0.0964059978723526,\ "Mean": 0.0964059978723526,\ "Min": 0.0964059978723526,\ "Name": "nomad.nomad.client.update_status",\ "Stddev": 0.0,\ "Sum": 0.0964059978723526\ }\ ] } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/metrics.mdx) --- # Plugins - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Plugins HTTP API ================ The `/plugin` endpoints are used to query for and interact with dynamic plugins. Currently only Container Storage Interface (CSI) plugins are dynamic. [](https://developer.hashicorp.com/nomad/api-docs/plugins#list-plugins) List Plugins ------------------------------------------------------------------------------------ This endpoint lists all dynamic plugins. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/plugins` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `plugin:read` | ### [](https://developer.hashicorp.com/nomad/api-docs/plugins#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/plugins#) [`type`](https://developer.hashicorp.com/nomad/api-docs/plugins#type) `(string: "")`\- Specifies the type of plugin to query. Currently only supports `csi`. This is specified as a query string parameter. Returns an empty list if omitted. ### [](https://developer.hashicorp.com/nomad/api-docs/plugins#sample-request) Sample Request $ curl \ https://localhost:4646/v1/plugins?type=csi ### [](https://developer.hashicorp.com/nomad/api-docs/plugins#sample-response) Sample Response [\ {\ "ID": "example",\ "Provider": "aws.ebs",\ "ControllerRequired": true,\ "ControllersHealthy": 2,\ "ControllersExpected": 3,\ "NodesHealthy": 14,\ "NodesExpected": 16,\ "CreateIndex": 52,\ "ModifyIndex": 93\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/plugins#read-plugin) Read Plugin ---------------------------------------------------------------------------------- Get details of a single plugin, including information about the plugin's job and client fingerprint data. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/plugin/csi/:plugin_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `plugin:read`, | | | `namespace:*`
Allocations listed are filtered by namespace | ### [](https://developer.hashicorp.com/nomad/api-docs/plugins#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/plugin/csi/example_plugin_id ### [](https://developer.hashicorp.com/nomad/api-docs/plugins#sample-response-1) Sample Response [\ {\ "ID": "example_plugin_id",\ "Provider": "aws.ebs",\ "Version": "1.0.1",\ "ControllersRequired": true,\ "ControllersHealthy": 1,\ "Controllers": {\ "example_node_id": {\ "PluginID": "example_plugin_id",\ "Provider": "aws.ebs",\ "ProviderVersion": "1.0.1",\ "AllocID": "alloc-id",\ "Healthy": true,\ "HealthDescription": "healthy",\ "UpdateTime": "2020-01-31T00:00:00.000Z",\ "RequiresControllerPlugin": true,\ "RequiresTopologies": true,\ "ControllerInfo": {\ "SupportsReadOnlyAttach": true,\ "SupportsAttachDetach": true,\ "SupportsListVolumes": true,\ "SupportsListVolumesAttachedNodes": false\ }\ }\ },\ "NodesHealthy": 1,\ "Nodes": {\ "example_node_id": {\ "PluginID": "example_plugin_id",\ "Provider": "aws.ebs",\ "ProviderVersion": "1.0.1",\ "AllocID": "alloc-id",\ "Healthy": true,\ "HealthDescription": "healthy",\ "UpdateTime": "2020-01-30T00:00:00.000Z",\ "RequiresControllerPlugin": true,\ "RequiresTopologies": true,\ "NodeInfo": {\ "ID": "example_node_id",\ "MaxVolumes": 51,\ "AccessibleTopology": {\ "key": "val2"\ },\ "RequiresNodeStageVolume": true\ }\ }\ }\ }\ ] [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/plugins.mdx) --- # Services - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Service HTTP API ================ The `/service` endpoints are used to query and interact with Nomad services. [](https://developer.hashicorp.com/nomad/api-docs/services#list-services) List Services --------------------------------------------------------------------------------------- This endpoint lists all the currently available Nomad services. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/services` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `YES` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/services#sample-request) Sample Request $ curl \ https://localhost:4646/v1/services $ curl \ https://localhost:4646/v1/services?namespace=* ### [](https://developer.hashicorp.com/nomad/api-docs/services#sample-response) Sample Response [\ {\ "Namespace": "default",\ "Services": [\ {\ "ServiceName": "example-cache-redis",\ "Tags": [\ "cache",\ "db"\ ]\ }\ ]\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/services#read-service) Read Service ------------------------------------------------------------------------------------- This endpoint reads a specific service. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/service/:service_name` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | Consistency Modes | ACL Required | | --- | --- | --- | | `YES` | `all` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/services#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`:service_name`](https://developer.hashicorp.com/nomad/api-docs/services#service_name) `(string: )` - Specifies the service name. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/services#namespace) `(string: "default")` - Specifies the target namespace. This parameter is used before any `filter` expression is applied. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`next_token`](https://developer.hashicorp.com/nomad/api-docs/services#next_token) `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which identifies the next expected service. This value can be obtained from the `X-Nomad-NextToken` header from the previous response. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`per_page`](https://developer.hashicorp.com/nomad/api-docs/services#per_page) `(int: 0)` - Specifies a maximum number of services to return for this request. If omitted, the response is not paginated. The value of the `X-Nomad-NextToken` header of the last response can be used as the `next_token` of the next request to fetch additional pages. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`filter`](https://developer.hashicorp.com/nomad/api-docs/services#filter) `(string: "")` - Specifies the [expression](https://developer.hashicorp.com/nomad/api-docs#filtering) used to filter the results. Consider using pagination or a query parameter to reduce resource used to serve the request. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`choose`](https://developer.hashicorp.com/nomad/api-docs/services#choose) `(string: "")` - Specifies the number of services to return and a hash key. Must be in the form `|`. Nomad uses [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing) to deliver consistent results for a given key, and stable results when the number of services changes. ### [](https://developer.hashicorp.com/nomad/api-docs/services#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/service/example-cache-redis ### [](https://developer.hashicorp.com/nomad/api-docs/services#sample-response-1) Sample Response [\ {\ "Address": "127.0.0.1",\ "AllocID": "177160af-26f6-619f-9c9f-5e46d1104395",\ "CreateIndex": 14,\ "Datacenter": "dc1",\ "ID": "_nomad-task-177160af-26f6-619f-9c9f-5e46d1104395-redis-example-cache-redis-db",\ "JobID": "example",\ "ModifyIndex": 24,\ "Namespace": "default",\ "NodeID": "7406e90b-de16-d118-80fe-60d0f2730cb3",\ "Port": 29702,\ "ServiceName": "example-cache-redis",\ "Tags": [\ "db",\ "cache"\ ]\ },\ {\ "Address": "127.0.0.1",\ "AllocID": "ba731da0-6df9-9858-ef23-806e9758a899",\ "CreateIndex": 35,\ "Datacenter": "dc1",\ "ID": "_nomad-task-ba731da0-6df9-9858-ef23-806e9758a899-redis-example-cache-redis-db",\ "JobID": "example",\ "ModifyIndex": 35,\ "Namespace": "default",\ "NodeID": "7406e90b-de16-d118-80fe-60d0f2730cb3",\ "Port": 27232,\ "ServiceName": "example-cache-redis",\ "Tags": [\ "db",\ "cache"\ ]\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/services#delete-service-registration) Delete Service Registration ------------------------------------------------------------------------------------------------------------------- This endpoint is used to delete an individual service registration. | Method | Path | Produces | | --- | --- | --- | | `DELETE` | `/v1/service/:service_name/:service_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `namespace:submit-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/services#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`:service_name`](https://developer.hashicorp.com/nomad/api-docs/services#service_name-1) `(string: )` - Specifies the service name. This is specified as part of the path. * [](https://developer.hashicorp.com/nomad/api-docs/services#) [`:service_id`](https://developer.hashicorp.com/nomad/api-docs/services#service_id) `(string: )` - Specifies the service ID. This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/services#sample-request-2) Sample Request $ curl \ --request DELETE \ https://localhost:4646/v1/service/example-cache-redis/_nomad-task-ba731da0-6df9-9858-ef23-806e9758a899-redis-example-cache-redis-db [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/services.mdx) --- # Scaling Policies - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Scaling Policies HTTP API ========================= The `/scaling/policies` and `/scaling/policy/` endpoints are used to list and view scaling policies. [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#list-scaling-policies) List Scaling Policies --------------------------------------------------------------------------------------------------------------- This endpoint returns the scaling policies from all jobs. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/scaling/policies` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | Consistency Modes | ACL Required | | --- | --- | --- | | `YES` | `all` | `namespace:list-scaling-policies` | ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#) [`job`](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#job) `(string: "")`\- Specifies the job ID to filter policies by. * [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#) [`type`](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#type) `(string: "")` - Specifies the type of scaling policy to filter by. In the open source version of Nomad, `horizontal` is the only supported value. Within Nomad Enterprise, `vertical_mem` and `vertical_cpu` are supported along with `vertical`. The latter returns policies matching both `vertical_mem` and `vertical_cpu`. ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#sample-request) Sample Request $ curl \ https://localhost:4646/v1/scaling/policies $ curl \ https://localhost:4646/v1/scaling/policies?job=example $ curl \ https://localhost:4646/v1/scaling/policies?type=vertical ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#sample-response) Sample Response [\ {\ "ID": "b2c64295-4315-2fdc-6158-a27156808729",\ "Enabled": true,\ "Type": "vertical_cpu",\ "Target": {\ "Namespace": "default",\ "Job": "example",\ "Group": "cache",\ "Task": "redis"\ },\ "CreateIndex": 1340,\ "ModifyIndex": 1340\ },\ {\ "ID": "c355d0ec-7aa1-2604-449d-4ec79c813d2c",\ "Enabled": true,\ "Type": "vertical_mem",\ "Target": {\ "Job": "example",\ "Group": "cache",\ "Task": "redis",\ "Namespace": "default"\ },\ "CreateIndex": 1340,\ "ModifyIndex": 1340\ },\ {\ "ID": "31a53813-24df-b2ad-77dc-1b4bad4e7dca",\ "Enabled": true,\ "Type": "horizontal",\ "Target": {\ "Job": "example",\ "Group": "cache",\ "Namespace": "default"\ },\ "CreateIndex": 1358,\ "ModifyIndex": 1358\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#read-scaling-policy) Read Scaling Policy ----------------------------------------------------------------------------------------------------------- This endpoint reads a specific scaling policy. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/scaling/policy/:policy_id` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | Consistency Modes | ACL Required | | --- | --- | --- | | `YES` | `all` | `namespace:read-scaling-policy` | ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#) [`:policy_id`](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#policy_id) `(string: )` - Specifies the ID of the scaling policy (as returned by the scaling policy list endpoint). This is specified as part of the path. ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#sample-request-1) Sample Request $ curl \ https://localhost:4646/v1/scaling/policy/5e9f9ef2-5223-6d35-bac1-be0f3cb974ad ### [](https://developer.hashicorp.com/nomad/api-docs/scaling-policies#sample-response-1) Sample Response { "CreateIndex": 10, "Enabled": true, "ID": "5e9f9ef2-5223-6d35-bac1-be0f3cb974ad", "Type": "horizontal", "Max": 10, "Min": 0, "ModifyIndex": 10, "Policy": { "engage": true, "foo": "bar", "howdy": "doody", "value": 6.0 }, "Target": { "Group": "cache", "Job": "example", "Namespace": "default" } } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/scaling-policies.mdx) --- # Events - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Events HTTP API =============== The `/event/stream` endpoint is used to stream events generated by Nomad. [](https://developer.hashicorp.com/nomad/api-docs/events#event-stream) Event Stream ----------------------------------------------------------------------------------- This endpoint streams a server's backlog of events as well as new events as they occur. The stream will be kept alive until the connection is closed. The format of the response body will be valid [ndjson](https://github.com/ndjson/ndjson-spec) . This means splitting the streaming response at every `\n` character will guarantee each message is a valid JSON object. Note that each JSON object may include multiple events (high server activity) or no events (heartbeating to keep the connection open). | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/event/stream` | `application/json` | The table below shows this endpoint's [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . Due to the nature of this endpoint individual topics require specific policies. Note that if you do not include a `topic` parameter all topics will be included by default, requiring a management token. | Topic | ACL Required | | --- | --- | | `*` | `management` | | `ACLPolicy` | `management` | | `ACLRole` | `management` | | `ACLToken` | `management` | | `Allocation` | `namespace:read-job` | | `CSIPlugin` | `namespace:read-job` | | `CSIVolume` | `namespace:csi-read-volume` | | `Deployment` | `namespace:read-job` | | `Evaluation` | `namespace:read-job` | | `HostVolume` | `namespace:host-volume-read` | | `Job` | `namespace:read-job` | | `NodePool` | `management` | | `Node` | `node:read` | | `Operator` | `operator:read` | | `Service` | `namespace:read-job` | ### [](https://developer.hashicorp.com/nomad/api-docs/events#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/events#) [`index`](https://developer.hashicorp.com/nomad/api-docs/events#index) `(int: 0)` - Specifies the index to start streaming events from. If the requested index is no longer in the buffer the stream will start at the next available index. * [](https://developer.hashicorp.com/nomad/api-docs/events#) [`namespace`](https://developer.hashicorp.com/nomad/api-docs/events#namespace) `(string: "default")` - Specifies the target namespace to filter on. Specifying `*` includes all namespaces for event types that support namespaces. If you specify all namespaces (`*`) you'll either need a management token, or an ACL Policy that explicitly applies to all namespaces (`*`). * [](https://developer.hashicorp.com/nomad/api-docs/events#) [`topic`](https://developer.hashicorp.com/nomad/api-docs/events#topic) `(topic:filter_key: "*:*")` - Specifies a topic to subscribe to and filter on. The default is to subscribe to all topics. Multiple topics may be specified by passing multiple `topic` parameters. A valid topic parameter includes a `topic` type and an optional `filter_key` separated by a colon `:`. As an example `?topic=Deployment:redis` would subscribe to all `Deployment` events for a job redis. an additional topic `&topic=Deployment:web` would include deployment events for redis and web. To only subscribe to `Node` events a topic parameter of `?topic=Node` without a separator value would be used. `?topic=Node:*` is also valid. ### [](https://developer.hashicorp.com/nomad/api-docs/events#event-topics) Event Topics | Topic | Output | | --- | --- | | ACLPolicy | ACLPolicy | | ACLRoles | ACLRole | | ACLToken | ACLToken | | Allocation | Allocation (no job information) | | CSIPlugin | CSIPlugin | | CSIVolume | CSIVolume | | Deployment | Deployment | | Evaluation | Evaluation | | HostVolume | HostVolume (dynamic host volumes only) | | Job | Job | | Node | Node | | NodeDrain | Node | | NodePool | NodePool | | Operator | UtilizationSnapshot

Enterprise | | Service | Service Registrations | ### [](https://developer.hashicorp.com/nomad/api-docs/events#event-types) Event Types | Type | | --- | | ACLPolicyDeleted | | ACLPolicyUpserted | | ACLRoleDeleted | | ACLRoleUpserted | | ACLTokenDeleted | | ACLTokenUpserted | | AllocationCreated | | AllocationUpdateDesiredStatus | | AllocationUpdated | | CSIVolumeDeregistered | | CSIVolumeRegistered | | DeploymentAllocHealth | | DeploymentPromotion | | DeploymentStatusUpdate | | EvaluationUpdated | | HostVolumeDeleted | | HostVolumeRegistered | | JobBatchDeregistered | | JobDeregistered | | JobRegistered | | NodeDeregistration | | NodeDrain | | NodeEligibility | | NodeEvent | | NodePoolDeleted | | NodePoolUpserted | | NodeRegistration | | PlanResult | | ServiceDeregistration | | ServiceRegistration | | UtilizationSnapshotUpserted | ### [](https://developer.hashicorp.com/nomad/api-docs/events#sample-request) Sample Request # Subscribe to all events and topics in the default namespace $ curl -s -v -N http://127.0.0.1:4646/v1/event/stream # Subscribe to all events and topics in all namespaces $ curl -s -v -N http://127.0.0.1:4646/v1/event/stream?namespace=* # Start at index 100 and subscribe to all Evaluation events $ curl -s -v -N http://127.0.0.1:4646/v1/event/stream?index=100&topic=Evaluation $ curl -G -s -v -N \ --data-urlencode "topic=Node:ccc4ce56-7f0a-4124-b8b1-a4015aa82c40" \ --data-urlencode "topic=Deployment" \ --data-urlencode "topic=Job:web" \ http://127.0.0.1:4646/v1/event/stream ### [](https://developer.hashicorp.com/nomad/api-docs/events#sample-response) Sample Response { "Index": 7, "Events": [\ {\ "Topic": "Node",\ "Type": "NodeRegistration",\ "Key": "ccc4ce56-7f0a-4124-b8b1-a4015aa82c40",\ "Namespace": "",\ "FilterKeys": null,\ "Index": 7,\ "Payload": {\ "Node": {\ "ID": "ccc4ce56-7f0a-4124-b8b1-a4015aa82c40",\ "Datacenter": "dc1",\ "Name": "nomad-4",\ "HTTPAddr": "127.0.0.1:4646",\ "TLSEnabled": false,\ "Attributes": {\ "cpu.arch": "amd64",\ "cpu.frequency": "4200",\ "cpu.modelname": "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz",\ "cpu.numcores": "8",\ "cpu.totalcompute": "33600",\ "driver.docker": "1",\ "driver.docker.bridge_ip": "172.17.0.1",\ "driver.docker.os_type": "linux",\ "driver.docker.runtimes": "runc",\ "driver.docker.version": "19.03.13",\ "driver.mock_driver": "1",\ "driver.raw_exec": "1",\ "kernel.name": "linux",\ "kernel.version": "5.4.0-48-generic",\ "memory.totalbytes": "16525733888",\ "nomad.revision": "8c88f29bff0849720e33b0cc73af87495358f3b8",\ "nomad.version": "0.13.0-dev",\ "os.name": "ubuntu",\ "os.signals": "SIGBUS,SIGFPE,SIGTRAP,SIGTTOU,SIGWINCH,SIGXFSZ,SIGHUP,SIGILL,SIGALRM,SIGCHLD,SIGSYS,SIGXCPU,SIGPROF,SIGQUIT,SIGTERM,SIGUSR2,SIGCONT,SIGIO,SIGSEGV,SIGTTIN,SIGIOT,SIGKILL,SIGPIPE,SIGABRT,SIGINT,SIGSTOP,SIGTSTP,SIGURG,SIGUSR1",\ "os.version": "20.04",\ "unique.advertise.address": "127.0.0.1:4646",\ "unique.cgroup.mountpoint": "/sys/fs/cgroup/systemd",\ "unique.hostname": "x1c",\ "unique.network.ip-address": "127.0.0.1",\ "unique.storage.bytesfree": "299488927744",\ "unique.storage.bytestotal": "502468108288",\ "unique.storage.volume": "/dev/nvme0n1p2"\ },\ "NodeResources": {\ "Cpu": {\ "CpuShares": 33600\ },\ "Memory": {\ "MemoryMB": 15760\ },\ "Disk": {\ "DiskMB": 285614\ },\ "Networks": [\ {\ "Mode": "bridge",\ "Device": "",\ "CIDR": "",\ "IP": "",\ "MBits": 0,\ "DNS": null,\ "ReservedPorts": null,\ "DynamicPorts": null\ },\ {\ "Mode": "host",\ "Device": "lo",\ "CIDR": "127.0.0.1/32",\ "IP": "127.0.0.1",\ "MBits": 1000,\ "DNS": null,\ "ReservedPorts": null,\ "DynamicPorts": null\ },\ {\ "Mode": "host",\ "Device": "lo",\ "CIDR": "::1/128",\ "IP": "::1",\ "MBits": 1000,\ "DNS": null,\ "ReservedPorts": null,\ "DynamicPorts": null\ }\ ],\ "NodeNetworks": [\ {\ "Mode": "bridge",\ "Device": "",\ "MacAddress": "",\ "Speed": 0,\ "Addresses": null\ },\ {\ "Mode": "host",\ "Device": "lo",\ "MacAddress": "",\ "Speed": 1000,\ "Addresses": [\ {\ "Family": "ipv4",\ "Alias": "default",\ "Address": "127.0.0.1",\ "ReservedPorts": "",\ "Gateway": ""\ },\ {\ "Family": "ipv6",\ "Alias": "default",\ "Address": "::1",\ "ReservedPorts": "",\ "Gateway": ""\ }\ ]\ }\ ],\ "Devices": null\ },\ "ReservedResources": {\ "Cpu": {\ "CpuShares": 0\ },\ "Memory": {\ "MemoryMB": 0\ },\ "Disk": {\ "DiskMB": 0\ },\ "Networks": {\ "ReservedHostPorts": ""\ }\ },\ "Links": null,\ "Meta": {\ "connect.gateway_image": "envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09",\ "connect.log_level": "info",\ "connect.sidecar_image": "envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09"\ },\ "NodeClass": "",\ "ComputedClass": "v1:9803688035578634002",\ "Drain": false,\ "DrainStrategy": null,\ "SchedulingEligibility": "eligible",\ "Status": "initializing",\ "StatusDescription": "",\ "StatusUpdatedAt": 1602770857,\ "Events": [\ {\ "Message": "Node registered",\ "Subsystem": "Cluster",\ "Details": null,\ "Timestamp": "2020-10-15T10:07:37-04:00",\ "CreateIndex": 0\ }\ ],\ "Drivers": {\ "docker": {\ "Attributes": {\ "driver.docker": "true",\ "driver.docker.bridge_ip": "172.17.0.1",\ "driver.docker.os_type": "linux",\ "driver.docker.runtimes": "runc",\ "driver.docker.version": "19.03.13"\ },\ "Detected": true,\ "Healthy": true,\ "HealthDescription": "Healthy",\ "UpdateTime": "2020-10-15T10:07:37.904159516-04:00"\ },\ "exec": {\ "Attributes": null,\ "Detected": false,\ "Healthy": false,\ "HealthDescription": "Driver must run as root",\ "UpdateTime": "2020-10-15T10:07:37.445083368-04:00"\ },\ "java": {\ "Attributes": null,\ "Detected": false,\ "Healthy": false,\ "HealthDescription": "Driver must run as root",\ "UpdateTime": "2020-10-15T10:07:37.445601605-04:00"\ },\ "qemu": {\ "Attributes": null,\ "Detected": false,\ "Healthy": false,\ "HealthDescription": "",\ "UpdateTime": "2020-10-15T10:07:37.445684857-04:00"\ },\ "raw_exec": {\ "Attributes": {\ "driver.raw_exec": "true"\ },\ "Detected": true,\ "Healthy": true,\ "HealthDescription": "Healthy",\ "UpdateTime": "2020-10-15T10:07:37.445431163-04:00"\ }\ },\ "CSIControllerPlugins": null,\ "CSINodePlugins": null,\ "HostVolumes": null,\ "CreateIndex": 7,\ "ModifyIndex": 7\ }\ }\ }\ ] } [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/events.mdx) --- # Nomad Integration Program | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Nomad Integration Program ========================= This page provides an overview of the HashiCorp Nomad Integration Program. [](https://developer.hashicorp.com/nomad/docs/partnerships#introduction) Introduction ------------------------------------------------------------------------------------- The HashiCorp Nomad Integration Program allows vendors to integrate their products to work with Nomad. As a scheduler, Nomad encompasses a broad area. Thereby, a set of integrations may require vendor integration code. Vendors integrating their solutions via the Nomad Integration Process provide their customers a verified and seamless user experience. The Nomad Integration Program primarily supports coding with the Go programming language. This program is intended to be largely a self-service process with links and guidance to information sources, clearly defined steps, and checkpoints. [](https://developer.hashicorp.com/nomad/docs/partnerships#types-of-nomad-integrations) Types of Nomad Integrations ------------------------------------------------------------------------------------------------------------------- Nomad is a simple and flexible orchestrator to deploy and manage containers and non-containerized applications across on-premises and cloud environments at scale. Nomad is widely adopted and used in production by organizations like Cloudflare, Roblox, Q2, Pandora, and more. Refer to the [What is Nomad? page](https://developer.hashicorp.com/nomad/docs/what-is-nomad) for a full description of the current features. This diagram depicts the key Nomad integration categories and types. ![Integration Categories](https://web-unified-docs-hashicorp.vercel.app/api/assets/nomad/latest/img/nomad-workload.png) Main Nomad categories for partners to integrate with include: * CI/CD (Continuous Integration & Delivery) * Container Runtime * GPUs & Other Hardware Devices * Monitoring * Artifact Repo * Networking * Storage * Autoscaling [](https://developer.hashicorp.com/nomad/docs/partnerships#development-process) Development Process --------------------------------------------------------------------------------------------------- The Nomad integration development process is divided into six steps. By following these steps, Nomad integrations can be developed alongside HashiCorp to ensure that the integrations are able to be verified and supported in Nomad as quickly as possible. A visual representation of the self-guided steps is depicted below. ![Development Process](https://www.datocms-assets.com/2885/1618463733-nomad-integration-program-steps.png) The individual Nomad integration steps include: 1. Engage: Initial contact between vendor and HashiCorp 2. Enable: Information and articles to aid with the development of the integration 3. Dev/Test: Integration development and test process 4. Review: HashiCorp code review and verification of integration (iterative process) 5. Release: Verified integration made available and listed on the HashiCorp website once the HashiCorp technology partnership agreement has been executed 6. Support: Ongoing maintenance and support of the provider by the vendor. ### [](https://developer.hashicorp.com/nomad/docs/partnerships#1-engage) 1\. Engage Please begin by providing some basic information about the integration that is being built via a simple [webform](https://docs.google.com/forms/d/e/1FAIpQLSflmhg8lkY5QyH3CB5TUMi4I1ZFQFx_GP_TVDtsbxMaFEMWpw/viewform) . This information is recorded and used by HashiCorp to track the integration through various stages. The information is also used to notify the integration developer of any overlapping work, perhaps coming from the community so you may better focus resources. Nomad has a large and active community and ecosystem of partners that may have already started working on a similar integration. We'll do our best to connect similar parties to avoid duplicate work. ### [](https://developer.hashicorp.com/nomad/docs/partnerships#2-enable) 2\. Enable While not mandatory, HashiCorp encourages vendors to sign and MNDA (Mutual Non-Disclosure Agreement) to allow for open dialog and sharing of ideas during the integration process. In an effort to support our self-serve model we've included links to resources, documentation, examples and best practices to guide you through the Nomad integration development and testing process. * Contributing to Nomad [guidelines](https://github.com/hashicorp/nomad/tree/main/contributing) * [Nomad Developer Community Forum](https://groups.google.com/g/nomad-tool) * [Nomad's source code](https://github.com/hashicorp/nomad) We encourage vendors to closely follow the above guidance. Adopting the same structure and coding patterns helps expedite the review and release cycles. ### [](https://developer.hashicorp.com/nomad/docs/partnerships#3-dev-test) 3\. Dev & Test Nomad requires all code-level integrations to be written in the [Go](https://golang.org/) programming language and contain an [MPL-2.0](https://en.wikipedia.org/wiki/Mozilla_Public_License) open source license. The only knowledge necessary to write a plugin is basic command-line skills and knowledge of the Go programming language. When writing in Go-Language, HashiCorp has found the integration development process to be straightforward and simple when vendors pay close attention and follow the resources and by adopting the same structure and coding patterns helps expedite the review and release cycles. Please remember that all integration major steps should contain acceptance testing and the appropriate documentation. Container Runtime * [Task driver documentation](https://developer.hashicorp.com/nomad/docs/job-declare/task-driver) * [Guide to build, install, and maintaining a task driver plugin](https://developer.hashicorp.com/nomad/plugins/author/task-driver) * [Community examples of task drivers](https://developer.hashicorp.com/nomad/plugins/drivers/community) GPUs & Specialized Hardware Devices * [Device plugin documentation](https://developer.hashicorp.com/nomad/plugins/devices) * [Guide to build, install, and maintain a device plugin](https://developer.hashicorp.com/nomad/plugins/author/device) * [Template for writing a device plugin](https://github.com/hashicorp/nomad-skeleton-device-plugin) * [Example of a device plugin](https://github.com/hashicorp/nomad/tree/main/devices/gpu/nvidia) Autoscaling * [Autoscaling plugin documentation](https://developer.hashicorp.com/nomad/tools/autoscaling) Storage * [CSI plugins documentation](https://developer.hashicorp.com/nomad/docs/architecture/storage/csi) Observability & Analysis * [Telemetry documentation](https://developer.hashicorp.com/nomad/docs/monitor) [](https://developer.hashicorp.com/nomad/docs/partnerships#4-review) 4\. Review ------------------------------------------------------------------------------- During the review process, HashiCorp will provide feedback on the newly developed integration. This is an important step to allow HashiCorp to review and verify your Nomad integration. Please send the integration code and other relevant logs for verification to: [Nomad-integration-dev@hashicorp.com](mailto:Nomad-integration-dev@hashicorp.com) . For task, device, storage, and autoscaling plugins, please submit a GitHub pull request (PR) against the [Nomad project](https://github.com/hashicorp/nomad) . In some cases the vendor may need to provide HashiCorp with a permanent test account so that the integration can be verified on an ongoing basis. The review process can take a while to complete and may require some iterations through the code to address and problems identified by the HashiCorp team. ### [](https://developer.hashicorp.com/nomad/docs/partnerships#5-release) 5\. Release At this stage, it is expected that the integration is fully complete, the necessary documentation has been written, the acceptance tests have all passed, and that HashiCorp has reviewed the integration. Once the plugin has been validated and accepted by HashiCorp, the plugin can be hosted anywhere so it can easily be downloaded then installed within Nomad. Once the integration has been released the vendor is requested to sign the HashiCorp Technology Partner Agreement so that we can have their integration be listed on the HashiCorp website. ### [](https://developer.hashicorp.com/nomad/docs/partnerships#6-support) 6\. Support Many vendors view the release step to be the end of the journey, while at HashiCorp we view it to be the beginning of the journey. Getting the integration built is just the first step in enabling users to leverage it against their infrastructure. Once development is completed, on-going effort is required to support the developed integration, maintain the provider, and address any issues in a timely manner. The expectation from the vendor/partner is to create a mechanism for them to track and resolve all critical issues as soon as possible within 48 hours and all other issues within 5 business days. This is a requirement given the critical nature of Nomad to the customer's operation. Vendors who choose to not support their integration will not be considered a verified integration and cannot be listed on the website. [](https://developer.hashicorp.com/nomad/docs/partnerships#checklist) Checklist ------------------------------------------------------------------------------- Below is an ordered checklist of steps that should be followed during the integration process. This just reiterates the steps already documented in the sections above. * Fill out the Nomad integration [webform](https://docs.google.com/forms/d/e/1FAIpQLSflmhg8lkY5QyH3CB5TUMi4I1ZFQFx_GP_TVDtsbxMaFEMWpw/viewform) * Execute the HashiCorp MNDA (Mutual Non-Disclosure Agreement) if needed * Develop and test Nomad integration along with the acceptance tests and documentation * Send email to [Nomad-integration-dev@hashicorp.com](mailto:Nomad-integration-dev@hashicorp.com) to schedule an initial review * Address review feedback and finalize the development process * Provide HashiCorp with credentials for underlying infrastructure for test purposes * Demo the integration and/or send the test logs to HashiCorp at: [Nomad-integration-dev@hashicorp.com](mailto:Nomad-integration-dev@hashicorp.com) * Execute HashiCorp Partner Agreement Documents, review logo guidelines, partner listing and more * Plan to continue supporting the integration with additional functionality and responding to customer issues. [](https://developer.hashicorp.com/nomad/docs/partnerships#contact-us) Contact Us --------------------------------------------------------------------------------- For any questions or feedback, please contact us at: [Nomad-integration-dev@hashicorp.com](mailto:Nomad-integration-dev@hashicorp.com) . [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/partnerships.mdx) --- # Sentinel Policies - HTTP API | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Sentinel Policies HTTP API ========================== The `/sentinel/policies` and `/sentinel/policy/` endpoints are used to manage Sentinel policies. For more details about Sentinel policies, please see the [Sentinel Policy Guide](https://developer.hashicorp.com/nomad/docs/govern/sentinel) . Sentinel endpoints are only available when ACLs are enabled. For more details about ACLs, please see the [ACL Guide](https://developer.hashicorp.com/nomad/docs/secure/acl) . Enterprise This feature requires [Nomad Enterprise](https://www.hashicorp.com/products/nomad) (opens in new tab). [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#list-policies) List Policies ------------------------------------------------------------------------------------------------ This endpoint lists all Sentinel policies. This lists the policies that have been replicated to the region, and may lag behind the authoritative region. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/sentinel/policies` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | Consistency Modes | ACL Required | | --- | --- | --- | | `YES` | `all` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-request) Sample Request $ curl \ https://localhost:4646/v1/sentinel/policies ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-response) Sample Response [\ {\ "Name": "foo",\ "Description": "test policy",\ "Scope": "submit-job",\ "EnforcementLevel": "advisory",\ "Hash": "CIs8aNX5OfFvo4D7ihWcQSexEJpHp+Za+dHSncVx5+8=",\ "CreateIndex": 8,\ "ModifyIndex": 8\ }\ ] [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#create-or-update-policy) Create or Update Policy -------------------------------------------------------------------------------------------------------------------- This endpoint creates or updates an Sentinel Policy. This request is always forwarded to the authoritative region. | Method | Path | Produces | | --- | --- | --- | | `POST` | `/v1/sentinel/policy/:policy_name` | `(empty body)` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#parameters) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`Name`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#name) `(string: )` - Specifies the name of the policy. Creates the policy if the name does not exist, otherwise updates the existing policy. * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`Description`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#description) `(string: )` - Specifies a human readable description. * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`Scope`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#scope) `(string: )` - Specifies the scope of when this policy applies. Only `submit-job` is currently supported. * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`EnforcementLevel`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#enforcementlevel) `(string: )` - Specifies the enforcement level of the policy. Can be `advisory` which warns on failure, `hard-mandatory` which prevents an operation on failure, and `soft-mandatory` which is like `hard-mandatory` but can be overridden. * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`Policy`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#policy) `(string: )` - Specifies the Sentinel policy itself. ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-payload) Sample Payload { "Name": "my-policy", "Description": "This is a great policy", "Scope": "submit-job", "EnforcementLevel": "advisory", "Policy": "main = rule { true }" } ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-request-1) Sample Request $ curl \ --request POST \ --data @payload.json \ https://localhost:4646/v1/sentinel/policy/my-policy [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#read-policy) Read Policy -------------------------------------------------------------------------------------------- This endpoint reads a Sentinel policy with the given name. This queries the policy that have been replicated to the region, and may lag behind the authoritative region. | Method | Path | Produces | | --- | --- | --- | | `GET` | `/v1/sentinel/policy/:policy_name` | `application/json` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) , [consistency modes](https://developer.hashicorp.com/nomad/api-docs#consistency-modes) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | Consistency Modes | ACL Required | | --- | --- | --- | | `YES` | `all` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-request-2) Sample Request $ curl \ https://localhost:4646/v1/sentinel/policy/foo ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-response-1) Sample Response { "Name": "foo", "Description": "test policy", "Scope": "submit-job", "EnforcementLevel": "advisory", "Policy": "main = rule { true }\n", "Hash": "CIs8aNX5OfFvo4D7ihWcQSexEJpHp+Za+dHSncVx5+8=", "CreateIndex": 8, "ModifyIndex": 8 } [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#delete-policy) Delete Policy ------------------------------------------------------------------------------------------------ This endpoint deletes the named Sentinel policy. This request is always forwarded to the authoritative region. | Method | Path | Produces | | --- | --- | --- | | `DELETE` | `/v1/sentinel/policy/:policy_name` | `(empty body)` | The table below shows this endpoint's support for [blocking queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries) and [required ACLs](https://developer.hashicorp.com/nomad/api-docs#acls) . | Blocking Queries | ACL Required | | --- | --- | | `NO` | `management` | ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#parameters-1) Parameters * [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#) [`policy_name`](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#policy_name) `(string: )` - Specifies the policy name to delete. ### [](https://developer.hashicorp.com/nomad/api-docs/sentinel-policies#sample-request-3) Sample Request $ curl \ --request DELETE \ https://localhost:4646/v1/sentinel/policy/foo [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/api-docs/sentinel-policies.mdx) --- # Enable gossip encryption | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) Enable gossip encryption ======================== Nomad server's gossip protocol that is used to communicate membership and liveness information can be encrypted with symmetric keys. Enabling gossip encryption requires you to set an encryption key when starting the Nomad server. The key can be set via the [`encrypt`](https://developer.hashicorp.com/nomad/docs/configuration/server#encrypt) parameter or with the [`-encrypt` command line option](https://developer.hashicorp.com/nomad/commands/agent) . The key must be a base64-encoded string of 32 random bytes. The same encryption key should be used on every server in a region. Note To secure RPC and HTTP communication, you will need to configure TLS. You can learn how in the [Enable TLS encryption guide](https://developer.hashicorp.com/nomad/docs/secure/traffic/tls) . [](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption#generate-an-encryption-key) Generate an encryption key ------------------------------------------------------------------------------------------------------------------------------------- The Nomad CLI includes a `operator gossip keyring generate` command for generating a new secure gossip encryption key. $ nomad operator gossip keyring generate 4kRkFQfcc3LU0BazP1ca+z== Current and older versions of `nomad operator gossip keyring generate` return 16 bytes; however, Nomad supports gossip encryption keys of 32 bytes as well. Supplying a 32 byte key enables AES-256 mode, where supplying a 16 byte key enables AES-128 mode. Alternatively, you can use any method that can create 32 random bytes encoded in base64. $ openssl rand -base64 32 4YwLQm6ZMwYgfldNBT5P76tAWMdcBmu+FPYRvCxvsHc= $ dd if=/dev/urandom bs=32 count=1 status=none | base64 IisA4F7Mu/RwGfBZelcsFzMlJ4+twnO5Z7eoTzD0T6c= [](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption#configure-the-server-to-use-the-key) Configure the server to use the key ------------------------------------------------------------------------------------------------------------------------------------------------------- Put the same generated key into every server's configuration file or command line arguments: server { enabled = true # Self-elect, should be 3 or 5 for production. This is only for single node # clusters which are strictly for development/demo. bootstrap_expect = 1 # Encrypt gossip communication encrypt = "+p7iF56z0EWoSIvhpYHWXZrSAAtnjR9l6XHRzHqQKlg=" } [](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption#restart-the-server-to-enable-encryption) Restart the server to enable encryption --------------------------------------------------------------------------------------------------------------------------------------------------------------- You can perform a rolling restart of the Nomad process on each of your server nodes to enable encryption. Restart your servers one at a time in order to maintain a quorum of nodes on one side or the other of this soft partition. Once all of the nodes have been restarted all gossip traffic will be encrypted between all of your server nodes. [](https://developer.hashicorp.com/nomad/docs/secure/traffic/gossip-encryption#next-steps) Next steps ----------------------------------------------------------------------------------------------------- If you would like to learn more technical information about Nomad's gossip protocol, consult the [Serf library](https://www.serf.io/docs/internals/gossip.html) documentation. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.11.x/content/docs/secure/traffic/gossip-encryption.mdx) --- # Traffic encryption overview | Nomad | HashiCorp Developer [HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream](https://www.hashicorp.com/conferences/hashiconf#livestream) [Nomad Home](https://developer.hashicorp.com/nomad) You are viewing documentation for version v1.10.x. [View latest version](https://developer.hashicorp.com/nomad/docs/secure/traffic) . Traffic encryption overview =========================== The Nomad agent supports encrypting all of its network traffic. There are two separate encryption systems, one for gossip traffic, and one for HTTP and RPC. Once you complete this collection you will have configured transport encryption for all of Nomad's protocols. RPC and HTTP communication is secured with mTLS. Nomad servers also communicate with a gossip protocol, Serf, which is encrypted using symmetric encryption and preshared keys. To review Nomad uses two different encryption schemes for its traffic. * HTTP - Used to communicate between CLI and Nomad agents. Secured by mTLS. * RPC - Used to communicate between Nomad agents. Secured by mTLS. * Serf - Used to communicate between Nomad servers. Frequently referred to as "gossip". Secured by a shared key. Use the [Enable TLS Encryption for Nomad](https://developer.hashicorp.com/nomad/docs/v1.10.x/secure/traffic/tls) guide to configure Nomad for mTLS and the [Enable Gossip Encryption for Nomad](https://developer.hashicorp.com/nomad/docs/v1.10.x/secure/traffic/gossip-encryption) to configure gossip encryption using a pre-shared key. [Edit this page on GitHub](https://github.com/hashicorp/web-unified-docs/blob/main/content/nomad/v1.10.x/content/docs/secure/traffic/index.mdx) ---