# Table of Contents - [Overview - Baseten](#overview-baseten) - [Cancel a queued async request. - Baseten](#cancel-a-queued-async-request-baseten) - [Get the status of an async request. - Baseten](#get-the-status-of-an-async-request-baseten) - [Asynchronously call the development deployment of a chain. - Baseten](#asynchronously-call-the-development-deployment-of-a-chain-baseten) - [Asynchronously call the production environment of a chain. - Baseten](#asynchronously-call-the-production-environment-of-a-chain-baseten) - [Asynchronously call a specific deployment of a model. - Baseten](#asynchronously-call-a-specific-deployment-of-a-model-baseten) - [Asynchronously call the development deployment of a model. - Baseten](#asynchronously-call-the-development-deployment-of-a-model-baseten) - [Asynchronously call the production environment of a model. - Baseten](#asynchronously-call-the-production-environment-of-a-model-baseten) - [Asynchronously call a specific deployment of a chain. - Baseten](#asynchronously-call-a-specific-deployment-of-a-chain-baseten) - [Asynchronously call a named environment of a model. - Baseten](#asynchronously-call-a-named-environment-of-a-model-baseten) - [Asynchronously call a named environment of a chain. - Baseten](#asynchronously-call-a-named-environment-of-a-chain-baseten) - [AI tools - Baseten](#ai-tools-baseten) - [Wake the development deployment of a model. - Baseten](#wake-the-development-deployment-of-a-model-baseten) - [Call the development deployment of a model. - Baseten](#call-the-development-deployment-of-a-model-baseten) - [Wake a specific deployment of a model by deployment ID. - Baseten](#wake-a-specific-deployment-of-a-model-by-deployment-id-baseten) - [Wake a named environment of a model. - Baseten](#wake-a-named-environment-of-a-model-baseten) - [Call the chain deployment associated with a specified environment. - Baseten](#call-the-chain-deployment-associated-with-a-specified-environment-baseten) - [Get async queue status for a named environment. - Baseten](#get-async-queue-status-for-a-named-environment-baseten) - [Wake the production environment of a model. - Baseten](#wake-the-production-environment-of-a-model-baseten) - [Call the development deployment of a chain. - Baseten](#call-the-development-deployment-of-a-chain-baseten) - [Get async queue status for a specific deployment. - Baseten](#get-async-queue-status-for-a-specific-deployment-baseten) - [Call a specific chain deployment by deployment ID. - Baseten](#call-a-specific-chain-deployment-by-deployment-id-baseten) - [Call the production environment of a model. - Baseten](#call-the-production-environment-of-a-model-baseten) - [Call the production environment of a chain. - Baseten](#call-the-production-environment-of-a-chain-baseten) - [Call a specific deployment of a model by deployment ID. - Baseten](#call-a-specific-deployment-of-a-model-by-deployment-id-baseten) - [Call the model deployment associated with a specified environment. - Baseten](#call-the-model-deployment-associated-with-a-specified-environment-baseten) - [Get async queue status for the development deployment. - Baseten](#get-async-queue-status-for-the-development-deployment-baseten) - [Asynchronously call a regional environment of a chain. - Baseten](#asynchronously-call-a-regional-environment-of-a-chain-baseten) - [Get async queue status for the production environment. - Baseten](#get-async-queue-status-for-the-production-environment-baseten) - [Wake a regional environment of a model. - Baseten](#wake-a-regional-environment-of-a-model-baseten) - [Call a regional environment of a model. - Baseten](#call-a-regional-environment-of-a-model-baseten) - [Call a regional environment of a chain. - Baseten](#call-a-regional-environment-of-a-chain-baseten) - [Get async queue status for a regional environment. - Baseten](#get-async-queue-status-for-a-regional-environment-baseten) - [Asynchronously call a regional environment of a model. - Baseten](#asynchronously-call-a-regional-environment-of-a-model-baseten) - [Baseten platform status - Baseten](#baseten-platform-status-baseten) - [Truss SDK Reference - Baseten](#truss-sdk-reference-baseten) - [Reference documentation - Baseten](#reference-documentation-baseten) - [Rate limits - Baseten](#rate-limits-baseten) - [Get a session - Baseten](#get-a-session-baseten) - [Speech-to-text models - Baseten](#speech-to-text-models-baseten) - [Agentic models - Baseten](#agentic-models-baseten) - [Long context models - Baseten](#long-context-models-baseten) - [Reasoning models - Baseten](#reasoning-models-baseten) - [Tool calling models - Baseten](#tool-calling-models-baseten) - [Multimodal (image) models - Baseten](#multimodal-image-models-baseten) - [Loops SDK - Baseten](#loops-sdk-baseten) - [Chains SDK Reference - Baseten](#chains-sdk-reference-baseten) - [Upsert a secret - Baseten](#upsert-a-secret-baseten) - [Create an API key - Baseten](#create-an-api-key-baseten) - [Get environment - Baseten](#get-environment-baseten) - [Update model environment - Baseten](#update-model-environment-baseten) - [Create environment - Baseten](#create-environment-baseten) - [Deployments - Baseten](#deployments-baseten) - [Lifecycle - Baseten](#lifecycle-baseten) - [Serving your trained model - Baseten](#serving-your-trained-model-baseten) - [Management - Baseten](#management-baseten) - [Truss configuration - Baseten](#truss-configuration-baseten) - [Deploy with optimized inference engines - Baseten](#deploy-with-optimized-inference-engines-baseten) - [Loading checkpoints - Baseten](#loading-checkpoints-baseten) - [Get started - Baseten](#get-started-baseten) - [Training on Baseten - Baseten](#training-on-baseten-baseten) - [External packages - Baseten](#external-packages-baseten) - [Overview - Baseten](#overview-baseten) - [Training SDK - Baseten](#training-sdk-baseten) - [Inference - Baseten](#inference-baseten) - [Overview - Baseten](#overview-baseten) - [Any model deployment by ID - Baseten](#any-model-deployment-by-id-baseten) - [Create training job - Baseten](#create-training-job-baseten) - [SSH access - Baseten](#ssh-access-baseten) - [Promote to model environment - Baseten](#promote-to-model-environment-baseten) - [Storage and data ingestion - Baseten](#storage-and-data-ingestion-baseten) - [Overview - Baseten](#overview-baseten) - [Unknown](#unknown) - [Development model deployment - Baseten](#development-model-deployment-baseten) - [Get all environments - Baseten](#get-all-environments-baseten) - [Get all secrets - Baseten](#get-all-secrets-baseten) - [Update production deployment autoscaling settings - Baseten](#update-production-deployment-autoscaling-settings-baseten) - [All models - Baseten](#all-models-baseten) - [Create a team API key - Baseten](#create-a-team-api-key-baseten) - [Delete models - Baseten](#delete-models-baseten) - [Update chainlet environment's autoscaling settings - Baseten](#update-chainlet-environment-s-autoscaling-settings-baseten) - [Get training job - Baseten](#get-training-job-baseten) - [Resume rolling deployment - Baseten](#resume-rolling-deployment-baseten) - [All instance types - Baseten](#all-instance-types-baseten) - [List all teams - Baseten](#list-all-teams-baseten) - [Instance type prices - Baseten](#instance-type-prices-baseten) - [Create training project - Baseten](#create-training-project-baseten) - [Chain deployment - Baseten](#chain-deployment-baseten) - [By ID - Baseten](#by-id-baseten) - [All chains - Baseten](#all-chains-baseten) - [Delete training project - Baseten](#delete-training-project-baseten) - [Create a team training project - Baseten](#create-a-team-training-project-baseten) - [Development model deployment - Baseten](#development-model-deployment-baseten) - [Update chainlet environment's instance type - Baseten](#update-chainlet-environment-s-instance-type-baseten) - [Any model deployment by ID - Baseten](#any-model-deployment-by-id-baseten) - [Development deployment - Baseten](#development-deployment-baseten) - [Production deployment - Baseten](#production-deployment-baseten) - [Any deployment by ID - Baseten](#any-deployment-by-id-baseten) - [Deactivate environment deployment - Baseten](#deactivate-environment-deployment-baseten) - [Any deployment by ID - Baseten](#any-deployment-by-id-baseten) - [Deactivate production deployment - Baseten](#deactivate-production-deployment-baseten) - [Development deployment - Baseten](#development-deployment-baseten) - [Get training job checkpoint files - Baseten](#get-training-job-checkpoint-files-baseten) - [List training job checkpoints - Baseten](#list-training-job-checkpoints-baseten) - [Delete chains - Baseten](#delete-chains-baseten) - [Force cancel rolling deployment - Baseten](#force-cancel-rolling-deployment-baseten) - [Get training project cache summary - Baseten](#get-training-project-cache-summary-baseten) --- # Overview - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/overview#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten is a training and inference platform. Bring a model (an open-source LLM from Hugging Face, a fine-tuned checkpoint, or a custom model) and Baseten turns it into a production API endpoint with autoscaling, observability, and optimized serving infrastructure. Baseten handles containerization, GPU scheduling across multiple clouds, and engine-level optimizations like TensorRT-LLM compilation, so you can focus on your model and your application. If you want to skip deployment entirely and start making inference calls right now, [Model APIs](https://docs.baseten.co/inference/model-apis/overview) provide OpenAI-compatible endpoints for models like DeepSeek, Qwen, and GLM. Point the OpenAI SDK at Baseten’s URL to run inference in seconds. If you’re an AI lab serving your own hosted model to your own customers under a branded URL with federated keys and per-customer billing, [Frontier Gateway](https://docs.baseten.co/frontier-gateway/overview) is the managed gateway product for that. [Quickstart: Make your first inference call\ ------------------------------------------\ \ Call a model through Model APIs in under two minutes. No deployment, no setup, just an API key and a request.](https://docs.baseten.co/quickstart) [​](https://docs.baseten.co/overview#deploy-a-model) Deploy a model ---------------------------------------------------------------------- The most common way to deploy a model on Baseten is with [Truss](https://pypi.org/project/truss/) , an open-source framework that packages your model into a deployable container. For supported architectures (most popular open-source LLMs, embedding models, and image generators), you only need a `config.yaml` file. Specify the model, the hardware, and the engine, and Truss handles the rest. config.yaml model_name: Qwen-2.5-3B resources: accelerator: L4 trt_llm: build: base_model: decoder checkpoint_repository: source: HF repo: "Qwen/Qwen2.5-3B-Instruct" Run `truss push` and Baseten builds a TensorRT-optimized container, deploys it to GPU infrastructure, and provides an endpoint. The model serves an OpenAI-compatible API out of the box. When you need custom behavior like preprocessing, postprocessing, or a model architecture that the built-in engines don’t support, Truss also supports [custom Python model code](https://docs.baseten.co/development/model/custom-model-code) . Write a `Model` class with `load` and `predict` methods, and Truss packages it the same way. Most teams start with config-only deployments and add custom code only when they need it. [Your first model\ ----------------\ \ Deploy a model to Baseten with just a config file. No custom code needed.](https://docs.baseten.co/development/model/build-your-first-model) [​](https://docs.baseten.co/overview#inference-engines) Inference engines ---------------------------------------------------------------------------- Baseten optimizes every deployment with an inference engine tuned for your model’s architecture. Select the engine that best supports your use case, and it handles the low-level performance work: quantization, tensor parallelism, KV cache management, and batching. [Engine-Builder-LLM\ ------------------\ \ Dense text generation models compiled with TensorRT-LLM. Supports lookahead decoding and structured outputs.](https://docs.baseten.co/engines/engine-builder-llm/overview) [BIS-LLM\ -------\ \ Large mixture-of-experts models like DeepSeek R1 and Qwen3 MoE with KV-aware routing and distributed inference.](https://docs.baseten.co/engines/bis-llm/overview) [BEI\ ---\ \ Embedding, reranking, and classification models with up to 1,400 embeddings per second throughput.](https://docs.baseten.co/engines/bei/overview) Choose the engine through a field in your `config.yaml`, or Baseten selects it automatically based on your model architecture. [​](https://docs.baseten.co/overview#multi-step-workflows-with-chains) Multi-step workflows with Chains ---------------------------------------------------------------------------------------------------------- Some applications need more than a single model call. A RAG pipeline retrieves documents, embeds them, and generates a response. An image generation workflow runs a diffusion model, upscales the result, and applies safety filtering. [Chains](https://docs.baseten.co/development/chain/overview) is Baseten’s framework for orchestrating these multi-step pipelines. Each step runs on its own hardware with its own dependencies, and Chains manages the data flow between them. Define the pipeline in Python, and Chains deploys, scales, and monitors each step independently. [​](https://docs.baseten.co/overview#training) Training ---------------------------------------------------------- Baseten also provides [training infrastructure](https://docs.baseten.co/training/overview) for fine-tuning and pre-training. Bring your training scripts (Axolotl, TRL, Megatron, or custom code) and run jobs on H200 or H100 GPUs. Checkpoints sync automatically during training, and you can deploy a fine-tuned model from checkpoint to production endpoint in a single command with `truss train deploy_checkpoints`. [​](https://docs.baseten.co/overview#production-infrastructure) Production infrastructure -------------------------------------------------------------------------------------------- Every deployment on Baseten runs on autoscaling infrastructure that adjusts replicas based on traffic. Configure minimum and maximum replicas, concurrency targets, and scale-down delays. Or use the defaults, which handle most workloads well. Models scale to zero when idle, eliminating costs during quiet periods, and scale up within seconds when traffic arrives. Baseten schedules workloads across multiple cloud providers and regions through Multi-cloud Capacity Management (MCM). Your models stay available even during provider-level disruptions, and MCM routes traffic across regions to minimize latency. Built-in [observability](https://docs.baseten.co/observability/metrics) gives you real-time metrics, logs, and request traces for every deployment. Export data to tools like Datadog or Prometheus, and debug behavior with full visibility into inputs, outputs, and errors. [​](https://docs.baseten.co/overview#find-your-path) Find your path ---------------------------------------------------------------------- * Build AI applications * Deploy and optimize models * Train and fine-tune Start with Model APIs and explore features that support production use cases. * [Model APIs overview](https://docs.baseten.co/inference/model-apis/overview) * [Structured outputs](https://docs.baseten.co/inference/structured-outputs) * [Tool calling](https://docs.baseten.co/inference/function-calling) * [RAG pipeline example](https://docs.baseten.co/examples/chains-build-rag) Deploy models on dedicated infrastructure with a config-only Truss deployment and tune from there. * [Deploy your first model](https://docs.baseten.co/development/model/build-your-first-model) * [Engine selection](https://docs.baseten.co/engines) * [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview) * [Performance optimization](https://docs.baseten.co/development/model/performance-optimization) Run training jobs and deploy results directly to production endpoints. * [Training overview](https://docs.baseten.co/training/overview) * [Get started with training](https://docs.baseten.co/training/getting-started) * [Deploy from checkpoint](https://docs.baseten.co/training/deployment) [​](https://docs.baseten.co/overview#next-steps) Next steps -------------------------------------------------------------- [How Baseten works\ -----------------\ \ The build pipeline, request routing, autoscaling, and deployment lifecycle under the hood.](https://docs.baseten.co/concepts/howbasetenworks) [Examples\ --------\ \ End-to-end guides for deploying and optimizing popular models.](https://docs.baseten.co/examples/overview) [API reference\ -------------\ \ Reference for the inference API, management API, and Truss CLI.](https://docs.baseten.co/reference/overview#api-reference) Was this page helpful? YesNo [Why BasetenProduction training and inference on dedicated infrastructure, for teams that have outgrown shared API endpoints.\ \ Next](https://docs.baseten.co/concepts/whybaseten) Ctrl+I Assistant Responses are generated using AI and may contain mistakes. --- # Cancel a queued async request. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) DELETE https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / async\_request / {request\_id} Try it Cancel a queued async request. cURL curl --request DELETE \ --url https://model-{model_id}.api.baseten.co/async_request/{request_id} \ --header 'Authorization: Bearer ' 200 401 429 { "request_id": "", "canceled": true, "message": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#parameter-request-id) request\_id string required The ID of the async request. #### Response 200 application/json Result of an async request cancellation. [​](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#response-request-id) request\_id string required The ID of the async request. [​](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#response-canceled) canceled boolean required Whether the request was canceled. [​](https://docs.baseten.co/api-reference/cancel-a-queued-async-request#response-message) message string required Additional details about whether the request was canceled. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request) [Get async queue status for the production environment.Returns the number of queued and in-progress async requests for the deployment promoted to the production environment. Rate limited to 20 requests per second.\ \ Next](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment) ⌘I Cancel a queued async request. cURL curl --request DELETE \ --url https://model-{model_id}.api.baseten.co/async_request/{request_id} \ --header 'Authorization: Bearer ' 200 401 429 { "request_id": "", "canceled": true, "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get the status of an async request. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / async\_request / {request\_id} Try it Get the status of an async request. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/async_request/{request_id} \ --header 'Authorization: Bearer ' 200 401 429 { "request_id": "", "deployment_id": "", "created_at": "", "status_at": "", "errors": [], "model_id": "", "chain_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#parameter-request-id) request\_id string required The ID of the async request. #### Response 200 application/json Current status of an async request. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-request-id) request\_id string required The ID of the async request. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-deployment-id) deployment\_id string required The ID of the deployment that executed the request. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-status) status enum required The status of the async request. Available options: `QUEUED`, `IN_PROGRESS`, `SUCCEEDED`, `FAILED`, `EXPIRED`, `CANCELED`, `WEBHOOK_FAILED` [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-webhook-status) webhook\_status enum required The status of sending the prediction result to the provided webhook. Available options: `PENDING`, `SUCCEEDED`, `FAILED`, `CANCELED`, `NO_WEBHOOK_PROVIDED` [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-created-at) created\_at string required The time in UTC at which the async request was created. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-status-at) status\_at string required The time in UTC at which the async request's status was last updated. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-errors) errors object\[\] required Errors that occurred while processing the async request. Empty if no errors occurred. Show child attributes [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-model-id) model\_id string The ID of the model that executed the request. Present for model requests. [​](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request#response-chain-id) chain\_id string The ID of the chain that executed the request. Present for chain requests. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/inference-api/predict-endpoints/streaming-transcription-api) [Cancel a queued async request.Cancels an async request. Only requests with \`QUEUED\` status may be canceled. Rate limited to 20 requests per second.\ \ Next](https://docs.baseten.co/api-reference/cancel-a-queued-async-request) ⌘I Get the status of an async request. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/async_request/{request_id} \ --header 'Authorization: Bearer ' 200 401 429 { "request_id": "", "deployment_id": "", "created_at": "", "status_at": "", "errors": [], "model_id": "", "chain_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call the development deployment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / async\_run\_remote Try it Asynchronously call the development deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 201 application/json Async run remote request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain) [Asynchronously call a specific deployment of a chain.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain) ⌘I Asynchronously call the development deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call the production environment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / async\_run\_remote Try it Asynchronously call the production environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 201 application/json Async run remote request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model) [Asynchronously call a named environment of a chain.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain) ⌘I Asynchronously call the production environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a specific deployment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / async\_predict Try it Asynchronously call a specific deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Body application/json There is a 256 KiB size limit on async predict request payloads. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#body-model-input) model\_input object required JSON-serializable model input. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#body-webhook-endpoint) webhook\_endpoint string HTTPS URL to receive the prediction result via webhook. Both HTTP/2 and HTTP/1.1 are supported. If omitted, the model must save outputs so they can be accessed later. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#body-priority) priority integer default:0 Priority of the request. Lower values are higher priority. Required range: `0 <= x <= 2` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#body-max-time-in-queue-seconds) max\_time\_in\_queue\_seconds integer default:600 Maximum time in seconds a request will spend in the queue before expiring. Must be between 10 seconds and 72 hours. Required range: `10 <= x <= 259200` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#body-inference-retry-config) inference\_retry\_config object Exponential backoff parameters for retrying predict requests. Show child attributes #### Response 201 application/json Async predict request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model) [Asynchronously call the production environment of a chain.Enqueues an asynchronous request for the chain deployment promoted to the production environment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain) ⌘I Asynchronously call a specific deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call the development deployment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / async\_predict Try it Asynchronously call the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json There is a 256 KiB size limit on async predict request payloads. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#body-model-input) model\_input object required JSON-serializable model input. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#body-webhook-endpoint) webhook\_endpoint string HTTPS URL to receive the prediction result via webhook. Both HTTP/2 and HTTP/1.1 are supported. If omitted, the model must save outputs so they can be accessed later. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#body-priority) priority integer default:0 Priority of the request. Lower values are higher priority. Required range: `0 <= x <= 2` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#body-max-time-in-queue-seconds) max\_time\_in\_queue\_seconds integer default:600 Maximum time in seconds a request will spend in the queue before expiring. Must be between 10 seconds and 72 hours. Required range: `10 <= x <= 259200` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#body-inference-retry-config) inference\_retry\_config object Exponential backoff parameters for retrying predict requests. Show child attributes #### Response 201 application/json Async predict request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model) [Asynchronously call a specific deployment of a model.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model) ⌘I Asynchronously call the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call the production environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / async\_predict Try it Asynchronously call the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json There is a 256 KiB size limit on async predict request payloads. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#body-model-input) model\_input object required JSON-serializable model input. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#body-webhook-endpoint) webhook\_endpoint string HTTPS URL to receive the prediction result via webhook. Both HTTP/2 and HTTP/1.1 are supported. If omitted, the model must save outputs so they can be accessed later. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#body-priority) priority integer default:0 Priority of the request. Lower values are higher priority. Required range: `0 <= x <= 2` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#body-max-time-in-queue-seconds) max\_time\_in\_queue\_seconds integer default:600 Maximum time in seconds a request will spend in the queue before expiring. Must be between 10 seconds and 72 hours. Required range: `10 <= x <= 259200` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#body-inference-retry-config) inference\_retry\_config object Exponential backoff parameters for retrying predict requests. Show child attributes #### Response 201 application/json Async predict request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id) [Asynchronously call a named environment of a model.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model) ⌘I Asynchronously call the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a specific deployment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / async\_run\_remote Try it Asynchronously call a specific deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 201 application/json Async run remote request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain) [Websocket environmentConnect via WebSocket to the deployment associated with an environment.\ \ Next](https://docs.baseten.co/reference/inference-api/predict-endpoints/environments-websocket) ⌘I Asynchronously call a specific deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a named environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / async\_predict Try it Asynchronously call a named environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Body application/json There is a 256 KiB size limit on async predict request payloads. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#body-model-input) model\_input object required JSON-serializable model input. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#body-webhook-endpoint) webhook\_endpoint string HTTPS URL to receive the prediction result via webhook. Both HTTP/2 and HTTP/1.1 are supported. If omitted, the model must save outputs so they can be accessed later. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#body-priority) priority integer default:0 Priority of the request. Lower values are higher priority. Required range: `0 <= x <= 2` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#body-max-time-in-queue-seconds) max\_time\_in\_queue\_seconds integer default:600 Maximum time in seconds a request will spend in the queue before expiring. Must be between 10 seconds and 72 hours. Required range: `10 <= x <= 259200` [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#body-inference-retry-config) inference\_retry\_config object Exponential backoff parameters for retrying predict requests. Show child attributes #### Response 201 application/json Async predict request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model) [Asynchronously call the development deployment of a model.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model) ⌘I Asynchronously call a named environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a named environment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / async\_run\_remote Try it Asynchronously call a named environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 201 application/json Async run remote request enqueued. [​](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain) [Asynchronously call the development deployment of a chain.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain) ⌘I Asynchronously call a named environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # AI tools - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/ai-tools#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten docs are optimized for AI tools. Connect your assistants, coding tools, and agents directly to the docs so they have up-to-date context when helping you build on Baseten. Every page includes a contextual menu (the icon in the top-right corner of any page) with shortcuts to copy content and connect your MCP server. [​](https://docs.baseten.co/ai-tools#mcp-server) MCP server -------------------------------------------------------------- The Model Context Protocol (MCP) connects AI tools directly to Baseten documentation. When connected, your AI tool searches the docs in real time while generating responses, so you get answers grounded in current documentation rather than stale training data. The Baseten docs MCP server is available at: https://docs.baseten.co/mcp * Claude Code * Claude Desktop * Cursor * VS Code Add the MCP server to Claude Code: claude mcp add --transport http baseten-docs https://docs.baseten.co/mcp Claude Code searches Baseten docs automatically when relevant to your prompts. 1 [](https://docs.baseten.co/ai-tools#) Open Claude settings Navigate to the **Connectors** page in Claude settings. 2 [](https://docs.baseten.co/ai-tools#) Add the connector Select **Add custom connector**, then enter: * **Name:** Baseten Docs * **URL:** `https://docs.baseten.co/mcp` Select **Add**. 3 [](https://docs.baseten.co/ai-tools#) Use in conversations When starting a conversation, select the attachments button (the plus icon) and choose the Baseten Docs connector. Claude searches the docs as needed while responding. 1 [](https://docs.baseten.co/ai-tools#) Open MCP settings Use `Cmd + Shift + P` (macOS) or `Ctrl + Shift + P` (Windows/Linux) to open the command palette. Search for **“Open MCP settings”**. 2 [](https://docs.baseten.co/ai-tools#) Add the server Select **Add custom MCP**. This opens your `mcp.json` file. Add the Baseten docs server: mcp.json { "mcpServers": { "baseten-docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } } } Create or update `.vscode/mcp.json` in your project: .vscode/mcp.json { "servers": { "baseten-docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } } } ### [​](https://docs.baseten.co/ai-tools#other-mcp-clients) Other MCP clients Any MCP-compatible tool (Goose, ChatGPT, Windsurf, and others) can connect using the server URL `https://docs.baseten.co/mcp`. Refer to your tool’s documentation for how to add an MCP server. You can also use `npx add-mcp` to auto-detect supported AI tools on your system and configure them: npx add-mcp https://docs.baseten.co [​](https://docs.baseten.co/ai-tools#skills) Skills ------------------------------------------------------ The skills file describes what AI agents can accomplish with Baseten, including required inputs and constraints. AI coding tools use this file to understand Baseten capabilities without reading every documentation page. Install the Baseten docs skill into your AI coding tool: npx skills add https://docs.baseten.co This gives your AI tool structured knowledge of Baseten’s capabilities so it can help you deploy models, configure autoscaling, set up inference endpoints, and more with product-aware guidance. View the skill file directly at [docs.baseten.co/skill.md](https://docs.baseten.co/skill.md) . For skills that drive Baseten directly (deploying models, using the `truss` CLI, Chains, environments, and the inference and management APIs), see the open-source [`basetenlabs/baseten-skills`](https://github.com/basetenlabs/baseten-skills) repository. It currently includes the `baseten` skill. To add a skill, follow the authoring workflow in the repo’s `CONTRIBUTING.md`. Skills and MCP serve complementary purposes. **Skills** tell an AI tool _what Baseten can do_ and how to do it. **MCP** lets the tool _search current documentation_ for specific details. For the best results, install both. [​](https://docs.baseten.co/ai-tools#llms-txt) llms.txt ---------------------------------------------------------- The `llms.txt` file is an industry-standard directory that helps LLMs index documentation efficiently, similar to how `sitemap.xml` helps search engines. Baseten docs automatically host two versions: * [docs.baseten.co/llms.txt](https://docs.baseten.co/llms.txt) : a structured list of all pages with descriptions. * [docs.baseten.co/llms-full.txt](https://docs.baseten.co/llms-full.txt) : the full text content of all pages. These files stay up to date automatically and require no configuration. AI tools and search engines like ChatGPT, Perplexity, and Google AI Overviews use them to understand and cite Baseten documentation. [​](https://docs.baseten.co/ai-tools#markdown-access) Markdown access ------------------------------------------------------------------------ Every documentation page is available as Markdown by appending `.md` to the URL. For example: https://docs.baseten.co/quickstart.md AI agents receive page content as Markdown instead of HTML, which reduces token usage and improves processing speed. You can use this to quickly copy any page’s content into an AI conversation. [​](https://docs.baseten.co/ai-tools#contextual-menu-reference) Contextual menu reference -------------------------------------------------------------------------------------------- The contextual menu on each page provides one-click access to these integrations. Select the menu icon in the top-right corner of any page. | Option | Description | | --- | --- | | Copy page | Copies the page as Markdown for pasting into any AI tool. | | View as Markdown | Opens the page as raw Markdown in a new tab. | | Copy MCP server URL | Copies the MCP server URL to your clipboard. | | Connect to Cursor | Installs the MCP server in Cursor. | | Connect to VS Code | Installs the MCP server in VS Code. | Was this page helpful? YesNo [Previous](https://docs.baseten.co/concepts/howbasetenworks) [QuickstartStart running inference on Baseten.\ \ Next](https://docs.baseten.co/quickstart) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Wake the development deployment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / wake Try it Wake the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 202 Wake request accepted. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model) [Wake a specific deployment of a model by deployment ID.\ \ Next](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id) ⌘I Wake the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Call the development deployment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / predict Try it Call the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/jsonapplication/octet-streammultipart/form-dataapplication/jsonapplication/octet-streammultipart/form-data JSON-serializable model input. The shape is defined by the model's `predict` function. #### Response 200 application/json Successful synchronous prediction. JSON-serializable output. The shape is defined by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment) [Call a specific deployment of a model by deployment ID.Sends a synchronous predict request to the specified deployment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id) ⌘I Call the development deployment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Wake a specific deployment of a model by deployment ID. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / wake Try it Wake a specific deployment of a model by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Response 202 Wake request accepted. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model) [Call a regional environment of a model.Sends a synchronous predict request via a regional hostname. The environment is determined by the hostname, not the path.\ \ Next](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model) ⌘I Wake a specific deployment of a model by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Wake a named environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / wake Try it Wake a named environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Response 202 Wake request accepted. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model) [Wake the development deployment of a model.\ \ Next](https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model) ⌘I Wake a named environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Call the chain deployment associated with a specified environment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / run\_remote Try it Call the chain deployment associated with a specified environment. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 200 application/json Successful synchronous chain execution. JSON-serializable output. The shape is defined by the chain. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain) [Call the development deployment of a chain.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain) ⌘I Call the chain deployment associated with a specified environment. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Get async queue status for a named environment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / async\_queue\_status Try it Get async queue status for a named environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Response 200 application/json Async queue status for a deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#response-model-id) model\_id string required The ID of the model. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#response-deployment-id) deployment\_id string required The ID of the deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#response-num-queued-requests) num\_queued\_requests integer required Number of requests with QUEUED status awaiting processing. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment#response-num-in-progress-requests) num\_in\_progress\_requests integer required Number of requests currently being processed by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment) [Get async queue status for the development deployment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment) ⌘I Get async queue status for a named environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # Wake the production environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / wake Try it Wake the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 202 Wake request accepted. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment) [Wake a named environment of a model.\ \ Next](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model) ⌘I Wake the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Call the development deployment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / run\_remote Try it Call the development deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 200 application/json Successful synchronous chain execution. JSON-serializable output. The shape is defined by the chain. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment) [Call a specific chain deployment by deployment ID.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id) ⌘I Call the development deployment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/development/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Get async queue status for a specific deployment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / async\_queue\_status Try it Get async queue status for a specific deployment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Response 200 application/json Async queue status for a deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#response-model-id) model\_id string required The ID of the model. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#response-deployment-id) deployment\_id string required The ID of the deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#response-num-queued-requests) num\_queued\_requests integer required Number of requests with QUEUED status awaiting processing. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment#response-num-in-progress-requests) num\_in\_progress\_requests integer required Number of requests currently being processed by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment) [Wake the production environment of a model.Triggers a wake for the deployment promoted to the production environment. Returns immediately with 202 Accepted.\ \ Next](https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model) ⌘I Get async queue status for a specific deployment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # Call a specific chain deployment by deployment ID. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / run\_remote Try it Call a specific chain deployment by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 200 application/json Successful synchronous chain execution. JSON-serializable output. The shape is defined by the chain. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain) [Asynchronously call the production environment of a model.Enqueues an asynchronous predict request for the deployment promoted to the production environment. Returns a request ID that can be used to poll for status or cancel the request.\ \ Next](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model) ⌘I Call a specific chain deployment by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Call the production environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / predict Try it Call the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/jsonapplication/octet-streammultipart/form-dataapplication/jsonapplication/octet-streammultipart/form-data JSON-serializable model input. The shape is defined by the model's `predict` function. #### Response 200 application/json Successful synchronous prediction. JSON-serializable output. The shape is defined by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/inference-api/messages) [Call the model deployment associated with a specified environment.Sends a synchronous predict request to the deployment promoted to the specified environment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment) ⌘I Call the production environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Call the production environment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / run\_remote Try it Call the production environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 200 application/json Successful synchronous chain execution. JSON-serializable output. The shape is defined by the chain. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id) [Call the chain deployment associated with a specified environment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment) ⌘I Call the production environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/production/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Call a specific deployment of a model by deployment ID. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / deployment / {deployment\_id} / predict Try it Call a specific deployment of a model by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id#parameter-deployment-id) deployment\_id string required The alphanumeric ID of the deployment. #### Body application/jsonapplication/octet-streammultipart/form-dataapplication/jsonapplication/octet-streammultipart/form-data JSON-serializable model input. The shape is defined by the model's `predict` function. #### Response 200 application/json Successful synchronous prediction. JSON-serializable output. The shape is defined by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model) [Call the production environment of a chain.Sends a synchronous request to the chain deployment promoted to the production environment. The request body is forwarded to the chain's \`run\_remote\` entrypoint.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain) ⌘I Call a specific deployment of a model by deployment ID. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Call the model deployment associated with a specified environment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / environments / {env\_name} / predict Try it Call the model deployment associated with a specified environment. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment#parameter-env-name) env\_name string required The name of the environment (e.g. `production`, `staging`). #### Body application/jsonapplication/octet-streammultipart/form-dataapplication/jsonapplication/octet-streammultipart/form-data JSON-serializable model input. The shape is defined by the model's `predict` function. #### Response 200 application/json Successful synchronous prediction. JSON-serializable output. The shape is defined by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model) [Call the development deployment of a model.Sends a synchronous predict request to the development deployment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model) ⌘I Call the model deployment associated with a specified environment. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/environments/{env_name}/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Get async queue status for the development deployment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / development / async\_queue\_status Try it Get async queue status for the development deployment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/development/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 application/json Async queue status for a deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#response-model-id) model\_id string required The ID of the model. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#response-deployment-id) deployment\_id string required The ID of the deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#response-num-queued-requests) num\_queued\_requests integer required Number of requests with QUEUED status awaiting processing. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment#response-num-in-progress-requests) num\_in\_progress\_requests integer required Number of requests currently being processed by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment) [Get async queue status for a specific deployment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment) ⌘I Get async queue status for the development deployment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/development/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a regional environment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / async\_run\_remote Try it Asynchronously call a regional environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 201 application/json Async run remote request enqueued. [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model) [Wake a regional environment of a model.\ \ Next](https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model) ⌘I Asynchronously call a regional environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/async_run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 201 400 401 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get async queue status for the production environment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / production / async\_queue\_status Try it Get async queue status for the production environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/production/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } #### Authorizations [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 application/json Async queue status for a deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#response-model-id) model\_id string required The ID of the model. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#response-deployment-id) deployment\_id string required The ID of the deployment. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#response-num-queued-requests) num\_queued\_requests integer required Number of requests with QUEUED status awaiting processing. [​](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment#response-num-in-progress-requests) num\_in\_progress\_requests integer required Number of requests currently being processed by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/cancel-a-queued-async-request) [Get async queue status for a named environment.\ \ Next](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment) ⌘I Get async queue status for the production environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/production/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # Wake a regional environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / wake Try it Wake a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 202 Wake request accepted. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain) [Get async queue status for a regional environment.\ \ Next](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment) ⌘I Wake a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/wake \ --header 'Authorization: Bearer ' 401 { "error": "", "detail": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Call a regional environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / predict Try it Call a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/jsonapplication/octet-streammultipart/form-dataapplication/jsonapplication/octet-streammultipart/form-data JSON-serializable model input. The shape is defined by the model's `predict` function. #### Response 200 application/json Successful synchronous prediction. JSON-serializable output. The shape is defined by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id) [Call a regional environment of a chain.Sends a synchronous run\_remote request via a regional hostname. The environment is determined by the hostname, not the path.\ \ Next](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain) ⌘I Call a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Call a regional environment of a chain. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / run\_remote Try it Call a regional environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} #### Authorizations [​](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json JSON input matching the chain's `run_remote` method signature. #### Response 200 application/json Successful synchronous chain execution. JSON-serializable output. The shape is defined by the chain. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model) [Asynchronously call a regional environment of a model.Enqueues an asynchronous predict request via a regional hostname. The environment is determined by the hostname, not the path.\ \ Next](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model) ⌘I Call a regional environment of a chain. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/run_remote \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{}' 200 400 401 429 502 503 504 {} Assistant Responses are generated using AI and may contain mistakes. --- # Get async queue status for a regional environment. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / async\_queue\_status Try it Get async queue status for a regional environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } #### Authorizations [​](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 application/json Async queue status for a deployment. [​](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#response-model-id) model\_id string required The ID of the model. [​](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#response-deployment-id) deployment\_id string required The ID of the deployment. [​](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#response-num-queued-requests) num\_queued\_requests integer required Number of requests with QUEUED status awaiting processing. [​](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment#response-num-in-progress-requests) num\_in\_progress\_requests integer required Number of requests currently being processed by the model. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model) [OverviewManage models and deployments with the Baseten management API. It supports monitoring, CI/CD, and automation at both the model and workspace levels.\ \ Next](https://docs.baseten.co/reference/management-api/overview) ⌘I Get async queue status for a regional environment. cURL curl --request GET \ --url https://model-{model_id}.api.baseten.co/async_queue_status \ --header 'Authorization: Bearer ' 200 401 429 { "model_id": "", "deployment_id": "", "num_queued_requests": 123, "num_in_progress_requests": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # Asynchronously call a regional environment of a model. - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST https://model-{model\_id}.api.baseten.cohttps://chain-{chain\_id}.api.baseten.cohttps://model-{model\_id}-{env\_name}.api.baseten.cohttps://chain-{chain\_id}-{env\_name}.api.baseten.co / async\_predict Try it Asynchronously call a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } #### Authorizations [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json There is a 256 KiB size limit on async predict request payloads. [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#body-model-input) model\_input object required JSON-serializable model input. [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#body-webhook-endpoint) webhook\_endpoint string HTTPS URL to receive the prediction result via webhook. Both HTTP/2 and HTTP/1.1 are supported. If omitted, the model must save outputs so they can be accessed later. [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#body-priority) priority integer default:0 Priority of the request. Lower values are higher priority. Required range: `0 <= x <= 2` [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#body-max-time-in-queue-seconds) max\_time\_in\_queue\_seconds integer default:600 Maximum time in seconds a request will spend in the queue before expiring. Must be between 10 seconds and 72 hours. Required range: `10 <= x <= 259200` [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#body-inference-retry-config) inference\_retry\_config object Exponential backoff parameters for retrying predict requests. Show child attributes #### Response 201 application/json Async predict request enqueued. [​](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model#response-request-id) request\_id string required The ID of the async request. Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain) [Asynchronously call a regional environment of a chain.Enqueues an asynchronous run\_remote request via a regional hostname. The environment is determined by the hostname, not the path.\ \ Next](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain) ⌘I Asynchronously call a regional environment of a model. cURL curl --request POST \ --url https://model-{model_id}.api.baseten.co/async_predict \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data ' { "model_input": {}, "webhook_endpoint": "", "priority": 0, "max_time_in_queue_seconds": 600, "inference_retry_config": { "max_attempts": 3, "initial_delay_ms": 1000, "max_delay_ms": 5000 } } ' 201 400 401 413 429 503 { "request_id": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Baseten platform status - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/status/status#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) This page automatically refreshes with real-time data from our status monitoring system. All systems are operational. Model Inference --------------- Normal Management API -------------- Normal Web Application --------------- Normal Last updated: Loading… Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Truss SDK Reference - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/sdk/truss#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/reference/sdk/truss#authentication) Authentication --------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/reference/sdk/truss#truss-login-api_key-str-%E2%86%92-none) `truss.login(api_key: str) → None` Authenticates with Baseten using an API key. **Parameters:** | Name | Type | Description | | --- | --- | --- | | `api_key` | _str_ | Baseten API Key. | * * * [​](https://docs.baseten.co/reference/sdk/truss#deploying-a-model) Deploying a Model --------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/reference/sdk/truss#truss-push-target_directory-str-kwargs-%E2%86%92-modeldeployment) `truss.push(target_directory: str, **kwargs) → ModelDeployment` Deploys a **Truss** model to Baseten. **Parameters:** | Name | Type | Description | | --- | --- | --- | | `target_directory` | _str_ | Path to the Truss directory to push. | | `remote` | _Optional\[str\]_ | Name of the remote in `.trussrc` to push to. | | `model_name` | _Optional\[str\]_ | Temporarily override the model name for this deployment without updating `config.yaml`. | | `publish` | _bool_ | Deploy as **published**. If no production deployment exists, promote it to production. | | `promote` | _bool_ | Deploy as **published** and promote to production, even if a production deployment exists. | | `preserve_previous_production_deployment` | _bool_ | Preserve the previous production deployment’s **autoscaling settings** (only with `promote`). | | `trusted` | _bool_ | **Deprecated.** All models are trusted by default. This parameter is ignored. | | `deployment_name` | _Optional\[str\]_ | Custom deployment name (must contain only alphanumeric, `.`, `-`, or `_` characters). (Requires `publish` or `promote`.) | | `disable_truss_download` | _bool_ | Disable downloading of the Truss directory from the UI. | | `environment` | _Optional\[str\]_ | Name of a stable environment to deploy to. | | `progress_bar` | _Optional\[Progress\]_ | Optional `rich.progress.Progress` instance to render deploy progress. | | `include_git_info` | _bool_ | Attach git versioning info (sha, branch, tag) to deployments made from within a git repo. Defaults to `False`. | | `preserve_env_instance_type` | _bool_ | When deploying to an `environment`, whether to resolve the instance type from the Truss config’s `resources` (`False`) or preserve the instance type already configured on the environment. Defaults to `True`. | | `deploy_timeout_minutes` | _Optional\[int\]_ | Timeout in minutes for the deployment operation. | | `labels` | _Optional\[dict\]_ | JSON-serializable dictionary of label key-value pairs to attach to the deployment. | | `team` | _Optional\[str\]_ | Name of the team to push the model to. | **Returns:** [ModelDeployment](https://docs.baseten.co/reference/sdk/truss#class-truss-api-definitions-modeldeployment) – An object representing the deployed model. * * * [​](https://docs.baseten.co/reference/sdk/truss#model-deployment-object) Model Deployment Object --------------------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/reference/sdk/truss#class-truss-api-definitions-modeldeployment) _class_ `truss.api.definitions.ModelDeployment` Represents a deployed model (returned by `truss.push()`). **Attributes** `model_id` → `str`: Unique ID of the deployed model. `model_deployment_id` → `str`: Unique ID of the model deployment. **Methods** `wait_for_active(timeout_seconds: int = 600)` → bool Waits for the deployment to become **active**. | Name | Type | Description | | --- | --- | --- | | `timeout_seconds` | _int_ | Maximum time to wait in seconds. Defaults to `600`. | **Returns**: `true` when deployment is ready. **Raises**: `TimeoutError` if deployment doesn’t become active within `timeout_seconds`. `ValueError` if deployment fails. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/cli/loops/loops-cli) [ChainsPython SDK Reference for Chains\ \ Next](https://docs.baseten.co/reference/sdk/chains) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Reference documentation - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/overview#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) This reference section documents our API, CLI, and Python SDK for deploying models, managing inference chains, and calling endpoints in production. [​](https://docs.baseten.co/reference/overview#api-reference) API Reference ------------------------------------------------------------------------------ Baseten provides two sets of API endpoints: ![inference-api](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/inference-api.png?fit=max&auto=format&n=W3NbEem9OZkF5rdB&q=85&s=db298df57b8b5162245c0288b629fdfe) Inference API ------------- For calling deployed models and chains. ![management-api](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/management-api.png?fit=max&auto=format&n=W3NbEem9OZkF5rdB&q=85&s=54a53f56d9b3798b3473734288242684) Management API -------------- For managing models, workspaces, and training jobs. [​](https://docs.baseten.co/reference/overview#cli-reference) CLI Reference ------------------------------------------------------------------------------ The CLI provides a command-line interface for managing deployments, running local inference, and configuring Truss models. * [Truss CLI reference](https://docs.baseten.co/reference/cli/truss/overview) : Commands for initializing, deploying, and managing models. * [Chains CLI reference](https://docs.baseten.co/reference/cli/chains/chains-cli) : Commands for orchestrating multi-model workflows. * [Training CLI reference](https://docs.baseten.co/reference/cli/training/training-cli) : Commands for managing training jobs. * * * [​](https://docs.baseten.co/reference/overview#sdk-reference) SDK Reference ------------------------------------------------------------------------------ The Python SDK provides an abstraction for deploying models, managing deployments, and interacting with models via code. * [Truss SDK reference](https://docs.baseten.co/reference/sdk/truss) : Deploy and interact with Truss models using Python. * [Chains SDK reference](https://docs.baseten.co/reference/sdk/chains) : Build and manage inference chains programmatically. * [Training SDK reference](https://docs.baseten.co/reference/sdk/training) : Deploy and interact with trained models using Python. Was this page helpful? YesNo [Truss configurationSet your model resources, dependencies, and more\ \ Next](https://docs.baseten.co/reference/truss-configuration) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Rate limits - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/rate-limits#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten enforces per-API-key rate limits on the management API. The limits protect shared infrastructure, such as the build queue, from bursty automation. [​](https://docs.baseten.co/reference/management-api/rate-limits#limits) Limits ---------------------------------------------------------------------------------- The default limit applies to every `/v1/*` endpoint. A few endpoints that touch shared build and deployment infrastructure have stricter limits. | Endpoint | Limit | | --- | --- | | `POST /v1/models/{model_id}/deployments/development/activate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/production/activate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/{deployment_id}/activate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/development/deactivate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/production/deactivate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/{deployment_id}/deactivate` | 20 requests/minute | | `POST /v1/models/{model_id}/deployments/development/retry` | 10 requests/minute | | `POST /v1/models/{model_id}/deployments/production/retry` | 10 requests/minute | | `POST /v1/models/{model_id}/deployments/{deployment_id}/retry` | 10 requests/minute | | `POST /v1/models/{model_id}/deployments/{deployment_id}/logs` | 30 requests/second | | All other `/v1/*` endpoints | 100 requests/second | Baseten tracks each endpoint separately. [​](https://docs.baseten.co/reference/management-api/rate-limits#rate-limited-responses) Rate-limited responses ------------------------------------------------------------------------------------------------------------------ A request over the limit returns `429 Too Many Requests`: { "error": "Rate limit exceeded. Please try again later.", "retry_after": 37 } `retry_after` is the number of seconds until the current rate-limit window resets. Wait at least that long before retrying. [​](https://docs.baseten.co/reference/management-api/rate-limits#retry-handling) Retry handling -------------------------------------------------------------------------------------------------- For CI pipelines or scripts that call the management API in a loop, handle `429` explicitly: import time import requests def post_with_retry(url, headers, max_attempts=5): for _ in range(max_attempts): response = requests.post(url, headers=headers) if response.status_code != 429: return response retry_after = response.json().get("retry_after", 1) time.sleep(retry_after) return response Back off on `retry_after` instead of retrying immediately. A tight retry loop wastes API calls; the server rejects every request until the window resets. [​](https://docs.baseten.co/reference/management-api/rate-limits#request-higher-limits) Request higher limits ---------------------------------------------------------------------------------------------------------------- If your workload needs sustained throughput above the default limits, [contact support](https://www.baseten.co/talk-to-us/increase-rate-limits/) to request per-endpoint increases for your organization. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/overview) [All models\ \ Next](https://docs.baseten.co/reference/management-api/models/gets-all-models) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Get a session - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/loops-api/sessions/get-a-session#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / loops / sessions / {session\_id} Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/loops/sessions/{session_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "session": { "id": "" } } #### Authorizations [​](https://docs.baseten.co/reference/loops-api/sessions/get-a-session#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/loops-api/sessions/get-a-session#parameter-session-id) session\_id string required #### Response 200 - application/json Response for `GET /v1/loops/sessions/`. [​](https://docs.baseten.co/reference/loops-api/sessions/get-a-session#response-session) session LoopsSessionV1 · object required The Loops session. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/loops-api/sessions/create-a-session) [Create a runCreates a Loops run with an associated sampler in the given session.\ \ Next](https://docs.baseten.co/reference/loops-api/runs/create-a-run) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/loops/sessions/{session_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "session": { "id": "" } } Assistant Responses are generated using AI and may contain mistakes. --- # Speech-to-text models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/speech-to-text#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/speech-to-text#transcription) Transcription ------------------------------------------------------------------------------------------------------- Qwen3-ASR --------- Deploy Qwen3-ASR with a prebuilt Truss config. Voxtral ------- Deploy Voxtral with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Agentic models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/agentic#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/agentic#llms) LLMs ------------------------------------------------------------------------------ DeepSeek V4 ----------- Deploy DeepSeek V4 with a prebuilt Truss config. GPT-OSS ------- Deploy GPT-OSS with a prebuilt Truss config. Nemotron 3 ---------- Deploy Nemotron 3 with a prebuilt Truss config. Qwen3.5 ------- Deploy Qwen3.5 with a prebuilt Truss config. Qwen3.6 ------- Deploy Qwen3.6 with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Long context models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/long-context#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/long-context#llms) LLMs ----------------------------------------------------------------------------------- DeepSeek V4 ----------- Deploy DeepSeek V4 with a prebuilt Truss config. Gemma 4 ------- Deploy Gemma 4 with a prebuilt Truss config. GLM-4.7 ------- Deploy GLM-4.7 with a prebuilt Truss config. GLM-5 ----- Deploy GLM-5 with a prebuilt Truss config. GPT-OSS ------- Deploy GPT-OSS with a prebuilt Truss config. Llama 3.1 --------- Deploy Llama 3.1 with a prebuilt Truss config. Llama 3.2 --------- Deploy Llama 3.2 with a prebuilt Truss config. Llama 3.3 --------- Deploy Llama 3.3 with a prebuilt Truss config. Llama 4 ------- Deploy Llama 4 with a prebuilt Truss config. MiniMax M2.5 ------------ Deploy MiniMax M2.5 with a prebuilt Truss config. Qwen3 ----- Deploy Qwen3 with a prebuilt Truss config. Qwen3.5 ------- Deploy Qwen3.5 with a prebuilt Truss config. Qwen3.6 ------- Deploy Qwen3.6 with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Reasoning models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/reasoning#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/reasoning#llms) LLMs -------------------------------------------------------------------------------- DeepSeek V4 ----------- Deploy DeepSeek V4 with a prebuilt Truss config. Gemma 4 ------- Deploy Gemma 4 with a prebuilt Truss config. GLM-4.7 ------- Deploy GLM-4.7 with a prebuilt Truss config. GLM-5 ----- Deploy GLM-5 with a prebuilt Truss config. GPT-OSS ------- Deploy GPT-OSS with a prebuilt Truss config. MiniMax M2.5 ------------ Deploy MiniMax M2.5 with a prebuilt Truss config. Nemotron 3 ---------- Deploy Nemotron 3 with a prebuilt Truss config. Qwen3 ----- Deploy Qwen3 with a prebuilt Truss config. Qwen3.5 ------- Deploy Qwen3.5 with a prebuilt Truss config. Qwen3.6 ------- Deploy Qwen3.6 with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Tool calling models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/tool-calling#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/tool-calling#llms) LLMs ----------------------------------------------------------------------------------- DeepSeek V4 ----------- Deploy DeepSeek V4 with a prebuilt Truss config. Gemma 4 ------- Deploy Gemma 4 with a prebuilt Truss config. GLM-4.7 ------- Deploy GLM-4.7 with a prebuilt Truss config. GLM-5 ----- Deploy GLM-5 with a prebuilt Truss config. GPT-OSS ------- Deploy GPT-OSS with a prebuilt Truss config. Llama 3.1 --------- Deploy Llama 3.1 with a prebuilt Truss config. Llama 3.2 --------- Deploy Llama 3.2 with a prebuilt Truss config. Llama 3.3 --------- Deploy Llama 3.3 with a prebuilt Truss config. Llama 4 ------- Deploy Llama 4 with a prebuilt Truss config. MiniMax M2.5 ------------ Deploy MiniMax M2.5 with a prebuilt Truss config. Nemotron 3 ---------- Deploy Nemotron 3 with a prebuilt Truss config. Qwen3 ----- Deploy Qwen3 with a prebuilt Truss config. Qwen3.5 ------- Deploy Qwen3.5 with a prebuilt Truss config. Qwen3.6 ------- Deploy Qwen3.6 with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Multimodal (image) models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/examples/models/capabilities/multimodal-image#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/examples/models/capabilities/multimodal-image#llms) LLMs --------------------------------------------------------------------------------------- Gemma 4 ------- Deploy Gemma 4 with a prebuilt Truss config. Llama 4 ------- Deploy Llama 4 with a prebuilt Truss config. Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Loops SDK - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/sdk/loops#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) The Loops Python SDK exposes three classes used in training scripts. `ServiceClient` provisions trainer and sampling servers on the Baseten control plane and manages the session that ties them together. `TrainingClient` runs forward passes, backward passes, and optimizer steps against a live trainer server. `SamplingClient` generates completions from the current weights, and the version pinning it carries means you always sample from exactly the checkpoint you trained. These three classes mirror the Tinker shapes; the methods you call are the same names. [​](https://docs.baseten.co/reference/sdk/loops#installation) Installation ----------------------------------------------------------------------------- The main client package is `baseten-loops`. The Tinker compatibility shim ships as the `[tinker]` extra, which pulls in the `baseten-loops-tinker` wheel and provides the `tinker` namespace. From a uv project (see the [quickstart](https://docs.baseten.co/loops/quickstart#install) for project setup), install both with the extra: uv add 'baseten-loops[tinker]' from baseten.loops import ServiceClient, TrainingClient, SamplingClient # Tinker-compatible namespace (provided by baseten-loops[tinker]) import tinker [​](https://docs.baseten.co/reference/sdk/loops#minimal-example) Minimal example ----------------------------------------------------------------------------------- One LoRA round trip: provision a trainer, run a forward and backward pass over a single masked prompt-and-answer pair, step the optimizer, publish the weights, and sample from them. from baseten.loops import ( ServiceClient, Datum, ModelInput, TensorData, AdamParams, SamplingParams, ) service_client = ServiceClient() training_client = service_client.create_lora_training_client( base_model="Qwen/Qwen3.5-2B", rank=16, ) # Tokenize one prompt/answer pair and mask the prompt positions from the loss. tokenizer = training_client.get_tokenizer() prompt = tokenizer.encode("What is the capital of France?\nAnswer:", add_special_tokens=False) answer = tokenizer.encode(" Paris", add_special_tokens=False) tokens = prompt + answer targets = [-100] * len(prompt) + answer datum = Datum( model_input=ModelInput.from_ints(tokens), loss_fn_inputs={"target_tokens": TensorData(data=targets, dtype="int64", shape=[len(targets)])}, ) training_client.forward_backward(data=[datum]).result(timeout=600.0) training_client.optim_step(AdamParams(learning_rate=4e-5)).result(timeout=600.0) sampling_client = training_client.save_weights_and_get_sampling_client(name="step-1").result(timeout=600.0) result = sampling_client.sample( prompt=ModelInput.from_ints(prompt), num_samples=1, sampling_params=SamplingParams(max_tokens=16), ) print(result.sequences[0].tokens) Each long-running call is submit-then-`.result()`: the submit validates and returns immediately, and `.result()` long-polls until the operation finishes. Provisioning the trainer can take several minutes on a fresh base model, so the first call blocks the longest. The `sampling_client` returned by `save_weights_and_get_sampling_client` is pinned to the version you just trained, so the sample reflects this step’s weights. For the end-to-end walkthrough with expected output, see the [Loops quickstart](https://docs.baseten.co/loops/quickstart) . [​](https://docs.baseten.co/reference/sdk/loops#serviceclient-provision) ServiceClient: provision ---------------------------------------------------------------------------------------------------- `ServiceClient` is the entry point for every session. It calls the Baseten control plane to create a `TrainerSession`, then provisions trainer and sampling servers within that session on demand. [​](https://docs.baseten.co/reference/sdk/loops#param-service-client-training-project-id-none-api-key-none-base-url-none-reuse-from-session-id-none) ServiceClient(training\_project\_id=None, \*, api\_key=None, base\_url=None, reuse\_from\_session\_id=None) ServiceClient Construct a `ServiceClient` and create a new `TrainerSession` on the Baseten control plane. Omit `training_project_id` to use the default project for the org, or pass one to target a specific training project. `api_key` defaults to the `BASETEN_API_KEY` environment variable.Pass `reuse_from_session_id` to reuse a prior session’s trainer and sampler for `create_lora_training_client` and `create_sampling_client` calls instead of provisioning fresh. The named session must belong to the same team. `ServiceClient` reads the `LOOPS_REUSE_FROM_SESSION_ID` environment variable when no kwarg is passed; the kwarg wins when both are set. Reuse is best-effort: if the prior deployment is stopped, failed, or unhealthy, a fresh one is provisioned and the call still succeeds. See [Reusing infrastructure across sessions](https://docs.baseten.co/loops/concepts#reusing-infrastructure-across-sessions) . [​](https://docs.baseten.co/reference/sdk/loops#param-service-client-local-trainer-url-sampler-url) ServiceClient.local(\*, trainer\_url, sampler\_url) ServiceClient Bind to already-running local trainer and sampler processes without contacting the control plane. Pass `trainer_url` and `sampler_url` as the base URLs of local server processes. Useful for end-to-end testing. [​](https://docs.baseten.co/reference/sdk/loops#param-create-lora-training-client-base-model-rank-32-seed-none-timeout-600-0-ready-timeout-3600-0-wandb-config-none) create\_lora\_training\_client(base\_model, rank=32, seed=None, timeout=600.0, ready\_timeout=3600.0, wandb\_config=None) TrainingClient Provision a `TrainerServer` for the given Hugging Face `base_model` and return a connected `TrainingClient`. The control plane also provisions a paired sampling server in the same call; `save_weights_and_get_sampling_client` uses that paired URL to gate on version readiness. Pass a `WandbConfig` instance to stream training metrics to a Weights & Biases run. [​](https://docs.baseten.co/reference/sdk/loops#param-create-sampling-client-base-model-timeout-300-0-ready-timeout-3600-0-model-path-none) create\_sampling\_client(base\_model, timeout=300.0, ready\_timeout=3600.0, model\_path=None) SamplingClient Provision a standalone `SamplingServer` for `base_model` and return a connected `SamplingClient`. Use this when you want to sample from a base model independently of a training run. The `model_path` argument is reserved and not yet implemented; passing it raises `NotImplementedError`. To sample from a specific checkpoint, use `TrainingClient.create_sampling_client(model_path=...)` on a live run instead. [​](https://docs.baseten.co/reference/sdk/loops#param-get-server-capabilities) get\_server\_capabilities() ServerCapabilities Return the control plane’s view of supported base models and the GPU classes it can provision them on. Useful for confirming a base model is available before calling `create_lora_training_client`. [​](https://docs.baseten.co/reference/sdk/loops#param-list-checkpoints-run-id) list\_checkpoints(run\_id) list\[Checkpoint\] List checkpoints saved by the run identified by `run_id`. Calls the Baseten API, not the trainer server directly. [​](https://docs.baseten.co/reference/sdk/loops#param-get-checkpoint-archive-url-checkpoint-id-page-size-1000-page-token-0) get\_checkpoint\_archive\_url(checkpoint\_id, page\_size=1000, page\_token=0) CheckpointFilesResponse Return presigned URLs for every file in the specified checkpoint folder. Checkpoint IDs are globally unique, so no run scoping is required. The Loops stack writes checkpoints as unzipped directories rather than archives, so this returns a file list instead of a single archive URL. [​](https://docs.baseten.co/reference/sdk/loops#param-session-id) session\_id str Property. The session ID assigned by the control plane. Available after construction. [​](https://docs.baseten.co/reference/sdk/loops#trainingclient-train) TrainingClient: train ---------------------------------------------------------------------------------------------- `TrainingClient` talks directly to a `dp_worker` instance. Long-running operations use a submit-and-retrieve protocol: the submit fires immediately on the calling thread (so validation errors surface at call time) and `.result()` long-polls the server until the operation finishes. You can submit multiple operations before awaiting any of them. [​](https://docs.baseten.co/reference/sdk/loops#param-forward-backward-data-loss-fn-cross-entropy-loss-fn-config-none) forward\_backward(data, loss\_fn="cross\_entropy", loss\_fn\_config=None) ForwardBackwardFuture Run a forward and backward pass over `data` (a list of `Datum` objects) using the specified loss function. Returns a `ForwardBackwardFuture`; call `.result()` to block until the pass completes and retrieve the loss. [​](https://docs.baseten.co/reference/sdk/loops#param-forward-data-loss-fn-cross-entropy-loss-fn-config-none) forward(data, loss\_fn="cross\_entropy", loss\_fn\_config=None) ForwardBackwardFuture Run a forward pass without gradient computation. Same inputs and output shape as `forward_backward`, but the gradient buffer is left untouched, so it is safe to interleave with gradient accumulation steps. [​](https://docs.baseten.co/reference/sdk/loops#param-optim-step-adam-params) optim\_step(adam\_params) OperationFuture\[OptimStepResponse\] Apply the accumulated gradients using the Adam optimizer configured by `adam_params`. Call this after one or more `forward_backward` calls. [​](https://docs.baseten.co/reference/sdk/loops#param-save-state-name-ttl-seconds-none) save\_state(name, ttl\_seconds=None) OperationFuture\[SaveWeightsResponse\] Persist a local training checkpoint under `name`. When a weight sync URI is configured server-side, `save_state` also publishes the LoRA adapter so a polling sampler can hot-swap to the new weights. [​](https://docs.baseten.co/reference/sdk/loops#param-save-weights-for-sampler-name-ttl-seconds-none) save\_weights\_for\_sampler(name, ttl\_seconds=None) OperationFuture\[SaveWeightsResponse\] Publish the LoRA adapter to the paired sampling server under `name` without returning a snapshot-pinned `SamplingClient`. Use this when you don’t need the version gate that `save_weights_and_get_sampling_client` provides. [​](https://docs.baseten.co/reference/sdk/loops#param-save-weights-and-get-sampling-client-name) save\_weights\_and\_get\_sampling\_client(name) \_ComposedFuture\[SamplingClient\] Publish the LoRA adapter to the paired sampling server under `name` and return a future that resolves to a `SamplingClient` pinned to the newly published version. Calling `.result()` runs two stages: the trainer publishes weights, then the SDK polls the sampler until at least one replica reports the new version loaded. The sampler-wait phase has a fixed 600-second ceiling independent of the `timeout=` you pass to `.result()`; if no replica reports the new version by then, the call raises `RuntimeError`. The returned `SamplingClient` carries `X-Min-Policy-Version` on every subsequent `sample()` call, so requests only land on replicas that have the right weights. [​](https://docs.baseten.co/reference/sdk/loops#param-load-state-path) load\_state(path) OperationFuture\[LoadWeightsResponse\] Load weights from a `bt://loops:/weights/` URI into this trainer. Use to resume training from a checkpoint. [​](https://docs.baseten.co/reference/sdk/loops#param-load-state-with-optimizer-path) load\_state\_with\_optimizer(path) OperationFuture\[LoadWeightsResponse\] Same as `load_state` but also restores Adam moments. Use when you want bit-exact resumption. [​](https://docs.baseten.co/reference/sdk/loops#param-list-checkpoints) list\_checkpoints() list\[Checkpoint\] List checkpoints for the run bound to this client. Requires that this client was constructed via `ServiceClient.create_lora_training_client` (which populates the necessary session and run IDs automatically). [​](https://docs.baseten.co/reference/sdk/loops#param-get-checkpoint-archive-url-checkpoint-id-page-size-1000-page-token-0-1) get\_checkpoint\_archive\_url(checkpoint\_id, page\_size=1000, page\_token=0) CheckpointFilesResponse Return presigned URLs for every file in a checkpoint folder. Same semantics as `ServiceClient.get_checkpoint_archive_url`. [​](https://docs.baseten.co/reference/sdk/loops#param-create-sampling-client-model-path) create\_sampling\_client(model\_path) SamplingClient Return a `SamplingClient` bound to the paired sampler, loading the weights at `model_path` (a `bt://loops:/sampler_weights/` URI). Distinct from `ServiceClient.create_sampling_client`, which provisions a fresh sampler. [​](https://docs.baseten.co/reference/sdk/loops#param-get-tokenizer) get\_tokenizer() PreTrainedTokenizer Return the Hugging Face `PreTrainedTokenizer` for the base model. Cached after the first load. [​](https://docs.baseten.co/reference/sdk/loops#param-get-info) get\_info() GetInfoResponse Return the model configuration for this training session (base model name, LoRA rank, and max sequence length) without a server round-trip. [​](https://docs.baseten.co/reference/sdk/loops#param-run-id) run\_id str | None Property. The run ID this client is bound to. Use this when filtering checkpoints or making HTTP API calls against the same run. [​](https://docs.baseten.co/reference/sdk/loops#param-policy-version) policy\_version int Property. The current policy version the trainer has published. Incremented on each `save_weights_and_get_sampling_client` (or `save_weights_for_sampler`) call. [​](https://docs.baseten.co/reference/sdk/loops#samplingclient-sample) SamplingClient: sample ------------------------------------------------------------------------------------------------ `SamplingClient` generates text completions from the model the sampler currently has loaded. There are two creation paths with different version semantics: `ServiceClient.create_sampling_client` returns an auto-updating client that follows whatever weights the sampler currently holds, while `TrainingClient.save_weights_and_get_sampling_client` returns a snapshot client pinned to the trained version. Both clients expose the same `sample` method. [​](https://docs.baseten.co/reference/sdk/loops#param-sample-prompt-num-samples-1-sampling-params-none-include-prompt-logprobs-false-topk-prompt-logprobs-0) sample(prompt, num\_samples=1, sampling\_params=None, include\_prompt\_logprobs=False, topk\_prompt\_logprobs=0) SampleResult Generate `num_samples` completions from `prompt` (a `ModelInput`). Pass a `SamplingParams` instance to control temperature, top-p, top-k, max tokens, seed, and stop sequences; omit it to use defaults. Set `include_prompt_logprobs=True` to get per-token log-probabilities for the input tokens alongside the output, and set `topk_prompt_logprobs` above `0` to also return the top-k alternatives at each prompt position. The sampler resolves which adapter or base model to serve from the version headers the client carries, so there is no per-call model override. [​](https://docs.baseten.co/reference/sdk/loops#param-compute-logprobs-prompt) compute\_logprobs(prompt) list\[float | None\] Return the per-token log-probabilities for `prompt` without generating any new tokens. Index 0 is always `None` because the first token has no preceding context to score against. Other positions may also be `None` if the sampler can’t compute a log-probability for that token. [​](https://docs.baseten.co/reference/sdk/loops#param-discover-base-model-name) discover\_base\_model\_name() str Return the base model ID from the sampler’s `/v1/models` list, specifically the entry with no parent. Retries with backoff while the sampler is still deploying. [​](https://docs.baseten.co/reference/sdk/loops#param-discover-adapter-name) discover\_adapter\_name() str | None Return the currently registered LoRA adapter ID (the first `/v1/models` entry with a non-null parent), or `None` if no adapter is loaded. [​](https://docs.baseten.co/reference/sdk/loops#param-get-base-model) get\_base\_model() str Return the base model ID this sampling client was created with, without contacting the server. [​](https://docs.baseten.co/reference/sdk/loops#param-get-tokenizer-1) get\_tokenizer() PreTrainedTokenizer Return the Hugging Face `PreTrainedTokenizer` for the base model this client was created with. [​](https://docs.baseten.co/reference/sdk/loops#types) Types --------------------------------------------------------------- ### [​](https://docs.baseten.co/reference/sdk/loops#training-inputs) Training inputs [​](https://docs.baseten.co/reference/sdk/loops#param-datum) Datum A single training example: a `ModelInput` paired with a dict of `TensorData` loss function inputs. [​](https://docs.baseten.co/reference/sdk/loops#param-model-input) ModelInput A tokenized prompt, represented as a list of `ModelInputChunk` objects. Construct with `ModelInput.from_ints(token_ids)` for the common case. [​](https://docs.baseten.co/reference/sdk/loops#param-model-input-chunk) ModelInputChunk A discriminated union of `EncodedTextChunk` (a list of token IDs) and `ImageChunk` (a base64-encoded image with an expected token count). [​](https://docs.baseten.co/reference/sdk/loops#param-tensor-data) TensorData A serializable tensor with a flat data list, a dtype string, and a shape. Convert to and from `torch.Tensor` with `TensorData.to_torch()` and `TensorData.from_torch(tensor)`. ### [​](https://docs.baseten.co/reference/sdk/loops#configuration) Configuration [​](https://docs.baseten.co/reference/sdk/loops#param-sampling-params) SamplingParams Controls for text generation: `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`, and `stop`. [​](https://docs.baseten.co/reference/sdk/loops#param-adam-params) AdamParams Optimizer hyperparameters: `learning_rate`, `beta1`, `beta2`, `eps`, `weight_decay`, and `grad_clip_norm`. [​](https://docs.baseten.co/reference/sdk/loops#param-wandb-config) WandbConfig Optional Weights & Biases settings (`project` and an optional run `name`) passed to `create_lora_training_client` to stream training metrics. ### [​](https://docs.baseten.co/reference/sdk/loops#results-and-handles) Results and handles [​](https://docs.baseten.co/reference/sdk/loops#param-sample-result) SampleResult The full response from `sample()`: a list of `SampledSequence` objects in `sequences`, the `policy_version` the sampler replica was running, and `prompt_logprobs` / `topk_prompt_logprobs` populated when the matching `sample()` flags are set. [​](https://docs.baseten.co/reference/sdk/loops#param-sampled-sequence) SampledSequence A single generated sequence: a list of output token IDs, optional per-token log-probabilities, and a stop reason. [​](https://docs.baseten.co/reference/sdk/loops#param-checkpoint) Checkpoint Metadata for a saved checkpoint, populated by `list_checkpoints()`. [​](https://docs.baseten.co/reference/sdk/loops#param-checkpoint-files-response) CheckpointFilesResponse A paginated list of presigned file URLs for a checkpoint, populated by `get_checkpoint_archive_url()`. [​](https://docs.baseten.co/reference/sdk/loops#param-checkpoint-file) CheckpointFile One entry in a `CheckpointFilesResponse.presigned_urls` list: a presigned URL plus `relative_file_name`, `node_rank`, `size_bytes`, and `last_modified` metadata. [​](https://docs.baseten.co/reference/sdk/loops#param-server-capabilities-supported-model) ServerCapabilities, SupportedModel Returned by `ServiceClient.get_server_capabilities()`; describe which base models the control plane can provision and on which GPU classes. [​](https://docs.baseten.co/reference/sdk/loops#param-operation-futuret) OperationFuture\[T\] A handle to a long-running training operation. Call `.result()` or `.result(timeout=seconds)` to block until the operation completes and return the result. The `forward` and `forward_backward` methods return a `ForwardBackwardFuture` subclass, and `save_weights_and_get_sampling_client` returns a composed future; both expose the same `.result()` contract. [​](https://docs.baseten.co/reference/sdk/loops#param-forward-backward-output-optim-step-response-save-weights-response-save-weights-for-sampler-response-load-weights-response-init-trainer-server-response-sample-response) ForwardBackwardOutput, OptimStepResponse, SaveWeightsResponse, SaveWeightsForSamplerResponse, LoadWeightsResponse, InitTrainerServerResponse, SampleResponse Response payloads returned by the matching `TrainingClient` and `SamplingClient` methods. [​](https://docs.baseten.co/reference/sdk/loops#errors) Errors ----------------------------------------------------------------- [​](https://docs.baseten.co/reference/sdk/loops#param-remote-op-error) RemoteOpError The server reports that an async operation failed. The `error_class` attribute carries the server-side exception class name (for example, `"ValueError"` or `"DispatcherError"`), which is useful for routing in caller code. [​](https://docs.baseten.co/reference/sdk/loops#param-unknown-request-error) UnknownRequestError The server returned 404 for an operation ID. This can mean the server has no record of the operation (after a pod restart, for example) or that the result was TTL-evicted. Resubmit the operation if the work is still needed; the server’s idempotency-key deduplication prevents double-execution. [​](https://docs.baseten.co/reference/sdk/loops#param-server-shutdown-error) ServerShutdownError The server is shutting down (503 response). Retry the request against a different replica. [​](https://docs.baseten.co/reference/sdk/loops#tinker-compatibility-shim) Tinker compatibility shim ------------------------------------------------------------------------------------------------------- Install with the `[tinker]` extra (`uv add 'baseten-loops[tinker]'`) and import `tinker`. The shim wheel maps Tinker’s client interface onto the Loops SDK, so existing training scripts that import from `tinker` run without modification. The underlying classes (`ServiceClient`, `TrainingClient`, `SamplingClient`) and every method on them are the same; only the import path changes. For the full list of mapped names and any behavioral differences, see the [Tinker compatibility guide](https://docs.baseten.co/loops/tinker-compatibility) . Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/sdk/training) [Truss Push GitHub ActionDeploy and validate a Truss model or chain on Baseten from GitHub Actions.\ \ Next](https://docs.baseten.co/reference/ci/github-action) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Chains SDK Reference - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/sdk/chains#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/reference/sdk/chains#chainlet-classes) Chainlet classes ====================================================================================== APIs for creating user-defined Chainlets. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-chainletbase) _class_ `truss_chains.ChainletBase` Base class for all chainlets. Inheriting from this class adds validations to make sure subclasses adhere to the chainlet pattern and facilitates remote chainlet deployment. Refer to [the docs](https://docs.baseten.co/development/chain/getting-started) and this [example chainlet](https://github.com/basetenlabs/truss/blob/main/truss-chains/truss_chains/reference_code/reference_chainlet.py) for more guidance on how to create subclasses. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-modelbase) _class_ `truss_chains.ModelBase` Base class for all standalone models. Inheriting from this class adds validations to make sure subclasses adhere to the truss model pattern. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-enginebuilderllmchainlet) _class_ `truss_chains.EngineBuilderLLMChainlet` #### [​](https://docs.baseten.co/reference/sdk/chains#method-final-async-run_remote-llm_input) _method final async_ run\_remote(llm\_input) **Parameters:** | Name | Type | Description | | --- | --- | --- | | `llm_input` | _EngineBuilderLLMInput_ | OpenAI compatible request. | * **Returns:** _AsyncIterator_\[str\] ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-depends) _function_ `truss_chains.depends` Sets a “symbolic marker” to indicate to the framework that a chainlet is a dependency of another chainlet. The return value of `depends` is intended to be used as a default argument in a chainlet’s `__init__`\-method. When deploying a chain remotely, a corresponding stub to the remote is injected in its place. In [`run_local`](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-run_local) mode an instance of a local chainlet is injected. Refer to [the docs](https://docs.baseten.co/development/chain/getting-started) and this [example chainlet](https://github.com/basetenlabs/truss/blob/main/truss-chains/truss_chains/reference_code/reference_chainlet.py) for more guidance on how make one chainlet depend on another chainlet. Despite the type annotation, this does _not_ immediately provide a chainlet instance. Only when deploying remotely or using `run_local` a chainlet instance is provided. **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `chainlet_cls` | _Type\[[ChainletBase](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-chainletbase)
\]_ | | The chainlet class of the dependency. | | `retries` | _int_ | `1` | The number of times to retry the remote chainlet in case of failures (for example, due to transient network issues). For streaming, retries are only made if the request fails before streaming any results back. Failures mid-stream not retried. | | `timeout_sec` | _float_ | `600.0` | Timeout for the HTTP request to this chainlet. | | `use_binary` | _bool_ | `False` | Whether to send data in binary format. This can give a parsing speedup and message size reduction (~25%) for numpy arrays. Use `NumpyArrayField` as a field type on pydantic models for integration and set this option to `True`. For simple text data, there is no significant benefit. | | `concurrency_limit` | _int_ | `300` | The maximum number of concurrent requests to send to the remote chainlet. Excessive requests will be queued and a warning will be shown. Try to design your algorithm in a way that spreads requests evenly over time so that this the default value can be used. | * **Returns:** A “symbolic marker” to be used as a default argument in a chainlet’s initializer. ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-depends_context) _function_ `truss_chains.depends_context` Sets a “symbolic marker” for injecting a context object at runtime. Refer to [the docs](https://docs.baseten.co/development/chain/getting-started) and this [example chainlet](https://github.com/basetenlabs/truss/blob/main/truss-chains/truss_chains/reference_code/reference_chainlet.py) for more guidance on the `__init__`\-signature of chainlets. Despite the type annotation, this does _not_ immediately provide a context instance. Only when deploying remotely or using `run_local` a context instance is provided. * **Returns:** A “symbolic marker” to be used as a default argument in a chainlet’s initializer. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deploymentcontext) _class_ `truss_chains.DeploymentContext` Bases: `pydantic.BaseModel` Bundles config values and resources needed to instantiate Chainlets. The context can optionally be added as a trailing argument in a Chainlet’s `__init__` method and then used to set up the chainlet (for example, using a secret as an access token for downloading model weights). **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `chainlet_to_service` | _Mapping\[str,[DeployedServiceDescriptor](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployedservicedescriptor)
\]_ | | A mapping from chainlet names to service descriptors. This is used to create RPC sessions to dependency chainlets. It contains only the chainlet services that are dependencies of the current chainlet. | | `secrets` | _Mapping\[str,str\]_ | | A mapping from secret names to secret values. It contains only the secrets that are listed in `remote_config.assets.secret_keys` of the current chainlet. | | `data_dir` | _Path\|None_ | `None` | The directory where the chainlet can store and access data, for example, for downloading model weights. | | `environment` | _[Environment](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-environment)
\|None_ | `None` | The environment that the chainlet is deployed in. None if the chainlet is not associated with an environment. | #### [​](https://docs.baseten.co/reference/sdk/chains#method-get_baseten_api_key) _method_ get\_baseten\_api\_key() * **Returns:** str #### [​](https://docs.baseten.co/reference/sdk/chains#method-get_service_descriptor-chainlet_name) _method_ get\_service\_descriptor(chainlet\_name) **Parameters:** | Name | Type | Description | | --- | --- | --- | | `chainlet_name` | _str_ | The name of the chainlet. | * **Returns:** [_DeployedServiceDescriptor_](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployedservicedescriptor) ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-environment) _class_ `truss_chains.Environment` Bases: `pydantic.BaseModel` The environment the chainlet is deployed in. * **Parameters:** **name** (_str_) – The name of the environment. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-chainletoptions) _class_ `truss_chains.ChainletOptions` Bases: `pydantic.BaseModel` **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `enable_b10_tracing` | _bool_ | `False` | enables baseten-internal trace data collection. This helps baseten engineers better analyze chain performance in case of issues. It is independent of a potentially user-configured tracing instrumentation. Turning this on, could add performance overhead. | | `enable_debug_logs` | _bool_ | `False` | Sets log level to debug in deployed server. | | `env_variables` | _Mapping\[str,str\]_ | `{}` | static environment variables available to the deployed chainlet. | | `health_checks` | _HealthChecks_ | `truss.base.truss_config.HealthChecks()` | Configures health checks for the chainlet. See [guide](https://docs.baseten.co/truss/guides/custom-health-checks#chains)
. | | `metadata` | _JsonValue\|None_ | `None` | Arbitrary JSON object to describe chainlet. | | `streaming_read_timeout` | _int_ | `60` | Amount of time (in seconds) between each streamed chunk before a timeout is triggered. | | `transport` | _Union\[HTTPOptions\|WebsocketOptions\|GRPCOptions\]‘_ | `None` | Allows to customize certain transport protocols, for example, websocket pings. | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-rpcoptions) _class_ `truss_chains.RPCOptions` Bases: `pydantic.BaseModel` Options to customize RPCs to dependency chainlets. **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `retries` | _int_ | `1` | The number of times to retry the remote chainlet in case of failures (for example, due to transient network issues). For streaming, retries are only made if the request fails before streaming any results back. Failures mid-stream not retried. | | `timeout_sec` | _float_ | `600.0` | Timeout for the HTTP request to this chainlet. | | `use_binary` | _bool_ | `False` | Whether to send data in binary format. This can give a parsing speedup and message size reduction (~25%) for numpy arrays. Use `NumpyArrayField` as a field type on pydantic models for integration and set this option to `True`. For simple text data, there is no significant benefit. | | `concurrency_limit` | _int_ | `300` | The maximum number of concurrent requests to send to the remote chainlet. Excessive requests will be queued and a warning will be shown. Try to design your algorithm in a way that spreads requests evenly over time so that this the default value can be used. | ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-mark_entrypoint) _function_ `truss_chains.mark_entrypoint` Decorator to mark a chainlet as the entrypoint of a chain. This decorator can be applied to _one_ chainlet in a source file and then the CLI push command simplifies: only the file, not the class within, must be specified. Optionally a display name for the Chain (not the Chainlet) can be set (effectively giving a custom default value for the `name` arg of the CLI push command). Example usage: import truss_chains as chains @chains.mark_entrypoint class MyChainlet(ChainletBase): ... # OR with custom Chain name. @chains.mark_entrypoint("My Chain Name") class MyChainlet(ChainletBase): ... [​](https://docs.baseten.co/reference/sdk/chains#remote-configuration) Remote Configuration ============================================================================================== These data structures specify for each chainlet how it gets deployed remotely, for example, dependencies and compute resources. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-remoteconfig) _class_ `truss_chains.RemoteConfig` Bases: `pydantic.BaseModel` Bundles config values needed to deploy a chainlet remotely. This is specified as a class variable for each chainlet class, for example, : import truss_chains as chains class MyChainlet(chains.ChainletBase): remote_config = chains.RemoteConfig( docker_image=chains.DockerImage( pip_requirements=["torch==2.0.1", ...] ), compute=chains.Compute(cpu_count=2, gpu="A10G", ...), assets=chains.Assets(secret_keys=["hf_access_token"], ...), ) **Parameters:** | Name | Type | Default | | --- | --- | --- | | `docker_image` | _[DockerImage](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-dockerimage)
_ | `truss_chains.DockerImage()` | | `compute` | _[Compute](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-compute)
_ | `truss_chains.Compute()` | | `assets` | _[Assets](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-assets)
_ | `truss_chains.Assets()` | | `name` | _str\|None_ | `None` | | `options` | _[ChainletOptions](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-chainletoptions)
_ | `truss_chains.ChainletOptions()` | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-dockerimage) _class_ `truss_chains.DockerImage` Bases: `pydantic.BaseModel` Configures the docker image in which a remote chainlet is deployed. Any paths are relative to the source file where `DockerImage` is defined and must be created with the helper function \[`make_abs_path_here`\] (#function-truss\_chains-make\_abs\_path\_here). This allows you for example organize chainlets in different (potentially nested) modules and keep their requirement files right next their python source files. **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `base_image` | _[BasetenImage](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-basetenimage)
\|[CustomImage](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-customimage)
_ | `truss_chains.BasetenImage()` | The base image used by the chainlet. Other dependencies and assets are included as additional layers on top of that image. You can choose a Baseten default image for a supported python version (for example, `BasetenImage.PY311`), this will also include GPU drivers if needed, or provide a custom image (for example, `CustomImage(image=”python:3.11-slim”)`). | | `pip_requirements_file` | _AbsPath\|None_ | `None` | **Deprecated.** Use `requirements_file` instead. Path to a file containing pip requirements. The file content is naively concatenated with `pip_requirements`. | | `pip_requirements` | _list\[str\]_ | `[]` | A list of pip requirements to install. Only supported with pip-style requirements files. Cannot be used with `pyproject.toml` or `uv.lock` requirements files. | | `apt_requirements` | _list\[str\]_ | `[]` | A list of apt requirements to install. | | `requirements_file` | _AbsPath\|None_ | `None` | Path to a requirements file. Supports `requirements.txt` (pip format), `pyproject.toml`, and `uv.lock`. The file type is auto-detected from the filename. For pip-style files, the content is concatenated with `pip_requirements`. For `pyproject.toml` and `uv.lock`, the file is used as-is for installing dependencies. | | `data_dir` | _AbsPath\|None_ | `None` | Data from this directory is copied into the docker image and accessible to the remote chainlet at runtime. | | `external_package_dirs` | _list\[AbsPath\]\|None_ | `None` | A list of directories containing additional python packages outside the chain’s workspace dir, for example, a shared library. This code is copied into the docker image and importable at runtime. | | `truss_server_version_override` | _str\|None_ | `None` | By default, deployed Chainlets use the truss server implementation corresponding to the truss version of the user’s CLI. To use a specific version, for example, pinning it for exact reproducibility, the version can be overridden here. Valid versions correspond to truss releases on PyPi: [https://pypi.org/project/truss/#history](https://pypi.org/project/truss/#history)
, for example, “0.9.80”. | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-basetenimage) _class_ `truss_chains.BasetenImage` Bases: `Enum` Default images, curated by baseten, for different python versions. If a Chainlet uses GPUs, drivers will be included in the image. | Enum Member | Value | | --- | --- | | `PY39` | _py39_ | | `PY310` | _py310_ | | `PY311` | _py311_ | | `PY312` | _py312_ | | `PY313` | _py313_ | | `PY314` | _py314_ | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-customimage) _class_ `truss_chains.CustomImage` Bases: `pydantic.BaseModel` Configures the usage of a custom image hosted on dockerhub. **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `image` | _str_ | | Reference to image on dockerhub. | | `python_executable_path` | _str\|None_ | `None` | Absolute path to python executable (if default `python` is ambiguous). | | `docker_auth` | _DockerAuthSettings\|None_ | `None` | See [corresponding truss config](https://docs.baseten.co/development/model/base-images#example%3A-docker-hub-authentication)
. | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-compute) _class_ `truss_chains.Compute` Specifies which compute resources a chainlet has in the _remote_ deployment. Not all combinations can be exactly satisfied by available hardware, in some cases more powerful machine types are chosen to make sure requirements are met or over-provisioned. Refer to the [baseten instance reference](https://docs.baseten.co/deployment/resources) . **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `cpu_count` | _int_ | `1` | Minimum number of CPUs to allocate. | | `memory` | _str_ | `'2Gi'` | Minimum memory to allocate, for example, “2Gi” (2 gibibytes). | | `gpu` | _str\|Accelerator\|None_ | `None` | GPU accelerator type, for example, “A10G”, “A100”, refer to the [truss config](https://docs.baseten.co/deployment/resources)
for more choices. | | `gpu_count` | _int_ | `1` | Number of GPUs to allocate. | | `predict_concurrency` | _int\|Literal\[‘cpu\_count’\]_ | `1` | Number of concurrent requests a single replica of a deployed chainlet handles. | Concurrency concepts are explained in the [autoscaling guide](https://docs.baseten.co/deployment/autoscaling/overview#scaling-triggers) . It is important to understand the difference between predict\_concurrency and the concurrency target (used for autoscaling, that is, adding or removing replicas). Furthermore, the `predict_concurrency` of a single instance is implemented in two ways: * Via python’s `asyncio`, if `run_remote` is an async def. This requires that `run_remote` yields to the event loop. * With a threadpool if it’s a synchronous function. This requires that the threads don’t have significant CPU load (due to the GIL). ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-assets) _class_ `truss_chains.Assets` Specifies which assets a chainlet can access in the remote deployment. For example, model weight caching can be used like this: import truss_chains as chains from truss.base import truss_config mistral_cache = truss_config.ModelRepo( repo_id="mistralai/Mistral-7B-Instruct-v0.2", allow_patterns=["*.json", "*.safetensors", ".model"] ) chains.Assets(cached=[mistral_cache], ...) **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `cached` | _Iterable\[ModelRepo\]_ | `()` | One or more `truss_config.ModelRepo` objects. | | `secret_keys` | _Iterable\[str\]_ | `()` | Names of secrets stored on baseten, that the chainlet should have access to. You can manage secrets on baseten [here](https://app.baseten.co/settings/secrets)
. | | `external_data` | _Iterable\[ExternalDataItem\]_ | `()` | Data to be downloaded from public URLs and made available in the deployment (via `context.data_dir`). | [​](https://docs.baseten.co/reference/sdk/chains#core) Core ============================================================== General framework and helper functions. ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-push) _function_ `truss_chains.push` Deploys a chain remotely (with all dependent chainlets). **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `entrypoint` | _Type\[ChainletT\]_ | | The chainlet class that serves as the entrypoint to the chain. | | `chain_name` | _str_ | | The name of the chain. | | `publish` | _bool_ | `True` | Whether to publish the chain as a published deployment (it is a draft deployment otherwise) | | `promote` | _bool_ | `True` | Whether to promote the chain to be the production deployment (this implies publishing as well). | | `only_generate_trusses` | _bool_ | `False` | Used for debugging purposes. If set to True, only the underlying truss models for the chainlets are generated in `/tmp/.chains_generated`. | | `remote` | _str_ | `'baseten'` | name of a remote config in .trussrc. If not provided, it will be inquired. | | `environment` | _str\|None_ | `None` | The name of an environment to promote deployment into. | | `progress_bar` | _Type\[progress.Progress\]\|None_ | `None` | Optional rich.progress.Progress if output is desired. | | `include_git_info` | _bool_ | `False` | Whether to attach git versioning info (sha, branch, tag) to deployments made from within a git repo. If set to True in .trussrc, it will always be attached. | * **Returns:** [_ChainService_](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployment-deployment_client-chainservice) : A chain service handle to the deployed chain. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployment-deployment_client-chainservice) _class_ `truss_chains.deployment.deployment_client.ChainService` Handle for a deployed chain. A `ChainService` is created and returned when using `push`. It bundles the individual services for each chainlet in the chain, and provides utilities to query their status, invoke the entrypoint etc. #### [​](https://docs.baseten.co/reference/sdk/chains#method-get_info) _method_ get\_info() Queries the statuses of all chainlets in the chain. * **Returns:** List of `DeployedChainlet`, `(name, is_entrypoint, status, logs_url)` for each chainlet. #### [​](https://docs.baseten.co/reference/sdk/chains#property-name--str) _property_ name _: str_ #### [​](https://docs.baseten.co/reference/sdk/chains#method-run_remote-json) _method_ run\_remote(json) Invokes the entrypoint with JSON data. **Parameters:** | Name | Type | Description | | --- | --- | --- | | `json` | _JSON dict_ | Input data to the entrypoint | * **Returns:** The JSON response. #### [​](https://docs.baseten.co/reference/sdk/chains#property-run_remote_url--str) _property_ run\_remote\_url _: str_ URL to invoke the entrypoint. #### [​](https://docs.baseten.co/reference/sdk/chains#property-status_page_url--str) _property_ status\_page\_url _: str_ Link to status page on Baseten. ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-make_abs_path_here) _function_ `truss_chains.make_abs_path_here` Helper to specify file paths relative to the _immediately calling_ module. For example, in you have a project structure like this: root/ chain.py common_requirements.text sub_package/ chainlet.py chainlet_requirements.txt You can now in `root/sub_package/chainlet.py` point to the requirements file like this: shared = make_abs_path_here("../common_requirements.text") specific = make_abs_path_here("chainlet_requirements.text") This helper uses the directory of the immediately calling module as an absolute reference point for resolving the file location. Therefore, you MUST NOT wrap the instantiation of `make_abs_path_here` into a function (for example, applying decorators) or use dynamic code execution.Ok: def foo(path: AbsPath): abs_path = path.abs_path foo(make_abs_path_here("./somewhere")) Not Ok: def foo(path: str): dangerous_value = make_abs_path_here(path).abs_path foo("./somewhere") **Parameters:** | Name | Type | Description | | --- | --- | --- | | `file_path` | _str_ | Absolute or relative path. | * **Returns:** _AbsPath_ ### [​](https://docs.baseten.co/reference/sdk/chains#function-truss_chains-run_local) _function_ `truss_chains.run_local` Context manager local debug execution of a chain. The arguments only need to be provided if the chainlets explicitly access any the corresponding fields of [`DeploymentContext`](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deploymentcontext) . **Parameters:** | Name | Type | Default | Description | | --- | --- | --- | --- | | `secrets` | _Mapping\[str,str\]\|None_ | `None` | A dict of secrets keys and values to provide to the chainlets. | | `data_dir` | _Path\|str\|None_ | `None` | Path to a directory with data files. | | `chainlet_to_service` | _Mapping\[str,[DeployedServiceDescriptor](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployedservicedescriptor)
\]_ | `None` | A dict of chainlet names to service descriptors. | Example usage (as trailing main section in a chain file): import os import truss_chains as chains class HelloWorld(chains.ChainletBase): ... if __name__ == "__main__": with chains.run_local( secrets={"some_token": os.environ["SOME_TOKEN"]}, chainlet_to_service={ "SomeChainlet": chains.DeployedServiceDescriptor( name="SomeChainlet", display_name="SomeChainlet", predict_url="https://...", options=chains.RPCOptions(), ) }, ): hello_world_chain = HelloWorld() result = hello_world_chain.run_remote(max_value=5) print(result) Refer to the [local debugging guide](https://docs.baseten.co/development/chain/localdev) for more details. ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployedservicedescriptor) _class_ `truss_chains.DeployedServiceDescriptor` Bases: `pydantic.BaseModel` Bundles values to establish an RPC session to a dependency chainlet, specifically with `StubBase`. **Parameters:** | Name | Type | Default | | --- | --- | --- | | `name` | _str_ | | | `display_name` | _str_ | | | `options` | _[RPCOptions](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-rpcoptions)
_ | | | `predict_url` | _str\|None_ | `None` | | `internal_url` | _InternalURL_ | `None` | ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-stubbase) _class_ `truss_chains.StubBase` Bases: `BasetenSession`, `ABC` Base class for stubs that invoke remote chainlets. Extends `BasetenSession` with methods for data serialization, de-serialization and invoking other endpoints. It is used internally for RPCs to dependency chainlets, but it can also be used in user-code for wrapping a deployed truss model into the Chains framework. It flexibly supports JSON and pydantic inputs and output. Example usage: import pydantic import truss_chains as chains class WhisperOutput(pydantic.BaseModel): ... class DeployedWhisper(chains.StubBase): # Input JSON, output JSON. async def run_remote(self, audio_b64: str) -> Any: return await self.predict_async( inputs={"audio": audio_b64}) # resp == {"text": ..., "language": ...} # OR Input JSON, output pydantic model. async def run_remote(self, audio_b64: str) -> WhisperOutput: return await self.predict_async( inputs={"audio": audio_b64}, output_model=WhisperOutput) # OR Input and output are pydantic models. async def run_remote(self, data: WhisperInput) -> WhisperOutput: return await self.predict_async(data, output_model=WhisperOutput) class MyChainlet(chains.ChainletBase): def __init__(self, ..., context=chains.depends_context()): ... self._whisper = DeployedWhisper.from_url( WHISPER_URL, context, options=chains.RPCOptions(retries=3), ) async def run_remote(self, ...): await self._whisper.run_remote(...) **Parameters:** | Name | Type | Description | | --- | --- | --- | | `service_descriptor` | _[DeployedServiceDescriptor](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deployedservicedescriptor)
\]_ | Contains the URL and other configuration. | | `api_key` | _str_ | A baseten API key to authorize requests. | #### [​](https://docs.baseten.co/reference/sdk/chains#classmethod-from_url-predict_url-context_or_api_key-options=none) _classmethod_ from\_url(predict\_url, context\_or\_api\_key, options=None) Factory method, convenient to be used in chainlet’s `__init__`\-method. **Parameters:** | Name | Type | Description | | --- | --- | --- | | `predict_url` | _str_ | URL to predict endpoint of another chain / truss model. | | `context_or_api_key` | _[DeploymentContext](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-deploymentcontext)
_ | Deployment context object, obtained in the chainlet’s `__init__` or Baseten API key. | | `options` | _[RPCOptions](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-rpcoptions)
_ | RPC options, for example, retries. | #### [​](https://docs.baseten.co/reference/sdk/chains#invocation-methods) Invocation Methods * `async predict_async(inputs: PydanticModel, output_model: Type[PydanticModel]) → PydanticModel` * `async predict_async(inputs: JSON, output_model: Type[PydanticModel]) → PydanticModel` * `async predict_async(inputs: JSON) → JSON` * `async predict_async_stream(inputs: PydanticModel | JSON) -> AsyncIterator[bytes]` Deprecated synchronous methods: * `predict_sync(inputs: PydanticModel, output_model: Type[PydanticModel]) → PydanticModel` * `predict_sync(inputs: JSON, output_model: Type[PydanticModel]) → PydanticModel` * `predict_sync(inputs: JSON) → JSON` ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-remoteerrordetail) _class_ `truss_chains.RemoteErrorDetail` Bases: `pydantic.BaseModel` When a remote chainlet raises an exception, this pydantic model contains information about the error and stack trace and is included in JSON form in the error response. **Parameters:** | Name | Type | | --- | --- | | `exception_cls_name` | _str_ | | `exception_module_name` | _str\|None_ | | `exception_message` | _str_ | | `user_stack_trace` | _list\[StackFrame\]_ | #### [​](https://docs.baseten.co/reference/sdk/chains#method-format) _method_ format() Format the error for printing, similar to how Python formats exceptions with stack traces. * **Returns:** str ### [​](https://docs.baseten.co/reference/sdk/chains#class-truss_chains-genericremoteexception) _class_ `truss_chains.GenericRemoteException` Bases: `Exception` Raised when calling a remote chainlet results in an error and it is not possible to re-raise the same exception that was raise remotely in the caller. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/sdk/truss) [Training SDKAPI reference for the Baseten training SDK.\ \ Next](https://docs.baseten.co/reference/sdk/training) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Upsert a secret - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / secrets Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/secrets \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my_secret", "value": "my_secret_value" }' 200 { "created_at": "2023-11-07T05:31:56Z", "name": "", "team_name": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json A request to create or update a Baseten secret by name. [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#body-name) name string required Name of the new or existing secret Example: `"my_secret"` [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#body-value) value string required Value of the secret Example: `"my_secret_value"` #### Response 200 - application/json A Baseten secret. Note that we do not support retrieving secret values. [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#response-created-at) created\_at string required Time the secret was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#response-name) name string required Name of the secret [​](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret#response-team-name) team\_name string required Name of the team the secret belongs to Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets) [Upsert a team secretCreates a new secret or updates an existing secret if one with the provided name already exists. The name and creation date of the created or updated secret is returned. This secret belongs to the specified team\ \ Next](https://docs.baseten.co/reference/management-api/teams/upserts-a-team-secret) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/secrets \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my_secret", "value": "my_secret_value" }' 200 { "created_at": "2023-11-07T05:31:56Z", "name": "", "team_name": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Create an API key - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / api\_keys Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/api_keys \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my-api-key", "type": "PERSONAL" }' 200 { "api_key": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json Request to create an API key. [​](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#body-type) type enum required Type of the API key. Available options: `PERSONAL`, `WORKSPACE_MANAGE_ALL`, `WORKSPACE_EXPORT_METRICS`, `WORKSPACE_INVOKE` Examples: `"PERSONAL"` `"WORKSPACE_EXPORT_METRICS"` `"WORKSPACE_INVOKE"` `"WORKSPACE_MANAGE_ALL"` [​](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#body-name-one-of-0) name string | null Optional name for the API key Example: `"my-api-key"` [​](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#body-model-ids-one-of-0) model\_ids string\[\] | null List of model IDs to scope the API key to, only present if type is 'WORKSPACE\_EXPORT\_METRICS' or 'WORKSPACE\_INVOKE' Example: ["aaaaaaaa"] #### Response 200 - application/json Represents an API key. [​](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key#response-api-key) api\_key string required The API key string Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/api-keys/lists-the-users-api-keys) [Delete an API keyDeletes an API key by prefix and returns info about the API key.\ \ Next](https://docs.baseten.co/reference/management-api/api-keys/delete-an-api-key) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/api_keys \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my-api-key", "type": "PERSONAL" }' 200 { "api_key": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get environment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / models / {model\_id} / environments / {env\_name} Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "name": "", "created_at": "2023-11-07T05:31:56Z", "model_id": "", "current_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "promotion_settings": { "redeploy_on_promotion": true, "rolling_deploy": true, "promotion_cleanup_strategy": "SCALE_TO_ZERO", "rolling_deploy_config": { "rolling_deploy_strategy": "REPLICA", "max_surge_percent": 10, "max_unavailable_percent": 0, "stabilization_time_seconds": 0, "replica_overhead_percent": 0 }, "ramp_up_while_promoting": true, "ramp_up_duration_seconds": 600 }, "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "candidate_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "in_progress_promotion": { "percent_traffic_to_new_version": 123, "error_message": "", "rolling_deploy": true } } #### Authorizations [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#parameter-env-name) env\_name string required #### Response 200 - application/json Environment for oracles. [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-name) name string required Name of the environment [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-created-at) created\_at string required Time the environment was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-model-id) model\_id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-current-deployment-one-of-0) current\_deployment DeploymentV1 · object required Current deployment of the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-autoscaling-settings) autoscaling\_settings AutoscalingSettingsV1 · object required Autoscaling settings for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-promotion-settings) promotion\_settings PromotionSettingsV1 · object required Promotion settings for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-instance-type) instance\_type InstanceTypeV1 · object required Instance type for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-candidate-deployment-one-of-0) candidate\_deployment DeploymentV1 · object Candidate deployment being promoted to the environment, if a promotion is in progress Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details#response-in-progress-promotion-one-of-0) in\_progress\_promotion InProgressPromotionV1 · object Details of the in-progress promotion, if any Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/environments/get-all-environments) [Update environmentAsynchronously updates an environment's settings. Poll the GET endpoint for the applied state.\ \ Next](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "name": "", "created_at": "2023-11-07T05:31:56Z", "model_id": "", "current_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "promotion_settings": { "redeploy_on_promotion": true, "rolling_deploy": true, "promotion_cleanup_strategy": "SCALE_TO_ZERO", "rolling_deploy_config": { "rolling_deploy_strategy": "REPLICA", "max_surge_percent": 10, "max_unavailable_percent": 0, "stabilization_time_seconds": 0, "replica_overhead_percent": 0 }, "ramp_up_while_promoting": true, "ramp_up_duration_seconds": 600 }, "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "candidate_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "in_progress_promotion": { "percent_traffic_to_new_version": 123, "error_message": "", "rolling_deploy": true } } Assistant Responses are generated using AI and may contain mistakes. --- # Update model environment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) PATCH / v1 / models / {model\_id} / environments / {env\_name} Try it cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name} \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "autoscaling_settings": { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "promotion_settings": { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": null, "rolling_deploy_config": null } }' 200 { "message": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#parameter-env-name) env\_name string required #### Body application/json A request to update an environment. [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#body-autoscaling-settings-one-of-0) autoscaling\_settings UpdateAutoscalingSettingsV1 · object Autoscaling settings for the environment Show child attributes Example: { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null} [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#body-promotion-settings-one-of-0) promotion\_settings UpdatePromotionSettingsV1 · object Promotion settings for the environment Show child attributes Example: { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": null, "rolling_deploy_config": null} #### Response 200 - application/json The response to a request to update autoscaling settings. [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#response-status) status enum required Status of the request to update autoscaling settings Available options: `ACCEPTED`, `QUEUED`, `UNCHANGED` [​](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings#response-message) message string required A message describing the status of the request to update autoscaling settings Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details) [Create Chain environmentCreate a chain environment. Returns the resulting environment.\ \ Next](https://docs.baseten.co/reference/management-api/environments/create-a-chain-environment) ⌘I cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name} \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "autoscaling_settings": { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "promotion_settings": { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": null, "rolling_deploy_config": null } }' 200 { "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Create environment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/environments/create-an-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / environments Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "staging", "autoscaling_settings": { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "promotion_settings": { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": true, "rolling_deploy_config": null } }' 200 { "name": "", "created_at": "2023-11-07T05:31:56Z", "model_id": "", "current_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "promotion_settings": { "redeploy_on_promotion": true, "rolling_deploy": true, "promotion_cleanup_strategy": "SCALE_TO_ZERO", "rolling_deploy_config": { "rolling_deploy_strategy": "REPLICA", "max_surge_percent": 10, "max_unavailable_percent": 0, "stabilization_time_seconds": 0, "replica_overhead_percent": 0 }, "ramp_up_while_promoting": true, "ramp_up_duration_seconds": 600 }, "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "candidate_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "in_progress_promotion": { "percent_traffic_to_new_version": 123, "error_message": "", "rolling_deploy": true } } #### Authorizations [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#parameter-model-id) model\_id string required #### Body application/json A request to create an environment. [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#body-name) name string required Name of the environment Example: `"staging"` [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#body-autoscaling-settings-one-of-0) autoscaling\_settings UpdateAutoscalingSettingsV1 · object Autoscaling settings for the environment Show child attributes Example: { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null} [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#body-promotion-settings-one-of-0) promotion\_settings UpdatePromotionSettingsV1 · object Promotion settings for the environment Show child attributes Example: { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": true, "rolling_deploy_config": null} #### Response 200 - application/json Environment for oracles. [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-name) name string required Name of the environment [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-created-at) created\_at string required Time the environment was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-model-id) model\_id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-current-deployment-one-of-0) current\_deployment DeploymentV1 · object required Current deployment of the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-autoscaling-settings) autoscaling\_settings AutoscalingSettingsV1 · object required Autoscaling settings for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-promotion-settings) promotion\_settings PromotionSettingsV1 · object required Promotion settings for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-instance-type) instance\_type InstanceTypeV1 · object required Instance type for the environment Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-candidate-deployment-one-of-0) candidate\_deployment DeploymentV1 · object Candidate deployment being promoted to the environment, if a promotion is in progress Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/create-an-environment#response-in-progress-promotion-one-of-0) in\_progress\_promotion InProgressPromotionV1 · object Details of the in-progress promotion, if any Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/terminates-deployment-replica) [Get all environmentsGets all environments for a given model\ \ Next](https://docs.baseten.co/reference/management-api/environments/get-all-environments) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "staging", "autoscaling_settings": { "autoscaling_window": 800, "concurrency_target": 3, "max_replica": 2, "max_scale_down_rate": null, "min_replica": 1, "scale_down_delay": 60, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "promotion_settings": { "promotion_cleanup_strategy": null, "ramp_up_duration_seconds": 600, "ramp_up_while_promoting": true, "redeploy_on_promotion": true, "rolling_deploy": true, "rolling_deploy_config": null } }' 200 { "name": "", "created_at": "2023-11-07T05:31:56Z", "model_id": "", "current_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "promotion_settings": { "redeploy_on_promotion": true, "rolling_deploy": true, "promotion_cleanup_strategy": "SCALE_TO_ZERO", "rolling_deploy_config": { "rolling_deploy_strategy": "REPLICA", "max_surge_percent": 10, "max_unavailable_percent": 0, "stabilization_time_seconds": 0, "replica_overhead_percent": 0 }, "ramp_up_while_promoting": true, "ramp_up_duration_seconds": 600 }, "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "candidate_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "in_progress_promotion": { "percent_traffic_to_new_version": 123, "error_message": "", "rolling_deploy": true } } Assistant Responses are generated using AI and may contain mistakes. --- # Deployments - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/troubleshooting/deployments#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/troubleshooting/deployments#issue-truss-push-can%E2%80%99t-find-config-yaml) Issue: `truss push` can’t find `config.yaml` -------------------------------------------------------------------------------------------------------------------------------------------------------- [Errno 2] No such file or directory: '/path/to/your-truss/config.yaml' ### [​](https://docs.baseten.co/troubleshooting/deployments#fix-set-correct-target-directory) Fix: set correct target directory The directory `truss push` is looking at is not a Truss. Make sure you’re giving `truss push` access to the correct directory by: * Running `truss push` from the directory containing the Truss. You should see the file `config.yaml` when you run `ls` in your working directory. * Or passing the target directory as an argument, such as `truss push /path/to/my-truss`. [​](https://docs.baseten.co/troubleshooting/deployments#issue-unexpected-failure-during-model-build) Issue: unexpected failure during model build ---------------------------------------------------------------------------------------------------------------------------------------------------- During the model build step, there can be unexpected failures from temporary circumstances. An example is a network error while downloading model weights from Hugging Face or installing a Python package from PyPi. ### [​](https://docs.baseten.co/troubleshooting/deployments#fix-restart-deploy-from-baseten-ui) Fix: restart deploy from Baseten UI First, check your model logs to determine the exact cause of the error. If it’s an error during model download, package installation, or similar, you can try restarting the deploy from the model dashboard in your workspace. * * * [​](https://docs.baseten.co/troubleshooting/deployments#autoscaling-issues) Autoscaling issues ------------------------------------------------------------------------------------------------- Before troubleshooting, review [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview) for parameter details, [Traffic patterns](https://docs.baseten.co/deployment/autoscaling/traffic-patterns) for pattern-specific recommendations, and [Request lifecycle](https://docs.baseten.co/deployment/autoscaling/request-lifecycle) for HTTP status codes and timeout behavior. ### [​](https://docs.baseten.co/troubleshooting/deployments#latency-spikes-during-scaling-events) Latency spikes during scaling events **Symptoms**: TTFT (time to first token) or p95/p99 latency degrades when replicas are added or removed. **Causes**: * Replicas terminated while handling in-flight requests * Cold start delays while new replicas initialize **Solutions** (in order of priority): 1. Increase [**scale-down delay**](https://docs.baseten.co/deployment/autoscaling/overview#scale-down-delay) (for example, 300s → 900s) to reduce how often replicas are removed. 2. Increase [**min replicas**](https://docs.baseten.co/deployment/autoscaling/overview#minimum-replicas) to reduce cold start frequency. 3. Lower [**target utilization**](https://docs.baseten.co/deployment/autoscaling/overview#target-utilization) to provide more headroom during scaling. ### [​](https://docs.baseten.co/troubleshooting/deployments#replicas-oscillating-thrash) Replicas oscillating (thrash) **Symptoms**: Replica count bounces repeatedly (for example, 8↔9) even with relatively stable traffic. **Causes**: Autoscaler reacting to short-term traffic noise or internal model fluctuations. **Solutions** (in order of priority): 1. Increase **scale-down delay**: this is the primary lever for oscillation. 2. Increase [**autoscaling window**](https://docs.baseten.co/deployment/autoscaling/overview#autoscaling-window) to smooth out noise. 3. Only then consider lowering **target utilization** for more headroom. Don’t use target utilization as the primary fix for thrash. Scale-down delay is more effective and doesn’t waste capacity. ### [​](https://docs.baseten.co/troubleshooting/deployments#slow-scale-up-/-%E2%80%9Cscaling-up-replicas%E2%80%9D-persists) Slow scale-up / “Scaling up replicas” persists **Symptoms**: New replicas take many minutes (or longer) to become ready. The deployment shows “Scaling up replicas” for an extended period. **Causes**: * GPU capacity not available in your region * Slow model initialization (large weights, slow downloads) **Solutions**: 1. **Pre-warm** by bumping min replicas via API before expected load spikes. 2. Contact support about capacity pool availability. 3. Check if optimized images are being used (look for “streaming-enabled image” in logs). ### [​](https://docs.baseten.co/troubleshooting/deployments#model-scales-to-zero-before-testing) Model scales to zero before testing **Symptoms**: A newly deployed model scales down to zero before you can send your first test request. **Solution**: Set `min_replica = 1` during testing. After testing, you can set it back to 0 if you want scale-to-zero behavior. ### [​](https://docs.baseten.co/troubleshooting/deployments#async-queue-growing-without-bound) Async queue growing without bound **Symptoms**: The async queue size keeps increasing and requests are not being processed fast enough. **Cause**: Requests are arriving faster than the deployment can process them. **Solutions**: 1. Increase [**max replicas**](https://docs.baseten.co/deployment/autoscaling/overview#maximum-replicas) to add more processing capacity. 2. Increase [**concurrency target**](https://docs.baseten.co/deployment/autoscaling/overview#concurrency-target) if your model can handle more concurrent requests. 3. Lower **target utilization** to trigger scaling earlier. ### [​](https://docs.baseten.co/troubleshooting/deployments#bill-higher-than-expected) Bill higher than expected **Symptoms**: GPU costs are higher than anticipated, especially during low-traffic periods. **Solutions**: 1. Raise **concurrency target** to squeeze more throughput from each replica. 2. Monitor **p95 latency** as you raise concurrency. If latency stays stable, keep raising; if it rises sharply, you’ve gone too far. 3. Enable **scale-to-zero** (min replicas = 0) for intermittent workloads. 4. Review your traffic patterns and adjust settings accordingly. See [Traffic patterns](https://docs.baseten.co/deployment/autoscaling/traffic-patterns) . ### [​](https://docs.baseten.co/troubleshooting/deployments#cold-starts-taking-too-long) Cold starts taking too long **Symptoms**: First request after scale-from-zero takes several minutes. Logs show extended time in model loading or container initialization. **Causes**: * Large model weights (10s–100s of GB) * Slow network downloads from model registries * Heavy initialization code in `load()` method **Solutions**: 1. Look for “streaming-enabled image” in logs. This confirms image streaming is active. 2. Keep `min_replica ≥ 1` to avoid cold starts entirely. 3. Pre-warm before expected traffic spikes using the [autoscaling API](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings) . See [Cold starts](https://docs.baseten.co/deployment/autoscaling/cold-starts) for detailed optimization strategies. ### [​](https://docs.baseten.co/troubleshooting/deployments#development-deployment-won%E2%80%99t-scale) Development deployment won’t scale **Symptoms**: Development deployment stays at 1 replica regardless of traffic. Can’t change autoscaling settings. **Cause**: Development deployments have fixed autoscaling settings that cannot be modified. Max replicas is locked at 1. **Solution**: Promote to a production deployment to enable full autoscaling. Development deployments are optimized for iteration with live reload, not traffic handling. See [Development deployments](https://docs.baseten.co/deployment/autoscaling/overview#development-deployments) for the fixed settings. ### [​](https://docs.baseten.co/troubleshooting/deployments#not-sure-which-traffic-pattern-i-have) Not sure which traffic pattern I have **Symptoms**: Unsure how to configure autoscaling because traffic behavior is unclear. **Solution**: 1. Go to your model’s **Metrics** tab in the Baseten dashboard. 2. Look at **Inference volume** and **Replicas** over the past week. 3. Identify your pattern: | You see… | Pattern | Key settings to adjust | | --- | --- | --- | | Frequent small spikes returning to baseline | Noisy/jittery | Longer autoscaling window | | Sharp jumps that stay high | Bursty | Short window, long delay, lower utilization | | Long flat periods with occasional bursts | Batch/scheduled | Scale-to-zero, pre-warming | | Gradual rises and falls | Smooth/steady | Higher utilization is safe | See [Traffic patterns](https://docs.baseten.co/deployment/autoscaling/traffic-patterns) for detailed recommendations. ### [​](https://docs.baseten.co/troubleshooting/deployments#concurrency-target-misconfigured) Concurrency target misconfigured **Symptoms**: Either unexpectedly high costs OR high latency despite having replicas available. **Diagnosis**: * **Too low** (common): Running many more replicas than needed. Default of 1 is conservative but expensive. * **Too high**: Requests queue at replicas, causing latency even when replica count looks healthy. **Solutions**: 1. Benchmark your model to find actual throughput capacity. 2. Use starting points by model type: | Model type | Starting concurrency | | --- | --- | | Standard Truss | 1 | | vLLM / LLM inference | 32–128 | | Text embeddings (TEI) | 32 | | Image generation (SDXL) | 1 | 3. Gradually increase while monitoring p95 latency. Stop when latency rises sharply. See [Concurrency target](https://docs.baseten.co/deployment/autoscaling/overview#concurrency-target) for full guidance. For detailed autoscaling configuration, see [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview) . For pattern-specific recommendations, see [Traffic patterns](https://docs.baseten.co/deployment/autoscaling/traffic-patterns) . Was this page helpful? YesNo [Previous](https://docs.baseten.co/observability/usage) [InferenceTroubleshoot common problems during model inference\ \ Next](https://docs.baseten.co/troubleshooting/inference) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Lifecycle - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/lifecycle#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) A training job in Baseten progresses through several states from creation to completion. Understanding these states helps you monitor and manage your training jobs effectively. [​](https://docs.baseten.co/training/lifecycle#job-states) Job states ------------------------------------------------------------------------ | State | Description | Active | Terminal | | --- | --- | --- | --- | | `TRAINING_JOB_PENDING` | The job is queued, waiting for GPU capacity to free up. | ✅ | | | `TRAINING_JOB_CREATED` | Initial state when a job is first created. Baseten has received the training configuration and persisted it to our records. | ✅ | | | `TRAINING_JOB_DEPLOYING` | Baseten is deploying the job, including provisioning compute resources and installing dependencies. | ✅ | | | `TRAINING_JOB_RUNNING` | The training code is actively executing. | ✅ | | | `TRAINING_JOB_COMPLETED` | The job has successfully finished execution. Any checkpoints or artifacts have been saved and uploaded. | | ✅ | | `TRAINING_JOB_DEPLOY_FAILED` | The job failed to deploy. This is likely due to a bad image or a resource allocation issue. | | ✅ | | `TRAINING_JOB_FAILED` | The job encountered an error and could not complete successfully. Check the logs for error details. | | ✅ | | `TRAINING_JOB_STOPPED` | The job was manually stopped by a user. | | ✅ | [​](https://docs.baseten.co/training/lifecycle#state-transitions) State transitions -------------------------------------------------------------------------------------- Jobs typically progress through states in the following order: 1. `TRAINING_JOB_PENDING` → `TRAINING_JOB_CREATED`: Automatic transition once GPU capacity is available 2. `TRAINING_JOB_CREATED` → `TRAINING_JOB_DEPLOYING`: Automatic transition once resources are allocated 3. `TRAINING_JOB_DEPLOYING` → `TRAINING_JOB_RUNNING`: Automatic transition once environment setup is complete 4. `TRAINING_JOB_RUNNING` → `TRAINING_JOB_COMPLETED`: Automatic transition upon successful completion A job may enter `TRAINING_JOB_FAILED` from any state if an error occurs. Similarly, `TRAINING_JOB_STOPPED` can be entered from any active state (`PENDING`, `DEPLOYING`, or `RUNNING`) when manually stopped. You can monitor these state transitions using the CLI command: truss train view # shows all active jobs truss train view --job-id # shows a specific job Or track a specific job’s progress with: truss train logs --job-id --tail Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/concepts/multinode) [ManagementHow to monitor, manage, and interact with your Baseten Training projects and jobs.\ \ Next](https://docs.baseten.co/training/management) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Serving your trained model - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten Training seamlessly integrates with Baseten’s model deployment capabilities. Once your `TrainingJob` has produced model checkpoints, you can deploy them as fully operational model endpoints. **This feature works with HuggingFace compatible LLMs**, allowing you to easily deploy fine-tuned language models directly from your training checkpoints with a single command. For optimized inference performance with TensorRT-LLM, BEI and Baseten Inference Stack, see [Deploy with optimized inference engines](https://docs.baseten.co/training/deploy-with-engine-builder) . To deploy checkpoints, first ensure you have a `TrainingJob` that’s running with a `checkpointing_config` enabled. runtime = definitions.Runtime( start_commands=[\ "/bin/sh -c './run.sh'",\ ], checkpointing_config=definitions.CheckpointingConfig( enabled=True, ), ) In your training code or configuration, ensure that your checkpoints are being written to the checkpointing directory, which can be referenced via [`$BT_CHECKPOINT_DIR`](https://docs.baseten.co/reference/sdk/training#baseten-provided-environment-variables) . The contents of this directory are uploaded to Baseten’s storage and made immediately available for deployment. _(You can optionally specify a `checkpoint_path` in your `checkpointing_config` if you prefer to write to a specific directory)._ The default location is “/tmp/training\_checkpoints”. To deploy your checkpoint(s) as a `Deployment`, you can: ### [​](https://docs.baseten.co/training/deployment#cli-deployment) CLI Deployment truss train deploy_checkpoints [OPTIONS] **Options:** | Option | Type | Description | | --- | --- | --- | | `--job-id` | TEXT | Job ID to deploy checkpoints from. If not specified, deploys from the most recent training job. | | `--project` | TEXT | Project name or project ID. | | `--project-id` | TEXT | Project ID. | | `--trainer-id` | TEXT | Trainer ID. Deploy checkpoints from a trainer instead of a training job. Mutually exclusive with `--project`, `--project-id`, and `--job-id`. | | `--config` | TEXT | Path to a Python file that defines a `DeployCheckpointsConfig` (see [Advanced CLI Deployment](https://docs.baseten.co/training/deployment#advanced-cli-deployment)
). | | `--dry-run` | FLAG | Generate a Truss config without deploying. Useful for inspecting or customizing the config before deployment. | | `--truss-config-output-dir` | TEXT | Path to output the Truss config to. Defaults to `truss_configs/_`, or `truss_configs/dry_run_` when using `--dry-run`. | | `--remote` | TEXT | Remote to use. | This will deploy the most recent checkpoint from your training job as an inference endpoint. ### [​](https://docs.baseten.co/training/deployment#ui-deployment) UI Deployment You can also deploy checkpoints directly from the Baseten UI by pressing the dropdown menu on your completed training job and selecting “Deploy” on your selected checkpoint. ### [​](https://docs.baseten.co/training/deployment#advanced-cli-deployment) Advanced CLI Deployment You can also: * run `truss train deploy_checkpoints [--job-id ]` and follow the setup wizard. * define an instance of a `DeployCheckpointsConfig` class (this is helpful for small changes that aren’t provided by the wizard) and run `truss train deploy_checkpoints --config `. When `deploy_checkpoints` is run, `truss` will construct a deployment `config.yml` and store it on disk. By default, the config is written to `truss_configs/_`. You can control the output location with `--truss-config-output-dir`. To inspect or customize the config before deploying, use `--dry-run` to generate the config without deploying: truss train deploy_checkpoints --job-id --dry-run If you’d like to modify the resulting deployment config, you can copy it into a permanent directory and customize it as needed. This file defines the source of truth for the deployment and can be deployed independently via `truss push`. See [deployments](https://docs.baseten.co/deployment/deployments) for more details. You can also reference training checkpoints using the `bt://` URI scheme in your [weights configuration](https://docs.baseten.co/development/model/bdn#baseten-training) . After successful deployment, your model will be deployed on Baseten, where you can run inference requests and evaluate performance. See [Calling Your Model](https://docs.baseten.co/inference/calling-your-model) for more details. To download the files you saved to the checkpointing directory or understand the file structure, you can run `truss train get_checkpoint_urls [--job-id=]` to get a JSON file containing presigned URLs for each training job. The JSON file contains the following structure: { "timestamp": "2025-06-23T13:44:16.485905+00:00", "job": { "id": "03yv1l3", "created_at": "2025-06-18T14:30:30.480Z", "current_status": "TRAINING_JOB_COMPLETED", "error_message": null, "instance_type": { "id": "H200:2x8x128x1600", "name": "H200:2x8x128x1600 - 2 Nodes of 8 H200 GPUs, 1128 GiB VRAM, 128 vCPUs, 1600 GiB RAM", "memory_limit_mib": 1650000, "millicpu_limit": 127900, "gpu_count": 8, "gpu_type": "H200", "gpu_memory_limit_mib": 1155072 }, "updated_at": "2025-06-18T14:30:30.510Z", "training_project_id": "lqz9o34", "training_project": { "id": "lqz9o34", "name": "checkpointing" } }, "checkpoint_artifacts": [\ {\ "url": "https://bt-training-eqwnwwp-f815d6cd-19bf-4589-bfcb-da76cd8432c0.s3.amazonaws.com/training_projects/lqz9o34/jobs/03yv1l3/rank-0/checkpoint-24/tokenizer_config.json?AWSAccessKeyId=AKIARLZO4BEQO4Q2A5NH&Signature=0vdzJf0686wNE1d9bm4%2Bw9ik5lY%3D&Expires=1751291056",\ "relative_file_name": "checkpoint-24/tokenizer_config.json",\ "node_rank": 0\ }\ ...\ ] } **Important notes about the presigned URLs:** * The presigned URLs expire after **7 days** from generation * These URLs are primarily intended for **evaluation and testing purposes**, not for long-term inference deployments * For production deployments, consider copying the checkpoint files to your Truss model directory and downloading them in the model’s `load()` function [​](https://docs.baseten.co/training/deployment#complex-and-custom-use-cases) Complex and custom use cases ------------------------------------------------------------------------------------------------------------- * Custom Model Architectures * Weights Sharded Across Nodes Examine the structure of your files with `truss train get_checkpoint_urls --job-id=`. If a file looks like this: { "url": "https://bt-training-eqwnwwp-f815d6cd-19bf-4589-bfcb-da76cd8432c0.s3.amazonaws.com/training_projects/lqz9o34/jobs/03yv1l3/rank-4/checkpoint-10/weights.safetensors?AWSAccessKeyId=AKIARLZO4BEQO4Q2A5NH&Signature=0vdzJf0686wNE1d9bm4%2Bw9ik5lY%3D&Expires=1751291056", "relative_file_name": "checkpoint-10/weights.safetensors", "node_rank": 4 } In your Truss configuration, add a section like this: Wildcards `*` match to an arbitrary number of chars while `?` matches to one. training_checkpoints: download_folder: /tmp/training_checkpoints artifact_references: - training_job_id: paths: - rank-*/checkpoint-10/ # Pull in all the files for checkpoint-10 across all nodes When your model pod starts up, you can read the file from the path `/tmp/training_checkpoints/rank-[node-rank]/[relative_file_name]`. For the example above, the file can be read from: /tmp/training_checkpoints//rank-4/checkpoint-10/weights.safetensors Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/loading) [Deploy with optimized inference enginesDeploy model checkpoints from Baseten Training directly to an inference engine without downloading or re-uploading weights.\ \ Next](https://docs.baseten.co/training/deploy-with-engine-builder) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Management - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/management#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Once you’ve submitted training jobs, Baseten provides tools to manage your `TrainingProject`s and individual `TrainingJob`s. You can use the [CLI](https://docs.baseten.co/reference/cli/training/training-cli) or the [API](https://docs.baseten.co/reference/training-api/overview) to manage your jobs. [​](https://docs.baseten.co/training/management#trainingproject-management) `TrainingProject` management ----------------------------------------------------------------------------------------------------------- * **Listing Projects:** To view all your training projects: truss train view This command will list all `TrainingProject`s you have access to, typically showing their names and IDs. Additionally, this command will show all active jobs. * **Viewing Jobs within a Project:** To see all jobs associated with a specific project, use its `project` (obtained when creating the project or from `truss train view`): truss train view --project * **Deleting a `TrainingProject`:** You can delete a training project via the API, or through the dashboard. Using the API: curl -X DELETE https://api.baseten.co/v1/training_projects/ \ -H "Authorization: Bearer YOUR_API_KEY" From the Baseten dashboard: 1. Select the training project you want to delete. 2. Type the project name (for example, `demo/qwen3-0.6b`) to confirm. 3. Select **Delete**. When you delete a project, the following data is permanently deleted with no archival or recovery option: * All undeployed [checkpoints](https://docs.baseten.co/training/concepts/checkpoints) from every job in the project * All data in the project’s [training cache](https://docs.baseten.co/training/concepts/cache) (`$BT_PROJECT_CACHE_DIR`) Checkpoints that have been [deployed](https://docs.baseten.co/training/deployment) aren’t affected. [​](https://docs.baseten.co/training/management#trainingjob-management) `TrainingJob` management --------------------------------------------------------------------------------------------------- After submitting a job with `truss train push config.py`, you receive a `project_id` and `job_id`. * **Listing Jobs:** As shown above, you can list all jobs within a project using: truss train view --project This will typically show job IDs, statuses, creation times, etc. * **Checking Status and Retrieving Logs:** To view the logs for a specific job, you can tail them in real-time or fetch existing logs. * To view logs for the most recently submitted job in the current context (for example, if you just pushed a job from your current terminal directory): truss train logs --tail * To view logs for a specific job using its `job-id`: truss train logs --job-id [--tail] Add `--tail` to follow the logs live. * **Understanding Job Statuses:** The `truss train view` and `truss train logs` commands will help you track which status a job is in. For more on the job lifecycle, see the [Lifecycle](https://docs.baseten.co/training/lifecycle) page. * **Stopping a `TrainingJob`:** If you need to stop a running job, use the `stop` command with the job’s project ID and job ID: truss train stop --job-id truss train stop --all # Stops all active jobs; Will prompt the user for confirmation. This will transition the job to the `TRAINING_JOB_STOPPED` state. * **Deleting a `TrainingJob`:** You can delete a training job via the API, or through the dashboard. Using the API: curl -X DELETE https://api.baseten.co/v1/training_projects//jobs/ \ -H "Authorization: Bearer YOUR_API_KEY" From the Baseten dashboard: 1. Select the project containing the job. 2. Select the job you want to delete. 3. Type the job name (for example, `job-2`) to confirm. 4. Select **Delete**. When you delete a job, all undeployed checkpoints are deleted permanently. There’s no archival or recovery option. Checkpoints that have been [deployed](https://docs.baseten.co/training/deployment) aren’t affected. * **Understanding Job Outputs & Checkpoints:** * The primary outputs of a successful `TrainingJob` are model **checkpoints** (if checkpointing is enabled and configured). * These checkpoints are stored by Baseten. For more information on how `CheckpointingConfig` works, see [Checkpoints](https://docs.baseten.co/training/concepts/checkpoints) . * When you are ready to [deploy a model](https://docs.baseten.co/training/deployment) , you will specify which checkpoints to use. The `model_name` you assign during deployment (via `DeployCheckpointsConfig`) becomes the identifier for this trained model version derived from your specific job’s checkpoints. * You can see the available checkpoints for a job via the [Training API](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints) . Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/lifecycle) [OverviewConnect to running training jobs from your local machine to debug, inspect state, and develop interactively.\ \ Next](https://docs.baseten.co/training/remote-access) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Truss configuration - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/truss-configuration#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) The `config.yaml` file defines how your model runs on Baseten: its dependencies, compute resources, secrets, and runtime behavior. You specify what your model needs; Baseten handles the infrastructure. Every Truss includes a `config.yaml` in its root directory. Configuration is optional, every value has a sensible default. Common configuration tasks include: * [Allocate GPU and memory](https://docs.baseten.co/reference/truss-configuration#resources) : compute resources for your instance. * [Declare environment variables](https://docs.baseten.co/reference/truss-configuration#param-environment-variables) : environment variables for your model. * [Configure concurrency](https://docs.baseten.co/reference/truss-configuration#runtime) : parallel request handling. * [Use a custom Docker image](https://docs.baseten.co/reference/truss-configuration#base_image) : deploy pre-built inference servers. YAML syntax If you’re new to YAML, here’s a quick primer. The default config uses `[]` for empty lists and `{}` for empty dictionaries. When adding values, the syntax changes to indented lines: # Empty requirements: [] secrets: {} # With values requirements: - torch - transformers secrets: hf_access_token: null [​](https://docs.baseten.co/reference/truss-configuration#ide-support) IDE support ------------------------------------------------------------------------------------- Truss ships a JSON schema for `config.yaml`. Projects created with `truss init` include a schema reference automatically, giving you autocompletion, hover docs, and validation in any editor that supports the [YAML language server](https://github.com/redhat-developer/yaml-language-server) (VS Code, JetBrains, Neovim, and others). To add schema support to an existing `config.yaml`, add this comment as the first line: # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json [​](https://docs.baseten.co/reference/truss-configuration#example) Example ----------------------------------------------------------------------------- The following example shows a config file for a GPU-accelerated text generation model: config.yaml model_name: my-llm description: A text generation model. requirements: - torch - transformers - accelerate resources: cpu: "4" memory: 16Gi accelerator: L4 secrets: hf_access_token: null For more examples, see the [truss-examples](https://github.com/basetenlabs/truss-examples) repository. [​](https://docs.baseten.co/reference/truss-configuration#reference) Reference --------------------------------------------------------------------------------- [​](https://docs.baseten.co/reference/truss-configuration#param-model-name) model\_name string The name of your model. This is displayed in the model details page in the Baseten UI. [​](https://docs.baseten.co/reference/truss-configuration#param-description) description string A description of your model. [​](https://docs.baseten.co/reference/truss-configuration#param-model-class-name) model\_class\_name string default:"Model" The name of the class that defines your Truss model. This class must implement at least a `predict` method. [​](https://docs.baseten.co/reference/truss-configuration#param-model-module-dir) model\_module\_dir string default:"model" The folder containing your model class. [​](https://docs.baseten.co/reference/truss-configuration#param-data-dir) data\_dir string default:"data" The folder for data files in your Truss. Access it in your model: model/model.py class Model: def __init__(self, **kwargs): data_dir = kwargs["data_dir"] # ... [​](https://docs.baseten.co/reference/truss-configuration#param-bundled-packages-dir) bundled\_packages\_dir string default:"packages" The folder for custom packages in your Truss.Place your own code here to reference in `model.py`. For example, with this project structure: stable-diffusion/ packages/ package_1/ subpackage/ script.py model/ model.py __init__.py config.yaml Inside the `model.py` the package can be imported like this: model/model.py from package_1.subpackage.script import run_script class Model: def __init__(self, **kwargs): pass def load(self): run_script() ... [​](https://docs.baseten.co/reference/truss-configuration#param-external-package-dirs) external\_package\_dirs string\[\] Use `external_package_dirs` to access custom packages located outside your Truss. This lets multiple Trusses share the same package.The following example shows a project structure where `shared_utils/` is outside the Truss: my-model/ model/ model.py config.yaml shared_utils/ helpers.py Specify the path in your `config.yaml`: config.yaml external_package_dirs: - ../shared_utils/ Then import the package in your `model.py`: model.py from shared_utils.helpers import process_input class Model: def predict(self, model_input): return process_input(model_input) [​](https://docs.baseten.co/reference/truss-configuration#param-environment-variables) environment\_variables object Key-value pairs exposed to the environment that the model executes in. Many Python libraries can be customized with environment variables. Do not store secret values directly in environment variables (or anywhere in the config file). See the `secrets` field for information on properly managing secrets. environment_variables: ENVIRONMENT: Staging DB_URL: https://my_database.example.com/ Baseten reserves some variable names (for example, `PORT`, `HOSTNAME`, `PATH`) and drops them from this field at deploy time. For the full list and for variables Baseten injects at runtime, see [Runtime environment](https://docs.baseten.co/development/model/custom-server#runtime-environment) . [​](https://docs.baseten.co/reference/truss-configuration#param-model-metadata) model\_metadata object A flexible field for additional metadata. The entire config file is available to your model at runtime.**Reserved keys** that Baseten interprets: * `example_model_input`: Sample input that populates the Baseten playground. For example, to configure a model with playground input and custom vLLM settings, use the following: model_metadata: example_model_input: {"prompt": "What is the meaning of life?"} vllm_config: tensor_parallel_size: 1 max_model_len: 4096 [​](https://docs.baseten.co/reference/truss-configuration#param-requirements-file) requirements\_file string Path to a dependency file. Supports `requirements.txt`, `pyproject.toml`, and `uv.lock`. Truss detects the format by filename. Pin versions for reproducibility.When set to a `pyproject.toml`, Truss installs packages from `[project.dependencies]`. When set to a `uv.lock`, a sibling `pyproject.toml` must exist in the same directory. requirements_file: ./requirements.txt requirements_file: ./pyproject.toml requirements_file: ./uv.lock [​](https://docs.baseten.co/reference/truss-configuration#param-requirements) requirements string\[\] A list of Python dependencies in [pip requirements file format](https://pip.pypa.io/en/stable/reference/requirements-file-format/) . Mutually exclusive with `requirements_file`. Only one can be specified.For example, to install pinned versions of the dependencies, use the following: requirements: - scikit-learn==1.0.2 - threadpoolctl==3.0.0 - joblib==1.1.0 - numpy==1.20.3 - scipy==1.7.3 [​](https://docs.baseten.co/reference/truss-configuration#param-system-packages) system\_packages string\[\] System packages that you would typically install using `apt` on a Debian operating system. system_packages: - ffmpeg - libsm6 - libxext6 [​](https://docs.baseten.co/reference/truss-configuration#param-python-version) python\_version string default:"py39" The Python version to use. Supported versions: * `py39` * `py310` * `py311` * `py312` * `py313` * `py314` [​](https://docs.baseten.co/reference/truss-configuration#param-secrets) secrets object Declare secrets your model needs at runtime, such as API keys or access tokens. Store the actual values in your [organization settings](https://app.baseten.co/settings/secrets) . Never store actual secret values in config. Use `null` as a placeholder. The key name must match the secret name in your organization. secrets: hf_access_token: null For more information, see [Secrets](https://docs.baseten.co/development/model/secrets) . [​](https://docs.baseten.co/reference/truss-configuration#param-examples-filename) examples\_filename string default:"examples.yaml" The path to a file containing example inputs for your model. [​](https://docs.baseten.co/reference/truss-configuration#param-live-reload) live\_reload boolean default:"false" If true, changes to your model code are automatically reloaded without restarting the server. Useful for development. [​](https://docs.baseten.co/reference/truss-configuration#param-apply-library-patches) apply\_library\_patches boolean default:"true" Whether to apply library patches for improved compatibility. [​](https://docs.baseten.co/reference/truss-configuration#resources) resources --------------------------------------------------------------------------------- The `resources` section specifies the compute resources that your model needs, including CPU, memory, and GPU resources. You can configure resources in two ways: **Option 1: Specify individual resource fields** resources: accelerator: A10G cpu: "4" memory: 20Gi Baseten provisions the smallest instance that meets the specified constraints. **Option 2: Specify an exact instance type** resources: instance_type: "A10G:4x16" Using `instance_type` lets you select an exact SKU from the [instance type reference](https://docs.baseten.co/deployment/resources#instance-type-reference) . When `instance_type` is specified, other resource fields are ignored. [​](https://docs.baseten.co/reference/truss-configuration#param-cpu) cpu string default:"1" CPU resources needed, expressed as either a raw number or “millicpus”. For example, `1000m` and `1` are equivalent. Fractional CPU amounts can be requested using millicpus. For example, `500m` is half of a CPU core. [​](https://docs.baseten.co/reference/truss-configuration#param-memory) memory string default:"2Gi" CPU RAM needed, expressed as a number with units. Units include “Gi” (Gibibytes), “G” (Gigabytes), “Mi” (Mebibytes), and “M” (Megabytes). For example, `1Gi` and `1024Mi` are equivalent. `Gi` in `resources.memory` refers to **Gibibytes**, which are slightly larger than **Gigabytes**. [​](https://docs.baseten.co/reference/truss-configuration#param-accelerator) accelerator string The GPU type for your instance. Available GPUs: * `T4` * `L4` * `L40S` * `A10G` * `V100` * `A100` * `A100_40GB` * `H100` * `H100_40GB` ([fractional GPU details](https://www.baseten.co/blog/using-fractional-h100-gpus-for-efficient-model-serving/) ) * `H200` * `B200` To request multiple GPUs (for example, if the weights don’t fit in a single GPU), use the `:` operator: resources: accelerator: L4:4 # Requests 4 L4s For more information, see how to [Manage resources](https://docs.baseten.co/deployment/resources) . [​](https://docs.baseten.co/reference/truss-configuration#param-instance-type) instance\_type string The full SKU name for the instance type. When specified, `cpu`, `memory`, and `accelerator` fields are ignored.Use this field to select an exact instance type from the [instance type reference](https://docs.baseten.co/deployment/resources#instance-type-reference) . The format is `:x` for GPU instances, or the bare `x` SKU for CPU-only instances. resources: instance_type: "L4:4x16" Examples: * `L4:4x16`: L4 GPU with 4 vCPUs and 16 GiB RAM. * `H100:8x80`: H100 GPU with 8 vCPUs and 80 GiB RAM (the exact specs vary by GPU type). * `4x16`: CPU-only instance with 4 vCPUs and 16 GiB RAM. [​](https://docs.baseten.co/reference/truss-configuration#param-node-count) node\_count number The number of nodes for multi-node deployments. Each node gets the specified resources. [​](https://docs.baseten.co/reference/truss-configuration#runtime) runtime ----------------------------------------------------------------------------- Runtime settings for your model instance. For example, to configure a high-throughput inference server with concurrency and health checks, use the following: runtime: predict_concurrency: 256 streaming_read_timeout: 120 health_checks: restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 300 [​](https://docs.baseten.co/reference/truss-configuration#param-predict-concurrency) predict\_concurrency number default:"1" The number of concurrent requests that can run in your model’s predict method. Defaults to 1, meaning `predict` runs one request at a time. Increase this if your model supports parallelism.See [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview#scaling-triggers) for more detail. [​](https://docs.baseten.co/reference/truss-configuration#param-streaming-read-timeout) streaming\_read\_timeout number default:"60" The timeout in seconds for streaming read operations. [​](https://docs.baseten.co/reference/truss-configuration#param-enable-tracing-data) enable\_tracing\_data boolean default:"false" If true, enables trace data export with built-in OTEL instrumentation. By default, data is collected internally by Baseten for troubleshooting. You can also export to your own systems. See the [tracing guide](https://docs.baseten.co/observability/tracing) . May add performance overhead. [​](https://docs.baseten.co/reference/truss-configuration#param-enable-debug-logs) enable\_debug\_logs boolean default:"false" If true, sets the Truss server log level to `DEBUG` instead of `INFO`. [​](https://docs.baseten.co/reference/truss-configuration#param-transport) transport object The transport protocol for your model. Supports `http` (default), `websocket`, and `grpc`. runtime: transport: kind: websocket ping_interval_seconds: 30 ping_timeout_seconds: 10 [​](https://docs.baseten.co/reference/truss-configuration#param-health-checks) health\_checks object Custom health check configuration for your deployments. For details, see [health check configuration](https://docs.baseten.co/development/model/custom-health-checks#health-check-configuration) . runtime: health_checks: startup_threshold_seconds: 2400 restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 300 [​](https://docs.baseten.co/reference/truss-configuration#param-startup-threshold-seconds) startup\_threshold\_seconds number How long the startup phase runs before marking the replica as unhealthy. During startup, readiness and liveness probes don’t run. Values must be between `10` and `3000` seconds. Defaults to 30 minutes (`1800` seconds). See [health checks](https://docs.baseten.co/development/model/custom-health-checks) for details. [​](https://docs.baseten.co/reference/truss-configuration#param-stop-traffic-threshold-seconds) stop\_traffic\_threshold\_seconds number How long health checks must continuously fail before Baseten stops traffic to the replica. Defaults to 30 minutes (`1800` seconds). [​](https://docs.baseten.co/reference/truss-configuration#param-restart-threshold-seconds) restart\_threshold\_seconds number How long health checks must continuously fail before Baseten restarts the replica. Defaults to 30 minutes (`1800` seconds). [​](https://docs.baseten.co/reference/truss-configuration#param-restart-check-delay-seconds) restart\_check\_delay\_seconds number deprecated How long to wait before running health checks. Deprecated. Use `startup_threshold_seconds` instead. [​](https://docs.baseten.co/reference/truss-configuration#param-remote-ssh) remote\_ssh object SSH access configuration for model replicas. When `enabled` is `true`, Baseten installs an OpenSSH server in the model container so you can connect from a terminal after running [`truss ssh setup`](https://docs.baseten.co/reference/cli/truss/ssh#setup) once: ssh model--.ssh.baseten.co Enable it in your config: runtime: remote_ssh: enabled: true Not compatible with [`docker_server.run_as_user_id`](https://docs.baseten.co/reference/truss-configuration#param-run-as-user-id) . SSH requires the default `app` user (UID `60000`). [​](https://docs.baseten.co/reference/truss-configuration#base_image) base\_image ------------------------------------------------------------------------------------ Use `base_image` to deploy a custom Docker image. This is useful for running scripts at build time or installing complex dependencies. For more information, see [Deploy custom Docker images](https://docs.baseten.co/development/model/custom-server) . For example, to use the vLLM Docker image as your base, use the following: base_image: image: vllm/vllm-openai:v0.7.3 python_executable_path: /usr/bin/python # ... [​](https://docs.baseten.co/reference/truss-configuration#param-image) image string The path to the Docker image, for example: * `vllm/vllm-openai` * `lmsysorg/sglang` * `nvcr.io/nvidia/nemo:23.03` When using image tags like `:latest`, Baseten uses a cached copy and may not reflect updates to the image. To pull a specific version, use image digests like `your-image@sha256:abc123...`. [​](https://docs.baseten.co/reference/truss-configuration#param-python-executable-path) python\_executable\_path string A path to the Python executable on the image, for example `/usr/bin/python`. base_image: image: vllm/vllm-openai:v0.12.0 python_executable_path: /usr/bin/python [​](https://docs.baseten.co/reference/truss-configuration#param-docker-auth) docker\_auth object Authentication configuration for a private Docker registry. base_image: docker_auth: auth_method: GCP_SERVICE_ACCOUNT_JSON secret_name: gcp-service-account registry: us-west2-docker.pkg.dev For more information, see [Private Docker registries](https://docs.baseten.co/development/model/private-registries) . [​](https://docs.baseten.co/reference/truss-configuration#param-auth-method) auth\_method string The authentication method for the private registry. Supported values: * `GCP_SERVICE_ACCOUNT_JSON` - authenticate with a [GCP service account](https://cloud.google.com/iam/docs/service-account-overview) . Add your service account JSON blob as a Truss secret. * `AWS_IAM` - authenticate with an [AWS IAM service account](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) . Add `aws_access_key_id` and `aws_secret_access_key` to your Baseten secrets. * `AWS_OIDC` - authenticate using AWS OIDC federation. Requires `aws_oidc_role_arn` and `aws_oidc_region`. * `GCP_OIDC` - authenticate using GCP Workload Identity Federation. Requires `gcp_oidc_service_account` and `gcp_oidc_workload_id_provider`. For `GCP_SERVICE_ACCOUNT_JSON`: base_image: docker_auth: auth_method: GCP_SERVICE_ACCOUNT_JSON secret_name: gcp-service-account registry: us-east4-docker.pkg.dev For `AWS_IAM`: base_image: docker_auth: auth_method: AWS_IAM registry: .dkr.ecr..amazonaws.com secrets: aws_access_key_id: null aws_secret_access_key: null For `AWS_OIDC`: base_image: docker_auth: auth_method: AWS_OIDC registry: .dkr.ecr..amazonaws.com aws_oidc_role_arn: arn:aws:iam::123456789012:role/my-role aws_oidc_region: us-east-1 For `GCP_OIDC`: base_image: docker_auth: auth_method: GCP_OIDC registry: us-east4-docker.pkg.dev gcp_oidc_service_account: my-sa@my-project.iam.gserviceaccount.com gcp_oidc_workload_id_provider: projects/123/locations/global/workloadIdentityPools/my-pool/providers/my-provider [​](https://docs.baseten.co/reference/truss-configuration#param-secret-name) secret\_name string The Truss secret that stores the credential for authentication. Required for `GCP_SERVICE_ACCOUNT_JSON`. Ensure this secret is added to the `secrets` section. [​](https://docs.baseten.co/reference/truss-configuration#param-registry) registry string The registry to authenticate to, for example `us-east4-docker.pkg.dev`. [​](https://docs.baseten.co/reference/truss-configuration#param-aws-access-key-id-secret-name) aws\_access\_key\_id\_secret\_name string default:"aws\_access\_key\_id" The secret name for the AWS access key ID. Only used with `AWS_IAM` auth method. [​](https://docs.baseten.co/reference/truss-configuration#param-aws-secret-access-key-secret-name) aws\_secret\_access\_key\_secret\_name string default:"aws\_secret\_access\_key" The secret name for the AWS secret access key. Only used with `AWS_IAM` auth method. [​](https://docs.baseten.co/reference/truss-configuration#docker_server) docker\_server ------------------------------------------------------------------------------------------ Use `docker_server` to deploy a custom Docker image that has its own HTTP server, without writing a `Model` class. This is useful for deploying inference servers like vLLM or SGLang that provide their own endpoints. See [Deploy custom Docker images](https://docs.baseten.co/development/model/custom-server) for usage details. For example, to deploy vLLM serving Qwen 2.5 3B, use the following: base_image: image: vllm/vllm-openai:v0.7.3 docker_server: start_command: vllm serve Qwen/Qwen2.5-3B-Instruct --enable-prefix-caching readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/completions server_port: 8000 # ... [​](https://docs.baseten.co/reference/truss-configuration#param-start-command) start\_command string The command to start the server. Required when `no_build` is not set or is `false`. When `no_build` is `true`, `start_command` is optional; if omitted, the image’s original `ENTRYPOINT` runs. [​](https://docs.baseten.co/reference/truss-configuration#param-server-port) server\_port number required The port where the server runs. Port 8080 is reserved by Baseten’s internal reverse proxy and cannot be used. [​](https://docs.baseten.co/reference/truss-configuration#param-predict-endpoint) predict\_endpoint string required The endpoint for inference requests. This is mapped to Baseten’s `/predict` route. Has no effect when `no_build` is `true`. No-build deployments do not remap URLs, so all server paths are accessible directly. See [No-build deployment](https://docs.baseten.co/development/model/custom-server#no-build-deployment) for details. [​](https://docs.baseten.co/reference/truss-configuration#param-readiness-endpoint) readiness\_endpoint string required The endpoint for [readiness probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#readiness-probe) . Determines when the container can accept traffic. [​](https://docs.baseten.co/reference/truss-configuration#param-liveness-endpoint) liveness\_endpoint string required The endpoint for [liveness probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#liveness-probe) . Determines if the container needs to be restarted. [​](https://docs.baseten.co/reference/truss-configuration#param-run-as-user-id) run\_as\_user\_id number The Linux UID to run the server process as inside the container. Use this when your base image expects a specific non-root user (for example, NVIDIA NIM containers).The specified UID must already exist in the base image. Values `0` (root) and `60000` (platform default) are not allowed.Baseten automatically sets ownership of `/app`, `/workspace`, the packages directory, and `$HOME` to this UID. If your server writes to other directories, ensure they are writable by this UID in your base image or via `build_commands`. [​](https://docs.baseten.co/reference/truss-configuration#param-no-build) no\_build boolean Skip the build step and deploy the base image as-is. Baseten copies the image to its container registry without running `docker build` or modifying the image in any way. Only available for [custom server deployments](https://docs.baseten.co/development/model/custom-server) that use `docker_server`.When `no_build` is `true`: * `start_command` is optional. If omitted, the image’s original `ENTRYPOINT` runs. * Environment variables and secrets are available. * Development mode is not supported. Deploy with `truss push` (published deployments are the default). Use this for security-hardened images (for example, Chainguard) that must remain unmodified. [Contact support](mailto:support@baseten.co) to enable no-build deployments for your organization. config.yaml base_image: image: your-registry/your-hardened-image:latest docker_server: no_build: true server_port: 8000 predict_endpoint: /predict readiness_endpoint: /health liveness_endpoint: /health See [No-build deployment](https://docs.baseten.co/development/model/custom-server#no-build-deployment) for usage details. The `/app` directory is reserved by Baseten. `/app`, `/home/app`, `/tmp`, `/workspace`, and `/packages` are writable in the container. See [Container filesystem](https://docs.baseten.co/development/model/custom-server#container-filesystem) for the full runtime contract. If you need other directories to be writable, use `run_as_user_id` or `build_commands` to set permissions. [​](https://docs.baseten.co/reference/truss-configuration#external_data) external\_data ------------------------------------------------------------------------------------------ Use `external_data` to download remote files into your image at build time. This reduces cold-start time by making data available without downloading it at runtime. Each entry specifies a URL to fetch and a path relative to the data directory where the file is stored. external_data: - url: https://my-bucket.s3.amazonaws.com/my-data.tar.gz local_data_path: my-data.tar.gz [​](https://docs.baseten.co/reference/truss-configuration#param-url) url string required The URL to download data from. [​](https://docs.baseten.co/reference/truss-configuration#param-local-data-path) local\_data\_path string required Path relative to the data directory where the downloaded file is stored. For example, `my-data.tar.gz` is stored at `/app/data/my-data.tar.gz`. [​](https://docs.baseten.co/reference/truss-configuration#param-name) name string An optional name for the download entry. [​](https://docs.baseten.co/reference/truss-configuration#param-backend) backend string default:"http\_public" The download backend to use. [​](https://docs.baseten.co/reference/truss-configuration#build_commands) build\_commands -------------------------------------------------------------------------------------------- [​](https://docs.baseten.co/reference/truss-configuration#param-build-commands) build\_commands string\[\] A list of shell commands to run during Docker build. These commands execute after system packages and Python requirements are installed. Use them for any setup that can’t be handled by `requirements` or `system_packages` alone.For example, to clone a GitHub repository into the container, use the following: build_commands: - git clone https://github.com/comfyanonymous/ComfyUI.git You can also combine `build_commands` with `docker_server` to deploy third-party inference servers. The following example installs Ollama at build time and runs it as a Docker server: model_name: ollama-tinyllama base_image: image: python:3.11-slim build_commands: - curl -fsSL https://ollama.com/install.sh | sh docker_server: start_command: sh -c "ollama serve & sleep 5 && ollama pull tinyllama && wait" readiness_endpoint: /api/tags liveness_endpoint: /api/tags predict_endpoint: /api/generate server_port: 11434 resources: cpu: "4" memory: 8Gi For more information, see [Build commands](https://docs.baseten.co/development/model/custom-server) . [​](https://docs.baseten.co/reference/truss-configuration#build) build ------------------------------------------------------------------------- The `build` section handles secret access during Docker builds. Other build-time configuration options are: * [`build_commands`](https://docs.baseten.co/reference/truss-configuration#build_commands) : shell commands to run during build. * [`requirements`](https://docs.baseten.co/reference/truss-configuration#requirements) : Python packages to install. * [`system_packages`](https://docs.baseten.co/reference/truss-configuration#system_packages) : apt packages to install. * [`base_image`](https://docs.baseten.co/reference/truss-configuration#base_image) : custom Docker base image. [​](https://docs.baseten.co/reference/truss-configuration#param-secret-to-path-mapping) secret\_to\_path\_mapping object Grants access to secrets during the build. Provide a mapping between a secret and a path on the image. You can then access the secret in commands specified in `build_commands` by running `cat` on the file.For example, to install a pip package from a private GitHub repository, use the following: build_commands: - pip install git+https://$(cat /root/my-github-access-token)@github.com/path/to-private-repo.git build: secret_to_path_mapping: my-github-access-token: /root/my-github-access-token secrets: my-github-access-token: null Under the hood, this option mounts your secret as a build secret. The value of your secret will be secure and will not be exposed in your Docker history or logs. [​](https://docs.baseten.co/reference/truss-configuration#weights-preview) weights Preview --------------------------------------------------------------------------------------------- Use `weights` to configure Baseten Delivery Network (BDN) for model weight delivery with multi-tier caching. This is the recommended approach for optimizing cold starts. weights: - source: "hf://meta-llama/Llama-3.1-8B@main" mount_location: "/models/llama" allow_patterns: ["*.safetensors", "config.json"] `weights` replaces the deprecated `model_cache` configuration. Use `truss migrate` to automatically convert your configuration. [​](https://docs.baseten.co/reference/truss-configuration#param-source) source string required URI specifying where to fetch weights from. Supported schemes: * `hf://`: Hugging Face Hub, for example `hf://meta-llama/Llama-3.1-8B@main` * `s3://`: AWS S3, for example `s3://my-bucket/models/weights` * `gs://`: Google Cloud Storage, for example `gs://my-bucket/models/weights` * `r2://`: Cloudflare R2, for example `r2://account_id.bucket/path` [​](https://docs.baseten.co/reference/truss-configuration#param-mount-location) mount\_location string required Absolute path where weights will be mounted in your container. Must start with `/`. [​](https://docs.baseten.co/reference/truss-configuration#param-auth-secret-name) auth\_secret\_name string Name of a Baseten secret containing credentials for private weight sources. [​](https://docs.baseten.co/reference/truss-configuration#param-auth) auth object Authentication configuration for accessing private weight sources. Required for OIDC-based authentication. Supported `auth_method` values: * `CUSTOM_SECRET`: use a Baseten secret (specify `auth_secret_name`). * `AWS_OIDC`: use AWS OIDC federation (requires `aws_oidc_role_arn` and `aws_oidc_region`). * `GCP_OIDC`: use GCP Workload Identity Federation (requires `gcp_oidc_service_account` and `gcp_oidc_workload_id_provider`). For AWS OIDC: weights: - source: "s3://my-bucket/models/weights" mount_location: "/models/weights" auth: auth_method: AWS_OIDC aws_oidc_role_arn: arn:aws:iam::123456789012:role/my-role aws_oidc_region: us-east-1 For GCP OIDC: weights: - source: "gs://my-bucket/models/weights" mount_location: "/models/weights" auth: auth_method: GCP_OIDC gcp_oidc_service_account: my-sa@my-project.iam.gserviceaccount.com gcp_oidc_workload_id_provider: projects/123/locations/global/workloadIdentityPools/my-pool/providers/my-provider [​](https://docs.baseten.co/reference/truss-configuration#param-allow-patterns) allow\_patterns string\[\] File patterns to include. Uses `fnmatch`\-style wildcards. Patterns like `*.safetensors` only match at the root level; use `**/*.safetensors` for recursive matching across subdirectories. [​](https://docs.baseten.co/reference/truss-configuration#param-ignore-patterns) ignore\_patterns string\[\] File patterns to exclude. Uses `fnmatch`\-style wildcards. Patterns like `*.bin` only match at the root level; use `**/*.bin` for recursive matching across subdirectories. For full documentation, see [Baseten Delivery Network (BDN)](https://docs.baseten.co/development/model/bdn) . [​](https://docs.baseten.co/reference/truss-configuration#model_cache-deprecated) model\_cache Deprecated ------------------------------------------------------------------------------------------------------------ `model_cache` is deprecated. Use [`weights`](https://docs.baseten.co/reference/truss-configuration#weights) instead for faster cold starts through multi-tier caching. Use `model_cache` to bundle model weights into your image at build time, reducing cold start latency. For example, to cache Llama 2 7B weights from Hugging Face, use the following: model_cache: - repo_id: NousResearch/Llama-2-7b-chat-hf revision: main ignore_patterns: - "*.bin" use_volume: true volume_folder: llama-2-7b-chat-hf Despite the name `model_cache`, there are multiple backends supported, not just Hugging Face. You can also cache weights stored on GCS, S3, or Azure. [​](https://docs.baseten.co/reference/truss-configuration#param-repo-id) repo\_id string required The source path for your model weights. For example, to cache weights from a Hugging Face repo, use the following: model_cache: - repo_id: madebyollin/sdxl-vae-fp16-fix Or you can cache weights from buckets like GCS or S3, using the following options: model_cache: - repo_id: gcs://path-to-my-bucket kind: gcs - repo_id: s3://path-to-my-bucket kind: s3 [​](https://docs.baseten.co/reference/truss-configuration#param-kind) kind string default:"hf" The source kind for the model cache. Supported values: `hf` (Hugging Face), `gcs`, `s3`, `azure`. [​](https://docs.baseten.co/reference/truss-configuration#param-revision) revision string The revision of your Hugging Face repo. Required when `use_volume` is true for Hugging Face repos. [​](https://docs.baseten.co/reference/truss-configuration#param-use-volume) use\_volume boolean required If true, caches model artifacts outside the container image. Recommended: `true`. [​](https://docs.baseten.co/reference/truss-configuration#param-volume-folder) volume\_folder string The location of the mounted folder. Required when `use_volume` is true. For example, `volume_folder: myrepo` makes the model available under `/app/model_cache/myrepo` at runtime. [​](https://docs.baseten.co/reference/truss-configuration#param-allow-patterns-1) allow\_patterns string\[\] File patterns to include in the cache. Uses Unix shell-style wildcards. By default, all paths are included. [​](https://docs.baseten.co/reference/truss-configuration#param-ignore-patterns-1) ignore\_patterns string\[\] File patterns to ignore, streamlining the caching process. Use Unix shell-style wildcards. Example: `["*.onnx", "Readme.md"]`. By default, nothing is ignored. [​](https://docs.baseten.co/reference/truss-configuration#param-runtime-secret-name) runtime\_secret\_name string default:"hf\_access\_token" The secret name to use for runtime authentication, for example when accessing private Hugging Face repos. [​](https://docs.baseten.co/reference/truss-configuration#trt_llm) trt\_llm ------------------------------------------------------------------------------ Configure TensorRT-LLM for optimized LLM inference on Baseten. TRT-LLM supports two inference stacks: * **v1**: Best for dense models, small models, and embedding models. Supports lookahead speculative decoding and LoRA adapters. * **v2**: Best for MoE models (Qwen3-MoE, DeepSeek, Kimi) and multi-node setups. config.yaml trt_llm: inference_stack: v2 build: checkpoint_repository: source: HF repo: meta-llama/Llama-3.1-8B-Instruct quantization_type: fp8 runtime: max_batch_size: 256 max_num_tokens: 8192 tensor_parallel_size: 1 resources: accelerator: H100 [​](https://docs.baseten.co/reference/truss-configuration#param-inference-stack) inference\_stack string default:"v1" The inference stack version to use.Supported values: * `v1`: Use for dense models, small models, and embedding/reranking models. Supports lookahead speculative decoding and LoRA adapters. * `v2`: Use for MoE models and multi-node setups. The v2 runtime manages build parameters automatically; only `checkpoint_repository`, `quantization_type`, and `num_builder_gpus` can be set under `build`. ### [​](https://docs.baseten.co/reference/truss-configuration#build-2) build Build-time configuration for TRT-LLM engine compilation. [​](https://docs.baseten.co/reference/truss-configuration#param-base-model) base\_model string default:"decoder" The model architecture type.Supported values: * `decoder`: For generative causal LLMs (Llama, Qwen, Mistral, DeepSeek). Auto-detects architecture from the checkpoint. * `encoder`: For causal embedding models. Optimized for throughput with models like Qwen3-8B for embeddings. * `encoder_bert`: For BERT-based models (classification, reranking, embeddings). Optimized for throughput and cold-start latency of models under 4B parameters. [​](https://docs.baseten.co/reference/truss-configuration#param-checkpoint-repository) checkpoint\_repository object required The model checkpoint to compile. See [checkpoint\_repository](https://docs.baseten.co/reference/truss-configuration#checkpoint_repository) for sub-fields. [​](https://docs.baseten.co/reference/truss-configuration#param-quantization-type) quantization\_type string default:"no\_quant" The quantization method for the model weights. Use `no_quant` for fp16/bf16 (uses the precision from the model’s `config.json`).Supported values: * `no_quant`: No quantization (fp16 or bf16). * `fp8`: FP8 weights with 16-bit KV cache. * `fp8_kv`: FP8 weights with FP8 KV cache. Faster attention with FP8 context FMHA. Not compatible with models that use `bias=True` (for example, Qwen 2.5). * `fp4`: FP4 weights with 16-bit KV cache. Requires B200 or newer GPUs. * `fp4_kv`: FP4 weights with FP8 KV cache. Requires B200 or newer GPUs. * `fp4_mlp_only`: FP4 quantization applied only to MLP layers, with 16-bit KV cache. Requires B200 or newer GPUs. [​](https://docs.baseten.co/reference/truss-configuration#param-tensor-parallel-count) tensor\_parallel\_count number default:"1" Number of GPUs for tensor parallelism. Must equal the number of GPUs in your `resources.accelerator` setting for v1. [​](https://docs.baseten.co/reference/truss-configuration#param-max-seq-len) max\_seq\_len number Maximum sequence length the engine supports. Automatically inferred from the model checkpoint when not set. For encoder models, this is inferred from `max_position_embeddings` in the model’s config. [​](https://docs.baseten.co/reference/truss-configuration#param-max-batch-size) max\_batch\_size number default:"256" Maximum number of requests batched together in one forward pass. Range: 1 to 2048. [​](https://docs.baseten.co/reference/truss-configuration#param-max-num-tokens) max\_num\_tokens number default:"8192" Maximum number of tokens batched together in one forward pass. For encoder models and generative models without chunked prefill, this limits the max context length. Range: 65 to 1048576. [​](https://docs.baseten.co/reference/truss-configuration#param-num-builder-gpus) num\_builder\_gpus number Number of GPUs to use during engine compilation. Set this higher than the deployment GPU count if quantization causes out-of-memory errors during the build step. If you run out of CPU memory, add more memory in the `resources` section instead. [​](https://docs.baseten.co/reference/truss-configuration#param-lora-adapters) lora\_adapters object A mapping of LoRA adapter names to checkpoint repositories. Each key becomes the `model` name in OpenAI-compatible API requests. Only supported on the v1 inference stack. trt_llm: build: lora_adapters: my-adapter: source: HF repo: my-org/my-lora-adapter lora_configuration: max_lora_rank: 64 [​](https://docs.baseten.co/reference/truss-configuration#param-lora-configuration) lora\_configuration object LoRA configuration. See [lora\_configuration](https://docs.baseten.co/reference/truss-configuration#lora_configuration) for sub-fields. Only supported on the v1 inference stack. [​](https://docs.baseten.co/reference/truss-configuration#param-speculator) speculator object Speculative decoding configuration. See [speculator](https://docs.baseten.co/reference/truss-configuration#speculator) for sub-fields. Only supported on the v1 inference stack. [​](https://docs.baseten.co/reference/truss-configuration#param-moe-expert-parallel-option) moe\_expert\_parallel\_option number default:"-1" Expert parallelism setting for MoE models. Set to `-1` to let the runtime decide. When set explicitly, must be a positive number less than or equal to `tensor_parallel_count`, and `tensor_parallel_count` should be divisible by this value for optimal performance. #### [​](https://docs.baseten.co/reference/truss-configuration#checkpoint_repository) checkpoint\_repository The model checkpoint to compile. Specifies the source, repository path, and optional credentials. [​](https://docs.baseten.co/reference/truss-configuration#param-source-1) source string required Where to fetch the checkpoint from.Supported values: * `HF`: Hugging Face Hub. * `S3`: AWS S3 bucket (for example, `s3://my-bucket/path/to/checkpoint`). * `GCS`: Google Cloud Storage bucket (for example, `gcs://my-bucket/path/to/checkpoint`). * `AZURE`: Azure Blob Storage. * `REMOTE_URL`: HTTP URL to a tar.gzip archive (for example, a presigned URL). * `BASETEN_TRAINING`: Deploy from a Baseten training job. Use the training job ID as `repo` and the run revision as `revision`. [​](https://docs.baseten.co/reference/truss-configuration#param-repo) repo string required The repository path. For `HF`, this is the Hugging Face repo ID (for example, `meta-llama/Llama-3.1-8B-Instruct`). For `S3`/`GCS`/`AZURE`, this is the bucket path. The checkpoint must contain `config.json` and model files in safetensors format. [​](https://docs.baseten.co/reference/truss-configuration#param-revision-1) revision string The revision or version of the checkpoint. For `HF` sources, this is the branch, tag, or commit hash. Required for `BASETEN_TRAINING` sources. [​](https://docs.baseten.co/reference/truss-configuration#param-runtime-secret-name-1) runtime\_secret\_name string default:"hf\_access\_token" The name of the Baseten secret that stores the access credential. Must match a key in your organization’s [secret settings](https://app.baseten.co/settings/secrets) . #### [​](https://docs.baseten.co/reference/truss-configuration#quantization_config) quantization\_config Calibration settings for quantized models. Only relevant when `quantization_type` is not `no_quant`. [​](https://docs.baseten.co/reference/truss-configuration#param-calib-size) calib\_size number default:"1024" Size of the calibration dataset. Must be a multiple of 64, between 64 and 16384. Increase for production runs (for example, 1536) or decrease for quick testing (for example, 256). [​](https://docs.baseten.co/reference/truss-configuration#param-calib-dataset) calib\_dataset string default:"abisee/cnn\_dailymail" Hugging Face dataset to use for calibration. Uses the `train` split and quantizes based on the `text` column. [​](https://docs.baseten.co/reference/truss-configuration#param-calib-max-seq-length) calib\_max\_seq\_length number default:"1536" Maximum sequence length for calibration samples. Must be a multiple of 64, between 64 and 16384. ### [​](https://docs.baseten.co/reference/truss-configuration#runtime-v1) runtime (v1) Runtime configuration for the v1 inference stack. trt_llm: inference_stack: v1 runtime: kv_cache_free_gpu_mem_fraction: 0.9 enable_chunked_context: true batch_scheduler_policy: guaranteed_no_evict total_token_limit: 500000 # ... [​](https://docs.baseten.co/reference/truss-configuration#param-kv-cache-free-gpu-mem-fraction) kv\_cache\_free\_gpu\_mem\_fraction number default:"0.9" Fraction of free GPU memory to allocate for the KV cache. Higher values allow more context but leave less room for other operations. [​](https://docs.baseten.co/reference/truss-configuration#param-kv-cache-host-memory-bytes) kv\_cache\_host\_memory\_bytes number Bytes of host (CPU) memory to reserve for KV cache offloading. Set to a high value to enable KV cache offload to host memory when GPU memory is constrained. [​](https://docs.baseten.co/reference/truss-configuration#param-enable-chunked-context) enable\_chunked\_context boolean default:"true" Whether to process long contexts in chunks. Requires `paged_kv_cache` and `use_paged_context_fmha` to be enabled in the build plugin configuration. [​](https://docs.baseten.co/reference/truss-configuration#param-batch-scheduler-policy) batch\_scheduler\_policy string default:"guaranteed\_no\_evict" The batch scheduling strategy.Supported values: * `guaranteed_no_evict`: Guarantees scheduling with the requested number of tokens. May queue requests if memory is insufficient. Recommended for most use cases. * `max_utilization`: Schedules requests without checking available memory. May need to pause requests if memory fills up. [​](https://docs.baseten.co/reference/truss-configuration#param-request-default-max-tokens) request\_default\_max\_tokens number Default maximum number of tokens per request when not specified by the client. [​](https://docs.baseten.co/reference/truss-configuration#param-served-model-name) served\_model\_name string The model name returned in OpenAI-compatible API responses. Only for generative (decoder) models. [​](https://docs.baseten.co/reference/truss-configuration#param-total-token-limit) total\_token\_limit number default:"500000" Maximum number of tokens scheduled at once to the C++ engine. Only for generative (decoder) models. [​](https://docs.baseten.co/reference/truss-configuration#param-webserver-default-route) webserver\_default\_route string Default API route for the model. Auto-detected from the model architecture for encoder models.Supported values: * `/v1/embeddings`: For embedding models. * `/rerank`: For reranking models. * `/predict`: For sequence classification models. ### [​](https://docs.baseten.co/reference/truss-configuration#runtime-v2) runtime (v2) Runtime configuration for the v2 inference stack. trt_llm: inference_stack: v2 runtime: max_batch_size: 256 max_num_tokens: 8192 tensor_parallel_size: 1 # ... [​](https://docs.baseten.co/reference/truss-configuration#param-max-seq-len-1) max\_seq\_len number Maximum sequence length. Range: 1 to 1048576. [​](https://docs.baseten.co/reference/truss-configuration#param-max-batch-size-1) max\_batch\_size number default:"256" Maximum number of requests batched together in one forward pass. Range: 1 to 2048. [​](https://docs.baseten.co/reference/truss-configuration#param-max-num-tokens-1) max\_num\_tokens number default:"8192" Maximum number of tokens batched together in one forward pass. Range: 65 to 131072. [​](https://docs.baseten.co/reference/truss-configuration#param-tensor-parallel-size) tensor\_parallel\_size number default:"1" Number of GPUs for tensor parallelism. [​](https://docs.baseten.co/reference/truss-configuration#param-enable-chunked-prefill) enable\_chunked\_prefill boolean default:"true" Whether to enable chunked prefill for generative (decoder) models. [​](https://docs.baseten.co/reference/truss-configuration#param-served-model-name-1) served\_model\_name string The model name returned in OpenAI-compatible API responses. Only for generative (decoder) models. ### [​](https://docs.baseten.co/reference/truss-configuration#speculator) speculator Configure speculative decoding to speed up inference by predicting multiple tokens per step. Only supported on the v1 inference stack. trt_llm: build: speculator: speculative_decoding_mode: LOOKAHEAD_DECODING lookahead_windows_size: 7 lookahead_ngram_size: 5 lookahead_verification_set_size: 3 max_batch_size: 64 # ... Speculative decoding works best at lower batch sizes (under 64). For high-throughput use cases, tune concurrency settings for more aggressive autoscaling instead. [​](https://docs.baseten.co/reference/truss-configuration#param-speculative-decoding-mode) speculative\_decoding\_mode string The speculative decoding strategy.Supported values: * `LOOKAHEAD_DECODING`: N-gram based speculation built into the runtime. Recommended for most use cases, especially code editing workloads where n-gram patterns are common. [​](https://docs.baseten.co/reference/truss-configuration#param-lookahead-windows-size) lookahead\_windows\_size number Lookahead window size for the `LOOKAHEAD_DECODING` mode. Required when using lookahead decoding. Recommended values: 5 to 8. [​](https://docs.baseten.co/reference/truss-configuration#param-lookahead-ngram-size) lookahead\_ngram\_size number N-gram size for the `LOOKAHEAD_DECODING` mode. Required when using lookahead decoding. Recommended values: 3 to 5. [​](https://docs.baseten.co/reference/truss-configuration#param-lookahead-verification-set-size) lookahead\_verification\_set\_size number Verification set size for the `LOOKAHEAD_DECODING` mode. Required when using lookahead decoding. Recommended values: 3 to 5. [​](https://docs.baseten.co/reference/truss-configuration#param-num-draft-tokens) num\_draft\_tokens number Maximum number of speculative tokens per step. Auto-calculated from the lookahead parameters when using `LOOKAHEAD_DECODING`. Maximum: 2048. [​](https://docs.baseten.co/reference/truss-configuration#param-enable-b10-lookahead) enable\_b10\_lookahead boolean default:"false" Enable the Baseten-optimized lookahead algorithm. Requires `speculative_decoding_mode` to be `LOOKAHEAD_DECODING`. When enabled with `(window_size, 1, 1)` settings (for example, `(8, 1, 1)` or `(32, 1, 1)`), enables dynamic speculation. ### [​](https://docs.baseten.co/reference/truss-configuration#lora_configuration) lora\_configuration LoRA adapter settings for the v1 inference stack. Use with `lora_adapters` to serve multiple fine-tuned models from a single deployment. [​](https://docs.baseten.co/reference/truss-configuration#param-max-lora-rank) max\_lora\_rank number default:"64" Maximum LoRA rank across all adapters. [​](https://docs.baseten.co/reference/truss-configuration#param-lora-target-modules) lora\_target\_modules string\[\] default:"\[\]" List of model modules to apply LoRA to. [​](https://docs.baseten.co/reference/truss-configuration#bis_llm-preview) bis\_llm Preview ---------------------------------------------------------------------------------------------- Configuration for deploying [BIS-LLM](https://docs.baseten.co/engines/bis-llm/overview) models, Baseten’s next-generation engine for Mixture of Experts and other large LLMs. This is a preview API and may change in future releases. BIS-LLM deployments take a separate path from standard `trt_llm` deployments. Several `truss push` flags are not supported: development deployments (the deployment must be published), `--promote`, `--environment`, `--preserve-previous-production-deployment`, `--disable-truss-download`, `--deployment-name`, and `--deploy-timeout-minutes`. bis_llm: version: "1.0.0" config: # BIS-LLM stack configuration (key/value pairs accepted by the chosen version) additional_autoscaling_config: metrics: - name: in_flight_tokens target: 1000 [​](https://docs.baseten.co/reference/truss-configuration#param-version) version string The version of the BIS-LLM deployment stack to use. [​](https://docs.baseten.co/reference/truss-configuration#param-config) config object Stack configuration passed through to the BIS-LLM deployment. The accepted keys depend on the chosen `version`. [​](https://docs.baseten.co/reference/truss-configuration#param-additional-autoscaling-config) additional\_autoscaling\_config object Additional autoscaling configuration for in-flight token metrics. [​](https://docs.baseten.co/reference/truss-configuration#param-metrics) metrics object\[\] required List of metric targets for autoscaling. Each entry has a `name` (string) and `target` (number). [​](https://docs.baseten.co/reference/truss-configuration#training_checkpoints) training\_checkpoints -------------------------------------------------------------------------------------------------------- Configuration for deploying models from training checkpoints. For example, to deploy a model using checkpoints from a training job, use the following: training_checkpoints: download_folder: /tmp/training_checkpoints artifact_references: - training_job_id: tr_abc123 paths: - "checkpoint-*" [​](https://docs.baseten.co/reference/truss-configuration#param-download-folder) download\_folder string default:"/tmp/training\_checkpoints" The folder to download the checkpoints to. [​](https://docs.baseten.co/reference/truss-configuration#param-artifact-references) artifact\_references object\[\] A list of artifact references to download. [​](https://docs.baseten.co/reference/truss-configuration#param-training-job-id) training\_job\_id string required The training job ID that the artifact reference belongs to. [​](https://docs.baseten.co/reference/truss-configuration#param-paths) paths string\[\] The paths of the files to download, which can contain `*` or `?` wildcards. The following environment variables are reserved by the platform and will be overwritten at runtime: `PORT`, `HOSTNAME`. You’ll see a warning if you attempt to set these in your config. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/overview) [OverviewBaseten provides two ways to call models: Model APIs for managed LLMs and deployed model endpoints for custom models and chains.\ \ Next](https://docs.baseten.co/reference/inference-api/overview) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Deploy with optimized inference engines - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/deploy-with-engine-builder#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) When a [Baseten Training](https://docs.baseten.co/training/overview) job completes, Baseten automatically saves your checkpoints to Baseten storage. You can deploy any of them to an inference engine without downloading or re-uploading anything. [Engine-Builder-LLM](https://docs.baseten.co/engines/engine-builder-llm/overview) , [BEI](https://docs.baseten.co/engines/bei/overview) , and [BIS-LLM](https://docs.baseten.co/engines/bis-llm/overview) all support this workflow. For deploying weights from external cloud storage (GCS, S3, Azure), see [Deploy from cloud storage](https://docs.baseten.co/engines/performance-concepts/cloud-storage-deployment) . [​](https://docs.baseten.co/training/deploy-with-engine-builder#checkpoint-reference) Checkpoint reference ------------------------------------------------------------------------------------------------------------- The `repo` and `revision` fields in `checkpoint_repository` specify which training project and checkpoint to deploy. * `repo`: Your Baseten Training project name. * `revision`: Which job and checkpoint to target. The following formats are supported: | `revision` value | Deploys | | --- | --- | | `/` | A specific checkpoint from a specific job (for example, `abc123/checkpoint-100`) | | `` | The latest checkpoint from a specific job | | `latest` or omitted | The latest checkpoint from the latest job | To look up checkpoint names for a job, run: truss train get_checkpoint_urls --job-id=YOUR_TRAINING_JOB_ID [​](https://docs.baseten.co/training/deploy-with-engine-builder#llm-deployment) LLM deployment ------------------------------------------------------------------------------------------------- Use [Engine-Builder-LLM](https://docs.baseten.co/engines/engine-builder-llm/overview) or [BIS-LLM](https://docs.baseten.co/engines/bis-llm/overview) to deploy a fine-tuned language model. Set `base_model` to `decoder`: config.yaml model_name: My Fine-Tuned LLM resources: accelerator: H100 use_gpu: true trt_llm: build: base_model: decoder checkpoint_repository: source: BASETEN_TRAINING repo: YOUR_TRAINING_PROJECT_NAME revision: YOUR_TRAINING_JOB_ID/checkpoint-100 Once deployed, call the model using the OpenAI-compatible chat completions endpoint: curl -X POST https://model-YOUR_MODEL_ID.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "1", "messages": [{"role": "user", "content": "Hello"}]}' See [Call your model](https://docs.baseten.co/inference/calling-your-model) for full inference options including streaming and the OpenAI SDK. [​](https://docs.baseten.co/training/deploy-with-engine-builder#embeddings-deployment) Embeddings deployment --------------------------------------------------------------------------------------------------------------- Use [BEI](https://docs.baseten.co/engines/bei/overview) to deploy a fine-tuned embedding or reranker model. Use `encoder_bert` for BERT-based models (sentence-transformers, rerankers, classifiers) or `encoder` for causal embedding models: config.yaml model_name: My Fine-Tuned Embeddings resources: accelerator: A10G use_gpu: true trt_llm: build: base_model: encoder_bert checkpoint_repository: source: BASETEN_TRAINING repo: YOUR_TRAINING_PROJECT_NAME revision: YOUR_TRAINING_JOB_ID/checkpoint-100 max_num_tokens: 16384 runtime: webserver_default_route: /v1/embeddings Encoder models have specific requirements: * **No tensor parallelism**: Omit `tensor_parallel_count` or set it to `1`. * **Fast tokenizer required**: Your checkpoint must include a `tokenizer.json` file. Models using only the legacy `vocab.txt` format aren’t supported. * **Embedding model files**: For sentence-transformer models, include `modules.json` and `1_Pooling/config.json` in your checkpoint. The `webserver_default_route` field sets the inference endpoint path: * `/v1/embeddings`: For embedding models. * `/rerank`: For rerankers. * `/predict`: For classifiers. * `/predict_tokens`: For token-level prediction. Once deployed, call the model using the embeddings endpoint: curl -X POST https://model-YOUR_MODEL_ID.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "1", "input": "Your text here"}' See [Call your model](https://docs.baseten.co/inference/calling-your-model) for full inference options. [​](https://docs.baseten.co/training/deploy-with-engine-builder#related) Related ----------------------------------------------------------------------------------- * [Engine-Builder-LLM configuration](https://docs.baseten.co/engines/engine-builder-llm/engine-builder-config) : Complete build and runtime options for LLMs. * [BEI reference configuration](https://docs.baseten.co/engines/bei/bei-reference) : Complete configuration for encoder models. * [Deploy from cloud storage](https://docs.baseten.co/engines/performance-concepts/cloud-storage-deployment) : GCS, S3, and Azure deployment using `checkpoint_repository`. * [Baseten Training overview](https://docs.baseten.co/training/overview) : Training jobs, checkpoints, and the full training workflow. * [Secrets management](https://docs.baseten.co/development/model/secrets) : Configure credentials for private storage. Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/deployment) [OverviewA training SDK that supports long sequence length, async RL, and one-click checkpoint deploys on the Baseten Inference Stack.\ \ Next](https://docs.baseten.co/loops/overview) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Loading checkpoints - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/loading#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Checkpoint loading lets you resume training from previously saved model states. When enabled, Baseten automatically downloads your specified checkpoints to the training environment before your training code starts. **Use cases:** * Resume failed training jobs. * Incremental training and fine-tuning. [​](https://docs.baseten.co/training/loading#accessing-downloaded-checkpoints) Accessing downloaded checkpoints ------------------------------------------------------------------------------------------------------------------ Checkpoints are available through the `BT_LOAD_CHECKPOINT_DIR` environment variable. For single-node training, they’re located in `BT_LOAD_CHECKPOINT_DIR/rank-0/`. For multi-node training, each node’s checkpoints are in `BT_LOAD_CHECKPOINT_DIR/rank-/`. [​](https://docs.baseten.co/training/loading#checkpoint-reference) Checkpoint reference ------------------------------------------------------------------------------------------ Create references to checkpoints using the `BasetenCheckpoint` factory: ### [​](https://docs.baseten.co/training/loading#from-latest) From latest # Load the latest checkpoint from a project BasetenCheckpoint.from_latest_checkpoint(project_name="my-training-project") # Load the latest checkpoint from a previous job BasetenCheckpoint.from_latest_checkpoint(job_id="gvpql31") **Parameters:** * `project_name`: Load the latest checkpoint from the most recent job in this project. * `job_id`: Load the latest checkpoint from this specific job. * Both parameters: Load the latest checkpoint from that specific job in that project. ### [​](https://docs.baseten.co/training/loading#from-named) From named # Pin your starting point to a specific checkpoint BasetenCheckpoint.from_named_checkpoint(checkpoint_name="checkpoint-20", job_id="gvpql31") **Parameters:** * `checkpoint_name`: The name of the specific checkpoint to load. * `job_id`: The job that contains the named checkpoint. * Both parameters: Load the named checkpoint from that specific job in that project. [​](https://docs.baseten.co/training/loading#configuration-examples) Configuration examples ---------------------------------------------------------------------------------------------- Here are practical examples of how to configure checkpoint loading in your training jobs: ### [​](https://docs.baseten.co/training/loading#from-latest-2) From latest # Latest checkpoint from project load_config = LoadCheckpointConfig( enabled=True, checkpoints=[\ BasetenCheckpoint.from_latest_checkpoint(project_name="gpt-finetuning")\ ] ) # Latest checkpoint from specific job load_config = LoadCheckpointConfig( enabled=True, checkpoints=[\ BasetenCheckpoint.from_latest_checkpoint(job_id="gvpql31")\ ] ) ### [​](https://docs.baseten.co/training/loading#from-named-2) From named # Specific named checkpoint load_config = LoadCheckpointConfig( enabled=True, checkpoints=[\ BasetenCheckpoint.from_named_checkpoint(\ checkpoint_name="checkpoint-20",\ job_id="gvpql31"\ )\ ] ) # Named checkpoint with custom download location load_config = LoadCheckpointConfig( enabled=True, download_folder="/tmp/my_checkpoints", checkpoints=[\ BasetenCheckpoint.from_named_checkpoint(\ checkpoint_name="checkpoint-20",\ job_id="rwnojdq"\ )\ ] ) **Configuration parameters:** * `enabled`: Set to `True` to enable checkpoint loading. * `checkpoints`: List containing checkpoint references. * `download_folder`: Optional custom download location (defaults to `/tmp/loaded_checkpoints`). [​](https://docs.baseten.co/training/loading#complete-trainingjob-setup) Complete TrainingJob setup ------------------------------------------------------------------------------------------------------ from truss_train import LoadCheckpointConfig, BasetenCheckpoint, CheckpointingConfig, TrainingJob, Image, Runtime, TrainingProject from truss_train.definitions import CacheConfig # Configure checkpoint loading load_checkpoint_config = LoadCheckpointConfig( enabled=True, download_folder="/tmp/loaded_checkpoints", checkpoints=[\ BasetenCheckpoint.from_latest_checkpoint(job_id="previous_job_id")\ ] ) # Configure checkpointing for saving new checkpoints checkpointing_config = CheckpointingConfig( enabled=True, checkpoint_path="/tmp/training_checkpoints" ) # Create TrainingJob job = TrainingJob( image=Image(base_image="your-base-image"), runtime=Runtime( checkpointing_config=checkpointing_config, load_checkpoint_config=load_checkpoint_config, start_commands=["chmod +x ./run.sh && ./run.sh"], cache_config=CacheConfig(enabled=True) ), ) project = TrainingProject(name="my-training-project", job=job) [​](https://docs.baseten.co/training/loading#using-checkpoints-in-your-training-code) Using checkpoints in your training code -------------------------------------------------------------------------------------------------------------------------------- Access loaded checkpoints using the `BT_LOAD_CHECKPOINT_DIR` environment variable: from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer from transformers.trainer_utils import get_last_checkpoint import os def train(): checkpoint_dir = os.environ.get("BT_LOAD_CHECKPOINT_DIR") last_checkpoint = None if checkpoint_dir: last_checkpoint = get_last_checkpoint(checkpoint_dir) if last_checkpoint: print(f"✅ Resuming from checkpoint: {last_checkpoint}") model = AutoModelForSequenceClassification.from_pretrained(last_checkpoint) tokenizer = AutoTokenizer.from_pretrained(checkpoint_dir) else: print("⚠️ No checkpoint found, starting from scratch") model = AutoModelForSequenceClassification.from_pretrained("your-base-model") tokenizer = AutoTokenizer.from_pretrained("your-base-model") else: print("ℹ️ No checkpoint loading configured") model = AutoModelForSequenceClassification.from_pretrained("your-base-model") tokenizer = AutoTokenizer.from_pretrained("your-base-model") training_args = TrainingArguments( output_dir=os.environ.get("BT_CHECKPOINT_DIR", "/tmp/training_checkpoints"), save_strategy="steps", save_steps=1000, load_best_model_at_end=True, ) trainer = Trainer(model=model, args=training_args) trainer.train(resume_from_checkpoint=last_checkpoint) Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/interactive-sessions) [Serving your trained modelHow to deploy checkpoints from Baseten Training jobs as usable models.\ \ Next](https://docs.baseten.co/training/deployment) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Get started - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/getting-started#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten Training runs your training code on managed cloud GPUs. You bring your own framework, point it at a GPU type, and submit. Baseten handles provisioning, syncs checkpoints as they’re saved, and deploys any checkpoint as a production endpoint in one command. This tutorial fine-tunes Qwen3-4B with LoRA on a single H100, from job submission to calling the deployed model. You’ll set up a project directory, define your infrastructure in a configuration file, and write the training scripts that run on an H100. [​](https://docs.baseten.co/training/getting-started#prerequisites) Prerequisites ------------------------------------------------------------------------------------ * **Baseten account**: [Sign up for Baseten](https://app.baseten.co/) . * **API key**: Generate an API key from [Settings > API keys](https://app.baseten.co/settings/account/api_keys) . * **[uv](https://docs.astral.sh/uv/) **: This guide uses `uvx` to run [Truss](https://pypi.org/project/truss/) commands without a separate install step. Log in to Baseten: uvx truss login [​](https://docs.baseten.co/training/getting-started#create-your-training-project) Create your training project ------------------------------------------------------------------------------------------------------------------ mkdir my-training-project && cd my-training-project ### [​](https://docs.baseten.co/training/getting-started#write-your-configuration-file) Write your configuration file Your configuration file uses the `truss_train` library to define your training infrastructure as Python objects: * [`TrainingProject`](https://docs.baseten.co/reference/sdk/training#trainingproject) : the top-level container for your project. * [`TrainingJob`](https://docs.baseten.co/reference/sdk/training#trainingjob) : a single job within a project, combining: * [`Image`](https://docs.baseten.co/reference/sdk/training#image) : what container to run. * [`Compute`](https://docs.baseten.co/reference/sdk/training#compute) : what hardware to provision. * [`Runtime`](https://docs.baseten.co/reference/sdk/training#runtime) : how to start training and what to persist. This is the file Baseten reads when you submit a job. It tells the platform which GPU to provision, which container image to use, and where to sync checkpoints. Create `config.py`: config.py from truss_train import ( TrainingProject, TrainingJob, Image, Compute, Runtime, CacheConfig, CheckpointingConfig, ) from truss.base.truss_config import AcceleratorSpec BASE_IMAGE = "pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime" training_runtime = Runtime( start_commands=[\ "chmod +x ./run.sh && ./run.sh",\ ], cache_config=CacheConfig(enabled=True), checkpointing_config=CheckpointingConfig(enabled=True), ) training_compute = Compute( accelerator=AcceleratorSpec(accelerator="H100", count=1), ) training_job = TrainingJob( image=Image(base_image=BASE_IMAGE), compute=training_compute, runtime=training_runtime, ) training_project = TrainingProject( name="qwen3-4b-lora-sft", job=training_job, ) `CacheConfig` avoids re-downloading models and datasets between jobs. `CheckpointingConfig` tells Baseten to sync your saved checkpoints so you can deploy them later. ### [​](https://docs.baseten.co/training/getting-started#write-your-training-scripts) Write your training scripts Create `run.sh` to install dependencies and launch training. This tutorial uses `pip install` in the start command, but you can also pre-install dependencies in a [custom base image](https://docs.baseten.co/training/concepts/basics#base-images) . run.sh #!/bin/bash set -eux pip install "trl>=0.20.0" "peft>=0.17.0" "transformers>=4.55.0" "datasets" python train.py Your `train.py` is your own training code. Baseten runs it as-is, so you can use any framework or training loop that works locally. In this example, we’ll fine-tune [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) on the [pirate-ultrachat-10k](https://huggingface.co/datasets/winglian/pirate-ultrachat-10k) dataset using LoRA with [TRL](https://huggingface.co/docs/trl) (Transformer Reinforcement Learning). The dataset teaches the model to respond in pirate dialect, so you’ll know fine-tuning worked when the deployed model starts saying “Ahoy, matey!” train.py import os from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM import torch from peft import LoraConfig from trl import SFTConfig, SFTTrainer MODEL_ID = "Qwen/Qwen3-4B" DATASET_ID = "winglian/pirate-ultrachat-10k" dataset = load_dataset(DATASET_ID, split="train") tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token model = AutoModelForCausalLM.from_pretrained( MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto", use_cache=False, ) peft_config = LoraConfig( r=8, lora_alpha=16, target_modules="all-linear", lora_dropout=0.05, task_type="CAUSAL_LM", ) training_args = SFTConfig( learning_rate=2e-4, num_train_epochs=1, max_steps=50, logging_steps=5, per_device_train_batch_size=4, gradient_accumulation_steps=4, gradient_checkpointing=True, max_length=1024, warmup_ratio=0.1, lr_scheduler_type="cosine", save_steps=25, bf16=True, output_dir=os.getenv("BT_CHECKPOINT_DIR", "./checkpoints"), ) trainer = SFTTrainer( model=model, args=training_args, train_dataset=dataset, processing_class=tokenizer, peft_config=peft_config, ) trainer.train() trainer.save_model(training_args.output_dir) print(f"Training complete. Model saved to {training_args.output_dir}") Save checkpoints to `$BT_CHECKPOINT_DIR` so Baseten can sync and deploy them. Baseten sets this variable automatically when checkpointing is enabled. With `save_steps=25` and `max_steps=50`, the trainer saves LoRA checkpoints at steps 25 and 50. [​](https://docs.baseten.co/training/getting-started#submit-your-training-job) Submit your training job ---------------------------------------------------------------------------------------------------------- Now that your project is set up, submit your training job. The CLI packages your files, creates the training project, and starts the job on your specified GPU. uvx truss train push config.py You’ll see: ✨ Training job successfully created! 🪵 View logs for your job via 'truss train logs --job-id --tail' 🔍 View metrics for your job via 'truss train metrics --job-id ' 🌐 View job in the UI: https://app.baseten.co/training//logs/ Copy the `job_id` to use in the next steps. [​](https://docs.baseten.co/training/getting-started#monitor-your-training-job) Monitor your training job ------------------------------------------------------------------------------------------------------------ Tail logs in real time with the job ID from the previous step. uvx truss train logs --job-id --tail You can also view logs, metrics, and job status in the [Baseten dashboard](https://app.baseten.co/training/) . [​](https://docs.baseten.co/training/getting-started#deploy-your-trained-model) Deploy your trained model ------------------------------------------------------------------------------------------------------------ When training finishes, Baseten syncs your checkpoints automatically. You’ll see: Training complete. Model saved to /mnt/ckpts Job has exited. Syncing checkpoints... * CLI * Dashboard Deploy your checkpoint to Baseten’s inference platform. The deployment downloads the base model weights and serves them with your LoRA adapter using vLLM. This step requires `hf_access_token` in [Baseten Secrets](https://docs.baseten.co/organization/secrets) because the serving layer downloads the base model separately. uvx truss train deploy_checkpoints Follow the interactive prompts to select a checkpoint, name your model, and choose a GPU. Fetching checkpoints for training job ... ? Use spacebar to select/deselect checkpoints to deploy. ○ . ○ checkpoint-50 ❯ ○ checkpoint-25 ? Enter the model name for your deployment: my-fine-tuned-model ? Select the GPU type to use for deployment: H100 ? Select the number of H100 GPUs to use for deployment: 1 ? Enter the huggingface secret name: hf_access_token Successfully created model version: deployment-1 Model version ID: Deploy from the [Baseten dashboard](https://app.baseten.co/training/) : 1. Select your training job. 2. Open the **Checkpoints** tab and choose a checkpoint. 3. Click **Deploy** and configure your model name, instance type, and scaling settings. ### [​](https://docs.baseten.co/training/getting-started#test-your-deployment) Test your deployment Call your deployed model using the OpenAI-compatible chat format. The `model` field matches the checkpoint name you selected during deployment. * cURL * Python * CLI export BASETEN_API_KEY="paste-your-api-key-here" curl -X POST https://model-.api.baseten.co/v1/chat/completions \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "checkpoint-25", "messages": [{"role": "user", "content": "What is the best way to learn Python programming?"}]}' import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-.api.baseten.co/environments/production/sync/v1" ) response = client.chat.completions.create( model="checkpoint-25", messages=[{"role": "user", "content": "What is the best way to learn Python programming?"}], ) print(response.choices[0].message.content) uvx truss predict --model --data '{"model": "checkpoint-25", "messages": [{"role": "user", "content": "What is the best way to learn Python programming?"}]}' The fine-tuned model responds in pirate dialect, confirming that the LoRA adapter is active: Ahoy there matey! Seeking knowledge of Python programming? Well, it's a treasure trove, but it takes patience and practice to find the gold... [​](https://docs.baseten.co/training/getting-started#next-steps) Next steps ------------------------------------------------------------------------------ * [Monitor and manage training jobs](https://docs.baseten.co/training/management) : for logs, metrics, and job lifecycle commands. * [Training SDK reference](https://docs.baseten.co/reference/sdk/training) : for all configuration options, including [base images](https://docs.baseten.co/reference/sdk/training#image) , [secrets](https://docs.baseten.co/reference/sdk/training#secretreference) , [private registries](https://docs.baseten.co/reference/sdk/training#dockerauth) , and [`.truss_ignore` syntax](https://docs.baseten.co/reference/cli/training/training-cli#ignoring-files-and-folders) . * Browse the [ML Cookbook](https://github.com/basetenlabs/ml-cookbook) : for framework examples and [advanced recipes](https://github.com/basetenlabs/ml-cookbook/tree/main/recipes) . Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/overview) [Building blocksLearn how to get up and running on Baseten Training\ \ Next](https://docs.baseten.co/training/concepts/basics) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Training on Baseten - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/overview#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten provides a flexible training platform that enables you to bring your own training scripts, use the latest training techniques, and fine-tune the newest models. Train models and serve them in production, all on one platform. Baseten automatically stores your checkpoints during training and makes them ready for deployment. You don’t need to download weights, re-upload them, or manage separate infrastructure. Your fine-tuned model goes from checkpoint to production endpoint in a single command. The core workflow requires just two commands: # Train your model truss train push config.py # Deploy from the checkpoint truss train deploy_checkpoints --job-id [​](https://docs.baseten.co/training/overview#choosing-between-truss-train-and-loops) Choosing between Truss Train and Loops ------------------------------------------------------------------------------------------------------------------------------- Baseten supports two training paths. **Truss Train** is the bring-your-own-container path documented on the rest of this page: package any training framework as a Truss, configure hardware, and run it on managed infrastructure. **[Loops](https://docs.baseten.co/loops/overview) ** is a Tinker-compatible managed path for fine-tuning and RL: write `import tinker` Python, and Baseten provisions a paired trainer and sampling server with weights that move live between them. If your team already uses Tinker, or wants SFT and RL without managing a training container, start with Loops. | | [Loops](https://docs.baseten.co/loops/overview) | `truss train` | | --- | --- | --- | | Training code | Tinker-compatible Python (`import tinker`) | Any container image | | Infrastructure setup | None. Baseten provisions trainer and sampling servers. | You define hardware in a Truss config | | Checkpoint format | Paginated presigned URLs, streamed to sampling server live | Your container’s output artifacts | | Inference path | Automatic: checkpoint deploys directly to a Baseten inference deployment | Manual: you move artifacts and deploy separately | | Session lifetime | Open until you call `truss loops deactivate` | Job runs to completion and exits | | Documentation | [Loops](https://docs.baseten.co/loops/overview) | This section | [​](https://docs.baseten.co/training/overview#train-and-serve-on-one-platform) Train and serve on one platform ----------------------------------------------------------------------------------------------------------------- The train-to-serve workflow is seamless: 1. **Set up your training project:** Bring any framework or start with a template. 2. **Configure your training job:** Define compute, runtime, and checkpointing settings. 3. **Run on managed infrastructure:** Use H200 or H100 GPUs, single-node or multi-node. 4. **Checkpoints sync automatically:** Baseten stores checkpoints as training progresses. 5. **Deploy your fine-tuned model:** Go from checkpoint to production endpoint in one command. Baseten handles infrastructure management and file transfers. Bring any framework (Axolotl, TRL, VeRL, Megatron, or your own training code) and your trained model serves traffic within minutes of training completion. [​](https://docs.baseten.co/training/overview#supported-frameworks) Supported frameworks ------------------------------------------------------------------------------------------- Baseten Training is framework-agnostic. Use whatever framework fits your workflow. | Framework | Best for | Example | | --- | --- | --- | | Axolotl | Configuration-driven fine-tuning with LoRA/QLoRA | [oss-gpt-20b-axolotl](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/oss-gpt-20b-axolotl) | | TRL | SFT, DPO, and GRPO with Hugging Face | [oss-gpt-20b-lora-trl](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/oss-gpt-20b-lora-trl) | | TRL | LoRA DPO fine-tuning | [qwen3-8b-lora-dpo-trl](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/qwen3-8b-lora-dpo-trl) | | VeRL | Reinforcement learning with custom rewards | [qwen3-8b-lora-verl](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/qwen3-8b-lora-verl) | | MS-Swift | Long-context and multilingual training | [qwen3-30b-mswift-multinode](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/qwen3-30b-mswift-multinode) | Browse the [ML Cookbook](https://github.com/basetenlabs/ml-cookbook) for more examples including multi-node training with FSDP and DeepSpeed. [​](https://docs.baseten.co/training/overview#key-features) Key features --------------------------------------------------------------------------- ### [​](https://docs.baseten.co/training/overview#checkpoint-management) Checkpoint management Checkpoints sync automatically to Baseten storage during training. You can: * **Deploy** any checkpoint as a production endpoint with [`truss train deploy_checkpoints`](https://docs.baseten.co/training/deployment) . * **Download** checkpoints for local evaluation and analysis. * **Resume** from any checkpoint if a job fails or you want to train further. Learn more about [checkpointing](https://docs.baseten.co/training/concepts/checkpoints) . ### [​](https://docs.baseten.co/training/overview#bdn-weight-and-data-loading) BDN weight and data loading Load model weights and training data through [Baseten Delivery Network (BDN)](https://docs.baseten.co/training/concepts/storage#load-weights-and-data-with-bdn) . Mount weights from Hugging Face, S3, GCS, Azure, R2, or any HTTPS URL directly into your training container with no download code needed. BDN mirrors weights before compute is provisioned, then caches them for faster mounting on subsequent jobs. See [storage and data ingestion](https://docs.baseten.co/training/concepts/storage) for setup details. ### [​](https://docs.baseten.co/training/overview#persistent-caching) Persistent caching Speed up training iterations by caching models, datasets, and preprocessed data between jobs. The cache persists across training runs, so you don’t re-download 70B models every time. See the [training cache](https://docs.baseten.co/training/concepts/cache) guide for configuration options. ### [​](https://docs.baseten.co/training/overview#multi-node-training) Multi-node training Scale training across multiple GPU nodes with InfiniBand networking. Baseten handles node orchestration, communication setup, and environment variables. You just set `node_count` in your configuration. Learn more about [multi-node training](https://docs.baseten.co/training/concepts/multinode) . ### [​](https://docs.baseten.co/training/overview#remote-access) Remote access Connect to running training containers to debug, inspect state, and iterate without resubmitting. Baseten offers two options: * **[SSH](https://docs.baseten.co/training/ssh) **: Connect from any OpenSSH client for terminal sessions and file transfer with `scp` or `sftp`. * **[VS Code & Cursor](https://docs.baseten.co/training/interactive-sessions) **: Connect from VS Code or Cursor Remote Tunnels for a full IDE experience. See the [Remote access overview](https://docs.baseten.co/training/remote-access) to choose between them. [​](https://docs.baseten.co/training/overview#next-steps) Next steps ----------------------------------------------------------------------- Get started ----------- Run your first training job and deploy the result. Loops ----- Tinker-compatible managed training with paired trainer and sampling servers. ML Cookbook ----------- Production-ready examples for frameworks and models. [​](https://docs.baseten.co/training/overview#reference) Reference --------------------------------------------------------------------- * [CLI reference](https://docs.baseten.co/reference/cli/training/training-cli) * [SDK reference](https://docs.baseten.co/reference/sdk/training) * [API reference](https://docs.baseten.co/reference/training-api/overview) Was this page helpful? YesNo [Previous](https://docs.baseten.co/frontier-gateway/calling-your-model) [Get startedRun your first training job and deploy it to production.\ \ Next](https://docs.baseten.co/training/getting-started) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # External packages - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/development/model/external-packages#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Truss allows you to include custom modules or third-party packages **not available on PyPi** using two methods: 1. **The `packages` directory**: For bundling small, Truss-specific packages. 2. **The `external_package_dirs` configuration**: For sharing packages across multiple Trusses. [​](https://docs.baseten.co/development/model/external-packages#1-using-the-packages-directory) 1\. Using the `packages` directory ------------------------------------------------------------------------------------------------------------------------------------- Each Truss includes a `packages/` directory where you can place Python modules to be included at build time. **Example directory structure:** stable-diffusion/ packages/ package_1/ subpackage/ script.py package_2/ utils.py model/ model.py __init__.py config.yaml **Importing a package in `model.py`:** model.py from package_1.subpackage.script import run_script from package_2.utils import RandomClass class Model: def __init__(self, **kwargs): self.random_class = RandomClass() def load(self): run_script() Use this method for lightweight, Truss-specific packages. [​](https://docs.baseten.co/development/model/external-packages#2-using-external_package_dirs) 2\. Using `external_package_dirs` ----------------------------------------------------------------------------------------------------------------------------------- If multiple Trusses need access to the same external package, define `external_package_dirs` in `config.yaml`: Note: Package here refers to an importable directory with Python source code. **Example directory structure:** stable-diffusion/ model/ model.py __init__.py config.yaml super_cool_awesome_plugin/ plugin1/ script.py plugin2/ run.py **Configuring external\_package\_dirs in config.yaml:** config.yaml external_package_dirs: - ../super_cool_awesome_plugin/ Paths must be relative to `config.yaml`. Note: Be sure to include any requirements for these packages in your truss configuration. **Referencing external packages in `model.py`:** model.py from plugin1.script import cool_constant from plugin2.run import AwesomeRunner class Model: def __init__(self, **kwargs): self.awesome_runner = AwesomeRunner() def load(self): self.awesome_runner.run(cool_constant) Was this page helpful? YesNo ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Overview - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/engines/index#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Baseten engines optimize model inference for specific architectures using [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) . All engines mirror build artifacts to the [Baseten Delivery Network](https://docs.baseten.co/development/model/bdn) automatically. * **[BEI](https://docs.baseten.co/engines/bei/overview) :** Embedding, reranking, and classification models on causal architectures with `FP8` and `FP4` quantization. * **[BEI-Bert](https://docs.baseten.co/engines/bei/bei-bert) :** Bidirectional BEI variant tuned for BERT-family encoders and cold-start-sensitive models under 4B parameters. * **[Engine-Builder-LLM](https://docs.baseten.co/engines/engine-builder-llm/overview) :** Dense text generation for Llama, Qwen, Mistral, and Gemma with lookahead decoding and multi-LoRA support. * **[BIS-LLM](https://docs.baseten.co/engines/bis-llm/overview) :** MoE and Enterprise serving with KV-aware routing, disaggregated prefill/decode, and Eagle/MTP speculation. [​](https://docs.baseten.co/engines/index#choose-an-engine) Choose an engine ------------------------------------------------------------------------------- Pick the row below that matches what you’re deploying. Cost, quality, and latency targets drive later choices (GPU, quantization, autoscaling) inside that engine. * **Embedding, reranking, classification, or NER models:** use [BEI](https://docs.baseten.co/engines/bei/overview) for decoder embedders (`Qwen3-Embedding`, `BAAI/bge`, `LlamaForSequenceClassification`) or [BEI-Bert](https://docs.baseten.co/engines/bei/bei-bert) for BERT-family encoders (`BERT`, `ModernBERT`, `EuroBERT`, `XLM-RoBERTa`). NER lives on [`BEI-Bert /predict_tokens`](https://docs.baseten.co/engines/bei/ner) . * **Dense text-generation LLMs** (`Llama 3` or `4`, `Qwen 3` or `3.5`, `Mistral`, `Gemma`, `Phi`, `GPT-OSS-20B`): use [Engine-Builder-LLM](https://docs.baseten.co/engines/engine-builder-llm/overview) , with [lookahead decoding](https://docs.baseten.co/engines/engine-builder-llm/lookahead-decoding) and [multi-LoRA](https://docs.baseten.co/engines/engine-builder-llm/lora-support) available. * **MoE models** (`GLM 5.x`, `Kimi K2.5` or `K2.6`, `DeepSeek V3`, `R1`, or `V4`, `MiniMax 2.5`, `Qwen3 MoE`, `GPT-OSS-120B`) **or workloads that need KV-cache-aware routing or disaggregated prefill/decode:** use [BIS-LLM](https://docs.baseten.co/engines/bis-llm/overview) . Currently a co-engineering pilot. * **Speech, image, video, or custom Python models:** ship a custom Truss. Browse [model examples](https://docs.baseten.co/examples/overview) for Whisper, Orpheus, Flux, and other pre-built deployments, or see [build your first model](https://docs.baseten.co/development/model/build-your-first-model) for custom inference logic. If your workload doesn’t fit one of the rows above (custom architectures, hybrid pipelines, BIS-LLM pilot access, sizing for unusual traffic shapes), email [support@baseten.co](mailto:support@baseten.co) and an engineer will route you. [​](https://docs.baseten.co/engines/index#performance-and-operations) Performance and operations --------------------------------------------------------------------------------------------------- * [Quantization guide](https://docs.baseten.co/engines/performance-concepts/quantization-guide) : `FP8` and `FP4` trade-offs, GPU support, and per-engine options. * [Autoscaling engines](https://docs.baseten.co/engines/performance-concepts/autoscaling-engines) : Token-based and request-based scaling for engine deployments. * [Cloud storage deployment](https://docs.baseten.co/engines/performance-concepts/cloud-storage-deployment) : Deploy engines from S3 or GCS instead of Hugging Face. * [Specialized model examples](https://docs.baseten.co/examples/overview) : Pre-built Truss examples for Whisper, Orpheus, Flux, and other dedicated deployments. [​](https://docs.baseten.co/engines/index#compare-engines) Compare engines ----------------------------------------------------------------------------- | Feature | BIS-LLM | Engine-Builder-LLM | BEI | BEI-Bert | Notes | | --- | --- | --- | --- | --- | --- | | **Quantization** | ✅ | ✅ | ✅ | ❌ | BEI-Bert: `FP16`/`BF16` only. | | **KV quantization** | ✅ | ✅ | ⚠️ | ⚠️ | `FP8_KV`, `FP4_KV` supported. | | **Lookahead decoding** | ❌ | ✅ | ❌ | ❌ | Engine-Builder-LLM (v1) only; BIS-LLM uses MTP/Eagle/N-gram speculation instead. | | **Self-serviceable** | 🔒 | ✅ | ✅ | ✅ | BIS-LLM requires Enterprise; other engines are self-serve. | | **KV-routing** | 🔒 | ❌ | ❌ | ❌ | BIS-LLM only. | | **Disaggregated serving** | 🔒 | ❌ | ❌ | ❌ | BIS-LLM Enterprise. | | **Tool calling & structured output** | ✅ | ✅ | ❌ | ❌ | Function calling support. | | **Classification models** | ❌ | ❌ | ✅ | ✅ | Sequence classification. | | **Embedding models** | ❌ | ❌ | ✅ | ✅ | Embedding generation. | | **Mixture-of-experts** | ✅ | ⚠️ (Qwen3MoE only) | ❌ | ❌ | MoE models like `DeepSeek-R1`. | | **MTP / Eagle / N-gram speculation** | 🔒 | ❌ | ❌ | ❌ | v2 speculative decoding via `speculative_config`. | | **HTTP request cancellation** | ✅ | ⚠️ | ✅ | ✅ | Engine-Builder-LLM: within the first 10ms only. | | **MultiModal Inputs** | 🔒 | ❌ | ⚠️ | ❌ | Selected architectures only. | Was this page helpful? YesNo [Previous](https://docs.baseten.co/deployment/ci-cd) [OverviewProduction-grade embeddings, reranking, and classification models\ \ Next](https://docs.baseten.co/engines/bei/overview) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Training SDK - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/sdk/training#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/reference/sdk/training#installation) Installation -------------------------------------------------------------------------------- Truss includes the training SDK: * uv (recommended) * pip (macOS/Linux) * pip (Windows) [uv](https://docs.astral.sh/uv/) is a fast Python package manager. Create a virtual environment and install Truss: uv venv && source .venv/bin/activate uv pip install truss Create a virtual environment and install Truss with pip: python3 -m venv .venv && source .venv/bin/activate pip install --upgrade truss Create a virtual environment and install Truss with pip: python3 -m venv .venv && .venv\Scripts\activate pip install --upgrade truss Define your training job in a configuration file (typically `config.py`). Import the SDK and accelerator config: config.py from truss_train import definitions from truss.base import truss_config You can also import classes directly from `truss_train` (for example, `from truss_train import Compute, Runtime`). * * * [​](https://docs.baseten.co/reference/sdk/training#complete-example) Complete example ---------------------------------------------------------------------------------------- Copy this `config.py` as a starting point for your training project. It configures [caching](https://docs.baseten.co/training/concepts/cache) to persist pip packages between jobs, [checkpointing](https://docs.baseten.co/training/concepts/checkpoints) to save model weights, and GPU compute on a single H200 node. Modify the `start_commands`, `environment_variables`, and `accelerator` fields for your use case. For more examples, see [ml-cookbook](https://github.com/basetenlabs/ml-cookbook/tree/main/examples) . config.py from truss_train import definitions from truss.base import truss_config # The Docker image your training code runs in. BASE_IMAGE = "pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime" # Runtime controls what happens when the container starts: which commands # run, which secrets are injected, and whether caching and checkpointing # are enabled. training_runtime = definitions.Runtime( start_commands=[\ "pip install transformers datasets accelerate",\ "torchrun --nproc-per-node=2 train.py",\ ], environment_variables={ "HF_TOKEN": definitions.SecretReference(name="hf_access_token"), "WANDB_API_KEY": definitions.SecretReference(name="wandb_api_key"), }, # Cache persists pip packages and downloaded models between jobs. cache_config=definitions.CacheConfig(enabled=True), # Checkpointing writes model weights to $BT_CHECKPOINT_DIR for # deployment or resuming later. checkpointing_config=definitions.CheckpointingConfig(enabled=True), ) # Compute defines the hardware allocated to each node. training_compute = definitions.Compute( node_count=1, accelerator=truss_config.AcceleratorSpec( accelerator=truss_config.Accelerator.H200, count=2, ), ) # TrainingJob combines the image, compute, and runtime into a single # unit that Baseten provisions and runs. training_job = definitions.TrainingJob( image=definitions.Image(base_image=BASE_IMAGE), compute=training_compute, runtime=training_runtime, ) # TrainingProject groups related jobs under one name. Pushing this # config creates the project (or reuses it) and submits a new job. training_project = definitions.TrainingProject( name="llm-fine-tuning", job=training_job, ) * * * [​](https://docs.baseten.co/reference/sdk/training#push) push ---------------------------------------------------------------- Submits a training job to Baseten. Every config you define with the classes below does nothing until you call `push()`. When you call `push()`, Baseten: 1. Authenticates with your Baseten account. 2. Creates the [training project](https://docs.baseten.co/training/overview) if one with the given name doesn’t already exist, or reuses the existing project. 3. Archives your source directory (your training script, data files, and any other local files) and uploads it. 4. Submits a new training job. Baseten provisions the hardware, pulls the container image, mounts any [BDN weights](https://docs.baseten.co/reference/sdk/training#weightssource) , extracts your source files into the container, and runs your [start\_commands](https://docs.baseten.co/reference/sdk/training#runtime) . The job then progresses through the [training lifecycle](https://docs.baseten.co/training/lifecycle) : * `CREATED`: Baseten has received the training configuration. * `DEPLOYING`: Baseten is provisioning compute resources and installing dependencies. * `RUNNING`: Your training code is actively executing. * `COMPLETED`: The job has finished. Checkpoints and artifacts have been saved. * `DEPLOY_FAILED`: The job failed to deploy, likely due to a bad image or resource allocation issue. * `FAILED`: The job encountered an error. Check the logs for details. * `STOPPED`: The job was manually stopped. The CLI command `uvx truss train push config.py` performs the same steps with additional options for team selection and flag overrides. The `push` function accepts either a file path or a `TrainingProject` object. config.py from truss_train import push # Pass a config file path: def push( config: Path, *, remote: str = "baseten", ) -> dict # Pass a TrainingProject object: def push( config: TrainingProject, *, remote: str = "baseten", source_dir: Optional[Path] = None, ) -> dict ### [​](https://docs.baseten.co/reference/sdk/training#parameters) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-config) config Path | TrainingProject required Path to a `config.py` file or a [TrainingProject](https://docs.baseten.co/reference/sdk/training#trainingproject) instance. When you pass a `Path`, Baseten imports the module and scans for an instance of `TrainingProject`. The module must contain exactly one. [​](https://docs.baseten.co/reference/sdk/training#param-remote) remote string Remote provider to push to. Defaults to `baseten`. [​](https://docs.baseten.co/reference/sdk/training#param-source-dir) source\_dir Path Root directory whose contents Baseten uploads as the job’s working directory. Baseten archives this directory and extracts it into the container before running [start\_commands](https://docs.baseten.co/reference/sdk/training#runtime) . Only applies when `config` is a `TrainingProject`. Defaults to the current directory. ### [​](https://docs.baseten.co/reference/sdk/training#return-value) Return value Returns a dictionary containing the created training job. Use the `id` and `training_project.id` values to monitor the job, stream logs, and list checkpoints. Output { "id": "gvpql31", "training_project_id": "aghi527", "training_project": { "id": "aghi527", "name": "llm-fine-tuning" }, "current_status": "TRAINING_JOB_CREATED", "instance_type": { ... }, "name": "fine-tune-v1", ... } For example, to submit a training job programmatically, pass a `TrainingProject` object to `push()`: submit\_job.py from pathlib import Path from truss.base import truss_config from truss_train import push, definitions project = definitions.TrainingProject( name="llm-fine-tuning", job=definitions.TrainingJob( image=definitions.Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=definitions.Compute( accelerator=truss_config.AcceleratorSpec( accelerator=truss_config.Accelerator.H200, count=2, ) ), runtime=definitions.Runtime( start_commands=["python train.py"], environment_variables={ "HF_TOKEN": definitions.SecretReference(name="hf_access_token"), }, ), ), ) result = push(config=project, source_dir=Path("./training")) print(f"Project ID: {result['training_project']['id']}") print(f"Job ID: {result['id']}") Output Project ID: aghi527 Job ID: gvpql31 ### [​](https://docs.baseten.co/reference/sdk/training#after-submitting) After submitting Once `push()` returns, Baseten queues your job and begins provisioning. Use the returned job ID to track progress: * **Stream logs:** `uvx truss train logs --job-id --tail` * **Check status:** `uvx truss train view --job-id ` * **List checkpoints:** Use the [get training job checkpoints](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints) API. * **Deploy a checkpoint:** For more information, see [deploy checkpoints](https://docs.baseten.co/reference/sdk/training#deploy-checkpoints) . For a complete working example, see the [programmatic training API recipe](https://github.com/basetenlabs/ml-cookbook/tree/main/recipes/programmatic-training-api) . For `config.py`\-based submission with the CLI, see the [training getting started guide](https://docs.baseten.co/training/getting-started) . * * * [​](https://docs.baseten.co/reference/sdk/training#trainingproject) TrainingProject -------------------------------------------------------------------------------------- Groups related training jobs under a single named project. When you [push](https://docs.baseten.co/reference/sdk/training#push) a `TrainingProject`, Baseten creates the project if it doesn’t exist, then submits the attached [TrainingJob](https://docs.baseten.co/reference/sdk/training#trainingjob) . All jobs in a project share the same [project-level cache](https://docs.baseten.co/training/concepts/cache) and appear together in the dashboard. config.py from truss_train import definitions project = definitions.TrainingProject( name="llm-fine-tuning", job=training_job, team_name="my-team", ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-2) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-name) name string required Project name. Reusing a name adds jobs to the existing project. [​](https://docs.baseten.co/reference/sdk/training#param-job) job TrainingJob required Training job to submit. Defines the container image, compute resources, runtime commands, and optional weights. For more information, see [TrainingJob](https://docs.baseten.co/reference/sdk/training#trainingjob) . [​](https://docs.baseten.co/reference/sdk/training#param-team-name) team\_name string Team that owns this project. Controls access and team-level cache scope. [​](https://docs.baseten.co/reference/sdk/training#trainingjob) TrainingJob ------------------------------------------------------------------------------ Represents a single training run. Baseten provisions the hardware specified in [Compute](https://docs.baseten.co/reference/sdk/training#compute) , pulls the container [Image](https://docs.baseten.co/reference/sdk/training#image) , uploads your source directory, mounts any [WeightsSource](https://docs.baseten.co/reference/sdk/training#weightssource) volumes, then executes the [Runtime](https://docs.baseten.co/reference/sdk/training#runtime) start commands. For more information, see the [training lifecycle](https://docs.baseten.co/training/lifecycle) . config.py from truss_train import definitions, WeightsSource from truss.base import truss_config training_job = definitions.TrainingJob( name="fine-tune-v1", image=definitions.Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=definitions.Compute( accelerator=truss_config.AcceleratorSpec( accelerator=truss_config.Accelerator.H200, count=4, ) ), runtime=definitions.Runtime( start_commands=["chmod +x ./run.sh && ./run.sh"], checkpointing_config=definitions.CheckpointingConfig(enabled=True), cache_config=definitions.CacheConfig(enabled=True), ), weights=[\ WeightsSource(\ source="hf://meta-llama/Llama-3.1-8B@main",\ mount_location="/app/models/llama",\ ),\ ], ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-3) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-image) image Image required Docker image that provides the training environment, including the OS, CUDA drivers, and pre-installed libraries. For more information, see [Image](https://docs.baseten.co/reference/sdk/training#image) . [​](https://docs.baseten.co/reference/sdk/training#param-compute) compute Compute Hardware allocation for each node. Set the GPU type and count via `accelerator`, and increase `node_count` for distributed training. Defaults to `Compute()`. For more information, see [Compute](https://docs.baseten.co/reference/sdk/training#compute) . [​](https://docs.baseten.co/reference/sdk/training#param-runtime) runtime Runtime Controls container startup: shell commands to execute, environment variables to inject, and whether to enable caching or checkpointing. Defaults to `Runtime()`. For more information, see [Runtime](https://docs.baseten.co/reference/sdk/training#runtime) . [​](https://docs.baseten.co/reference/sdk/training#param-name-1) name string Display name for this job in the dashboard and API responses. [​](https://docs.baseten.co/reference/sdk/training#param-interactive-session) interactive\_session InteractiveSession Opens a remote tunnel so you can attach VS Code or Cursor to the running container for live debugging. For more information, see [InteractiveSession](https://docs.baseten.co/reference/sdk/training#interactivesession) . [​](https://docs.baseten.co/reference/sdk/training#param-workspace) workspace Workspace Controls which local files Baseten uploads to the container. Use this to exclude large directories, include files from outside the root, or change the root entirely. For more information, see [Workspace](https://docs.baseten.co/reference/sdk/training#workspace) . [​](https://docs.baseten.co/reference/sdk/training#param-weights) weights WeightsSource\[\] default:\[\] Model weights that BDN mirrors and mounts read-only in the container. Supports Hugging Face, S3, GCS, Azure, R2, and direct URLs. For more information, see [WeightsSource](https://docs.baseten.co/reference/sdk/training#weightssource) . [​](https://docs.baseten.co/reference/sdk/training#param-enable-baseten-workdir) enable\_baseten\_workdir bool default:true When `True` (default), Baseten uses `/b10/workspace` as the container’s working directory, extracting your source files and base image working directory contents there. The `BT_WORKING_DIR` environment variable points to `/b10/workspace`. When `False`, the Docker image’s default working directory is used and `BT_WORKING_DIR` is unset. [​](https://docs.baseten.co/reference/sdk/training#weightssource) WeightsSource ---------------------------------------------------------------------------------- Mounts pre-trained model weights into the training container as a read-only volume. Baseten mirrors the weights through [BDN](https://docs.baseten.co/development/model/bdn) before provisioning compute, so the data is on disk before your `start_commands` run. On subsequent jobs, BDN serves the cached copy from a cluster- or node-local cache, which avoids re-downloading. For the full delivery behavior, see [how BDN serves training jobs](https://docs.baseten.co/training/concepts/storage#how-bdn-serves-training-jobs) . * Hugging Face * S3 with auth * File filtering config.py from truss_train import WeightsSource WeightsSource( source="hf://Qwen/Qwen3-0.6B", mount_location="/app/models/Qwen/Qwen3-0.6B", ) config.py from truss_train import WeightsSource WeightsSource( source="s3://my-bucket/training-data", mount_location="/app/data/training-data", auth={"auth_method": "CUSTOM_SECRET", "auth_secret_name": "aws_credentials"}, ) config.py from truss_train import WeightsSource WeightsSource( source="hf://meta-llama/Llama-3.1-8B@main", mount_location="/app/models/llama", allow_patterns=["*.safetensors", "config.json", "tokenizer.*"], ignore_patterns=["*.md", "*.txt"], ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-4) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-source) source string required URI with scheme prefix. | Scheme | Example | Description | | --- | --- | --- | | `hf://` | `hf://meta-llama/Llama-3.1-8B@main` | Hugging Face Hub. | | `s3://` | `s3://my-bucket/path/to/data` | Amazon S3. | | `gs://` | `gs://my-bucket/path/to/data` | Google Cloud Storage. | | `r2://` | `r2://account_id.bucket/path` | Cloudflare R2. | For Hugging Face sources, pin to a specific revision with the `@revision` suffix (branch, tag, or commit SHA). [​](https://docs.baseten.co/reference/sdk/training#param-mount-location) mount\_location string required Absolute path where Baseten mounts the weights in the container. [​](https://docs.baseten.co/reference/sdk/training#param-auth) auth WeightsAuth Authentication configuration. See the [BDN configuration reference](https://docs.baseten.co/development/model/bdn#configuration-reference) . [​](https://docs.baseten.co/reference/sdk/training#param-auth-secret-name) auth\_secret\_name string Baseten secret name for credentials. [​](https://docs.baseten.co/reference/sdk/training#param-allow-patterns) allow\_patterns string\[\] File patterns to include during download. [​](https://docs.baseten.co/reference/sdk/training#param-ignore-patterns) ignore\_patterns string\[\] File patterns to exclude during download. [​](https://docs.baseten.co/reference/sdk/training#image) Image ------------------------------------------------------------------ Sets the Docker image that Baseten pulls to create the training container. The image provides the OS, CUDA drivers, Python version, and any pre-installed libraries your training code needs. Use a public image from Docker Hub or a private image with [DockerAuth](https://docs.baseten.co/reference/sdk/training#dockerauth) . config.py image = definitions.Image( base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime" ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-5) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-base-image) base\_image string required Full Docker image tag, such as `"pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"`. [​](https://docs.baseten.co/reference/sdk/training#param-docker-auth) docker\_auth DockerAuth Credentials for pulling from private registries like AWS ECR or Google Container Registry. Store actual credentials as [Baseten secrets](https://docs.baseten.co/organization/secrets) . For more information, see [DockerAuth](https://docs.baseten.co/reference/sdk/training#dockerauth) . ### [​](https://docs.baseten.co/reference/sdk/training#dockerauth) DockerAuth Provides credentials for pulling images from private Docker registries (AWS ECR, Google Container Registry, etc.). Store the actual credential values as secrets in your [Baseten workspace](https://docs.baseten.co/organization/secrets) and reference them with [SecretReference](https://docs.baseten.co/reference/sdk/training#secretreference) . [​](https://docs.baseten.co/reference/sdk/training#param-auth-method) auth\_method DockerAuthType required Authentication method. [​](https://docs.baseten.co/reference/sdk/training#param-registry) registry string required Docker registry URL. [​](https://docs.baseten.co/reference/sdk/training#param-aws-iam-docker-auth) aws\_iam\_docker\_auth AWSIAMDockerAuth IAM credentials for authenticating with AWS ECR. Requires `access_key_secret_ref` and `secret_access_key_secret_ref`. For more information, see [AWSIAMDockerAuth](https://docs.baseten.co/reference/sdk/training#awsiamdockerauth) . [​](https://docs.baseten.co/reference/sdk/training#param-gcp-service-account-json-docker-auth) gcp\_service\_account\_json\_docker\_auth GCPServiceAccountJSONDockerAuth Service account JSON credentials for authenticating with Google Container Registry. For more information, see [GCPServiceAccountJSONDockerAuth](https://docs.baseten.co/reference/sdk/training#gcpserviceaccountjsondockerauth) . [​](https://docs.baseten.co/reference/sdk/training#param-registry-secret-docker-auth) registry\_secret\_docker\_auth RegistrySecretDockerAuth Username/password credentials for authenticating with registries that support static credentials (Docker Hub, GHCR, NGC). Not compatible with AWS ECR or GCP Artifact Registry. For more information, see [RegistrySecretDockerAuth](https://docs.baseten.co/reference/sdk/training#registrysecretdockerauth) . #### [​](https://docs.baseten.co/reference/sdk/training#awsiamdockerauth) AWSIAMDockerAuth Authenticates with AWS ECR using IAM credentials. config.py from truss.base import truss_config image = definitions.Image( base_image="123456789.dkr.ecr.us-east-1.amazonaws.com/my-image:latest", docker_auth=definitions.DockerAuth( auth_method=truss_config.DockerAuthType.AWS_IAM, registry="123456789.dkr.ecr.us-east-1.amazonaws.com", aws_iam_docker_auth=definitions.AWSIAMDockerAuth( access_key_secret_ref=definitions.SecretReference(name="aws_access_key"), secret_access_key_secret_ref=definitions.SecretReference(name="aws_secret_access_key"), ) ) ) [​](https://docs.baseten.co/reference/sdk/training#param-access-key-secret-ref) access\_key\_secret\_ref SecretReference required AWS access key ID, stored as a [Baseten secret](https://docs.baseten.co/organization/secrets) and referenced by name. [​](https://docs.baseten.co/reference/sdk/training#param-secret-access-key-secret-ref) secret\_access\_key\_secret\_ref SecretReference required AWS secret access key, stored as a [Baseten secret](https://docs.baseten.co/organization/secrets) and referenced by name. #### [​](https://docs.baseten.co/reference/sdk/training#gcpserviceaccountjsondockerauth) GCPServiceAccountJSONDockerAuth Authenticates with Google Container Registry using service account JSON. config.py from truss.base import truss_config image = definitions.Image( base_image="gcr.io/my-project/my-image:latest", docker_auth=definitions.DockerAuth( auth_method=truss_config.DockerAuthType.GCP_SERVICE_ACCOUNT_JSON, registry="gcr.io", gcp_service_account_json_docker_auth=definitions.GCPServiceAccountJSONDockerAuth( service_account_json_secret_ref=definitions.SecretReference(name="gcp_service_account_json"), ) ) ) [​](https://docs.baseten.co/reference/sdk/training#param-service-account-json-secret-ref) service\_account\_json\_secret\_ref SecretReference required GCP service account JSON, stored as a [Baseten secret](https://docs.baseten.co/organization/secrets) and referenced by name. #### [​](https://docs.baseten.co/reference/sdk/training#registrysecretdockerauth) RegistrySecretDockerAuth Authenticates with registries that support static username/password credentials, including Docker Hub, GHCR, and NGC. For AWS ECR or GCP Artifact Registry, use [AWSIAMDockerAuth](https://docs.baseten.co/reference/sdk/training#awsiamdockerauth) or [GCPServiceAccountJSONDockerAuth](https://docs.baseten.co/reference/sdk/training#gcpserviceaccountjsondockerauth) instead. config.py from truss.base import truss_config image = definitions.Image( base_image="your-registry/your-image:latest", docker_auth=definitions.DockerAuth( auth_method=truss_config.DockerAuthType.REGISTRY_SECRET, registry="docker.io", registry_secret_docker_auth=definitions.RegistrySecretDockerAuth( secret_ref=definitions.SecretReference(name="my_docker_cred") ) ) ) [​](https://docs.baseten.co/reference/sdk/training#param-secret-ref) secret\_ref SecretReference required Registry credentials in `username:password` format (plaintext, not Base64-encoded), stored as a [Baseten secret](https://docs.baseten.co/organization/secrets) and referenced by name. [​](https://docs.baseten.co/reference/sdk/training#compute) Compute ---------------------------------------------------------------------- Defines the hardware Baseten allocates for each training job. Set `node_count` above 1 for [multi-node distributed training](https://docs.baseten.co/training/concepts/multinode) , which provisions multiple identical nodes and injects coordination environment variables (`BT_LEADER_ADDR`, `BT_NODE_RANK`, `BT_GROUP_SIZE`). config.py from truss.base import truss_config compute = definitions.Compute( node_count=2, cpu_count=8, memory="64Gi", accelerator=truss_config.AcceleratorSpec( accelerator=truss_config.Accelerator.H200, count=4, ) ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-6) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-node-count) node\_count integer default:1 Number of nodes to provision. Each node gets the full CPU, memory, and GPU allocation. [​](https://docs.baseten.co/reference/sdk/training#param-cpu-count) cpu\_count integer default:1 CPU cores per node. [​](https://docs.baseten.co/reference/sdk/training#param-memory) memory string RAM per node (for example, `"64Gi"`). Defaults to `2Gi`. [​](https://docs.baseten.co/reference/sdk/training#param-accelerator) accelerator AcceleratorSpec GPU type and count per node. For more information, see [AcceleratorSpec](https://docs.baseten.co/reference/sdk/training#acceleratorspec) . ### [​](https://docs.baseten.co/reference/sdk/training#acceleratorspec) AcceleratorSpec Selects the GPU type and count per node. The `count` determines how many GPUs are available to your training script on each node (exposed as `$BT_NUM_GPUS`). [​](https://docs.baseten.co/reference/sdk/training#param-accelerator-1) accelerator Accelerator GPU type.Available options: * `H100`: NVIDIA H100. * `H200`: NVIDIA H200. [​](https://docs.baseten.co/reference/sdk/training#param-count) count integer default:1 Number of GPUs per node. [​](https://docs.baseten.co/reference/sdk/training#runtime) Runtime ---------------------------------------------------------------------- Controls what happens when the training container starts. Baseten executes `start_commands` in order inside the container. Use them to install dependencies, set up data, and launch your training script. Baseten injects environment variables before the first command runs; use [SecretReference](https://docs.baseten.co/reference/sdk/training#secretreference) for sensitive values like API keys so they aren’t stored in your config file. config.py runtime = definitions.Runtime( start_commands=["chmod +x ./run.sh && ./run.sh"], environment_variables={ "BATCH_SIZE": "32", "WANDB_API_KEY": definitions.SecretReference(name="wandb_api_key"), "HF_TOKEN": definitions.SecretReference(name="hf_access_token"), }, checkpointing_config=definitions.CheckpointingConfig(enabled=True), cache_config=definitions.CacheConfig(enabled=True), ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-7) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-start-commands) start\_commands string\[\] default:\[\] Shell commands that Baseten executes sequentially when the container starts. [​](https://docs.baseten.co/reference/sdk/training#param-environment-variables) environment\_variables object Key-value pairs that Baseten injects as env vars. Use [SecretReference](https://docs.baseten.co/reference/sdk/training#secretreference) for sensitive values. [​](https://docs.baseten.co/reference/sdk/training#param-checkpointing-config) checkpointing\_config CheckpointingConfig Enables writing model checkpoints to persistent storage. When enabled, Baseten mounts a volume and exports `$BT_CHECKPOINT_DIR`. Defaults to `CheckpointingConfig()`. For more information, see [CheckpointingConfig](https://docs.baseten.co/reference/sdk/training#checkpointingconfig) . [​](https://docs.baseten.co/reference/sdk/training#param-cache-config) cache\_config CacheConfig Enables a persistent read-write cache that survives across jobs for pip packages, model downloads, and preprocessed datasets. For more information, see [CacheConfig](https://docs.baseten.co/reference/sdk/training#cacheconfig) . [​](https://docs.baseten.co/reference/sdk/training#param-load-checkpoint-config) load\_checkpoint\_config LoadCheckpointConfig Downloads checkpoints from a previous job into the container before `start_commands` run. Use this to resume training or initialize weights from an earlier experiment. For more information, see [LoadCheckpointConfig](https://docs.baseten.co/reference/sdk/training#loadcheckpointconfig) . [​](https://docs.baseten.co/reference/sdk/training#param-enable-cache) enable\_cache boolean deprecated Use `cache_config` with `enabled=True` instead. ### [​](https://docs.baseten.co/reference/sdk/training#secretreference) SecretReference Injects a secret stored in your [Baseten workspace](https://docs.baseten.co/organization/secrets) as an environment variable at runtime. Baseten never writes the value to your config file or source code. Use this for API keys, tokens, and credentials. config.py secret_ref = definitions.SecretReference(name="wandb_api_key") [​](https://docs.baseten.co/reference/sdk/training#param-name-2) name string required Name of the secret as it appears in your workspace settings. ### [​](https://docs.baseten.co/reference/sdk/training#checkpointingconfig) CheckpointingConfig Enables persistent checkpoint storage for the training job. When `enabled` is true, Baseten mounts a persistent volume and exports `$BT_CHECKPOINT_DIR` as an environment variable pointing to it. Your training script writes model weights, optimizer state, or any artifacts to that directory. These checkpoints survive job termination and can be [deployed to inference](https://docs.baseten.co/training/deployment) or [loaded into future jobs](https://docs.baseten.co/reference/sdk/training#loadcheckpointconfig) . See the [checkpointing guide](https://docs.baseten.co/training/concepts/checkpoints) for best practices. config.py checkpointing = definitions.CheckpointingConfig( enabled=True, volume_size_gib=500, ) [​](https://docs.baseten.co/reference/sdk/training#param-enabled) enabled boolean default:false Set to `true` to mount a persistent checkpoint volume. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-path) checkpoint\_path string Override the default checkpoint directory path. [​](https://docs.baseten.co/reference/sdk/training#param-volume-size-gib) volume\_size\_gib integer Size of the checkpoint volume in GiB. Defaults to a platform-managed size. ### [​](https://docs.baseten.co/reference/sdk/training#cacheconfig) CacheConfig Enables a persistent read-write cache that survives across jobs. Use the cache for pip packages, downloaded model weights, preprocessed datasets, or any data you don’t want to re-download on every run. When `enabled` is true, Baseten mounts two shared directories into the container. When `require_cache_affinity` is true (the default), Baseten schedules the job on a node that already has cached data, which avoids cold starts. See the [cache guide](https://docs.baseten.co/training/concepts/cache) for usage patterns. config.py cache = definitions.CacheConfig( enabled=True, require_cache_affinity=True, ) When enabled, Baseten exports two cache directories as environment variables. | Environment variable | Description | | --- | --- | | `$BT_PROJECT_CACHE_DIR` | Shared across all jobs in the same [TrainingProject](https://docs.baseten.co/reference/sdk/training#trainingproject)
. Use for project-specific datasets or compiled artifacts. | | `$BT_TEAM_CACHE_DIR` | Shared across all jobs in the same team. Use for common model weights or shared libraries. | [​](https://docs.baseten.co/reference/sdk/training#param-enabled-1) enabled boolean default:false Set to `true` to mount persistent cache volumes. [​](https://docs.baseten.co/reference/sdk/training#param-enable-legacy-hf-mount) enable\_legacy\_hf\_mount boolean default:false Mount the Hugging Face cache at the legacy path for backward compatibility. [​](https://docs.baseten.co/reference/sdk/training#param-require-cache-affinity) require\_cache\_affinity boolean default:true Schedule the job on a node with existing cached data when possible. [​](https://docs.baseten.co/reference/sdk/training#param-mount-base-path) mount\_base\_path string Base path where Baseten mounts cache directories. Defaults to `/root/.cache`. ### [​](https://docs.baseten.co/reference/sdk/training#loadcheckpointconfig) LoadCheckpointConfig Downloads checkpoints from previous training jobs into the container before `start_commands` run. Use this to resume training from a saved state or to initialize weights from an earlier experiment. Baseten downloads the specified checkpoints to `download_folder` (also exported as `$BT_LOAD_CHECKPOINT_DIR`) and your training script reads them at startup. For more information, see the [loading checkpoints](https://docs.baseten.co/training/loading) walkthrough. config.py load_config = definitions.LoadCheckpointConfig( enabled=True, download_folder="/tmp/loaded_checkpoints", checkpoints=[\ definitions.BasetenCheckpoint.from_latest_checkpoint(project_name="my-project"),\ definitions.BasetenCheckpoint.from_named_checkpoint(\ checkpoint_name="checkpoint-24",\ job_id="abc123",\ )\ ] ) [​](https://docs.baseten.co/reference/sdk/training#param-enabled-2) enabled boolean default:false Set to `true` to download checkpoints before `start_commands` run. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoints) checkpoints BasetenCheckpoint\[\] required One or more checkpoint references to download. Create references with `BasetenCheckpoint.from_latest_checkpoint()` or `BasetenCheckpoint.from_named_checkpoint()`. For more information, see [BasetenCheckpoint](https://docs.baseten.co/reference/sdk/training#basetencheckpoint) . [​](https://docs.baseten.co/reference/sdk/training#param-download-folder) download\_folder string Directory where Baseten downloads checkpoints. Exported as `$BT_LOAD_CHECKPOINT_DIR`. Defaults to `/tmp/loaded_checkpoints`. ### [​](https://docs.baseten.co/reference/sdk/training#basetencheckpoint) BasetenCheckpoint Creates references to checkpoints saved by previous training jobs. Pass these references to [LoadCheckpointConfig](https://docs.baseten.co/reference/sdk/training#loadcheckpointconfig) to download checkpoint data into your container at job start. You can reference checkpoints by project name (gets the most recent), by job ID (gets the most recent from that job), or by exact checkpoint name and job ID. config.py latest = definitions.BasetenCheckpoint.from_latest_checkpoint( project_name="my-fine-tuning-project" ) specific = definitions.BasetenCheckpoint.from_named_checkpoint( checkpoint_name="checkpoint-100", job_id="abc123", ) runtime = definitions.Runtime( start_commands=["python train.py"], load_checkpoint_config=definitions.LoadCheckpointConfig( enabled=True, checkpoints=[latest, specific], ) ) #### [​](https://docs.baseten.co/reference/sdk/training#from_latest_checkpoint) from\_latest\_checkpoint Returns a reference to the most recent checkpoint from a project or job. At least one of `project_name` or `job_id` is required. BasetenCheckpoint.from_latest_checkpoint( project_name: Optional[str] = None, job_id: Optional[str] = None, ) [​](https://docs.baseten.co/reference/sdk/training#param-project-name) project\_name string Project name to get the latest checkpoint from. [​](https://docs.baseten.co/reference/sdk/training#param-job-id) job\_id string Job ID to get the latest checkpoint from. #### [​](https://docs.baseten.co/reference/sdk/training#from_named_checkpoint) from\_named\_checkpoint Returns a reference to a specific checkpoint by its name and job ID. BasetenCheckpoint.from_named_checkpoint( checkpoint_name: str, job_id: str, ) [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-name) checkpoint\_name string required Checkpoint name. [​](https://docs.baseten.co/reference/sdk/training#param-job-id-1) job\_id string required Job ID. [​](https://docs.baseten.co/reference/sdk/training#workspace) Workspace -------------------------------------------------------------------------- Controls which local files Baseten uploads to the training container. By default, Baseten archives the directory containing your `config.py` (or the `source_dir` you pass to [push](https://docs.baseten.co/reference/sdk/training#push) ) and extracts it into the container’s working directory. Use `Workspace` to customize this behavior: exclude large data directories, include files from outside the root, or change the root entirely. config.py training_job = definitions.TrainingJob( image=definitions.Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), workspace=definitions.Workspace( exclude_dirs=["data", ".git"], ), ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-8) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-workspace-root) workspace\_root string Override the root directory to archive. Defaults to the config file’s parent directory. [​](https://docs.baseten.co/reference/sdk/training#param-external-dirs) external\_dirs string\[\] default:\[\] Additional directories outside `workspace_root` to include in the upload. [​](https://docs.baseten.co/reference/sdk/training#param-exclude-dirs) exclude\_dirs string\[\] default:\[\] Directories to exclude from the upload (for example, `"data"`, `".git"`, `"__pycache__"`). [​](https://docs.baseten.co/reference/sdk/training#interactivesession) InteractiveSession -------------------------------------------------------------------------------------------- Enables interactive access to the training container for live debugging. Configure `session_provider` to choose between [VS Code and Cursor remote tunnels](https://docs.baseten.co/training/interactive-sessions) and [SSH](https://docs.baseten.co/training/ssh) , and `trigger` to control when the session starts. * SSH * VS Code & Cursor config.py from truss_train import definitions from truss_train.definitions import ( InteractiveSession, InteractiveSessionTrigger, InteractiveSessionProvider, ) from truss.base import truss_config training_job = definitions.TrainingJob( image=definitions.Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=definitions.Compute( accelerator=truss_config.AcceleratorSpec(accelerator="H200", count=2), ), runtime=definitions.Runtime( start_commands=["chmod +x ./run.sh && ./run.sh"], ), interactive_session=InteractiveSession( trigger=InteractiveSessionTrigger.ON_FAILURE, session_provider=InteractiveSessionProvider.SSH, ), ) See the [SSH guide](https://docs.baseten.co/training/ssh) for setup and connection instructions. config.py from truss_train import definitions from truss_train.definitions import ( InteractiveSession, InteractiveSessionTrigger, InteractiveSessionProvider, InteractiveSessionAuthProvider, ) from truss.base import truss_config training_job = definitions.TrainingJob( image=definitions.Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=definitions.Compute( accelerator=truss_config.AcceleratorSpec(accelerator="H200", count=2), ), runtime=definitions.Runtime( start_commands=["chmod +x ./run.sh && ./run.sh"], ), interactive_session=InteractiveSession( trigger=InteractiveSessionTrigger.ON_FAILURE, timeout_minutes=-1, session_provider=InteractiveSessionProvider.VS_CODE, auth_provider=InteractiveSessionAuthProvider.GITHUB, ), ) See the [VS Code and Cursor guide](https://docs.baseten.co/training/interactive-sessions) for connection instructions. ### [​](https://docs.baseten.co/reference/sdk/training#parameters-9) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-trigger) trigger InteractiveSessionTrigger Controls when to activate the session. Defaults to `ON_DEMAND`.Available options: * `ON_STARTUP`: active from job start. * `ON_FAILURE`: activates when training exits with a non-zero code. * `ON_DEMAND`: activates when you connect (SSH) or authenticate through the device code flow (remote tunnel), or when you change the trigger on a running job. [​](https://docs.baseten.co/reference/sdk/training#param-timeout-minutes) timeout\_minutes integer default:480 Minutes before the session expires. Set to `-1` to extend the expiry to 10 years. For remote tunnel sessions, the expiry resets on every reconnect. For SSH, the expiry is set once at session start. [​](https://docs.baseten.co/reference/sdk/training#param-session-provider) session\_provider InteractiveSessionProvider Connection method for the interactive session. Defaults to `VS_CODE`.Available options: * `VS_CODE`: VS Code Remote Tunnels. * `CURSOR`: Cursor Remote Tunnels. * `SSH`: Direct SSH access from your terminal. See the [SSH guide](https://docs.baseten.co/training/ssh) for setup. [​](https://docs.baseten.co/reference/sdk/training#param-auth-provider) auth\_provider InteractiveSessionAuthProvider Authentication provider for the remote tunnel device code flow. Defaults to `MICROSOFT`. Ignored when `session_provider=SSH`.Available options: * `GITHUB`: authenticate via GitHub. * `MICROSOFT`: authenticate via Microsoft. * * * [​](https://docs.baseten.co/reference/sdk/training#environment-variables) Environment variables -------------------------------------------------------------------------------------------------- Baseten automatically injects these environment variables into every training container. Your training script can read them to discover job metadata, locate scratch, checkpoint, and cache directories, and coordinate across nodes in [multi-node jobs](https://docs.baseten.co/training/concepts/multinode) . ### [​](https://docs.baseten.co/reference/sdk/training#standard-variables) Standard variables | Variable | Description | Example | | --- | --- | --- | | `BT_TRAINING_JOB_ID` | Training job ID. | `"gvpql31"` | | `BT_TRAINING_PROJECT_ID` | Training project ID. | `"aghi527"` | | `BT_TRAINING_JOB_NAME` | Training job name. | `"gpt-oss-20b-lora"` | | `BT_TRAINING_PROJECT_NAME` | Training project name. | `"gpt-oss-finetunes"` | | `BT_NUM_GPUS` | Number of GPUs per node. | `"4"` | | `BT_WORKING_DIR` | Container working directory. Set when [`enable_baseten_workdir`](https://docs.baseten.co/reference/sdk/training#param-enable-baseten-workdir)
is `True`. | `"/b10/workspace"` | | `BT_SCRATCH_DIR` | Ephemeral scratch directory backed by local NVMe storage. Cleared when the job completes. | `"/mnt/bt-scratch"` | | `BT_CHECKPOINT_DIR` | Checkpoint save directory. | `"/mnt/ckpts"` | | `BT_LOAD_CHECKPOINT_DIR` | Loaded checkpoints directory. | `"/tmp/loaded_checkpoints"` | | `BT_PROJECT_CACHE_DIR` | Project-level cache directory. | `"/root/.cache/user_artifacts"` | | `BT_TEAM_CACHE_DIR` | Team-level cache directory. | `"/root/.cache/team_artifacts"` | | `BT_RW_CACHE_DIR` | Base read-write cache directory. | `"/root/.cache"` | | `BT_RETRY_COUNT` | Job retry attempt count. | `"0"` | ### [​](https://docs.baseten.co/reference/sdk/training#multi-node-variables) Multi-node variables For distributed training across multiple nodes: | Variable | Description | Example | | --- | --- | --- | | `BT_GROUP_SIZE` | Number of nodes in deployment. | `"2"` | | `BT_LEADER_ADDR` | Leader node address. | `"10.0.0.1"` | | `BT_NODE_RANK` | Node rank (0 for leader). | `"0"` | * * * [​](https://docs.baseten.co/reference/sdk/training#deploy-checkpoints) Deploy checkpoints -------------------------------------------------------------------------------------------- Deploys trained model checkpoints from a completed training job to Baseten’s inference platform. Baseten downloads the checkpoint weights, packages them with a serving runtime, and creates a deployable model endpoint. See the [deployment guide](https://docs.baseten.co/training/deployment) for the full workflow. ### [​](https://docs.baseten.co/reference/sdk/training#deploy-with-cli-wizard) Deploy with CLI wizard Deploy checkpoints interactively with the CLI wizard: uvx truss train deploy_checkpoints --job-id The wizard guides you through selecting checkpoints and configuring deployment. Baseten automatically recognizes checkpoints for full fine-tunes and LoRAs for LLMs and Whisper models. The `deploy_checkpoints` command doesn’t support FSDP checkpoints. Configure these manually in the Truss config. For optimized inference with TensorRT-LLM, see [Deploy with optimized inference engines](https://docs.baseten.co/training/deploy-with-engine-builder) . ### [​](https://docs.baseten.co/reference/sdk/training#deploy-with-static-configuration) Deploy with static configuration Create a Python config file for repeatable deployments: uvx truss train deploy_checkpoints --config [​](https://docs.baseten.co/reference/sdk/training#deploycheckpointsconfig) DeployCheckpointsConfig ------------------------------------------------------------------------------------------------------ Defines how to deploy checkpoints from a completed training job to a Baseten inference endpoint. Baseten reads the checkpoint weights, selects the correct serving backend based on the model weights format (full, LoRA, or Whisper), and provisions the specified [Compute](https://docs.baseten.co/reference/sdk/training#compute) resources. deploy\_config.py from truss_train import definitions from truss.base import truss_config deploy_config = definitions.DeployCheckpointsConfig( model_name="fine-tuned-llm", checkpoint_details=definitions.CheckpointList( base_model_id="meta-llama/Llama-3.1-8B-Instruct", checkpoints=[\ definitions.LoRACheckpoint(\ training_job_id="gvpql31",\ checkpoint_name="checkpoint-100",\ lora_details=definitions.LoRADetails(rank=16),\ )\ ] ), compute=definitions.Compute( accelerator=truss_config.AcceleratorSpec( accelerator=truss_config.Accelerator.H200, count=1, ) ), ) ### [​](https://docs.baseten.co/reference/sdk/training#parameters-10) Parameters [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-details) checkpoint\_details CheckpointList Checkpoints to deploy, including the base model ID for LoRA and one or more checkpoint references. For more information, see [CheckpointList](https://docs.baseten.co/reference/sdk/training#checkpointlist) . [​](https://docs.baseten.co/reference/sdk/training#param-model-name) model\_name string Name for the deployed model in the Baseten dashboard. [​](https://docs.baseten.co/reference/sdk/training#param-runtime-1) runtime DeployCheckpointsRuntime Environment variables for the inference runtime, such as API keys or serving configuration. For more information, see [DeployCheckpointsRuntime](https://docs.baseten.co/reference/sdk/training#deploycheckpointsruntime) . [​](https://docs.baseten.co/reference/sdk/training#param-compute-1) compute Compute GPU and memory allocation for the inference endpoint. Uses the same [Compute](https://docs.baseten.co/reference/sdk/training#compute) configuration as training jobs. ### [​](https://docs.baseten.co/reference/sdk/training#deploycheckpointsruntime) DeployCheckpointsRuntime Sets environment variables for the deployed inference endpoint. Use this to inject API keys or configuration that the serving runtime needs. [​](https://docs.baseten.co/reference/sdk/training#param-environment-variables-1) environment\_variables object Key-value pairs that Baseten injects as env vars. Use [SecretReference](https://docs.baseten.co/reference/sdk/training#secretreference) for sensitive values. ### [​](https://docs.baseten.co/reference/sdk/training#checkpointlist) CheckpointList Groups one or more checkpoints for deployment. For LoRA deployments, set `base_model_id` to the Hugging Face model ID you trained the adapters on. [​](https://docs.baseten.co/reference/sdk/training#param-download-folder-1) download\_folder string Directory where Baseten downloads checkpoint files during deployment. Defaults to `/tmp/training_checkpoints`. [​](https://docs.baseten.co/reference/sdk/training#param-base-model-id) base\_model\_id string Hugging Face model ID for the base model. Required for LoRA deployments. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoints-1) checkpoints Checkpoint\[\] default:\[\] One or more [FullCheckpoint](https://docs.baseten.co/reference/sdk/training#fullcheckpoint) , [LoRACheckpoint](https://docs.baseten.co/reference/sdk/training#loracheckpoint) , or [WhisperCheckpoint](https://docs.baseten.co/reference/sdk/training#whispercheckpoint) instances. [​](https://docs.baseten.co/reference/sdk/training#param-trainer-checkpoint-ids) trainer\_checkpoint\_ids string\[\] default:\[\] Trainer checkpoint IDs to deploy. Use this when deploying checkpoints produced by a trainer rather than a training job. Mutually exclusive with `checkpoints` — set one or the other, not both. ### [​](https://docs.baseten.co/reference/sdk/training#checkpoint-types) Checkpoint types Baseten supports three checkpoint types. Use the type that matches how your model was trained. #### [​](https://docs.baseten.co/reference/sdk/training#fullcheckpoint) FullCheckpoint Deploys a complete set of model weights from a full fine-tune. [​](https://docs.baseten.co/reference/sdk/training#param-training-job-id) training\_job\_id string required Training job ID. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-name-1) checkpoint\_name string required Checkpoint name. [​](https://docs.baseten.co/reference/sdk/training#param-model-weight-format) model\_weight\_format string Auto-set to `full`. #### [​](https://docs.baseten.co/reference/sdk/training#loracheckpoint) LoRACheckpoint Deploys LoRA adapter weights on top of the base model you specify in [CheckpointList](https://docs.baseten.co/reference/sdk/training#checkpointlist) . [​](https://docs.baseten.co/reference/sdk/training#param-training-job-id-1) training\_job\_id string required Training job ID. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-name-2) checkpoint\_name string required Checkpoint name. [​](https://docs.baseten.co/reference/sdk/training#param-model-weight-format-1) model\_weight\_format string Auto-set to `lora`. [​](https://docs.baseten.co/reference/sdk/training#param-lora-details) lora\_details LoRADetails LoRA adapter configuration. Set `rank` to match the rank you used during training. Defaults to `LoRADetails()`. Valid values: * 8, 16, 32, 64, 128, 256, 320, 512. For more information, see [LoRADetails](https://docs.baseten.co/reference/sdk/training#loradetails) . #### [​](https://docs.baseten.co/reference/sdk/training#whispercheckpoint) WhisperCheckpoint Deploys fine-tuned Whisper model weights for speech-to-text inference. [​](https://docs.baseten.co/reference/sdk/training#param-training-job-id-2) training\_job\_id string required Training job ID. [​](https://docs.baseten.co/reference/sdk/training#param-checkpoint-name-3) checkpoint\_name string required Checkpoint name. [​](https://docs.baseten.co/reference/sdk/training#param-model-weight-format-2) model\_weight\_format string Auto-set to `whisper`. ### [​](https://docs.baseten.co/reference/sdk/training#loradetails) LoRADetails Sets the LoRA rank for adapter deployment. The rank must match the rank you set during training. [​](https://docs.baseten.co/reference/sdk/training#param-rank) rank integer default:16 LoRA rank. Valid values: 8, 16, 32, 64, 128, 256, 320, 512. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/sdk/chains) [Loops SDKPython client for Loops: ServiceClient, TrainingClient, SamplingClient, and the Tinker compatibility shim.\ \ Next](https://docs.baseten.co/reference/sdk/loops) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Inference - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/troubleshooting/inference#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) [​](https://docs.baseten.co/troubleshooting/inference#model-i/o-issues) Model I/O issues ------------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/troubleshooting/inference#error-jsondecodeerror) Error: JSONDecodeError json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) This error means you’re attempting to pass a model input that is not JSON-serializable. For example, you might have left out the double quotes required for a valid string: truss predict -d 'This is not a string' # Wrong truss predict -d '"This is a string"' # Correct [​](https://docs.baseten.co/troubleshooting/inference#model-version-issues) Model version issues --------------------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/troubleshooting/inference#error-no-oracleversion-matches-the-given-query) Error: No OracleVersion matches the given query Make sure that the model ID or deployment ID you’re passing is correct and that the associated model has not been deleted. Additionally, make sure you’re using the correct endpoint: * [Production environment endpoint](https://docs.baseten.co/reference/inference-api/predict-endpoints/environments-predict) . * [Development deployment endpoint](https://docs.baseten.co/reference/inference-api/predict-endpoints/development-predict) . * [Deployment endpoint](https://docs.baseten.co/reference/inference-api/predict-endpoints/deployment-predict) . [​](https://docs.baseten.co/troubleshooting/inference#authentication-issues) Authentication issues ----------------------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/troubleshooting/inference#error-service-provider-not-found) Error: Service provider not found ValueError: Service provider example-service-provider not found in ~/.trussrc This error means your `~/.trussrc` is incomplete or incorrect. It should be formatted as follows: [baseten] remote_provider = baseten api_key = YOUR.API_KEY remote_url = https://app.baseten.co ### [​](https://docs.baseten.co/troubleshooting/inference#error-you-have-to-log-in-to-perform-the-request) Error: You have to log in to perform the request This error occurs on `truss predict` when the API key in `~/.trussrc` for a given host is missing or incorrect. To fix it, update your API key in the `~/.trussrc` file. ### [​](https://docs.baseten.co/troubleshooting/inference#error-please-check-the-api-key-you-provided) Error: Please check the API key you provided { "error": "please check the api-key you provided" } This error occurs when using `curl` or similar to call the model via its API endpoint when the API key passed in the request header is not valid. Make sure you’re using a valid API key then try again. Was this page helpful? YesNo [Previous\ \ DeploymentsTroubleshoot common problems during model deployment](https://docs.baseten.co/troubleshooting/deployments) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Overview - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/overview#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) The Training API manages training projects, jobs, and related resources through a RESTful interface. Use this API to: * Monitor training job metrics and logs * Manage training jobs * Manage checkpoints and artifacts [​](https://docs.baseten.co/reference/training-api/overview#authentication) Authentication --------------------------------------------------------------------------------------------- All Training API requests require authentication with an API key: Authorization: Bearer EMPTY [​](https://docs.baseten.co/reference/training-api/overview#base-url) Base URL --------------------------------------------------------------------------------- All Training API endpoints are relative to: https://api.baseten.co/v1 [​](https://docs.baseten.co/reference/training-api/overview#available-endpoints) Available Endpoints ------------------------------------------------------------------------------------------------------- ### [​](https://docs.baseten.co/reference/training-api/overview#training-projects) Training Projects | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/training_projects`](https://docs.baseten.co/reference/training-api/get-training-projects) | List all training projects | | `GET` | [`/training_projects/{training_project_id}`](https://docs.baseten.co/reference/training-api/get-training-project) | Get a training project | | `POST` | [`/training_projects`](https://docs.baseten.co/reference/training-api/create-training-project) | Create a training project | | `DELETE` | [`/training_projects/{training_project_id}`](https://docs.baseten.co/reference/training-api/delete-training-project) | Delete a training project | | `GET` | [`/training_projects/{training_project_id}/cache/summary`](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary) | Get training project cache summary | ### [​](https://docs.baseten.co/reference/training-api/overview#training-jobs) Training Jobs The following endpoints use the relative base path: `/training_projects/{training_project_id}/jobs` | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`.../`](https://docs.baseten.co/reference/training-api/create-training-job) | Create a training job | | `GET` | [`.../`](https://docs.baseten.co/reference/training-api/list-training-jobs) | List all jobs in a project | | `GET` | [`.../{training_job_id}`](https://docs.baseten.co/reference/training-api/get-training-job) | Get a specific training job | | `POST` | [`.../{training_job_id}/stop`](https://docs.baseten.co/reference/training-api/stop-training-job) | Stop a training job | | `DELETE` | [`.../{training_job_id}`](https://docs.baseten.co/reference/training-api/delete-training-job) | Delete a training job | | `POST` | [`.../{training_job_id}/recreate`](https://docs.baseten.co/reference/training-api/recreate-training-job) | Recreate a training job | | `GET` | [`.../{training_job_id}/logs`](https://docs.baseten.co/reference/training-api/get-training-job-logs) | Get training job logs | | `GET` | [`.../{training_job_id}/metrics`](https://docs.baseten.co/reference/training-api/get-training-job-metrics) | Get training job metrics | | `GET` | [`.../{training_job_id}/checkpoints`](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints) | List job checkpoints | | `GET` | [`.../{training_job_id}/checkpoint_files`](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files) | Get training job checkpoint files | | `GET` | [`.../{training_job_id}/download`](https://docs.baseten.co/reference/training-api/download-training-job) | Download training job artifacts | | `GET` | [`.../{training_job_id}/auth_codes`](https://docs.baseten.co/reference/training-api/get-auth-codes-for-training-job) | Get auth codes for a training job | Search endpoint: | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/training_jobs/search`](https://docs.baseten.co/reference/training-api/search-training-jobs) | Search across all training jobs | Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/billing/gets-billing-usage-summary-for-a-date-range) [Create training projectUpserts a training project with the specified metadata.\ \ Next](https://docs.baseten.co/reference/training-api/create-training-project) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Any model deployment by ID - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) PATCH / v1 / models / {model\_id} / deployments / {deployment\_id} / autoscaling\_settings Try it cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } To update autoscaling settings at the environment level, use the [update environment settings](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings) endpoint. #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#parameter-deployment-id) deployment\_id string required #### Body application/json A request to update autoscaling settings for a deployment. All fields are optional, and we only update ones passed in. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-min-replica-one-of-0) min\_replica integer | null Minimum number of replicas Example: `0` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-max-replica-one-of-0) max\_replica integer | null Maximum number of replicas Example: `7` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-autoscaling-window-one-of-0) autoscaling\_window integer | null Timeframe of traffic considered for autoscaling decisions Example: `600` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-scale-down-delay-one-of-0) scale\_down\_delay integer | null Waiting period before scaling down any active replica Example: `120` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-concurrency-target-one-of-0) concurrency\_target integer | null Number of requests per replica before scaling up Example: `2` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-target-utilization-percentage-one-of-0) target\_utilization\_percentage integer | null Target utilization percentage for scaling up/down. Example: `70` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-target-in-flight-tokens-one-of-0) target\_in\_flight\_tokens integer | null Target number of in-flight tokens for autoscaling decisions. Early access only. Example: `40000` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#body-max-scale-down-rate-one-of-0) max\_scale\_down\_rate number | null Maximum rate at which replicas can scale down (e.g. 2.0 means at most halve replicas per window). Required range: `1 < x <= 2` Example: `2` #### Response 200 - application/json The response to a request to update autoscaling settings. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#response-status) status enum required Status of the request to update autoscaling settings Available options: `ACCEPTED`, `QUEUED`, `UNCHANGED` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings#response-message) message string required A message describing the status of the request to update autoscaling settings Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings) [Production autoscalingUpdates a production deployment's autoscaling settings and returns the update status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings) ⌘I cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Create training job - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/create-training-job#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / training\_projects / {training\_project\_id} / jobs Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_job": { "image": { "base_image": "hello-world", "docker_auth": null }, "compute": { "node_count": 1, "cpu_count": 1, "memory": "2Gi", "accelerator": { "accelerator": "H100", "count": 2 } }, "runtime": { "start_commands": [\ "python main.py"\ ], "environment_variables": { "API_KEY": "your_api_key_here", "PATH": "/usr/bin" }, "enable_cache": true, "cache_config": { "enable_legacy_hf_mount": true, "enabled": true, "mount_base_path": "/root/.cache", "require_cache_affinity": true }, "checkpointing_config": { "enabled": true, "checkpoint_path": "/mnt/ckpts", "volume_size_gib": 10 }, "load_checkpoint_config": null }, "name": "gpt-oss-job", "truss_user_env": null, "interactive_session": null, "weights": [\ {\ "allow_patterns": null,\ "auth": null,\ "auth_secret_name": null,\ "ignore_patterns": null,\ "mount_location": "/app/models/base",\ "source": "hf://meta-llama/Llama-3-8B@main"\ }\ ], "enable_baseten_workdir": false, "priority": 0 } }' 200 { "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } } } #### Authorizations [​](https://docs.baseten.co/reference/training-api/create-training-job#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/create-training-job#parameter-training-project-id) training\_project\_id string required #### Body application/json A request to create a training job. [​](https://docs.baseten.co/reference/training-api/create-training-job#body-training-job) training\_job CreateTrainingJobV1 · object required The training job to create. Show child attributes #### Response 200 - application/json A response to creating a training job. [​](https://docs.baseten.co/reference/training-api/create-training-job#response-training-job) training\_job TrainingJobV1 · object required The created training job. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/get-auth-codes-for-training-job) [Loops API referenceHTTP routes for Loops sessions, runs, samplers, checkpoints, and deployments.\ \ Next](https://docs.baseten.co/reference/loops-api/overview) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_job": { "image": { "base_image": "hello-world", "docker_auth": null }, "compute": { "node_count": 1, "cpu_count": 1, "memory": "2Gi", "accelerator": { "accelerator": "H100", "count": 2 } }, "runtime": { "start_commands": [\ "python main.py"\ ], "environment_variables": { "API_KEY": "your_api_key_here", "PATH": "/usr/bin" }, "enable_cache": true, "cache_config": { "enable_legacy_hf_mount": true, "enabled": true, "mount_base_path": "/root/.cache", "require_cache_affinity": true }, "checkpointing_config": { "enabled": true, "checkpoint_path": "/mnt/ckpts", "volume_size_gib": 10 }, "load_checkpoint_config": null }, "name": "gpt-oss-job", "truss_user_env": null, "interactive_session": null, "weights": [\ {\ "allow_patterns": null,\ "auth": null,\ "auth_secret_name": null,\ "ignore_patterns": null,\ "mount_location": "/app/models/base",\ "source": "hf://meta-llama/Llama-3-8B@main"\ }\ ], "enable_baseten_workdir": false, "priority": 0 } }' 200 { "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } } } Assistant Responses are generated using AI and may contain mistakes. --- # SSH access - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/ssh#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) SSH (Secure Shell) is a protocol for encrypted, authenticated access to a remote machine. With training jobs on Baseten, you can SSH into any running job and get a full terminal inside the training container to debug, inspect files, run commands, edit code, or transfer data with `scp` and `sftp`. This gives you the same control you’d have on a local GPU box, just running on Baseten-managed hardware. Unlike [VS Code and Cursor remote tunnels](https://docs.baseten.co/training/interactive-sessions) , SSH is terminal-first and works with any OpenSSH-compatible tool. [​](https://docs.baseten.co/training/ssh#prerequisites) Prerequisites ------------------------------------------------------------------------ SSH sessions must be enabled for your Baseten workspace. [Contact support](mailto:support@baseten.co) to request access. * **Baseten account**: [Sign up](https://app.baseten.co/) and generate an [API key](https://app.baseten.co/settings/account/api_keys) . * **[uv](https://docs.astral.sh/uv/) **: This guide uses `uvx` to run [Truss](https://pypi.org/project/truss/) commands without a separate install step. Log in to Baseten: uvx truss login * **OpenSSH client**: Pre-installed on macOS and Linux. On Windows, use the OpenSSH optional feature or WSL. [​](https://docs.baseten.co/training/ssh#configuration) Configuration ------------------------------------------------------------------------ To enable SSH access on a training job, set `session_provider` to `SSH` in your `config.py`: config.py from truss_train import TrainingProject, TrainingJob, Image, Compute, Runtime from truss_train.definitions import ( InteractiveSession, InteractiveSessionTrigger, InteractiveSessionProvider, ) from truss.base.truss_config import AcceleratorSpec training_job = TrainingJob( image=Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=Compute( accelerator=AcceleratorSpec(accelerator="H200", count=1), ), runtime=Runtime( start_commands=["python train.py"], ), interactive_session=InteractiveSession( trigger=InteractiveSessionTrigger.ON_STARTUP, session_provider=InteractiveSessionProvider.SSH, ), ) training_project = TrainingProject(name="my-training-project", job=training_job) This example sets `trigger=ON_STARTUP` so the session is available as soon as the job starts running. The default is `ON_DEMAND`, which activates the session on your first SSH connection; see [Trigger modes](https://docs.baseten.co/training/remote-access#trigger-modes) for all options. Training jobs support H200 and H100 GPUs. For all `InteractiveSession` fields, see the [SDK reference](https://docs.baseten.co/reference/sdk/training#interactivesession) . For supported compute, see [Compute resources](https://docs.baseten.co/training/concepts/basics#compute-resources) . [​](https://docs.baseten.co/training/ssh#quick-start-with-cli) Quick start with CLI -------------------------------------------------------------------------------------- To create an SSH-enabled workstation without writing a `config.py`, use the `truss train workstation` command. It configures an interactive session with `InteractiveSessionTrigger.ON_STARTUP` and runs `sleep infinity` to keep the container alive. # Default: 1x H100 uvx truss train workstation # 4x H200 with custom project name uvx truss train workstation --accelerator H200 --gpu-count 4 --project-id my-dev-box # Custom base image uvx truss train workstation --image pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime Once the workstation is running, connect using the SSH command provided in the output. See the [CLI reference](https://docs.baseten.co/reference/cli/training/training-cli#workstation) for all available options. [​](https://docs.baseten.co/training/ssh#quick-start) Quick start -------------------------------------------------------------------- This walkthrough uses the [MNIST PyTorch example](https://github.com/basetenlabs/ml-cookbook/tree/main/examples/mnist-pytorch/training) to push a training job with SSH enabled, then connects to the container from your terminal. ### [​](https://docs.baseten.co/training/ssh#1-set-up-your-machine) 1\. Set up your machine Run `uvx truss ssh setup` once to configure OpenSSH: uvx truss ssh setup This generates an SSH keypair, installs a `ProxyCommand` helper, and adds a wildcard `Host` entry to `~/.ssh/config`. You only need to do this once per machine. The expected output is: SSH keypair: /Users//.ssh/baseten/id_ed25519 Proxy script: /Users//.ssh/baseten/proxy-command.py SSH config updated: ~/.ssh/config Default remote: SSH access configured. Connect to a running workload with: Training job: ssh training-job--.ssh.baseten.co Inference model: ssh model--.ssh.baseten.co Where: * `/Users/` is your local home directory. * `` is your default remote from `~/.trussrc` (typically `baseten`). If you’ve already run setup on this machine, the first line instead reads `WARNING: Existing SSH keypair found at , reusing it.` You can safely ignore it. ### [​](https://docs.baseten.co/training/ssh#2-clone-the-example) 2\. Clone the example git clone https://github.com/basetenlabs/ml-cookbook.git cd ml-cookbook/examples/mnist-pytorch/training ### [​](https://docs.baseten.co/training/ssh#3-configure-and-push-the-job) 3\. Configure and push the job Edit `config.py` to add an `interactive_session` with `session_provider=SSH`, as shown in [Configuration](https://docs.baseten.co/training/ssh#configuration) . Then push: uvx truss train push config.py The expected output is: ✨ Training job successfully created! 🪵 View logs for your job via 'truss train logs --job-id --tail' 🔍 View metrics for your job via 'truss train metrics --job-id ' 🌐 View job in the UI: https://app.baseten.co/training//logs/ Where: * `` is your new training job’s ID. Note it down; you’ll use it to connect in the next step. * `` is the Baseten-assigned ID for the training project. ### [​](https://docs.baseten.co/training/ssh#4-connect) 4\. Connect Once the job is running, find its ID with `uvx truss train view`, then SSH in: ssh training-job--0.ssh.baseten.co For example, to connect to node 0 of job `abc1234`: ssh training-job-abc1234-0.ssh.baseten.co You’re connected when you see a shell prompt like `root@baseten-training-job--multinode-0:~#`. By default, your source files are extracted to `/b10/workspace` (available as `$BT_WORKING_DIR`). If you set [`enable_baseten_workdir=False`](https://docs.baseten.co/reference/sdk/training#param-enable-baseten-workdir) , Baseten uses your base image’s `WORKDIR` instead. [​](https://docs.baseten.co/training/ssh#how-it-works) How it works ---------------------------------------------------------------------- When you run `ssh training-job--.ssh.baseten.co`, Baseten authenticates you using the API key stored by `uvx truss ssh setup`, issues a short-lived SSH certificate, and routes the connection to the correct training job. Certificates refresh automatically on every connection, so you never need to manage keys or tokens manually. [​](https://docs.baseten.co/training/ssh#hostname-format) Hostname format ---------------------------------------------------------------------------- training-job--.ssh.baseten.co | Segment | Description | Example | | --- | --- | --- | | `job_id` | Training job ID. Find it with `uvx truss train view` or in the Baseten dashboard. | `abc1234` | | `node` | Node index, starting at 0. | `0` | Examples: # Single-node job ssh training-job-abc1234-0.ssh.baseten.co # Second node of a multi-node job ssh training-job-xyz5678-1.ssh.baseten.co [​](https://docs.baseten.co/training/ssh#ide-integration) IDE integration ---------------------------------------------------------------------------- Because `uvx truss ssh setup` configures standard OpenSSH, tools that speak SSH can connect with the same hostname: * **VS Code**: Install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension, then connect to `training-job--.ssh.baseten.co`. * **Cursor**: Use the built-in SSH remote feature with `training-job--.ssh.baseten.co`. [​](https://docs.baseten.co/training/ssh#multi-node-jobs) Multi-node jobs ---------------------------------------------------------------------------- For [multi-node training jobs](https://docs.baseten.co/training/concepts/multinode) , specify the node index in the hostname. Node 0 is the leader: # Leader node ssh training-job-abc1234-0.ssh.baseten.co # Worker node 1 ssh training-job-abc1234-1.ssh.baseten.co [​](https://docs.baseten.co/training/ssh#file-transfer) File transfer ------------------------------------------------------------------------ Use `scp` or `sftp` with the same hostname to transfer files: # Copy a file to the training container scp ./data.csv training-job-abc1234-0.ssh.baseten.co:/workspace/data.csv # Copy results from the container scp training-job-abc1234-0.ssh.baseten.co:/workspace/results.json ./results.json # Interactive file browser sftp training-job-abc1234-0.ssh.baseten.co [​](https://docs.baseten.co/training/ssh#multiple-remotes) Multiple remotes ------------------------------------------------------------------------------ If you only have one remote configured in `~/.trussrc`, you can skip this section. Baseten uses it automatically. If you have multiple remotes, include the remote name in the hostname so the proxy script knows which credentials to use: training-job--..ssh.baseten.co For example, to connect using the `baseten-dev` remote: ssh training-job-abc1234-0.baseten-dev.ssh.baseten.co [​](https://docs.baseten.co/training/ssh#session-management) Session management ---------------------------------------------------------------------------------- For how to view session status, change triggers, and extend session expiry, see the [Remote access overview](https://docs.baseten.co/training/remote-access#session-management) . [​](https://docs.baseten.co/training/ssh#troubleshooting) Troubleshooting ---------------------------------------------------------------------------- ### [​](https://docs.baseten.co/training/ssh#%E2%80%9Dinvalid-job-id-must-be-a-valid-hash-id%E2%80%9D) ”Invalid job id: must be a valid hash id” Check that the job ID in the hostname is correct. Find your job ID with `uvx truss train view` or in the Baseten dashboard. ### [​](https://docs.baseten.co/training/ssh#%E2%80%9Dssh-keypair-not-found%E2%80%9D-or-%E2%80%9Ccommand-not-found%E2%80%9D) ”SSH keypair not found” or “command not found” Run `uvx truss ssh setup` to configure your machine. ### [​](https://docs.baseten.co/training/ssh#connection-refused-or-job-unreachable) Connection refused or job unreachable SSH requires the training job to be in the `RUNNING` state. Check with: uvx truss train view --job-id If the job is running but SSH still fails, the job may not have SSH enabled. Confirm `session_provider=InteractiveSessionProvider.SSH` is set in your [Configuration](https://docs.baseten.co/training/ssh#configuration) . ### [​](https://docs.baseten.co/training/ssh#tls-errors) TLS errors The proxy script requires Python 3.10 or newer. If you see TLS errors, re-run setup with a newer Python interpreter: uvx truss ssh setup --python $(which python3.12) Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/remote-access) [VS Code & CursorConnect to training containers for remote debugging and development via VS Code or Cursor Remote Tunnels.\ \ Next](https://docs.baseten.co/training/interactive-sessions) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Promote to model environment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / environments / {env\_name} / promote Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/promote \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "scale_down_previous_deployment": true, "deployment_id": null, "preserve_env_instance_type": true }' 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#parameter-env-name) env\_name string required #### Body application/json A request to promote a deployment to a environment. [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#body-deployment-id) deployment\_id string required The id of the deployment to promote [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#body-scale-down-previous-deployment) scale\_down\_previous\_deployment boolean default:true Whether to scale down the previous deployment after promoting Example: `true` [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#body-preserve-env-instance-type) preserve\_env\_instance\_type boolean default:true Whether to use the promoting deployment's instance type or preserve target environment's instance type Example: `true` #### Response 200 - application/json A deployment of a model. [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-id) id string required Unique identifier of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-created-at) created\_at string required Time the deployment was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-name) name string required Name of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-model-id) model\_id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-is-production) is\_production boolean required Whether the deployment is the production deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-is-development) is\_development boolean required Whether the deployment is the development deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-status) status enum required Status of the deployment Available options: `BUILDING`, `DEPLOYING`, `DEPLOY_FAILED`, `LOADING_MODEL`, `ACTIVE`, `UNHEALTHY`, `BUILD_FAILED`, `BUILD_STOPPED`, `DEACTIVATING`, `INACTIVE`, `FAILED`, `UPDATING`, `SCALED_TO_ZERO`, `WAKING_UP` [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-active-replica-count) active\_replica\_count integer required Number of active replicas [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-autoscaling-settings-one-of-0) autoscaling\_settings AutoscalingSettingsV1 · object required Autoscaling settings for the deployment. If null, the model has not finished deploying Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-instance-type-name-one-of-0) instance\_type\_name string | null required Name of the instance type the model deployment is running on [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-environment-one-of-0) environment string | null required The environment associated with the deployment [​](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment#response-labels-one-of-0) labels Labels · object User-provided key-value labels for the deployment Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment) [Cancel model promotionCancels an ongoing promotion to an environment and returns the cancellation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/promote/cancel-promotion) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/promote \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "scale_down_previous_deployment": true, "deployment_id": null, "preserve_env_instance_type": true }' 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } Assistant Responses are generated using AI and may contain mistakes. --- # Storage and data ingestion - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/training/concepts/storage#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) Training jobs need model weights, training datasets, and configuration files. Baseten provides multiple ways to get data into your training container, from cached delivery through [Baseten Delivery Network (BDN)](https://docs.baseten.co/development/model/bdn) to direct downloads in your training script. [​](https://docs.baseten.co/training/concepts/storage#load-weights-and-data-with-bdn) Load weights and data with BDN ----------------------------------------------------------------------------------------------------------------------- Use the [`weights`](https://docs.baseten.co/reference/sdk/training#weightssource) parameter on [`TrainingJob`](https://docs.baseten.co/reference/sdk/training#trainingjob) to mount model weights and training data into your container through BDN. BDN mirrors your data once and serves it from multi-tier caches, so subsequent jobs start faster. BDN mirrors your weights to Baseten storage during the `CREATED` state, before any compute is provisioned. Once your job is scheduled on a node, BDN places the weights on local disk before your `start_commands` run. Weight delivery never overlaps with workload execution, so BDN has no effect on training throughput. The only difference between a cache hit and a cache miss is how long the deploy phase takes. Each weight source specifies a remote URI and a local mount path. When your container starts, the data is already available at the `mount_location`. No download code needed in your training script. ### [​](https://docs.baseten.co/training/concepts/storage#hugging-face-and-s3-example) Hugging Face and S3 example Load model weights from Hugging Face and training data from S3, mounted into the training container before your code runs: config.py from truss_train import TrainingProject, TrainingJob, Image, Compute, Runtime, WeightsSource from truss.base.truss_config import AcceleratorSpec training_job = TrainingJob( image=Image(base_image="pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"), compute=Compute( accelerator=AcceleratorSpec(accelerator="H200", count=1), ), runtime=Runtime( start_commands=["python train.py"], ), weights=[\ WeightsSource(\ source="hf://Qwen/Qwen3-0.6B",\ mount_location="/app/models/Qwen/Qwen3-0.6B",\ ),\ WeightsSource(\ source="s3://my-bucket/training-data",\ mount_location="/app/data/training-data",\ ),\ ], ) training_project = TrainingProject(name="qwen3-finetune", job=training_job) In your training script, reference the mount paths directly: train.py from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("/app/models/Qwen/Qwen3-0.6B") tokenizer = AutoTokenizer.from_pretrained("/app/models/Qwen/Qwen3-0.6B") # Training data is available at /app/data/training-data/ ### [​](https://docs.baseten.co/training/concepts/storage#supported-sources) Supported sources BDN supports these URI schemes: | Scheme | Example | Description | | --- | --- | --- | | `hf://` | `hf://meta-llama/Llama-3.1-8B@main` | Hugging Face Hub. | | `s3://` | `s3://my-bucket/path/to/data` | Amazon S3. | | `gs://` | `gs://my-bucket/path/to/data` | Google Cloud Storage. | | `r2://` | `r2://account_id.bucket/path` | Cloudflare R2. | For Hugging Face sources, pin to a specific revision with the `@revision` suffix (branch, tag, or commit SHA). ### [​](https://docs.baseten.co/training/concepts/storage#authentication) Authentication Private or gated sources require authentication. Add an `auth` block to your `WeightsSource`: * Hugging Face * S3 (IAM credentials) Store a [Hugging Face token](https://huggingface.co/settings/tokens) as a [Baseten secret](https://docs.baseten.co/development/model/secrets) : WeightsSource( source="hf://meta-llama/Llama-3.1-8B@main", mount_location="/app/models/llama", auth={"auth_method": "CUSTOM_SECRET", "auth_secret_name": "hf_access_token"}, ) Store AWS credentials as a JSON [Baseten secret](https://docs.baseten.co/development/model/secrets) : WeightsSource( source="s3://my-bucket/training-data", mount_location="/app/data/training-data", auth={"auth_method": "CUSTOM_SECRET", "auth_secret_name": "aws_credentials"}, ) The secret value must contain `aws_access_key_id`, `aws_secret_access_key`, and `aws_region`. For the full list of authentication options and source-specific configuration, see the [BDN configuration reference](https://docs.baseten.co/development/model/bdn#configuration-reference) . ### [​](https://docs.baseten.co/training/concepts/storage#filtering-files) Filtering files Use `allow_patterns` and `ignore_patterns` to download only the files you need: WeightsSource( source="hf://meta-llama/Llama-3.1-8B@main", mount_location="/app/models/llama", allow_patterns=["*.safetensors", "config.json", "tokenizer.*"], ignore_patterns=["*.md", "*.txt"], ) ### [​](https://docs.baseten.co/training/concepts/storage#how-bdn-serves-training-jobs) How BDN serves training jobs When you submit a training job, BDN compares your `weights` config to what’s already in Baseten storage, pulls anything missing from the upstream source, and stages the full set on the node before your `start_commands` run. Data delivery happens entirely during the `CREATED` and `DEPLOYING` phases. Two cache tiers sit in front of Baseten’s mirror: * **Cluster-local cache:** shared across nodes in a GPU cluster. Populated the first time a job in that cluster pulls a given set of files. * **Node-local cache:** lives on the node itself. Populated when a job lands on that node. Both caches evict with LRU. On a **node-local hit**, the node mounts the data directly and your job starts almost immediately. On a **cluster-local hit**, BDN transfers the data from the cluster cache to the node, which adds a small amount of deploy time. On a **full miss**, BDN pulls from its mirror, which adds more deploy time. None of these affect training throughput. ### [​](https://docs.baseten.co/training/concepts/storage#bdn-or-training-cache) BDN or training cache? Use BDN for read-only inputs that are known at job start, like model weights and frozen datasets. Baseten delivers them before training begins, so you never pay for IO or compute time while they load. Use the [training cache](https://docs.baseten.co/training/concepts/cache) when you need read-write storage that persists across jobs, or when one job produces data that a later job consumes. Common examples: pip package installs, compiled artifacts, and preprocessed datasets you build once and reuse. * * * [​](https://docs.baseten.co/training/concepts/storage#storage-types-overview) Storage types overview ------------------------------------------------------------------------------------------------------- Baseten Training provides four ways to move data in and out of a job: | Storage type | Persistence | Use case | | --- | --- | --- | | [BDN (`weights`)](https://docs.baseten.co/training/concepts/storage#load-weights-and-data-with-bdn) | Mirrored once; cluster- and node-local LRU caches | Read-only model weights and datasets known at job start. | | [Training cache](https://docs.baseten.co/training/concepts/cache) | Read-write, persistent between jobs | Pip packages, compiled artifacts, preprocessed datasets. | | [Checkpointing](https://docs.baseten.co/training/concepts/checkpoints) | Backed up to cloud storage | Model checkpoints and artifacts you want to deploy or download. | | Ephemeral storage | Cleared after job completes | Temporary files, intermediate outputs. | Training cache is scoped to a single GPU cluster. Data cached on one cluster (for example, H100) is not available on a different cluster (for example, H200). To use the same data on multiple clusters, duplicate it to each cluster’s cache or load it through BDN. ### [​](https://docs.baseten.co/training/concepts/storage#ephemeral-storage) Ephemeral storage Write temporary files to the `$BT_SCRATCH_DIR` directory. This path is backed by local NVMe storage on the node and is cleared when your job completes. Use it for: * Temporary files during training. * Intermediate outputs that don’t need to persist. * Scratch space for data processing. import os scratch = os.environ["BT_SCRATCH_DIR"] tmp_output = os.path.join(scratch, "processed_data") Do not write temporary files to arbitrary paths like `/tmp` or `/root`. Always use `$BT_SCRATCH_DIR` so Baseten can manage storage across hardware configurations. [​](https://docs.baseten.co/training/concepts/storage#loading-data-in-your-training-script) Loading data in your training script ----------------------------------------------------------------------------------------------------------------------------------- When data isn’t available through a BDN-supported URI scheme, download it directly in your training script. This works well for datasets loaded from framework-specific libraries or custom download logic. * Amazon S3 * Hugging Face * Google Cloud Storage Use [Baseten secrets](https://docs.baseten.co/organization/secrets) to authenticate to your S3 bucket. 1. Add your AWS credentials as secrets in your Baseten account. 2. Reference the secrets in your job configuration: from truss_train import definitions runtime = definitions.Runtime( environment_variables={ "AWS_ACCESS_KEY_ID": definitions.SecretReference(name="aws_access_key_id"), "AWS_SECRET_ACCESS_KEY": definitions.SecretReference(name="aws_secret_access_key"), }, ) 3. Download from S3 in your training script: import boto3 s3 = boto3.client('s3') s3.download_file('my-bucket', 'training-data.tar.gz', '/path/to/local/file') To avoid re-downloading large datasets on each job, download to the [training cache](https://docs.baseten.co/training/concepts/cache) and check if files exist before downloading. Reference a Hugging Face dataset in your training code: from datasets import load_dataset ds = load_dataset("your-username/your-dataset", split="train") For private datasets, authenticate using a Hugging Face token stored in [Baseten secrets](https://docs.baseten.co/organization/secrets) : runtime = definitions.Runtime( environment_variables={ "HF_TOKEN": definitions.SecretReference(name="hf_access_token"), }, ) Authenticate via [Baseten secrets](https://docs.baseten.co/organization/secrets) and download in your training code: from google.cloud import storage client = storage.Client() bucket = client.bucket('my-bucket') blob = bucket.blob('training-data.tar.gz') blob.download_to_filename('/path/to/local/file') [​](https://docs.baseten.co/training/concepts/storage#data-size-and-limits) Data size and limits --------------------------------------------------------------------------------------------------- | Size | Description | | --- | --- | | Small | A few GBs. | | Medium | Up to 1 TB (most common). | | Large | 1-10 TB. | The default training cache is 1 TB. [Contact support](mailto:support@baseten.co) to increase the cache size for larger datasets. [​](https://docs.baseten.co/training/concepts/storage#data-security) Data security ------------------------------------------------------------------------------------- Data transfer happens within Baseten’s VPC using secure connections. Baseten doesn’t share customer data across tenants. When you enable [training cache](https://docs.baseten.co/training/concepts/cache) , data persists between jobs until you delete the project. Ephemeral storage is cleared when your job completes. For self-hosted deployments, training can use storage buckets in your own AWS or GCP account. To learn more and access official policies and certifications, visit the [Baseten Trust Center](https://trust.baseten.co/) . [​](https://docs.baseten.co/training/concepts/storage#storage-performance) Storage performance ------------------------------------------------------------------------------------------------- Read and write speeds vary by cluster and storage configuration: | Storage type | Write speed | Read speed | | --- | --- | --- | | Node storage | 1.2-1.8 GB/s | 1.7-2.1 GB/s | | Training cache | 340 MB/s - 1.0 GB/s | 470 MB/s - 1.6 GB/s | For workloads with high I/O requirements or large storage requirements, [contact support](mailto:support@baseten.co) . [​](https://docs.baseten.co/training/concepts/storage#next-steps) Next steps ------------------------------------------------------------------------------- * **[BDN configuration reference](https://docs.baseten.co/development/model/bdn#configuration-reference) **: Full list of weight source options, authentication methods, and supported URI schemes. * **[Cache](https://docs.baseten.co/training/concepts/cache) **: Persist data between jobs and speed up training iterations. * **[Checkpointing](https://docs.baseten.co/training/concepts/checkpoints) **: Save and manage model checkpoints during training. * **[Multinode training](https://docs.baseten.co/training/concepts/multinode) **: Scale training across multiple nodes with shared cache access. Was this page helpful? YesNo [Previous](https://docs.baseten.co/training/concepts/checkpoints) [Multinode trainingLearn how to configure and run multinode training jobs with Baseten Training.\ \ Next](https://docs.baseten.co/training/concepts/multinode) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Overview - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/overview#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) The full OpenAPI spec is available at [api.baseten.co/v1/spec](https://api.baseten.co/v1/spec) for generating API clients. Management API requests are rate limited per API key. See [Rate limits](https://docs.baseten.co/reference/management-api/rate-limits) for the per-endpoint limits and how to handle `429` responses. [​](https://docs.baseten.co/reference/management-api/overview#model-endpoints) Model endpoints ------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/models`](https://docs.baseten.co/reference/management-api/models/gets-all-models) | Get all models | | `GET` | [`/v1/models/{model_id}`](https://docs.baseten.co/reference/management-api/models/gets-a-model-by-id) | Get models by ID | | `DEL` | [`/v1/models/{model_id}`](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id) | Delete models | [​](https://docs.baseten.co/reference/management-api/overview#chain-endpoints) Chain endpoints ------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/chains`](https://docs.baseten.co/reference/management-api/chains/gets-all-chains) | Get all Chains | | `GET` | [`/v1/chains/{chain_id}`](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id) | Get a Chain by ID | | `DEL` | [`/v1/chains/{chain_id}`](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id) | Delete Chains | [​](https://docs.baseten.co/reference/management-api/overview#deployment-endpoints) Deployment endpoints ----------------------------------------------------------------------------------------------------------- * Models * Chains ### [​](https://docs.baseten.co/reference/management-api/overview#activate-a-model-deployment) Activate a model deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/activate`](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-deployment-associated-with-an-environment) | **Activate** an environment | | `POST` | [`/v1/models/{model_id}/deployments/development/activate`](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-development-deployment) | **Activate** development | | `POST` | [`/v1/models/{model_id}/deployments/{deployment_id}/activate`](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-deployment) | **Activate** a deployment | | `POST` | [`/v1/models/{model_id}/deployments/production/activate`](https://docs.baseten.co/reference/management-api/deployments/activate/activates-production-deployment) | **Activate** production | ### [​](https://docs.baseten.co/reference/management-api/overview#deactivate-a-model-deployment) Deactivate a model deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/deactivate`](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment) | **Deactivate** an environment | | `POST` | [`/v1/models/{model_id}/deployments/development/deactivate`](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment) | **Deactivate** development | | `POST` | [`/v1/models/{model_id}/deployments/{deployment_id}/deactivate`](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment) | **Deactivate** a deployment | | `POST` | [`/v1/models/{model_id}/deployments/production/deactivate`](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment) | **Deactivate** production | ### [​](https://docs.baseten.co/reference/management-api/overview#retry-a-model-deployment) Retry a model deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/models/{model_id}/deployments/development/retry`](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment) | **Retry** development | | `POST` | [`/v1/models/{model_id}/deployments/{deployment_id}/retry`](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment) | **Retry** a deployment | | `POST` | [`/v1/models/{model_id}/deployments/production/retry`](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment) | **Retry** production | ### [​](https://docs.baseten.co/reference/management-api/overview#promote-a-model-deployment) Promote a model deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/promote`](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment) | **Promote** to model **environment** | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/cancel_promotion`](https://docs.baseten.co/reference/management-api/deployments/promote/cancel-promotion) | **Cancel** a promotion to an environment | | `POST` | [`/v1/models/{model_id}/deployments/development/promote`](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-development-deployment-to-production) | **Promote** development deployment | | `POST` | [`/v1/models/{model_id}/deployments/{deployment_id}/promote`](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-production) | **Promote** any deployment | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/pause_promotion`](https://docs.baseten.co/reference/management-api/deployments/promote/pause-promotion) | **Pause** rolling deployment | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/resume_promotion`](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion) | **Resume** rolling deployment | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/force_cancel_promotion`](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion) | **Force cancel** rolling deployment | | `POST` | [`/v1/models/{model_id}/environments/{env_name}/force_roll_forward_promotion`](https://docs.baseten.co/reference/management-api/deployments/promote/force-roll-forward-promotion) | **Force roll forward** promotion | ### [​](https://docs.baseten.co/reference/management-api/overview#autoscaling) Autoscaling | Method | Endpoint | Description | | --- | --- | --- | | `PATCH` | [`.../deployments/development/autoscaling_settings`](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings) | Updates **development’s autoscaling** settings | | `PATCH` | [`.../deployments/{deployment_id}/autoscaling_settings`](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings) | Updates a **deployment’s autoscaling** settings | | `PATCH` | [`.../deployments/production/autoscaling_settings`](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings) | Updates **production’s autoscaling** settings | ### [​](https://docs.baseten.co/reference/management-api/overview#manage-deployment-endpoints) Manage deployment endpoints | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/models/{model_id}/deployments`](https://docs.baseten.co/reference/management-api/deployments/gets-all-deployments-of-a-model) | Get all model deployments | | `GET` | [`/v1/models/{model_id}/deployments/production`](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-production-deployment) | Production model deployment | | `GET` | [`/v1/models/{model_id}/deployments/development`](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment) | Development model deployment | | `GET` | [`/v1/models/{model_id}/deployments/{deployment_id}`](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id) | Any model deployment by ID | | `GET` | [`/v1/models/{model_id}/deployments/{deployment_id}/logs`](https://docs.baseten.co/reference/management-api/deployments/get-deployment-logs) | Get model deployment logs | | `DEL` | [`/v1/models/{model_id}/deployments/{deployment_id}`](https://docs.baseten.co/reference/management-api/deployments/deletes-a-models-deployment-by-id) | Delete model deployments | | `DEL` | [`/v1/models/{model_id}/deployments/{deployment_id}/replicas/{replica_id}`](https://docs.baseten.co/reference/management-api/deployments/terminates-deployment-replica) | Terminate deployment replica | ### [​](https://docs.baseten.co/reference/management-api/overview#deactivate-a-chain-deployment) Deactivate a Chain deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/chains/{chain_id}/deployments/{chain_deployment_id}/deactivate`](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment) | **Deactivate** a Chain deployment | ### [​](https://docs.baseten.co/reference/management-api/overview#promote-a-chain-deployment) Promote a Chain deployment | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/chains/{chain_id}/environments/{env_name}/promote`](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-chain-deployment-to-an-environment) | Promote to chain environment | ### [​](https://docs.baseten.co/reference/management-api/overview#autoscaling-2) Autoscaling | Method | Endpoint | Description | | --- | --- | --- | | `PATCH` | [`.../chainlet_settings/autoscaling_settings`](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings) | **Update chainlet** environment’s autoscaling settings | ### [​](https://docs.baseten.co/reference/management-api/overview#manage-chain-deployments) Manage Chain deployments | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/chains/{chain_id}/deployments`](https://docs.baseten.co/reference/management-api/deployments/gets-all-chain-deployments) | Get all chain deployments | | `GET` | [`/v1/chains/{chain_id}/deployments/{chain_deployment_id}`](https://docs.baseten.co/reference/management-api/deployments/gets-a-chain-deployment-by-id) | Any chain deployment by ID | | `DEL` | [`/v1/chains/{chain_id}/deployments/{chain_deployment_id}`](https://docs.baseten.co/reference/management-api/deployments/deletes-a-chain-deployment-by-id) | **Delete** chain deployments | [​](https://docs.baseten.co/reference/management-api/overview#environment-endpoints) Environment endpoints ------------------------------------------------------------------------------------------------------------- * Models * Chains | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/models/{model_id}/environments`](https://docs.baseten.co/reference/management-api/environments/create-an-environment) | Create environment | | `GET` | [`/v1/models/{model_id}/environments`](https://docs.baseten.co/reference/management-api/environments/get-all-environments) | Get all environments | | `GET` | [`/v1/models/{model_id}/environments/{env_name}`](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details) | Get an environment details | | `PATCH` | [`/v1/models/{model_id}/environments/{env_name}`](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings) | Update model environment | | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/chains/{chain_id}/environments`](https://docs.baseten.co/reference/management-api/environments/create-a-chain-environment) | Create chain environment | | `GET` | [`/v1/chains/{chain_id}/environments`](https://docs.baseten.co/reference/management-api/environments/get-all-chain-environments) | Get all chain environments | | `GET` | [`/v1/chains/{chain_id}/environments/{env_name}`](https://docs.baseten.co/reference/management-api/environments/get-a-chain-environments-details) | Get a chain environment | | `PATCH` | [`/v1/chains/{chain_id}/environments/{env_name}`](https://docs.baseten.co/reference/management-api/environments/update-a-chain-environments-settings) | Update chain environment | | `POST` | [`/v1/chains/{chain_id}/environments/{env_name}/chainlet_settings/instance_types/update`](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings) | Update chainlet environment’s instance type | [​](https://docs.baseten.co/reference/management-api/overview#instance-type-endpoints) Instance type endpoints ----------------------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/instance_types`](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types) | Get all instance types | | `GET` | [`/v1/instance_type_prices`](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices) | Get instance type prices | [​](https://docs.baseten.co/reference/management-api/overview#team-endpoints) Team endpoints ----------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/teams`](https://docs.baseten.co/reference/management-api/teams/lists-all-teams) | Get all teams | [​](https://docs.baseten.co/reference/management-api/overview#secret-endpoints) Secret endpoints --------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/secrets`](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets) | Get all secrets | | `POST` | [`/v1/secrets`](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret) | Create or update a secret | | `GET` | [`/v1/teams/{team_id}/secrets`](https://docs.baseten.co/reference/management-api/teams/gets-all-team-secrets) | Get all team secrets | | `POST` | [`/v1/teams/{team_id}/secrets`](https://docs.baseten.co/reference/management-api/teams/upserts-a-team-secret) | Create or update a team secret | [​](https://docs.baseten.co/reference/management-api/overview#api-key-endpoints) API Key endpoints ----------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/api_keys`](https://docs.baseten.co/reference/management-api/api-keys/lists-the-users-api-keys) | Get all API keys | | `POST` | [`/v1/api_keys`](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key) | Create an API key | | `DELETE` | [`/v1/api_keys/{api_key_prefix}`](https://docs.baseten.co/reference/management-api/api-keys/delete-an-api-key) | Delete an API key | | `POST` | [`/v1/teams/{team_id}/api_keys`](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key) | Create a team API key | [​](https://docs.baseten.co/reference/management-api/overview#billing-endpoints) Billing endpoints ----------------------------------------------------------------------------------------------------- | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/billing/usage_summary`](https://docs.baseten.co/reference/management-api/billing/gets-billing-usage-summary-for-a-date-range) | Get billing usage summary | [​](https://docs.baseten.co/reference/management-api/overview#training-endpoints) Training endpoints ------------------------------------------------------------------------------------------------------- For a complete reference and request schemas, see the [Training API overview](https://docs.baseten.co/reference/training-api/overview) . ### [​](https://docs.baseten.co/reference/management-api/overview#training-projects) Training projects | Method | Endpoint | Description | | --- | --- | --- | | `GET` | [`/v1/training_projects`](https://docs.baseten.co/reference/training-api/get-training-projects) | List all training projects | | `POST` | [`/v1/training_projects`](https://docs.baseten.co/reference/training-api/create-training-project) | Create a training project | | `GET` | [`/v1/training_projects/{training_project_id}`](https://docs.baseten.co/reference/training-api/get-training-project) | Get a training project | | `DELETE` | [`/v1/training_projects/{training_project_id}`](https://docs.baseten.co/reference/training-api/delete-training-project) | Delete a training project | | `POST` | [`/v1/teams/{team_id}/training_projects`](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project) | Create a team training project | | `GET` | [`/v1/training_projects/{training_project_id}/cache/summary`](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary) | Get training project cache summary | ### [​](https://docs.baseten.co/reference/management-api/overview#training-jobs) Training jobs | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/training_projects/{training_project_id}/jobs`](https://docs.baseten.co/reference/training-api/create-training-job) | Create a training job | | `GET` | [`/v1/training_projects/{training_project_id}/jobs`](https://docs.baseten.co/reference/training-api/list-training-jobs) | List all jobs in a project | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}`](https://docs.baseten.co/reference/training-api/get-training-job) | Get a training job by ID | | `DELETE` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}`](https://docs.baseten.co/reference/training-api/delete-training-job) | Delete a training job | | `POST` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/stop`](https://docs.baseten.co/reference/training-api/stop-training-job) | Stop a training job | | `POST` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/recreate`](https://docs.baseten.co/reference/training-api/recreate-training-job) | Recreate a training job | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/logs`](https://docs.baseten.co/reference/training-api/get-training-job-logs) | Get training job logs | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/metrics`](https://docs.baseten.co/reference/training-api/get-training-job-metrics) | Get training job metrics | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoints`](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints) | List job checkpoints | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoint_files`](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files) | Get checkpoint files | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/download`](https://docs.baseten.co/reference/training-api/download-training-job) | Download training job artifacts | | `GET` | [`/v1/training_projects/{training_project_id}/jobs/{training_job_id}/auth_codes`](https://docs.baseten.co/reference/training-api/get-auth-codes-for-training-job) | Get auth codes for a training job | | `POST` | [`/v1/training_jobs/search`](https://docs.baseten.co/reference/training-api/search-training-jobs) | Search across all training jobs | [​](https://docs.baseten.co/reference/management-api/overview#frontier-gateway-endpoints) Frontier Gateway endpoints ----------------------------------------------------------------------------------------------------------------------- For the conceptual guide and end-to-end examples, see the [Frontier Gateway overview](https://docs.baseten.co/frontier-gateway/overview) . ### [​](https://docs.baseten.co/reference/management-api/overview#groups) Groups | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/gateway/groups`](https://docs.baseten.co/reference/gateway/groups/create-a-group) | Create a group | | `GET` | [`/v1/gateway/groups`](https://docs.baseten.co/reference/gateway/groups/list-groups) | List groups | | `GET` | [`/v1/gateway/groups/{group_id}`](https://docs.baseten.co/reference/gateway/groups/get-a-group) | Get a group | | `PATCH` | [`/v1/gateway/groups/{group_id}`](https://docs.baseten.co/reference/gateway/groups/update-a-group) | Update a group | | `DELETE` | [`/v1/gateway/groups/{group_id}`](https://docs.baseten.co/reference/gateway/groups/delete-a-group) | Delete a group | ### [​](https://docs.baseten.co/reference/management-api/overview#api-keys) API keys | Method | Endpoint | Description | | --- | --- | --- | | `POST` | [`/v1/gateway/groups/{group_id}/api_keys`](https://docs.baseten.co/reference/gateway/api-keys/create-an-api-key) | Create an API key | | `GET` | [`/v1/gateway/groups/{group_id}/api_keys`](https://docs.baseten.co/reference/gateway/api-keys/list-api-keys-for-a-group) | List API keys for a group | | `GET` | [`/v1/gateway/groups/{group_id}/api_keys/{api_key_prefix}`](https://docs.baseten.co/reference/gateway/api-keys/get-an-api-key) | Get an API key | | `DELETE` | [`/v1/gateway/groups/{group_id}/api_keys/{api_key_prefix}`](https://docs.baseten.co/reference/gateway/api-keys/revoke-an-api-key) | Revoke an API key | Was this page helpful? YesNo [Previous](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment) [Rate limitsRate limits, response shape, and retry handling for the Baseten management API.\ \ Next](https://docs.baseten.co/reference/management-api/rate-limits) ⌘I Assistant Responses are generated using AI and may contain mistakes. --- # Unknown \# Baseten ## Docs - \[AI tools\](https://docs.baseten.co/ai-tools.md): Connect AI tools to Baseten documentation for context-aware assistance with deploying and serving models. - \[Cancel a queued async request.\](https://docs.baseten.co/api-reference/cancel-a-queued-async-request.md): Cancels an async request. Only requests with \`QUEUED\` status may be canceled. Rate limited to 20 requests per second. - \[Get the status of an async request.\](https://docs.baseten.co/api-reference/get-the-status-of-an-async-request.md): Returns the current status of an async model or chain request. Rate limited to 20 requests per second. - \[Asynchronously call a named environment of a chain.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain.md) - \[Asynchronously call a named environment of a model.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model.md) - \[Asynchronously call a specific deployment of a chain.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain.md) - \[Asynchronously call a specific deployment of a model.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model.md) - \[Asynchronously call the development deployment of a chain.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain.md) - \[Asynchronously call the development deployment of a model.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model.md) - \[Asynchronously call the production environment of a chain.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain.md): Enqueues an asynchronous request for the chain deployment promoted to the production environment. - \[Asynchronously call the production environment of a model.\](https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model.md): Enqueues an asynchronous predict request for the deployment promoted to the production environment. Returns a request ID that can be used to poll for status or cancel the request. - \[Call a specific chain deployment by deployment ID.\](https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id.md) - \[Call a specific deployment of a model by deployment ID.\](https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id.md): Sends a synchronous predict request to the specified deployment. - \[Call the chain deployment associated with a specified environment.\](https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment.md) - \[Call the development deployment of a chain.\](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain.md) - \[Call the development deployment of a model.\](https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model.md): Sends a synchronous predict request to the development deployment. - \[Call the model deployment associated with a specified environment.\](https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment.md): Sends a synchronous predict request to the deployment promoted to the specified environment. - \[Call the production environment of a chain.\](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain.md): Sends a synchronous request to the chain deployment promoted to the production environment. The request body is forwarded to the chain's \`run\_remote\` entrypoint. - \[Call the production environment of a model.\](https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model.md): Sends a synchronous predict request to the deployment promoted to the production environment. The request body is forwarded directly to the model's \`predict\` function. - \[Get async queue status for a named environment.\](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment.md) - \[Get async queue status for a specific deployment.\](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment.md) - \[Get async queue status for the development deployment.\](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment.md) - \[Get async queue status for the production environment.\](https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment.md): Returns the number of queued and in-progress async requests for the deployment promoted to the production environment. Rate limited to 20 requests per second. - \[Wake a named environment of a model.\](https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model.md) - \[Wake a specific deployment of a model by deployment ID.\](https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id.md) - \[Wake the development deployment of a model.\](https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model.md) - \[Wake the production environment of a model.\](https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model.md): Triggers a wake for the deployment promoted to the production environment. Returns immediately with 202 Accepted. - \[Asynchronously call a regional environment of a chain.\](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain.md): Enqueues an asynchronous run\_remote request via a regional hostname. The environment is determined by the hostname, not the path. - \[Asynchronously call a regional environment of a model.\](https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model.md): Enqueues an asynchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. - \[Call a regional environment of a chain.\](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain.md): Sends a synchronous run\_remote request via a regional hostname. The environment is determined by the hostname, not the path. - \[Call a regional environment of a model.\](https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model.md): Sends a synchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. - \[Get async queue status for a regional environment.\](https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment.md) - \[Wake a regional environment of a model.\](https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model.md) - \[How Baseten works\](https://docs.baseten.co/concepts/howbasetenworks.md): The moving parts behind training, deployment, request routing, autoscaling, and environment promotion on Baseten. - \[Why Baseten\](https://docs.baseten.co/concepts/whybaseten.md): Production training and inference on dedicated infrastructure, for teams that have outgrown shared API endpoints. - \[Cold starts\](https://docs.baseten.co/deployment/autoscaling/cold-starts.md): Understand cold starts and how to minimize their impact on your deployments. - \[Autoscaling\](https://docs.baseten.co/deployment/autoscaling/overview.md): Configure autoscaling to dynamically adjust replicas based on traffic while minimizing idle compute costs. - \[Request lifecycle\](https://docs.baseten.co/deployment/autoscaling/request-lifecycle.md): What happens to a request from submission to response, including routing, queuing, the 1200-second sync predict timeout, and error handling. - \[Traffic patterns\](https://docs.baseten.co/deployment/autoscaling/traffic-patterns.md): Identify your traffic pattern and configure autoscaling settings to match. - \[CI/CD\](https://docs.baseten.co/deployment/ci-cd.md): Automate Truss deployments with GitHub Actions. - \[Concepts\](https://docs.baseten.co/deployment/concepts.md): Deployments, environments, resources, autoscaling, and CI/CD on Baseten. - \[Deployments\](https://docs.baseten.co/deployment/deployments.md): Deploy, manage, and scale machine learning models with Baseten - \[Environments\](https://docs.baseten.co/deployment/environments.md): Manage your model's release cycles with environments. - \[Regional environments\](https://docs.baseten.co/deployment/regional-environments.md): Guarantee inference data stays in a specific geographic region with regional environments. - \[Resources\](https://docs.baseten.co/deployment/resources.md): Manage and configure model resources - \[Rolling deployments\](https://docs.baseten.co/deployment/rolling-deployments.md): Gradually shift traffic to a new deployment with replica-based rolling deployments. - \[Binary IO\](https://docs.baseten.co/development/chain/binaryio.md): Performant serialization of numeric data - \[Concepts\](https://docs.baseten.co/development/chain/concepts.md): Glossary of Chains concepts and terminology - \[Deploy\](https://docs.baseten.co/development/chain/deploy.md): Deploy your Chain on Baseten - \[Architecture and design\](https://docs.baseten.co/development/chain/design.md): How to structure your Chainlets - \[Engine-Builder LLM Models\](https://docs.baseten.co/development/chain/engine-builder-models.md): Engine-Builder LLM models are pre-trained models that are optimized for specific inference tasks. - \[Error Handling\](https://docs.baseten.co/development/chain/errorhandling.md): Understanding and handling Chains errors - \[Your first Chain\](https://docs.baseten.co/development/chain/getting-started.md): Build and deploy two example Chains - \[Invocation\](https://docs.baseten.co/development/chain/invocation.md): Call your deployed Chain - \[Local Development\](https://docs.baseten.co/development/chain/localdev.md): Iterating, Debugging, Testing, Mocking - \[Overview\](https://docs.baseten.co/development/chain/overview.md) - \[Streaming\](https://docs.baseten.co/development/chain/streaming.md): Streaming outputs, reducing latency, SSEs - \[Truss Integration\](https://docs.baseten.co/development/chain/stub.md): Integrate deployed Truss models with stubs - \[Subclassing\](https://docs.baseten.co/development/chain/subclassing.md): Modularize and re-use Chainlet implementations - \[Watch\](https://docs.baseten.co/development/chain/watch.md): Live-patch deployed code - \[Overview\](https://docs.baseten.co/development/concepts.md): Choose between self-deployed models and Chains, and learn the development cycle that applies to both. - \[b10cache\](https://docs.baseten.co/development/model/b10cache.md): Persist data across replicas or deployments - \[Base Docker images\](https://docs.baseten.co/development/model/base-images.md): A guide to configuring a base image for your truss - \[Baseten Delivery Network\](https://docs.baseten.co/development/model/bdn.md): Optimize cold starts with multi-tier caching and data delivery - \[Custom build commands\](https://docs.baseten.co/development/model/build-commands.md): How to run your own docker commands during the build stage - \[Build your first model\](https://docs.baseten.co/development/model/build-your-first-model.md): Deploy a model to Baseten with just a config file. Pick an open-source model from Hugging Face, choose a GPU, and get an endpoint in minutes. - \[Configuration\](https://docs.baseten.co/development/model/configuration.md): How to configure your model. - \[Custom health checks\](https://docs.baseten.co/development/model/custom-health-checks.md): Customize the health of your deployments. - \[Custom model code\](https://docs.baseten.co/development/model/custom-model-code.md): Deploy a model with custom Python using the Truss Model class. - \[Custom Docker containers\](https://docs.baseten.co/development/model/custom-server.md): Deploy custom Docker containers to run inference servers like vLLM, SGLang, Triton, or any containerized application. - \[Data and storage\](https://docs.baseten.co/development/model/data-directory.md): Get files into your deployment and persist files your model writes at runtime. - \[Deploy and iterate\](https://docs.baseten.co/development/model/deploy-and-iterate.md): Use development deployments with live patching for rapid iteration, then promote to production. - \[Access model environments\](https://docs.baseten.co/development/model/environments.md): Configure model behavior based on environment - \[gRPC\](https://docs.baseten.co/development/model/grpc.md): Invoke your model over gRPC. - \[HTTP endpoints\](https://docs.baseten.co/development/model/http-endpoints.md): Expose server HTTP endpoints from custom model code. - \[Implementation\](https://docs.baseten.co/development/model/implementation.md): How to implement your model. - \[Cached weights\](https://docs.baseten.co/development/model/model-cache.md): Accelerate cold starts and availability by prefetching and caching your weights. - \[Developing a model on Baseten\](https://docs.baseten.co/development/model/overview.md): This page introduces the key concepts and workflow you'll use to package, configure, and iterate on models using Baseten's developer tooling. - \[Performance optimization\](https://docs.baseten.co/development/model/performance-optimization.md): Optimize model latency, throughput, and cost with Baseten engines - \[Private Docker registries\](https://docs.baseten.co/development/model/private-registries.md): Pull images from private container registries in Baseten deployments. - \[Using request objects / cancellation\](https://docs.baseten.co/development/model/requests.md): Get more control by directly using the request object. - \[Custom responses\](https://docs.baseten.co/development/model/responses.md): Get more control by directly creating the response object. - \[Secrets\](https://docs.baseten.co/development/model/secrets.md): Use secrets securely in your models - \[Streaming output\](https://docs.baseten.co/development/model/streaming.md): Stream generative model output as it's produced - \[Torch compile caching\](https://docs.baseten.co/development/model/torch-compile-cache.md): Accelerate cold starts by loading in previous compilation artifacts. - \[WebSockets\](https://docs.baseten.co/development/model/websockets.md): Enable real-time, streaming, bidirectional communication using WebSockets for Truss models and Chains. - \[BEI-Bert\](https://docs.baseten.co/engines/bei/bei-bert.md): Bidirectional encoder embeddings with cold-start optimization - \[Configuration reference\](https://docs.baseten.co/engines/bei/bei-reference.md): Complete reference config for BEI and BEI-Bert engines - \[Named entity recognition\](https://docs.baseten.co/engines/bei/ner.md): Token-level entity classification on BEI-Bert with /predict\_tokens - \[Overview\](https://docs.baseten.co/engines/bei/overview.md): Production-grade embeddings, reranking, and classification models - \[Advanced features for BIS-LLM\](https://docs.baseten.co/engines/bis-llm/advanced-features.md): KV-aware routing, disaggregated serving, and speculative decoding - \[Configuration reference\](https://docs.baseten.co/engines/bis-llm/bis-llm-config.md): Complete reference config for v2 inference stack and MoE models - \[Migrate from Engine-Builder-LLM\](https://docs.baseten.co/engines/bis-llm/migrate-from-v1.md): Translate a v1 Engine-Builder-LLM configuration to BIS-LLM (v2), including the autoscaling, speculation, and routing changes that aren't just renames - \[Overview\](https://docs.baseten.co/engines/bis-llm/overview.md): Token-based autoscaling, KV-aware routing, disaggregated serving, and speculative decoding for MoE and large dense models - \[Custom engine builder\](https://docs.baseten.co/engines/engine-builder-llm/custom-engine-builder.md): Implement custom model.py for business logic, logging, and advanced inference patterns - \[Configuration reference\](https://docs.baseten.co/engines/engine-builder-llm/engine-builder-config.md): Complete reference config for dense text generation models - \[Speculative decoding\](https://docs.baseten.co/engines/engine-builder-llm/lookahead-decoding.md): Lookahead decoding on Engine-Builder-LLM (v1) for code generation and predictable content - \[LoRA support\](https://docs.baseten.co/engines/engine-builder-llm/lora-support.md): Multi-LoRA adapters for Engine-Builder-LLM engine - \[Overview\](https://docs.baseten.co/engines/engine-builder-llm/overview.md): Dense LLM text generation with lookahead decoding and structured outputs - \[Overview\](https://docs.baseten.co/engines/index.md): Inference engines for embeddings, dense LLMs, MoE models, and Enterprise serving - \[Autoscaling engines\](https://docs.baseten.co/engines/performance-concepts/autoscaling-engines.md): Engine-specific autoscaling settings for BEI, Engine-Builder-LLM, and BIS-LLM - \[Deploy from cloud storage\](https://docs.baseten.co/engines/performance-concepts/cloud-storage-deployment.md): Connect your S3 bucket, GCS bucket, Azure container, or Hugging Face repository to Baseten's TRT-LLM inference engines and deploy without re-uploading weights. - \[Quantization guide\](https://docs.baseten.co/engines/performance-concepts/quantization-guide.md): FP8 and FP4 trade-offs and hardware requirements for all engines - \[Serve embeddings with BEI\](https://docs.baseten.co/examples/bei.md): Deploy embedding, reranking, and classification models on Baseten Embeddings Inference. - \[Transcribe audio with Chains\](https://docs.baseten.co/examples/chains-audio-transcription.md): Process hours of audio in seconds using efficient chunking, distributed inference, and optimized GPU resources. - \[Build a RAG pipeline with Chains\](https://docs.baseten.co/examples/chains-build-rag.md): Combine retrieval and generation into a single compound workflow. - \[Customize a model\](https://docs.baseten.co/examples/customize-a-model.md): Deploy a model with custom Python code using the Truss Model class. - \[Deploy a Hugging Face model\](https://docs.baseten.co/examples/deploy-a-hugging-face-model.md): Deploy Gemma 4 26B on Baseten with vLLM, BDN-cached weights, EAGLE3 speculative decoding, and prefix caching. - \[Build and deploy an LLM\](https://docs.baseten.co/examples/deploy-a-llm.md): Package and deploy an LLM with Truss, from model setup to inference. - \[Deploy your first model\](https://docs.baseten.co/examples/deploy-your-first-model.md): Deploy an open-source LLM to Baseten with just a config file and get an OpenAI-compatible API endpoint. - \[Deploy a Dockerized model\](https://docs.baseten.co/examples/docker.md): Deploy any model in a pre-built Docker container. - \[Generate images with Flux\](https://docs.baseten.co/examples/image-generation.md): Deploy Flux Schnell as a text-to-image endpoint. - \[Qwen3 Embedding\](https://docs.baseten.co/examples/models/embedding/qwen3-embedding.md): Alibaba's Qwen3 Embedding is an 8B text embedding model that maps text into dense vectors for semantic search, retrieval-augmented generation, clustering, and classification. - \[Qwen3 Reranker\](https://docs.baseten.co/examples/models/embedding/qwen3-reranker.md): Alibaba's Qwen3 Reranker is an 8B cross-encoder for high-quality passage reranking in retrieval-augmented generation pipelines. - \[FLUX.1\](https://docs.baseten.co/examples/models/image-gen/flux1.md): FLUX.1 recipes: 2 variants (dev, schnell), diffusion-transformer architecture. - \[Gemma 4\](https://docs.baseten.co/examples/models/llm/gemma-4.md): Gemma 4 recipes: 4 variants (E2B, E4B, 26B A4B, 31B), Dense and MoE architectures. - \[GLM-4.7\](https://docs.baseten.co/examples/models/llm/glm-4.7.md): GLM-4.7 recipes: 2 variants (Standard, Flash), MoE architecture. - \[GLM-5\](https://docs.baseten.co/examples/models/llm/glm-5.md): Z.ai's GLM-5 frontier model, served from an FP8 checkpoint on B200:8. - \[GPT-OSS\](https://docs.baseten.co/examples/models/llm/gpt-oss.md): GPT-OSS recipes: 2 variants (20B, 120B), Dense and MoE architectures. - \[Llama 3.1\](https://docs.baseten.co/examples/models/llm/llama-3.1.md): Meta's Llama 3.1 8B instruction-tuned model. Runs on a single B200 from NVIDIA's FP8 checkpoint with EAGLE3 speculative decoding for high concurrent throughput. - \[Llama 3.2\](https://docs.baseten.co/examples/models/llm/llama-3.2.md): Meta's compact Llama 3.2 instruction-tuned model. Runs on a single H100 40GB for low-cost chat and edge-adjacent workloads. - \[Llama 3.3\](https://docs.baseten.co/examples/models/llm/llama-3.3.md): Meta's Llama 3.3 70B instruction-tuned model. Runs on H100:4 through Baseten Inference Stack from NVIDIA's FP8 checkpoint, tuned for low time-to-first-token. - \[Llama 4\](https://docs.baseten.co/examples/models/llm/llama-4.md): Meta's Llama 4 Scout is a 17B-active MoE with native multimodal support and a 10M token context window. - \[MiniMax M2.5\](https://docs.baseten.co/examples/models/llm/minimax-m2.5.md): Large MoE model with native reasoning and tool calling. Uses the MiniMax-specific append-think reasoning format. - \[Nemotron 3\](https://docs.baseten.co/examples/models/llm/nemotron-3.md): NVIDIA's Nemotron 3 Super 120B A12B Mixture-of-Experts model. Runs on B200:2 through Baseten Inference Stack with MTP speculative decoding and the NVFP4-quantized checkpoint, tuned for high-throughput reasoning. - \[Qwen3\](https://docs.baseten.co/examples/models/llm/qwen3.md): Sparse MoE model with 235B total parameters (22B active per token). FP8-quantized checkpoint for production-scale reasoning and agentic workflows. - \[Qwen3.5\](https://docs.baseten.co/examples/models/llm/qwen3.5.md): Qwen3.5 recipes: 4 variants (4B, 9B, 35B, 122B), Dense, Hybrid MoE, and MoE architectures. - \[Qwen3.6\](https://docs.baseten.co/examples/models/llm/qwen3.6.md): Qwen3.6 recipes: 2 variants (27B, 35B-A3B), Dense and Hybrid MoE architectures. - \[Qwen3-ASR\](https://docs.baseten.co/examples/models/transcription/qwen3-asr.md): Alibaba's Qwen3-ASR is a compact 1.7B speech-to-text model with multilingual transcription support. - \[Voxtral\](https://docs.baseten.co/examples/models/transcription/voxtral.md): Mistral's Voxtral Mini Realtime is a 4B speech-to-text model tuned for real-time streaming transcription. - \[Deploy LLMs with Ollama\](https://docs.baseten.co/examples/ollama.md): Run LLMs on Ollama as a custom Docker server. - \[Building with Baseten\](https://docs.baseten.co/examples/overview.md) - \[Deploy LLMs with SGLang\](https://docs.baseten.co/examples/sglang.md): Run LLMs on SGLang's high-performance serving framework. - \[Stream LLM responses\](https://docs.baseten.co/examples/streaming.md): Stream LLM output token by token. - \[Add system packages\](https://docs.baseten.co/examples/system-packages.md): Deploy a model with both Python and system dependencies. - \[Deploy LLMs with TensorRT-LLM\](https://docs.baseten.co/examples/tensorrt-llm.md): Optimize LLMs for low latency and high throughput. - \[Generate speech with Kokoro\](https://docs.baseten.co/examples/text-to-speech.md): Deploy Kokoro as a text-to-speech endpoint. - \[Deploy LLMs with vLLM\](https://docs.baseten.co/examples/vllm.md): Run any open-source LLM on vLLM's serving framework. - \[Manage groups and API keys\](https://docs.baseten.co/frontier-gateway/api-keys.md): Walk the full lifecycle: create groups, build a hierarchy, mint and revoke API keys, and delete groups when a customer churns. - \[Billing webhooks\](https://docs.baseten.co/frontier-gateway/billing-webhooks.md): Receive signed per-request usage events from Frontier Gateway and pipe them into your billing provider out-of-band from the inference path. - \[Calling your model\](https://docs.baseten.co/frontier-gateway/calling-your-model.md): Make your first inference call through Baseten Frontier Gateway with a federated API key issued by your AI lab. - \[Get started\](https://docs.baseten.co/frontier-gateway/get-started.md): Create a group, mint an API key, and call your model through the gateway. - \[Baseten Frontier Gateway\](https://docs.baseten.co/frontier-gateway/overview.md): A managed API gateway for AI labs to serve hosted models under a branded URL with hierarchical groups, inherited rate and usage limits, and billing webhooks. - \[Rate and usage limits\](https://docs.baseten.co/frontier-gateway/rate-limits.md): Per-group, per-model token and request limits, two inheritance modes, and how Frontier Gateway computes the effective limits the runtime enforces. - \[Async inference\](https://docs.baseten.co/inference/async.md): Run asynchronous inference on deployed models - \[Call your model\](https://docs.baseten.co/inference/calling-your-model.md): Run inference on deployed models - \[Function calling\](https://docs.baseten.co/inference/function-calling.md): Tool selection and structured function calls with LLMs - \[Configure HTTP clients\](https://docs.baseten.co/inference/http-client-configuration.md): Connection pooling, retries, and timeouts for reliable inference requests. Baseten's default request timeout is 20 minutes (1200 seconds) for sync predict and 60 minutes (3600 seconds) for async submit. - \[Integrations\](https://docs.baseten.co/inference/integrations.md): Integrate your models with tools and use Baseten anywhere - \[JSON mode\](https://docs.baseten.co/inference/json-mode.md): Constrain model output to syntactically valid JSON - \[Deprecation\](https://docs.baseten.co/inference/model-apis/deprecation.md): Baseten's deprecation policy for Model APIs - \[Model APIs\](https://docs.baseten.co/inference/model-apis/overview.md): OpenAI-compatible endpoints for high-performance LLMs - \[Rate limits and budgets\](https://docs.baseten.co/inference/model-apis/rate-limits-and-budgets.md): Rate limits and usage budgets for Model APIs - \[Reasoning\](https://docs.baseten.co/inference/model-apis/reasoning.md): Control extended thinking for reasoning-capable models - \[Vision\](https://docs.baseten.co/inference/model-apis/vision.md): Send images and videos alongside text to vision-capable models - \[Model I/O in binary\](https://docs.baseten.co/inference/output-format/binary.md): Decode and save binary model output - \[Model I/O with files\](https://docs.baseten.co/inference/output-format/files.md): Call models by passing a file or URL - \[Overview\](https://docs.baseten.co/inference/overview.md): Inference on Baseten: Model APIs, self-deployed models, how responses are delivered, structured outputs, tool calling, and client configuration. - \[Performance client\](https://docs.baseten.co/inference/performance-client.md): High-performance client library for embeddings, reranking, classification, and generic batch requests - \[SSH access\](https://docs.baseten.co/inference/ssh.md): Connect to running model deployments directly from your terminal with standard SSH. - \[Streaming\](https://docs.baseten.co/inference/streaming.md): Return model output token by token as it is generated. - \[Structured outputs\](https://docs.baseten.co/inference/structured-outputs.md): JSON schema validation and controlled text generation across all engines - \[Concepts\](https://docs.baseten.co/loops/concepts.md): How Loops sessions, trainer servers, sampling servers, and checkpoints fit together. - \[Loops\](https://docs.baseten.co/loops/overview.md): A training SDK that supports long sequence length, async RL, and one-click checkpoint deploys on the Baseten Inference Stack. - \[Quickstart\](https://docs.baseten.co/loops/quickstart.md): Create a Loops checkpoint and list it. - \[Supported base models\](https://docs.baseten.co/loops/supported-models.md): Hugging Face base models Loops accepts, with sequence-length limits. - \[Tinker compatibility\](https://docs.baseten.co/loops/tinker-compatibility.md): Most Tinker code runs on Loops with one install change, apart from paginated checkpoints, auth, and cluster routing. - \[Export to Datadog\](https://docs.baseten.co/observability/export-metrics/datadog.md): Export metrics from Baseten to Datadog - \[Export to Grafana Cloud\](https://docs.baseten.co/observability/export-metrics/grafana.md): Export metrics from Baseten to Grafana Cloud - \[Export to New Relic\](https://docs.baseten.co/observability/export-metrics/new-relic.md): Export metrics from Baseten to New Relic - \[Overview\](https://docs.baseten.co/observability/export-metrics/overview.md): Export metrics from Baseten to your observability stack - \[Export to Prometheus\](https://docs.baseten.co/observability/export-metrics/prometheus.md): Export metrics from Baseten to Prometheus - \[Metrics support matrix\](https://docs.baseten.co/observability/export-metrics/supported-metrics.md): Every metric you can export from Baseten, with its type and labels - \[Status and health\](https://docs.baseten.co/observability/health.md): Every model deployment in your Baseten workspace has a status to represent its activity and health. - \[Logs\](https://docs.baseten.co/observability/logs.md): Scope logs by environment or deployment, then filter by request ID for individual predictions. - \[Metrics\](https://docs.baseten.co/observability/metrics.md): Understand the load and performance of your model - \[Secure model inference\](https://docs.baseten.co/observability/security.md): Keeping your models safe and private - \[Tracing\](https://docs.baseten.co/observability/tracing.md): Investigate the prediction flow in detail - \[Billing and usage\](https://docs.baseten.co/observability/usage.md): How Baseten meters per-minute usage, and how to manage payment, credits, and invoices for your workspace. - \[Access control\](https://docs.baseten.co/organization/access.md): Manage access to your Baseten organization with role-based access control. - \[API keys\](https://docs.baseten.co/organization/api-keys.md): Authenticate requests to Baseten for deployment, inference, and management. - \[Audit logs\](https://docs.baseten.co/organization/audit-logs.md): Track configuration and access changes across your Baseten organization, and export audit events to your SIEM. - \[OpenID Connect (OIDC) authentication\](https://docs.baseten.co/organization/oidc.md): Use short-lived OIDC tokens to securely authenticate to cloud resources - \[Organization settings\](https://docs.baseten.co/organization/overview.md): Manage your Baseten organization's access, security, and resources. - \[Restricted environments\](https://docs.baseten.co/organization/restricted-environments.md): Control access to sensitive environments like production with environment-level permissions. - \[Secrets\](https://docs.baseten.co/organization/secrets.md): Store and access sensitive credentials in your deployed models. - \[SSO and SCIM\](https://docs.baseten.co/organization/sso-and-scim.md): Authenticate Baseten users through your identity provider and automatically provision accounts, directory groups, and roles. - \[Teams\](https://docs.baseten.co/organization/teams.md): Organize your organization into multiple teams with isolated resources and granular access control. - \[Overview\](https://docs.baseten.co/overview.md): Baseten helps you train, deploy, and serve AI models at scale with high performance and cost efficiency. - \[Quickstart\](https://docs.baseten.co/quickstart.md): Start running inference on Baseten. - \[Truss Push GitHub Action\](https://docs.baseten.co/reference/ci/github-action.md): Deploy and validate a Truss model or chain on Baseten from GitHub Actions. - \[Chains CLI reference\](https://docs.baseten.co/reference/cli/chains/chains-cli.md): Deploy, manage, and develop Chains using the Truss CLI. - \[Baseten command-line tools\](https://docs.baseten.co/reference/cli/index.md): Baseten ships two open-source CLIs: Truss for authoring model code and the Baseten CLI for managing your workspace. This page covers what each one is for, when they overlap, and how to use them together. - \[Loops CLI reference\](https://docs.baseten.co/reference/cli/loops/loops-cli.md): Deploy and inspect Loops sessions, runs, samplers, and checkpoints using the Truss CLI. - \[Training CLI reference\](https://docs.baseten.co/reference/cli/training/training-cli.md): Deploy, manage, and monitor training jobs using the Truss CLI. - \[truss auth\](https://docs.baseten.co/reference/cli/truss/auth.md): Manage authentication with Baseten remotes. - \[truss cleanup\](https://docs.baseten.co/reference/cli/truss/cleanup.md): Clean up Truss data. - \[truss configure\](https://docs.baseten.co/reference/cli/truss/configure.md): Configure Truss settings. - \[truss container\](https://docs.baseten.co/reference/cli/truss/container.md): Run and manage Truss containers locally. - \[truss download\](https://docs.baseten.co/reference/cli/truss/download.md): Download the Truss for a deployed model. - \[truss image\](https://docs.baseten.co/reference/cli/truss/image.md): Build and manage Truss Docker images. - \[truss init\](https://docs.baseten.co/reference/cli/truss/init.md): Create a new Truss project. - \[truss login\](https://docs.baseten.co/reference/cli/truss/login.md): Authenticate with Baseten. - \[truss migrate\](https://docs.baseten.co/reference/cli/truss/migrate.md): Migrate model\_cache and external\_data to the unified weights API. - \[truss model-config\](https://docs.baseten.co/reference/cli/truss/model-config.md): Fetch the config of a deployed model. - \[truss model-logs\](https://docs.baseten.co/reference/cli/truss/model-logs.md): Fetch logs for a deployed model. - \[Truss CLI reference\](https://docs.baseten.co/reference/cli/truss/overview.md): Deploy, manage, and develop models using the Truss CLI. - \[truss predict\](https://docs.baseten.co/reference/cli/truss/predict.md): Call the packaged model. - \[truss push\](https://docs.baseten.co/reference/cli/truss/push.md): Deploy a model to Baseten. - \[truss run-python\](https://docs.baseten.co/reference/cli/truss/run-python.md): Run a Python script in the Truss environment. - \[truss ssh\](https://docs.baseten.co/reference/cli/truss/ssh.md): SSH access to Baseten workloads. - \[truss upgrade\](https://docs.baseten.co/reference/cli/truss/upgrade.md): Upgrade the truss package to the latest or a specified version. - \[truss watch\](https://docs.baseten.co/reference/cli/truss/watch.md): Live reload during development. - \[truss whoami\](https://docs.baseten.co/reference/cli/truss/whoami.md): Show user information. - \[Create an API key\](https://docs.baseten.co/reference/gateway/api-keys/create-an-api-key.md): Mint a federated API key under a Frontier Gateway group. The plaintext key is returned exactly once. - \[Get an API key\](https://docs.baseten.co/reference/gateway/api-keys/get-an-api-key.md): Fetch metadata for one federated API key by its prefix. The plaintext key is never returned after creation. - \[List API keys for a group\](https://docs.baseten.co/reference/gateway/api-keys/list-api-keys-for-a-group.md): List the federated API keys minted under a Frontier Gateway group. Cursor-paginated. - \[Register an API key\](https://docs.baseten.co/reference/gateway/api-keys/register-an-api-key.md): Attach a caller-supplied API key to a Frontier Gateway group so downstream consumers can continue using a key they already issued. - \[Revoke an API key\](https://docs.baseten.co/reference/gateway/api-keys/revoke-an-api-key.md): Revoke a federated API key by its prefix. Other keys under the same group are unaffected. - \[Billing webhooks\](https://docs.baseten.co/reference/gateway/billing-webhooks.md): Payload, header, and signature reference for Frontier Gateway billing webhooks. - \[Create a group\](https://docs.baseten.co/reference/gateway/groups/create-a-group.md): Create a Frontier Gateway group with its model set, per-model limits, and a place in the hierarchy. - \[Delete a group\](https://docs.baseten.co/reference/gateway/groups/delete-a-group.md): Delete a Frontier Gateway group, recursively remove its descendants, and revoke every key in the subtree. - \[Get a group\](https://docs.baseten.co/reference/gateway/groups/get-a-group.md): Fetch a single Frontier Gateway group by its internal id, including its effective limits after inheritance. - \[Get group usage\](https://docs.baseten.co/reference/gateway/groups/get-group-usage.md): Read current-window consumption against the usage limits configured on a Frontier Gateway group. - \[List groups\](https://docs.baseten.co/reference/gateway/groups/list-groups.md): List Frontier Gateway groups in your workspace. Cursor-paginated, with optional lookup by external identifier. - \[Update a group\](https://docs.baseten.co/reference/gateway/groups/update-a-group.md): Update a Frontier Gateway group's display name or model configuration. Hierarchy and enforcement mode are immutable. - \[Chat Completions\](https://docs.baseten.co/reference/inference-api/chat-completions.md): Create chat completions using Baseten Model APIs, an OpenAI-compatible endpoint for managed LLMs. - \[Messages\](https://docs.baseten.co/reference/inference-api/messages.md): Create Anthropic Messages API requests against Baseten Model APIs. - \[Overview\](https://docs.baseten.co/reference/inference-api/overview.md): Baseten provides two ways to call models: Model APIs for managed LLMs and deployed model endpoints for custom models and chains. - \[Websocket deployment\](https://docs.baseten.co/reference/inference-api/predict-endpoints/deployment-websocket.md): Connect via WebSocket to a specific deployment. - \[Websocket development\](https://docs.baseten.co/reference/inference-api/predict-endpoints/development-websocket.md): Connect via WebSocket to the development deployment of a model or chain. - \[Websocket environment\](https://docs.baseten.co/reference/inference-api/predict-endpoints/environments-websocket.md): Connect via WebSocket to the deployment associated with an environment. - \[Transcribe Streaming Audio\](https://docs.baseten.co/reference/inference-api/predict-endpoints/streaming-transcription-api.md): Transcribe audio in real time over a WebSocket connection. - \[Transcribe Pre-Recorded Audio\](https://docs.baseten.co/reference/inference-api/predict-endpoints/transcription-api.md): Transcribe a pre-recorded audio file using a deployed transcription model. - \[Get checkpoint files\](https://docs.baseten.co/reference/loops-api/checkpoints/get-checkpoint-files.md): Get presigned URLs for the files under a Loops checkpoint. Returns a paginated list. - \[List checkpoints\](https://docs.baseten.co/reference/loops-api/checkpoints/list-checkpoints.md): List Loops checkpoints filtered by run id, base model, or bt:// URI. Provide exactly one filter. - \[Validate a checkpoint\](https://docs.baseten.co/reference/loops-api/checkpoints/validate-a-checkpoint.md): Whether the caller can manage and use this checkpoint. - \[Deactivate a deployment\](https://docs.baseten.co/reference/loops-api/deployments/deactivate-a-deployment.md): Shut down a Loops deployment by ID. Saved checkpoints remain accessible. Resolving base\_model -> deployment\_id is the caller's responsibility — list deployments and pick the active one. - \[Get a deployment\](https://docs.baseten.co/reference/loops-api/deployments/get-a-deployment.md): Fetch a Loops deployment by ID, including its latest status. - \[Get deployment metrics\](https://docs.baseten.co/reference/loops-api/deployments/get-deployment-metrics.md): Returns per-node GPU/CPU/memory utilization and Knative queue-proxy request rate / concurrency / latency for the trainer pods. The sampler half of a Loops deployment is an OracleVersion and uses the existing model-metrics endpoint. - \[List deployments\](https://docs.baseten.co/reference/loops-api/deployments/list-deployments.md): List the caller's Loops deployments. Returns every deployment regardless of status; clients filter terminal states. - \[Loops API reference\](https://docs.baseten.co/reference/loops-api/overview.md): HTTP routes for Loops sessions, runs, samplers, checkpoints, and deployments. - \[Create a run\](https://docs.baseten.co/reference/loops-api/runs/create-a-run.md): Creates a Loops run with an associated sampler in the given session. - \[Get a run\](https://docs.baseten.co/reference/loops-api/runs/get-a-run.md): Fetch a Loops run by ID. - \[List runs\](https://docs.baseten.co/reference/loops-api/runs/list-runs.md): List Loops runs visible to the requesting user, optionally filtered by run id and/or base model. - \[Create a sampler\](https://docs.baseten.co/reference/loops-api/samplers/create-a-sampler.md): Creates a standalone Loops sampler not linked to a run. - \[Get a sampler\](https://docs.baseten.co/reference/loops-api/samplers/get-a-sampler.md): Fetch a Loops sampler by ID. - \[List samplers\](https://docs.baseten.co/reference/loops-api/samplers/list-samplers.md): List Loops samplers visible to the requesting user. - \[Get server capabilities\](https://docs.baseten.co/reference/loops-api/server/get-capabilities.md): Returns the list of models supported by the Loops server, including each model's maximum context length. - \[Create a session\](https://docs.baseten.co/reference/loops-api/sessions/create-a-session.md): Creates a Loops session for the given training project. - \[Get a session\](https://docs.baseten.co/reference/loops-api/sessions/get-a-session.md): Fetch a Loops session by ID. - \[Create an API key\](https://docs.baseten.co/reference/management-api/api-keys/creates-an-api-key.md): Creates an API key with the provided name and type. The API key is returned in the response. - \[Delete an API key\](https://docs.baseten.co/reference/management-api/api-keys/delete-an-api-key.md): Deletes an API key by prefix and returns info about the API key. - \[Get all API keys\](https://docs.baseten.co/reference/management-api/api-keys/lists-the-users-api-keys.md): Lists all API keys your account has access to. - \[Get billing usage summary\](https://docs.baseten.co/reference/management-api/billing/gets-billing-usage-summary-for-a-date-range.md): Returns billing usage data within the specified date range. Includes dedicated model serving, training, and model APIs usage. The date range must not exceed 31 days. - \[Delete chains\](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id.md) - \[By ID\](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id.md) - \[All chains\](https://docs.baseten.co/reference/management-api/chains/gets-all-chains.md) - \[Any deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-deployment.md): Activates an inactive deployment and returns the activation status. - \[Activate environment deployment\](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-deployment-associated-with-an-environment.md): Activates an inactive deployment associated with an environment and returns the activation status. - \[Development deployment\](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-development-deployment.md): Activates an inactive development deployment and returns the activation status. - \[Activate production deployment\](https://docs.baseten.co/reference/management-api/deployments/activate/activates-production-deployment.md): Activates an inactive production deployment and returns the activation status. - \[Update chainlet environment's autoscaling settings\](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings.md): Updates a chainlet environment's autoscaling settings and returns the updated chainlet environment settings. - \[Any model deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings.md): Updates a deployment's autoscaling settings and returns the update status. - \[Development model deployment\](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings.md): Updates a development deployment's autoscaling settings and returns the update status. - \[Update production deployment autoscaling settings\](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings.md): Updates a production deployment's autoscaling settings and returns the update status. - \[Chain deployment\](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment.md): Deactivates a chain deployment and returns the deactivation status. - \[Any deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment.md): Deactivates a deployment and returns the deactivation status. - \[Deactivate environment deployment\](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment.md): Deactivates a deployment associated with an environment and returns the deactivation status. - \[Development deployment\](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment.md): Deactivates a development deployment and returns the deactivation status. - \[Deactivate production deployment\](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment.md): Deactivates a production deployment and returns the deactivation status. - \[Delete chain deployment\](https://docs.baseten.co/reference/management-api/deployments/deletes-a-chain-deployment-by-id.md) - \[Delete model deployments\](https://docs.baseten.co/reference/management-api/deployments/deletes-a-models-deployment-by-id.md): Deletes a model's deployment by ID and returns the tombstone of the deployment. - \[Get model deployment logs\](https://docs.baseten.co/reference/management-api/deployments/get-deployment-logs.md): Gets all the logs for a model deployment in the given time range. - \[Any chain deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/gets-a-chain-deployment-by-id.md) - \[Any model deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id.md): Gets a model's deployment by ID and returns the deployment. - \[Development model deployment\](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment.md): Gets a model's development deployment and returns the deployment. - \[Production model deployment\](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-production-deployment.md): Gets a model's production deployment and returns the deployment. - \[Get all chain deployments\](https://docs.baseten.co/reference/management-api/deployments/gets-all-chain-deployments.md) - \[Get all model deployments\](https://docs.baseten.co/reference/management-api/deployments/gets-all-deployments-of-a-model.md) - \[Cancel model promotion\](https://docs.baseten.co/reference/management-api/deployments/promote/cancel-promotion.md): Cancels an ongoing promotion to an environment and returns the cancellation status. - \[Force cancel rolling deployment\](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion.md): Immediately cancels an in-progress rolling promotion and triggers rollback to the previous version. - \[Force roll forward promotion\](https://docs.baseten.co/reference/management-api/deployments/promote/force-roll-forward-promotion.md): Immediately completes the rolling promotion, shifting all traffic to the new version. This works even if the promotion is in the process of rolling back. - \[Pause rolling deployment\](https://docs.baseten.co/reference/management-api/deployments/promote/pause-promotion.md): Pauses an in-progress rolling promotion after the current step completes. No further scaling changes are made until resumed. - \[Promote to chain environment\](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-chain-deployment-to-an-environment.md): Promotes an existing chain deployment to an environment and returns the promoted chain deployment. - \[Promote to model environment\](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment.md): Promotes an existing deployment to an environment and returns the promoted deployment. - \[Any model deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-production.md): Promotes an existing deployment to production and returns the same deployment. - \[Development model deployment\](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-development-deployment-to-production.md): Creates a new production deployment from the development deployment, the currently building deployment is returned. - \[Resume rolling deployment\](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion.md): Resumes a paused rolling promotion, continuing from where it was paused. - \[Any deployment by ID\](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment.md): Retries a failed deployment and returns the retry status and updated deployment. - \[Development deployment\](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment.md): Retries a failed development deployment and returns the retry status and updated deployment. - \[Production deployment\](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment.md): Retries a failed production deployment and returns the retry status and updated deployment. - \[Terminate deployment replica\](https://docs.baseten.co/reference/management-api/deployments/terminates-deployment-replica.md): Terminates a deployment replica and returns the termination status. - \[Create Chain environment\](https://docs.baseten.co/reference/management-api/environments/create-a-chain-environment.md): Create a chain environment. Returns the resulting environment. - \[Create environment\](https://docs.baseten.co/reference/management-api/environments/create-an-environment.md): Creates an environment for the specified model and returns the environment. - \[Get Chain environment\](https://docs.baseten.co/reference/management-api/environments/get-a-chain-environments-details.md): Gets a chain environment's details and returns the chain environment. - \[Get all Chain environments\](https://docs.baseten.co/reference/management-api/environments/get-all-chain-environments.md): Gets all chain environments for a given chain - \[Get all environments\](https://docs.baseten.co/reference/management-api/environments/get-all-environments.md): Gets all environments for a given model - \[Get environment\](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details.md): Gets an environment's details and returns the environment. - \[Update Chain environment\](https://docs.baseten.co/reference/management-api/environments/update-a-chain-environments-settings.md): Update a chain environment's settings and returns the chain environment. - \[Update chainlet environment's instance type\](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings.md): Updates a chainlet environment's instance type settings. The chainlet environment setting must exist. When updated, a new chain deployment is created and deployed. It is promoted to the chain environment according to promotion settings on the environment. - \[Update model environment\](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings.md): Asynchronously updates an environment's settings. Poll the GET endpoint for the applied state. - \[All instance types\](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types.md) - \[Instance type prices\](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices.md) - \[Delete models\](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id.md) - \[By ID\](https://docs.baseten.co/reference/management-api/models/gets-a-model-by-id.md) - \[All models\](https://docs.baseten.co/reference/management-api/models/gets-all-models.md) - \[Overview\](https://docs.baseten.co/reference/management-api/overview.md): Manage models and deployments with the Baseten management API. It supports monitoring, CI/CD, and automation at both the model and workspace levels. - \[Rate limits\](https://docs.baseten.co/reference/management-api/rate-limits.md): Rate limits, response shape, and retry handling for the Baseten management API. - \[Get all secrets\](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets.md) - \[Upsert a secret\](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret.md): Creates or updates a secret by name. Scoped to the caller's primary team — use the team-scoped variant to target a specific team. - \[Create a team API key\](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key.md): Creates a team API key with the provided name and type. The API key is returned in the response. - \[Create a team training project\](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project.md): Upserts a training project with the specified metadata for a team. - \[Get all team secrets\](https://docs.baseten.co/reference/management-api/teams/gets-all-team-secrets.md) - \[List all teams\](https://docs.baseten.co/reference/management-api/teams/lists-all-teams.md): Returns a list of all teams the authenticated user has access to. - \[Upsert a team secret\](https://docs.baseten.co/reference/management-api/teams/upserts-a-team-secret.md): Creates a new secret or updates an existing secret if one with the provided name already exists. The name and creation date of the created or updated secret is returned. This secret belongs to the specified team - \[Reference documentation\](https://docs.baseten.co/reference/overview.md): For deploying, managing, and interacting with machine learning models on Baseten. - \[Chains SDK Reference\](https://docs.baseten.co/reference/sdk/chains.md): Python SDK Reference for Chains - \[Loops SDK\](https://docs.baseten.co/reference/sdk/loops.md): Python client for Loops: ServiceClient, TrainingClient, SamplingClient, and the Tinker compatibility shim. - \[Training SDK\](https://docs.baseten.co/reference/sdk/training.md): API reference for the Baseten training SDK. - \[Truss SDK Reference\](https://docs.baseten.co/reference/sdk/truss.md): Python SDK for deploying and managing models with Truss. - \[Create training job\](https://docs.baseten.co/reference/training-api/create-training-job.md): Creates a training job with the specified configuration. - \[Create training project\](https://docs.baseten.co/reference/training-api/create-training-project.md): Upserts a training project with the specified metadata. - \[Delete training job\](https://docs.baseten.co/reference/training-api/delete-training-job.md): Deletes a training job. Stops it first if still running. - \[Delete training project\](https://docs.baseten.co/reference/training-api/delete-training-project.md): Deletes a training project and all associated training jobs. - \[Download training job source code\](https://docs.baseten.co/reference/training-api/download-training-job.md): Get the uploaded training job as a S3 Artifact - \[Get auth codes for training job\](https://docs.baseten.co/reference/training-api/get-auth-codes-for-training-job.md): Get authentication codes for all nodes of a training job's interactive sessions. - \[Get training job\](https://docs.baseten.co/reference/training-api/get-training-job.md): Get the details of an existing training job. - \[Get training job checkpoint files\](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files.md): Get presigned URLs for all checkpoint files for a training job. - \[List training job checkpoints\](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints.md): Get the checkpoints for a training job. - \[Get training job logs\](https://docs.baseten.co/reference/training-api/get-training-job-logs.md): Get the logs for a training job with the provided filters. - \[Get training job metrics\](https://docs.baseten.co/reference/training-api/get-training-job-metrics.md): Get the metrics for a training job. - \[Get training project\](https://docs.baseten.co/reference/training-api/get-training-project.md): Get the details of an existing training project. - \[Get training project cache summary\](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary.md): Get the cache summary for the most recent training job in the project. - \[List training projects\](https://docs.baseten.co/reference/training-api/get-training-projects.md): List all training projects for the organization. - \[List training jobs\](https://docs.baseten.co/reference/training-api/list-training-jobs.md): List all training jobs for the training project. - \[Overview\](https://docs.baseten.co/reference/training-api/overview.md): Programmatically manage Baseten Training resources. - \[Recreate training job\](https://docs.baseten.co/reference/training-api/recreate-training-job.md): Create a new training job with the same configuration as an existing training job. - \[Search training jobs\](https://docs.baseten.co/reference/training-api/search-training-jobs.md): Search training jobs for the organization. - \[Stop training job\](https://docs.baseten.co/reference/training-api/stop-training-job.md): Stops a training job. - \[Truss configuration\](https://docs.baseten.co/reference/truss-configuration.md): Set your model resources, dependencies, and more - \[Baseten platform status\](https://docs.baseten.co/status/status.md): Current operational status of Baseten's services. - \[Building blocks\](https://docs.baseten.co/training/concepts/basics.md): Learn how to get up and running on Baseten Training - \[Cache\](https://docs.baseten.co/training/concepts/cache.md): Learn how to use the training cache to speed up your training iterations by persisting data between jobs. - \[Checkpoints\](https://docs.baseten.co/training/concepts/checkpoints.md): Learn how to use Baseten's checkpointing feature to manage model checkpoints and avoid disk errors during training. - \[Multinode training\](https://docs.baseten.co/training/concepts/multinode.md): Learn how to configure and run multinode training jobs with Baseten Training. - \[Storage and data ingestion\](https://docs.baseten.co/training/concepts/storage.md): Load model weights and training data into Baseten training containers through BDN, S3, Hugging Face, and GCS. - \[Deploy with optimized inference engines\](https://docs.baseten.co/training/deploy-with-engine-builder.md): Deploy model checkpoints from Baseten Training directly to an inference engine without downloading or re-uploading weights. - \[Serving your trained model\](https://docs.baseten.co/training/deployment.md): How to deploy checkpoints from Baseten Training jobs as usable models. - \[Get started\](https://docs.baseten.co/training/getting-started.md): Run your first training job and deploy it to production. - \[VS Code and Cursor remote tunnels\](https://docs.baseten.co/training/interactive-sessions.md): Connect to training containers for remote debugging and development via VS Code or Cursor Remote Tunnels. - \[Lifecycle\](https://docs.baseten.co/training/lifecycle.md): Understanding the different states and transitions in a Baseten training job's lifecycle. - \[Loading checkpoints\](https://docs.baseten.co/training/loading.md): Resume training from existing checkpoints to continue where you left off. - \[Management\](https://docs.baseten.co/training/management.md): How to monitor, manage, and interact with your Baseten Training projects and jobs. - \[Training on Baseten\](https://docs.baseten.co/training/overview.md): Train custom models with developer-first training infrastructure on Baseten. - \[Remote access\](https://docs.baseten.co/training/remote-access.md): Connect to running training jobs from your local machine to debug, inspect state, and develop interactively. - \[SSH access\](https://docs.baseten.co/training/ssh.md): Connect to training containers directly from your terminal with standard SSH. - \[Deployments\](https://docs.baseten.co/troubleshooting/deployments.md): Troubleshoot common problems during model deployment - \[Inference\](https://docs.baseten.co/troubleshooting/inference.md): Troubleshoot common problems during model inference ## OpenAPI Specs - \[management-api-spec\](https://docs.baseten.co/reference/management-api-spec.json) - \[llm-openapi-spec\](https://docs.baseten.co/reference/inference-api/llm-openapi-spec.json) - \[messages-openapi-spec\](https://docs.baseten.co/reference/inference-api/messages-openapi-spec.json) - \[inference-api-spec\](https://docs.baseten.co/reference/inference-api/inference-api-spec.json) - \[meta\](https://docs.baseten.co/styles/proselint/meta.json) - \[Very\](https://docs.baseten.co/styles/proselint/Very.yml) - \[Uncomparables\](https://docs.baseten.co/styles/proselint/Uncomparables.yml) - \[Typography\](https://docs.baseten.co/styles/proselint/Typography.yml) - \[Spelling\](https://docs.baseten.co/styles/proselint/Spelling.yml) - \[Skunked\](https://docs.baseten.co/styles/proselint/Skunked.yml) - \[RASSyndrome\](https://docs.baseten.co/styles/proselint/RASSyndrome.yml) - \[P-Value\](https://docs.baseten.co/styles/proselint/P-Value.yml) - \[Oxymorons\](https://docs.baseten.co/styles/proselint/Oxymorons.yml) - \[Nonwords\](https://docs.baseten.co/styles/proselint/Nonwords.yml) - \[Needless\](https://docs.baseten.co/styles/proselint/Needless.yml) - \[Malapropisms\](https://docs.baseten.co/styles/proselint/Malapropisms.yml) - \[LGBTTerms\](https://docs.baseten.co/styles/proselint/LGBTTerms.yml) - \[LGBTOffensive\](https://docs.baseten.co/styles/proselint/LGBTOffensive.yml) - \[Jargon\](https://docs.baseten.co/styles/proselint/Jargon.yml) - \[Hyperbole\](https://docs.baseten.co/styles/proselint/Hyperbole.yml) - \[Hedging\](https://docs.baseten.co/styles/proselint/Hedging.yml) - \[GroupTerms\](https://docs.baseten.co/styles/proselint/GroupTerms.yml) - \[GenderBias\](https://docs.baseten.co/styles/proselint/GenderBias.yml) - \[Diacritical\](https://docs.baseten.co/styles/proselint/Diacritical.yml) - \[DenizenLabels\](https://docs.baseten.co/styles/proselint/DenizenLabels.yml) - \[DateSpacing\](https://docs.baseten.co/styles/proselint/DateSpacing.yml) - \[DateRedundancy\](https://docs.baseten.co/styles/proselint/DateRedundancy.yml) - \[DateMidnight\](https://docs.baseten.co/styles/proselint/DateMidnight.yml) - \[DateCase\](https://docs.baseten.co/styles/proselint/DateCase.yml) - \[Cursing\](https://docs.baseten.co/styles/proselint/Cursing.yml) - \[Currency\](https://docs.baseten.co/styles/proselint/Currency.yml) - \[CorporateSpeak\](https://docs.baseten.co/styles/proselint/CorporateSpeak.yml) - \[Cliches\](https://docs.baseten.co/styles/proselint/Cliches.yml) - \[But\](https://docs.baseten.co/styles/proselint/But.yml) - \[Archaisms\](https://docs.baseten.co/styles/proselint/Archaisms.yml) - \[Apologizing\](https://docs.baseten.co/styles/proselint/Apologizing.yml) - \[Annotations\](https://docs.baseten.co/styles/proselint/Annotations.yml) - \[AnimalLabels\](https://docs.baseten.co/styles/proselint/AnimalLabels.yml) - \[Airlinese\](https://docs.baseten.co/styles/proselint/Airlinese.yml) - \[WordList\](https://docs.baseten.co/styles/Google/WordList.yml) - \[Will\](https://docs.baseten.co/styles/Google/Will.yml) - \[We\](https://docs.baseten.co/styles/Google/We.yml) - \[Units\](https://docs.baseten.co/styles/Google/Units.yml) - \[Spacing\](https://docs.baseten.co/styles/Google/Spacing.yml) - \[Slang\](https://docs.baseten.co/styles/Google/Slang.yml) - \[Semicolons\](https://docs.baseten.co/styles/Google/Semicolons.yml) - \[Ranges\](https://docs.baseten.co/styles/Google/Ranges.yml) - \[Quotes\](https://docs.baseten.co/styles/Google/Quotes.yml) - \[Periods\](https://docs.baseten.co/styles/Google/Periods.yml) - \[Passive\](https://docs.baseten.co/styles/Google/Passive.yml) - \[Parens\](https://docs.baseten.co/styles/Google/Parens.yml) - \[OxfordComma\](https://docs.baseten.co/styles/Google/OxfordComma.yml) - \[Ordinal\](https://docs.baseten.co/styles/Google/Ordinal.yml) - \[OptionalPlurals\](https://docs.baseten.co/styles/Google/OptionalPlurals.yml) - \[LyHyphens\](https://docs.baseten.co/styles/Google/LyHyphens.yml) - \[Latin\](https://docs.baseten.co/styles/Google/Latin.yml) - \[Headings\](https://docs.baseten.co/styles/Google/Headings.yml) - \[HeadingPunctuation\](https://docs.baseten.co/styles/Google/HeadingPunctuation.yml) - \[Gender\](https://docs.baseten.co/styles/Google/Gender.yml) - \[FirstPerson\](https://docs.baseten.co/styles/Google/FirstPerson.yml) - \[Exclamation\](https://docs.baseten.co/styles/Google/Exclamation.yml) - \[EmDash\](https://docs.baseten.co/styles/Google/EmDash.yml) - \[Ellipses\](https://docs.baseten.co/styles/Google/Ellipses.yml) - \[DateFormat\](https://docs.baseten.co/styles/Google/DateFormat.yml) - \[Contractions\](https://docs.baseten.co/styles/Google/Contractions.yml) - \[Colons\](https://docs.baseten.co/styles/Google/Colons.yml) - \[Acronyms\](https://docs.baseten.co/styles/Google/Acronyms.yml) - \[AMPM\](https://docs.baseten.co/styles/Google/AMPM.yml) - \[package\](https://docs.baseten.co/package.json) - \[package-lock\](https://docs.baseten.co/package-lock.json) - \[settings\](https://docs.baseten.co/.vscode/settings.json) - \[spec\](https://docs.baseten.co/spec) --- # Development model deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) PATCH / v1 / models / {model\_id} / deployments / development / autoscaling\_settings Try it cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } To update autoscaling settings at the environment level, use the [update environment settings](https://docs.baseten.co/reference/management-api/environments/update-an-environments-settings) endpoint. #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#parameter-model-id) model\_id string required #### Body application/json A request to update autoscaling settings for a deployment. All fields are optional, and we only update ones passed in. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-min-replica-one-of-0) min\_replica integer | null Minimum number of replicas Example: `0` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-max-replica-one-of-0) max\_replica integer | null Maximum number of replicas Example: `7` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-autoscaling-window-one-of-0) autoscaling\_window integer | null Timeframe of traffic considered for autoscaling decisions Example: `600` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-scale-down-delay-one-of-0) scale\_down\_delay integer | null Waiting period before scaling down any active replica Example: `120` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-concurrency-target-one-of-0) concurrency\_target integer | null Number of requests per replica before scaling up Example: `2` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-target-utilization-percentage-one-of-0) target\_utilization\_percentage integer | null Target utilization percentage for scaling up/down. Example: `70` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-target-in-flight-tokens-one-of-0) target\_in\_flight\_tokens integer | null Target number of in-flight tokens for autoscaling decisions. Early access only. Example: `40000` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#body-max-scale-down-rate-one-of-0) max\_scale\_down\_rate number | null Maximum rate at which replicas can scale down (e.g. 2.0 means at most halve replicas per window). Required range: `1 < x <= 2` Example: `2` #### Response 200 - application/json The response to a request to update autoscaling settings. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#response-status) status enum required Status of the request to update autoscaling settings Available options: `ACCEPTED`, `QUEUED`, `UNCHANGED` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-development-deployments-autoscaling-settings#response-message) message string required A message describing the status of the request to update autoscaling settings Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-chain-deployment-to-an-environment) [DeploymentUpdates a deployment's autoscaling settings and returns the update status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings) ⌘I cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get all environments - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/environments/get-all-environments#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / models / {model\_id} / environments Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/environments \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "environments": [\ {\ "name": "",\ "created_at": "2023-11-07T05:31:56Z",\ "model_id": "",\ "current_deployment": {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "model_id": "",\ "is_production": true,\ "is_development": true,\ "active_replica_count": 123,\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "environment": "",\ "labels": {}\ },\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "promotion_settings": {\ "redeploy_on_promotion": true,\ "rolling_deploy": true,\ "promotion_cleanup_strategy": "SCALE_TO_ZERO",\ "rolling_deploy_config": {\ "rolling_deploy_strategy": "REPLICA",\ "max_surge_percent": 10,\ "max_unavailable_percent": 0,\ "stabilization_time_seconds": 0,\ "replica_overhead_percent": 0\ },\ "ramp_up_while_promoting": true,\ "ramp_up_duration_seconds": 600\ },\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ },\ "candidate_deployment": {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "model_id": "",\ "is_production": true,\ "is_development": true,\ "active_replica_count": 123,\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "environment": "",\ "labels": {}\ },\ "in_progress_promotion": {\ "percent_traffic_to_new_version": 123,\ "error_message": "",\ "rolling_deploy": true\ }\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/environments/get-all-environments#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/environments/get-all-environments#parameter-model-id) model\_id string required #### Response 200 - application/json list of environments [​](https://docs.baseten.co/reference/management-api/environments/get-all-environments#response-environments) environments EnvironmentV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/environments/create-an-environment) [Get environmentGets an environment's details and returns the environment.\ \ Next](https://docs.baseten.co/reference/management-api/environments/get-an-environments-details) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/environments \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "environments": [\ {\ "name": "",\ "created_at": "2023-11-07T05:31:56Z",\ "model_id": "",\ "current_deployment": {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "model_id": "",\ "is_production": true,\ "is_development": true,\ "active_replica_count": 123,\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "environment": "",\ "labels": {}\ },\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "promotion_settings": {\ "redeploy_on_promotion": true,\ "rolling_deploy": true,\ "promotion_cleanup_strategy": "SCALE_TO_ZERO",\ "rolling_deploy_config": {\ "rolling_deploy_strategy": "REPLICA",\ "max_surge_percent": 10,\ "max_unavailable_percent": 0,\ "stabilization_time_seconds": 0,\ "replica_overhead_percent": 0\ },\ "ramp_up_while_promoting": true,\ "ramp_up_duration_seconds": 600\ },\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ },\ "candidate_deployment": {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "model_id": "",\ "is_production": true,\ "is_development": true,\ "active_replica_count": 123,\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "environment": "",\ "labels": {}\ },\ "in_progress_promotion": {\ "percent_traffic_to_new_version": 123,\ "error_message": "",\ "rolling_deploy": true\ }\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Get all secrets - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / secrets Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/secrets \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "secrets": [\ {\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "team_name": ""\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of Baseten secrets. [​](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets#response-secrets) secrets SecretV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/teams/lists-all-teams) [Upsert a secretCreates or updates a secret by name. Scoped to the caller's primary team — use the team-scoped variant to target a specific team.\ \ Next](https://docs.baseten.co/reference/management-api/secrets/upserts-a-secret) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/secrets \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "secrets": [\ {\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "team_name": ""\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Update production deployment autoscaling settings - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) PATCH / v1 / models / {model\_id} / deployments / production / autoscaling\_settings Try it cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#parameter-model-id) model\_id string required #### Body application/json A request to update autoscaling settings for a deployment. All fields are optional, and we only update ones passed in. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-min-replica-one-of-0) min\_replica integer | null Minimum number of replicas Example: `0` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-max-replica-one-of-0) max\_replica integer | null Maximum number of replicas Example: `7` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-autoscaling-window-one-of-0) autoscaling\_window integer | null Timeframe of traffic considered for autoscaling decisions Example: `600` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-scale-down-delay-one-of-0) scale\_down\_delay integer | null Waiting period before scaling down any active replica Example: `120` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-concurrency-target-one-of-0) concurrency\_target integer | null Number of requests per replica before scaling up Example: `2` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-target-utilization-percentage-one-of-0) target\_utilization\_percentage integer | null Target utilization percentage for scaling up/down. Example: `70` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-target-in-flight-tokens-one-of-0) target\_in\_flight\_tokens integer | null Target number of in-flight tokens for autoscaling decisions. Early access only. Example: `40000` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#body-max-scale-down-rate-one-of-0) max\_scale\_down\_rate number | null Maximum rate at which replicas can scale down (e.g. 2.0 means at most halve replicas per window). Required range: `1 < x <= 2` Example: `2` #### Response 200 - application/json The response to a request to update autoscaling settings. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#response-status) status enum required Status of the request to update autoscaling settings Available options: `ACCEPTED`, `QUEUED`, `UNCHANGED` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-production-deployment-autoscaling-settings#response-message) message string required A message describing the status of the request to update autoscaling settings Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings) [All deployments\ \ Next](https://docs.baseten.co/reference/management-api/deployments/gets-all-deployments-of-a-model) ⌘I cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "min_replica": 0, "max_replica": 7, "autoscaling_window": 600, "scale_down_delay": 120, "concurrency_target": 2, "target_utilization_percentage": 70, "target_in_flight_tokens": 40000, "max_scale_down_rate": 2.0 }' 200 { "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # All models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/models/gets-all-models#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / models Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "models": [\ {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "deployments_count": 123,\ "production_deployment_id": "",\ "development_deployment_id": "",\ "instance_type_name": "",\ "team_name": ""\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/models/gets-all-models#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of models. [​](https://docs.baseten.co/reference/management-api/models/gets-all-models#response-models) models ModelV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/rate-limits) [By ID\ \ Next](https://docs.baseten.co/reference/management-api/models/gets-a-model-by-id) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "models": [\ {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "deployments_count": 123,\ "production_deployment_id": "",\ "development_deployment_id": "",\ "instance_type_name": "",\ "team_name": ""\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Create a team API key - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / teams / {team\_id} / api\_keys Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/teams/{team_id}/api_keys \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my-api-key", "type": "PERSONAL" }' 200 { "api_key": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#parameter-team-id) team\_id string required #### Body application/json Request to create an API key. [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#body-type) type enum required Type of the API key. Available options: `PERSONAL`, `WORKSPACE_MANAGE_ALL`, `WORKSPACE_EXPORT_METRICS`, `WORKSPACE_INVOKE` Examples: `"PERSONAL"` `"WORKSPACE_EXPORT_METRICS"` `"WORKSPACE_INVOKE"` `"WORKSPACE_MANAGE_ALL"` [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#body-name-one-of-0) name string | null Optional name for the API key Example: `"my-api-key"` [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#body-model-ids-one-of-0) model\_ids string\[\] | null List of model IDs to scope the API key to, only present if type is 'WORKSPACE\_EXPORT\_METRICS' or 'WORKSPACE\_INVOKE' Example: ["aaaaaaaa"] #### Response 200 - application/json Represents an API key. [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-api-key#response-api-key) api\_key string required The API key string Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/api-keys/delete-an-api-key) [Usage summaryReturns billing usage data within the specified date range. Includes dedicated model serving, training, and model APIs usage. The date range must not exceed 31 days.\ \ Next](https://docs.baseten.co/reference/management-api/billing/gets-billing-usage-summary-for-a-date-range) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/teams/{team_id}/api_keys \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "name": "my-api-key", "type": "PERSONAL" }' 200 { "api_key": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Delete models - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) DELETE / v1 / models / {model\_id} Try it cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/models/{model_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id#parameter-model-id) model\_id string required #### Response 200 - application/json A model tombstone. [​](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id#response-id) id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id#response-deleted) deleted boolean required Whether the model was deleted Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/models/gets-a-model-by-id) [All chains\ \ Next](https://docs.baseten.co/reference/management-api/chains/gets-all-chains) ⌘I cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/models/{model_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } Assistant Responses are generated using AI and may contain mistakes. --- # Update chainlet environment's autoscaling settings - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) PATCH / v1 / chains / {chain\_id} / environments / {env\_name} / chainlet\_settings / autoscaling\_settings Try it cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/chains/{chain_id}/environments/{env_name}/chainlet_settings/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "updates": [\ {\ "autoscaling_settings": {\ "autoscaling_window": 800,\ "concurrency_target": 4,\ "max_replica": 3,\ "max_scale_down_rate": null,\ "min_replica": 2,\ "scale_down_delay": 63,\ "target_in_flight_tokens": null,\ "target_utilization_percentage": null\ },\ "chainlet_name": "HelloWorld"\ }\ ] }' 200 { "message": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#parameter-chain-id) chain\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#parameter-env-name) env\_name string required #### Body application/json A request to update the autoscaling settings for a multiple chainlets in an environment. If a chainlet name doesn't exist, an error is returned. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#body-updates) updates ChainletEnvironmentAutoscalingSettingsUpdateV1 · object\[\] required Mapping of chainlet name to the desired chainlet autoscaling settings. If the chainlet name doesn't exist, an error is returned. Show child attributes Examples: [ { "autoscaling_settings": { "autoscaling_window": 800, "concurrency_target": 4, "max_replica": 3, "max_scale_down_rate": null, "min_replica": 2, "scale_down_delay": 63, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "chainlet_name": "HelloWorld" }] [ { "autoscaling_settings": { "autoscaling_window": null, "concurrency_target": null, "max_replica": null, "max_scale_down_rate": null, "min_replica": 0, "scale_down_delay": null, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "chainlet_name": "HelloWorld" }, { "autoscaling_settings": { "autoscaling_window": null, "concurrency_target": null, "max_replica": null, "max_scale_down_rate": null, "min_replica": 0, "scale_down_delay": null, "target_in_flight_tokens": null, "target_utilization_percentage": null }, "chainlet_name": "RandInt" }] #### Response 200 - application/json The response to a request to update autoscaling settings. [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#response-status) status enum required Status of the request to update autoscaling settings Available options: `ACCEPTED`, `QUEUED`, `UNCHANGED` [​](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings#response-message) message string required A message describing the status of the request to update autoscaling settings Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/environments/update-a-chain-environments-settings) [Update chainlet environment's instance typeUpdates a chainlet environment's instance type settings. The chainlet environment setting must exist. When updated, a new chain deployment is created and deployed. It is promoted to the chain environment according to promotion settings on the environment.\ \ Next](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings) ⌘I cURL cURL curl --request PATCH \ --url https://api.baseten.co/v1/chains/{chain_id}/environments/{env_name}/chainlet_settings/autoscaling_settings \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "updates": [\ {\ "autoscaling_settings": {\ "autoscaling_window": 800,\ "concurrency_target": 4,\ "max_replica": 3,\ "max_scale_down_rate": null,\ "min_replica": 2,\ "scale_down_delay": 63,\ "target_in_flight_tokens": null,\ "target_utilization_percentage": null\ },\ "chainlet_name": "HelloWorld"\ }\ ] }' 200 { "message": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get training job - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/get-training-job#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / training\_projects / {training\_project\_id} / jobs / {training\_job\_id} Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" }, "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } } } #### Authorizations [​](https://docs.baseten.co/reference/training-api/get-training-job#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/get-training-job#parameter-training-project-id) training\_project\_id string required [​](https://docs.baseten.co/reference/training-api/get-training-job#parameter-training-job-id) training\_job\_id string required #### Response 200 - application/json A response to fetch a training job. [​](https://docs.baseten.co/reference/training-api/get-training-job#response-training-project) training\_project TrainingProjectV1 · object required The training project. Show child attributes [​](https://docs.baseten.co/reference/training-api/get-training-job#response-training-job) training\_job TrainingJobV1 · object required The fetched training job. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/delete-training-job) [Get training job checkpoint filesGet presigned URLs for all checkpoint files for a training job.\ \ Next](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" }, "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } } } Assistant Responses are generated using AI and may contain mistakes. --- # Resume rolling deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / environments / {env\_name} / resume\_promotion Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/resume_promotion \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Resume continues the rolling deployment from where it was paused. The deployment returns to `RAMPING_UP` and proceeds with the remaining promotion steps. See [Deployment control actions](https://docs.baseten.co/deployment/rolling-deployments#deployment-control-actions) for details. #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion#parameter-env-name) env\_name string required #### Response 200 - application/json The response to a request to signal a rolling promotion. [​](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion#response-success) success boolean required Whether the signal was successfully sent Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/promote/pause-promotion) [Force cancel rolling deploymentImmediately cancels an in-progress rolling promotion and triggers rollback to the previous version.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/resume_promotion \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # All instance types - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / instance\_types Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/instance_types \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "instance_types": [\ {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of instance types. [​](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types#response-instance-types) instance\_types InstanceTypeV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings) [Instance type prices\ \ Next](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/instance_types \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "instance_types": [\ {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # List all teams - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/teams/lists-all-teams#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / teams Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/teams \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "teams": [\ {\ "id": "",\ "name": "",\ "default": true,\ "created_at": "2023-11-07T05:31:56Z"\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/teams/lists-all-teams#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of teams. [​](https://docs.baseten.co/reference/management-api/teams/lists-all-teams#response-teams) teams TeamV1 · object\[\] required A list of teams Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices) [Get all secrets\ \ Next](https://docs.baseten.co/reference/management-api/secrets/gets-all-secrets) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/teams \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "teams": [\ {\ "id": "",\ "name": "",\ "default": true,\ "created_at": "2023-11-07T05:31:56Z"\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Instance type prices - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / instance\_type\_prices Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/instance_type_prices \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "instance_types": [\ {\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ },\ "price": 123\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of instance types. [​](https://docs.baseten.co/reference/management-api/instance-types/gets-instance-type-prices#response-instance-types) instance\_types InstanceTypeWithPriceV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types) [List all teamsReturns a list of all teams the authenticated user has access to.\ \ Next](https://docs.baseten.co/reference/management-api/teams/lists-all-teams) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/instance_type_prices \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "instance_types": [\ {\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ },\ "price": 123\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Create training project - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/create-training-project#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / training\_projects Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/training_projects \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_project": { "name": "My Training Project" } }' 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" } } #### Authorizations [​](https://docs.baseten.co/reference/training-api/create-training-project#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Body application/json A request to upsert a training project. [​](https://docs.baseten.co/reference/training-api/create-training-project#body-training-project) training\_project UpsertTrainingProjectV1 · object required The training project to upsert. Show child attributes #### Response 200 - application/json A response to upserting a training project. [​](https://docs.baseten.co/reference/training-api/create-training-project#response-training-project) training\_project TrainingProjectV1 · object required The upserted training project. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/overview) [Delete training projectDeletes a training project and all associated training jobs.\ \ Next](https://docs.baseten.co/reference/training-api/delete-training-project) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/training_projects \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_project": { "name": "My Training Project" } }' 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" } } Assistant Responses are generated using AI and may contain mistakes. --- # Chain deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / chains / {chain\_id} / deployments / {chain\_deployment\_id} / deactivate Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/chains/{chain_id}/deployments/{chain_deployment_id}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment#parameter-chain-id) chain\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment#parameter-chain-deployment-id) chain\_deployment\_id string required #### Response 200 - application/json The response to a request to deactivate a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment#response-success) success boolean default:true Whether the deployment was successfully deactivated Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id) [EnvironmentActivates an inactive deployment associated with an environment and returns the activation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/activate/activates-a-deployment-associated-with-an-environment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/chains/{chain_id}/deployments/{chain_deployment_id}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # By ID - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / chains / {chain\_id} Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/chains/{chain_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "deployments_count": 123, "team_name": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#parameter-chain-id) chain\_id string required #### Response 200 - application/json A chain. [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#response-id) id string required Unique identifier of the chain [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#response-created-at) created\_at string required Time the chain was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#response-name) name string required Name of the chain [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#response-deployments-count) deployments\_count integer required Number of deployments of the chain [​](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id#response-team-name) team\_name string required Name of the team associated with the chain Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/chains/gets-all-chains) [Delete chains\ \ Next](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/chains/{chain_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "deployments_count": 123, "team_name": "" } Assistant Responses are generated using AI and may contain mistakes. --- # All chains - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/chains/gets-all-chains#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / chains Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/chains \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "chains": [\ {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "deployments_count": 123,\ "team_name": ""\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/chains/gets-all-chains#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Response 200 - application/json A list of chains. [​](https://docs.baseten.co/reference/management-api/chains/gets-all-chains#response-chains) chains ChainV1 · object\[\] required Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/models/deletes-a-model-by-id) [By ID\ \ Next](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/chains \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "chains": [\ {\ "id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "name": "",\ "deployments_count": 123,\ "team_name": ""\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Delete training project - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/delete-training-project#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) DELETE / v1 / training\_projects / {training\_project\_id} Try it cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/training_projects/{training_project_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } #### Authorizations [​](https://docs.baseten.co/reference/training-api/delete-training-project#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/delete-training-project#parameter-training-project-id) training\_project\_id string required #### Response 200 - application/json A training project tombstone. [​](https://docs.baseten.co/reference/training-api/delete-training-project#response-id) id string required Unique identifier of the training project [​](https://docs.baseten.co/reference/training-api/delete-training-project#response-deleted) deleted boolean required Whether the training project was deleted Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/create-training-project) [Get training projectGet the details of an existing training project.\ \ Next](https://docs.baseten.co/reference/training-api/get-training-project) ⌘I cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/training_projects/{training_project_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } Assistant Responses are generated using AI and may contain mistakes. --- # Create a team training project - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / teams / {team\_id} / training\_projects Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/teams/{team_id}/training_projects \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_project": { "name": "My Training Project" } }' 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" } } #### Authorizations [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project#parameter-team-id) team\_id string required #### Body application/json A request to upsert a training project. [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project#body-training-project) training\_project UpsertTrainingProjectV1 · object required The training project to upsert. Show child attributes #### Response 200 - application/json A response to upserting a training project. [​](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project#response-training-project) training\_project TrainingProjectV1 · object required The upserted training project. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/get-training-projects) [Get training project cache summaryGet the cache summary for the most recent training job in the project.\ \ Next](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/teams/{team_id}/training_projects \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "training_project": { "name": "My Training Project" } }' 200 { "training_project": { "id": "", "name": "", "created_at": "2023-11-07T05:31:56Z", "updated_at": "2023-11-07T05:31:56Z", "latest_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "team_name": "" } } Assistant Responses are generated using AI and may contain mistakes. --- # Development model deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / models / {model\_id} / deployments / development Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#parameter-model-id) model\_id string required #### Response 200 - application/json A deployment of a model. [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-id) id string required Unique identifier of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-created-at) created\_at string required Time the deployment was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-name) name string required Name of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-model-id) model\_id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-is-production) is\_production boolean required Whether the deployment is the production deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-is-development) is\_development boolean required Whether the deployment is the development deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-status) status enum required Status of the deployment Available options: `BUILDING`, `DEPLOYING`, `DEPLOY_FAILED`, `LOADING_MODEL`, `ACTIVE`, `UNHEALTHY`, `BUILD_FAILED`, `BUILD_STOPPED`, `DEACTIVATING`, `INACTIVE`, `FAILED`, `UPDATING`, `SCALED_TO_ZERO`, `WAKING_UP` [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-active-replica-count) active\_replica\_count integer required Number of active replicas [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-autoscaling-settings-one-of-0) autoscaling\_settings AutoscalingSettingsV1 · object required Autoscaling settings for the deployment. If null, the model has not finished deploying Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-instance-type-name-one-of-0) instance\_type\_name string | null required Name of the instance type the model deployment is running on [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-environment-one-of-0) environment string | null required The environment associated with the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment#response-labels-one-of-0) labels Labels · object User-provided key-value labels for the deployment Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-production-deployment) [Deployment by IDGets a model's deployment by ID and returns the deployment.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } Assistant Responses are generated using AI and may contain mistakes. --- # Update chainlet environment's instance type - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / chains / {chain\_id} / environments / {env\_name} / chainlet\_settings / instance\_types / update Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/chains/{chain_id}/environments/{env_name}/chainlet_settings/instance_types/update \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "updates": [\ {\ "chainlet_name": "HelloWorld",\ "instance_type_id": "1x4"\ },\ {\ "chainlet_name": "RandInt",\ "instance_type_id": "A10G:2x24x96"\ }\ ] }' 200 { "requires_redeployment": true, "chain_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "chain_id": "", "environment": "", "chainlets": [\ {\ "id": "",\ "name": "",\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "active_replica_count": 123\ }\ ] }, "chainlet_environment_settings": [\ {\ "chainlet_name": "",\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ }\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#parameter-chain-id) chain\_id string required [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#parameter-env-name) env\_name string required #### Body application/json A request to update the instance types for chainlets in an environment. Multiples updates can be made in one request. The updates will be processed in batch and a new deployment will be created, deployed and promoted into the environment. [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#body-updates) updates ChainletEnvironmentInstanceTypeUpdateV1 · object\[\] required Mapping of chainlet name to the desired chainlet instance type. If the chainlet name doesn't exist, an error is returned. Show child attributes Example: [ { "chainlet_name": "HelloWorld", "instance_type_id": "1x4" }, { "chainlet_name": "RandInt", "instance_type_id": "A10G:2x24x96" }] #### Response 200 - application/json A response to update the environment settings for a chainlet. If updating the instance type resulted in a re-deployment, `requires_redeployment` will be True and the resulting deployment will be returned in the `chain_deployment` field. [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#response-requires-redeployment) requires\_redeployment boolean required Whether the resource update requires a re-deployment to update the instance type. [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#response-chain-deployment-one-of-0) chain\_deployment ChainDeploymentV1 · object required The chain deployment resulting from the resource update, if any. Show child attributes [​](https://docs.baseten.co/reference/management-api/environments/update-a-chainlet-environments-instance-type-settings#response-chainlet-environment-settings) chainlet\_environment\_settings ChainletEnvironmentSettingsV1 · object\[\] required The updated chainlet environment settings Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/autoscaling/update-a-chainlet-environments-autoscaling-settings) [All instance types\ \ Next](https://docs.baseten.co/reference/management-api/instance-types/gets-all-instance-types) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/chains/{chain_id}/environments/{env_name}/chainlet_settings/instance_types/update \ --header "Authorization: Bearer $BASETEN_API_KEY" \ --data '{ "updates": [\ {\ "chainlet_name": "HelloWorld",\ "instance_type_id": "1x4"\ },\ {\ "chainlet_name": "RandInt",\ "instance_type_id": "A10G:2x24x96"\ }\ ] }' 200 { "requires_redeployment": true, "chain_deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "chain_id": "", "environment": "", "chainlets": [\ {\ "id": "",\ "name": "",\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type_name": "",\ "active_replica_count": 123\ }\ ] }, "chainlet_environment_settings": [\ {\ "chainlet_name": "",\ "autoscaling_settings": {\ "min_replica": 123,\ "max_replica": 123,\ "autoscaling_window": 123,\ "scale_down_delay": 123,\ "concurrency_target": 123,\ "target_utilization_percentage": 123,\ "target_in_flight_tokens": 123,\ "max_scale_down_rate": 123\ },\ "instance_type": {\ "id": "",\ "name": "",\ "memory_limit_mib": 123,\ "millicpu_limit": 123,\ "gpu_count": 123,\ "gpu_type": "",\ "gpu_memory_limit_mib": 123\ }\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Any model deployment by ID - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / models / {model\_id} / deployments / {deployment\_id} Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#parameter-deployment-id) deployment\_id string required #### Response 200 - application/json A deployment of a model. [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-id) id string required Unique identifier of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-created-at) created\_at string required Time the deployment was created in ISO 8601 format [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-name) name string required Name of the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-model-id) model\_id string required Unique identifier of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-is-production) is\_production boolean required Whether the deployment is the production deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-is-development) is\_development boolean required Whether the deployment is the development deployment of the model [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-status) status enum required Status of the deployment Available options: `BUILDING`, `DEPLOYING`, `DEPLOY_FAILED`, `LOADING_MODEL`, `ACTIVE`, `UNHEALTHY`, `BUILD_FAILED`, `BUILD_STOPPED`, `DEACTIVATING`, `INACTIVE`, `FAILED`, `UPDATING`, `SCALED_TO_ZERO`, `WAKING_UP` [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-active-replica-count) active\_replica\_count integer required Number of active replicas [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-autoscaling-settings-one-of-0) autoscaling\_settings AutoscalingSettingsV1 · object required Autoscaling settings for the deployment. If null, the model has not finished deploying Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-instance-type-name-one-of-0) instance\_type\_name string | null required Name of the instance type the model deployment is running on [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-environment-one-of-0) environment string | null required The environment associated with the deployment [​](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-deployment-by-id#response-labels-one-of-0) labels Labels · object User-provided key-value labels for the deployment Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/gets-a-models-development-deployment) [Get model deployment logsGets all the logs for a model deployment in the given time range.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/get-deployment-logs) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} } Assistant Responses are generated using AI and may contain mistakes. --- # Development deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / development / deactivate Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment#parameter-model-id) model\_id string required #### Response 200 - application/json The response to a request to deactivate a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment#response-success) success boolean default:true Whether the deployment was successfully deactivated Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment) [DeploymentDeactivates a deployment and returns the deactivation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # Production deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / production / retry Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#parameter-model-id) model\_id string required #### Response 200 - application/json The response to a request to retry a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#response-retried) retried boolean required Whether the retry was successfully initiated [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#response-deployment) deployment DeploymentV1 · object required The deployment that was retried Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment#response-reason-one-of-0) reason string | null Explanation of the result. Provided when retried is false to explain why retry was not possible. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment) [EnvironmentPromotes an existing deployment to an environment and returns the promoted deployment.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Any deployment by ID - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / {deployment\_id} / retry Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#parameter-deployment-id) deployment\_id string required #### Response 200 - application/json The response to a request to retry a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#response-retried) retried boolean required Whether the retry was successfully initiated [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#response-deployment) deployment DeploymentV1 · object required The deployment that was retried Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment#response-reason-one-of-0) reason string | null Explanation of the result. Provided when retried is false to explain why retry was not possible. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment) [ProductionRetries a failed production deployment and returns the retry status and updated deployment.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/retry/retries-production-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Deactivate environment deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / environments / {env\_name} / deactivate Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment#parameter-env-name) env\_name string required #### Response 200 - application/json The response to a request to deactivate a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment-associated-with-an-environment#response-success) success boolean default:true Whether the deployment was successfully deactivated Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/activate/activates-production-deployment) [DevelopmentDeactivates a development deployment and returns the deactivation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # Any deployment by ID - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / {deployment\_id} / deactivate Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment#parameter-deployment-id) deployment\_id string required #### Response 200 - application/json The response to a request to deactivate a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment#response-success) success boolean default:true Whether the deployment was successfully deactivated Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-development-deployment) [Deactivate production deploymentDeactivates a production deployment and returns the deactivation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # Deactivate production deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / production / deactivate Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment#parameter-model-id) model\_id string required #### Response 200 - application/json The response to a request to deactivate a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment#response-success) success boolean default:true Whether the deployment was successfully deactivated Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-deployment) [DevelopmentRetries a failed development deployment and returns the retry status and updated deployment.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/production/deactivate \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # Development deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / deployments / development / retry Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#parameter-model-id) model\_id string required #### Response 200 - application/json The response to a request to retry a deployment. [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#response-retried) retried boolean required Whether the retry was successfully initiated [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#response-deployment) deployment DeploymentV1 · object required The deployment that was retried Show child attributes [​](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-development-deployment#response-reason-one-of-0) reason string | null Explanation of the result. Provided when retried is false to explain why retry was not possible. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-production-deployment) [DeploymentRetries a failed deployment and returns the retry status and updated deployment.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/retry/retries-a-deployment) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/deployments/development/retry \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "retried": true, "deployment": { "id": "", "created_at": "2023-11-07T05:31:56Z", "name": "", "model_id": "", "is_production": true, "is_development": true, "active_replica_count": 123, "autoscaling_settings": { "min_replica": 123, "max_replica": 123, "autoscaling_window": 123, "scale_down_delay": 123, "concurrency_target": 123, "target_utilization_percentage": 123, "target_in_flight_tokens": 123, "max_scale_down_rate": 123 }, "instance_type_name": "", "environment": "", "labels": {} }, "reason": "" } Assistant Responses are generated using AI and may contain mistakes. --- # Get training job checkpoint files - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / training\_projects / {training\_project\_id} / jobs / {training\_job\_id} / checkpoint\_files Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoint_files \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "presigned_urls": [\ {\ "url": "",\ "relative_file_name": "",\ "node_rank": 123,\ "size_bytes": 123,\ "last_modified": ""\ }\ ], "total_count": 123, "next_page_token": 123 } #### Authorizations [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#parameter-training-project-id) training\_project\_id string required [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#parameter-training-job-id) training\_job\_id string required #### Query Parameters [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#parameter-page-size) page\_size integer default:1000 Max files per page (default 1000). Required range: `x >= 1` [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#parameter-page-token) page\_token integer default:0 Offset into the file list (default 0). Required range: `x >= 0` #### Response 200 - application/json A response to fetch presigned URLs for checkpoint files of a training job. [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#response-presigned-urls) presigned\_urls CheckpointFile · object\[\] required List of presigned URLs for checkpoint files. Show child attributes [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#response-total-count) total\_count integer required Total number of checkpoint files available. [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files#response-next-page-token-one-of-0) next\_page\_token integer | null Token to use for fetching the next page of results. None when there are no more results. Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/get-training-job) [List training job checkpointsGet the checkpoints for a training job.\ \ Next](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoint_files \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "presigned_urls": [\ {\ "url": "",\ "relative_file_name": "",\ "node_rank": 123,\ "size_bytes": 123,\ "last_modified": ""\ }\ ], "total_count": 123, "next_page_token": 123 } Assistant Responses are generated using AI and may contain mistakes. --- # List training job checkpoints - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / training\_projects / {training\_project\_id} / jobs / {training\_job\_id} / checkpoints Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoints \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "checkpoints": [\ {\ "checkpoint_id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "checkpoint_type": "",\ "base_model": "",\ "lora_adapter_config": {},\ "size_bytes": 123,\ "training_job_id": "",\ "sync_status": ""\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#parameter-training-project-id) training\_project\_id string required [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#parameter-training-job-id) training\_job\_id string required #### Response 200 - application/json A response to fetch checkpoints for a training job. [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#response-training-job) training\_job TrainingJobV1 · object required The training job. Show child attributes [​](https://docs.baseten.co/reference/training-api/get-training-job-checkpoints#response-checkpoints) checkpoints TrainingJobCheckpointV1 · object\[\] required The checkpoints for the training job. Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/training-api/get-training-job-checkpoint-files) [Get training job logsGet the logs for a training job with the provided filters.\ \ Next](https://docs.baseten.co/reference/training-api/get-training-job-logs) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/jobs/{training_job_id}/checkpoints \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "training_job": { "id": "", "created_at": "2023-11-07T05:31:56Z", "current_status": "", "instance_type": { "id": "", "name": "", "memory_limit_mib": 123, "millicpu_limit": 123, "gpu_count": 123, "gpu_type": "", "gpu_memory_limit_mib": 123 }, "updated_at": "2023-11-07T05:31:56Z", "training_project_id": "", "training_project": { "id": "", "name": "" }, "error_message": "", "name": "gpt-oss-job", "priority": 0, "user": { "email": "" } }, "checkpoints": [\ {\ "checkpoint_id": "",\ "created_at": "2023-11-07T05:31:56Z",\ "checkpoint_type": "",\ "base_model": "",\ "lora_adapter_config": {},\ "size_bytes": 123,\ "training_job_id": "",\ "sync_status": ""\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. --- # Delete chains - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) DELETE / v1 / chains / {chain\_id} Try it cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/chains/{chain_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id#parameter-chain-id) chain\_id string required #### Response 200 - application/json A chain tombstone. [​](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id#response-id) id string required Unique identifier of the chain [​](https://docs.baseten.co/reference/management-api/chains/deletes-a-chain-by-id#response-deleted) deleted boolean required Whether the chain was deleted Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/chains/gets-a-chain-by-id) [Chain deploymentDeactivates a chain deployment and returns the deactivation status.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/deactivate/deactivates-a-chain-deployment) ⌘I cURL cURL curl --request DELETE \ --url https://api.baseten.co/v1/chains/{chain_id} \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "id": "", "deleted": true } Assistant Responses are generated using AI and may contain mistakes. --- # Force cancel rolling deployment - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) POST / v1 / models / {model\_id} / environments / {env\_name} / force\_cancel\_promotion Try it cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/force_cancel_promotion \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } #### Authorizations [​](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion#parameter-model-id) model\_id string required [​](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion#parameter-env-name) env\_name string required #### Response 200 - application/json The response to a request to signal a rolling promotion. [​](https://docs.baseten.co/reference/management-api/deployments/promote/force-cancel-promotion#response-success) success boolean required Whether the signal was successfully sent Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/deployments/promote/resume-promotion) [Force roll forward promotionImmediately completes the rolling promotion, shifting all traffic to the new version. This works even if the promotion is in the process of rolling back.\ \ Next](https://docs.baseten.co/reference/management-api/deployments/promote/force-roll-forward-promotion) ⌘I cURL cURL curl --request POST \ --url https://api.baseten.co/v1/models/{model_id}/environments/{env_name}/force_cancel_promotion \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "success": true } Assistant Responses are generated using AI and may contain mistakes. --- # Get training project cache summary - Baseten > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.baseten.co/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#content-area) [Documentation](https://docs.baseten.co/overview) [Examples](https://docs.baseten.co/examples/overview) [Reference](https://docs.baseten.co/reference/overview) [Status](https://docs.baseten.co/status/status) GET / v1 / training\_projects / {training\_project\_id} / cache / summary Try it cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/cache/summary \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "timestamp": "", "project_id": "", "file_summaries": [\ {\ "path": "",\ "size_bytes": 123,\ "modified": "",\ "file_type": "",\ "permissions": ""\ }\ ] } #### Authorizations [​](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#authorization-authorization) Authorization string header required Pass your Baseten API key. Clients automatically send `Authorization: Bearer `. Direct callers can also use `Authorization: Api-Key `; both schemes are accepted. #### Path Parameters [​](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#parameter-training-project-id) training\_project\_id string required #### Response 200 - application/json Response for getting cache summary. [​](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#response-timestamp) timestamp string required Timestamp when the cache summary was captured [​](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#response-project-id) project\_id string required Project ID associated with the cache [​](https://docs.baseten.co/reference/training-api/get-training-project-cache-summary#response-file-summaries) file\_summaries FileSummary · object\[\] required List of files in the cache Show child attributes Was this page helpful? YesNo [Previous](https://docs.baseten.co/reference/management-api/teams/creates-a-team-training-project) [Download training job source codeGet the uploaded training job as a S3 Artifact\ \ Next](https://docs.baseten.co/reference/training-api/download-training-job) ⌘I cURL cURL curl --request GET \ --url https://api.baseten.co/v1/training_projects/{training_project_id}/cache/summary \ --header "Authorization: Bearer $BASETEN_API_KEY" 200 { "timestamp": "", "project_id": "", "file_summaries": [\ {\ "path": "",\ "size_bytes": 123,\ "modified": "",\ "file_type": "",\ "permissions": ""\ }\ ] } Assistant Responses are generated using AI and may contain mistakes. ---