# Table of Contents - [Welcome to Vellum | Vellum | Documentation](#welcome-to-vellum-vellum-documentation) - [Welcome to Vellum | Vellum | Documentation](#welcome-to-vellum-vellum-documentation) - [Resources | Vellum | Documentation](#resources-vellum-documentation) - [Vellum's Help Center | Vellum | Documentation](#vellum-s-help-center-vellum-documentation) - [Self-Hosted Vellum | Vellum | Documentation](#self-hosted-vellum-vellum-documentation) - [Version 0.2 | Vellum | Documentation](#version-0-2-vellum-documentation) - [Version 0.3 | Vellum | Documentation](#version-0-3-vellum-documentation) - [February 2026 | Vellum | Documentation](#february-2026-vellum-documentation) - [Changelog | February, 2025 | Vellum | Documentation](#changelog-february-2025-vellum-documentation) - [December 2025 | Vellum | Documentation](#december-2025-vellum-documentation) - [Changelog | April, 2025 | Vellum | Documentation](#changelog-april-2025-vellum-documentation) - [Changelog | December, 2024 | Vellum | Documentation](#changelog-december-2024-vellum-documentation) - [Changelog | May, 2025 | Vellum | Documentation](#changelog-may-2025-vellum-documentation) - [October 2025 | Vellum | Documentation](#october-2025-vellum-documentation) - [January 2026 | Vellum | Documentation](#january-2026-vellum-documentation) - [August 2025 | Vellum | Documentation](#august-2025-vellum-documentation) - [September 2025 | Vellum | Documentation](#september-2025-vellum-documentation) - [Changelog | September, 2024 | Vellum | Documentation](#changelog-september-2024-vellum-documentation) - [Changelog | January, 2024 | Vellum | Documentation](#changelog-january-2024-vellum-documentation) - [Changelog | August, 2024 | Vellum | Documentation](#changelog-august-2024-vellum-documentation) - [Changelog | January, 2025 | Vellum | Documentation](#changelog-january-2025-vellum-documentation) - [June 2025 | Vellum | Documentation](#june-2025-vellum-documentation) - [November 2025 | Vellum | Documentation](#november-2025-vellum-documentation) - [Changelog | May, 2024 | Vellum | Documentation](#changelog-may-2024-vellum-documentation) - [Changelog | February, 2024 | Vellum | Documentation](#changelog-february-2024-vellum-documentation) - [Changelog | March, 2025 | Vellum | Documentation](#changelog-march-2025-vellum-documentation) - [Changelog | April, 2024 | Vellum | Documentation](#changelog-april-2024-vellum-documentation) - [Changelog | March, 2024 | Vellum | Documentation](#changelog-march-2024-vellum-documentation) - [Changelog | October, 2024 | Vellum | Documentation](#changelog-october-2024-vellum-documentation) - [Changelog | November, 2024 | Vellum | Documentation](#changelog-november-2024-vellum-documentation) - [Changelog | July, 2024 | Vellum | Documentation](#changelog-july-2024-vellum-documentation) - [Push Feature Status | Vellum | Documentation](#push-feature-status-vellum-documentation) - [Datasets Overview | Vellum | Documentation](#datasets-overview-vellum-documentation) --- # Welcome to Vellum | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Quick Links ----------- [Product Documentation\ \ Learn about Vellum’s features, including Prompts, Workflows, and Deployments.](https://docs.vellum.ai/product/getting-started/overview) [Developer Documentation\ \ Integrate with Vellum using our SDKs, APIs, and developer tools.](https://docs.vellum.ai/developers/getting-started/overview) [Agent Builder Guide\ \ Practical tips and best practices for building Workflows with Agent Builder.](https://docs.vellum.ai/product/agent-builder/agent-builder-sme) [Workflows\ \ Build complex AI applications with Vellum’s visual Workflow builder.](https://docs.vellum.ai/product/workflows/introduction) [Evaluation\ \ Test and evaluate your AI applications with metrics and test suites.](https://docs.vellum.ai/product/evaluation/quantitative-evaluation) What is Vellum? --------------- Vellum is an end-to-end AI development platform that helps teams build and deploy AI-powered applications. It provides a collaborative environment where both technical and non-technical team members can contribute to AI projects using familiar tools and interfaces. ![Vellum Four Pillars of Development](https://storage.googleapis.com/vellum-public/help-docs/home/vellum_four_pillars.png) Vellum's Four Pillars of AI Development The Challenges of AI Development -------------------------------- AI development is often slowed down by silos between technical and business teams, leading to: * Slow development cycles * Communication gaps between stakeholders * AI systems being shipped with “vibe checks” and not rigorous evals * Insufficient monitoring of production use cases * Integration challenges between different tools Vellum addresses these problems by providing a unified platform where all team members play a crucial role in the AI development process. How Does Vellum Work? --------------------- Vellum combines several key capabilities into a single, integrated platform: ### For Technical Teams * **Developer Tools**: Full IDE support and API integrations * **Version Control**: Built-in systems for managing prompts, orchestration logic and models * **Testing Framework**: Quantitative testing and performance monitoring. Clear visualization of inputs and outputs at each step of your graphs * **RAG Pipeline**: Document search and retrieval-augmented generation capabilities ### For Non-Technical Teams * **Low-Code Interface**: Visual tools for prompt engineering and testing * **Spreadsheet Integration**: Familiar interfaces for content management * **Collaborative Features**: Shared workspaces and feedback tools ### For Mixed Teams There are many ways teams can collaborate in Vellum. Here are just a few common patterns: * Subject matter experts write prompts, measure quality, and deploy updates * Engineers build integrations and test control flow * Changes sync between code and UI to easily share assets and feedback This flexible approach lets each team member contribute using their preferred tools while maintaining a single source of truth. Getting Started --------------- Choose your path to get started with Vellum: [Developer Guide\ \ Get started with our SDKs, APIs, and developer tools to build and deploy AI applications.](https://docs.vellum.ai/developers/getting-started/overview) [Subject Matter Expert Guide\ \ Learn how to build AI products using our low-code interface, visual tools, and collaborative features.](https://docs.vellum.ai/product/getting-started/overview) Need help? Contact our support team at [support@vellum.ai](mailto:support@vellum.ai) Ask AI Assistant Responses are generated using AI and may contain mistakes. Hi, I'm an AI assistant with access to documentation and other content. Tip: You can toggle this pane with ⌘ + / ![Vellum Four Pillars of Development](https://docs.vellum.ai/) --- # Welcome to Vellum | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Quick Links ----------- [Product Documentation\ \ Learn about Vellum’s features, including Prompts, Workflows, and Deployments.](https://docs.vellum.ai/product/getting-started/overview) [Developer Documentation\ \ Integrate with Vellum using our SDKs, APIs, and developer tools.](https://docs.vellum.ai/developers/getting-started/overview) [Agent Builder Guide\ \ Practical tips and best practices for building Workflows with Agent Builder.](https://docs.vellum.ai/product/agent-builder/agent-builder-sme) [Workflows\ \ Build complex AI applications with Vellum’s visual Workflow builder.](https://docs.vellum.ai/product/workflows/introduction) [Evaluation\ \ Test and evaluate your AI applications with metrics and test suites.](https://docs.vellum.ai/product/evaluation/quantitative-evaluation) What is Vellum? --------------- Vellum is an end-to-end AI development platform that helps teams build and deploy AI-powered applications. It provides a collaborative environment where both technical and non-technical team members can contribute to AI projects using familiar tools and interfaces. ![Vellum Four Pillars of Development](https://storage.googleapis.com/vellum-public/help-docs/home/vellum_four_pillars.png) Vellum's Four Pillars of AI Development The Challenges of AI Development -------------------------------- AI development is often slowed down by silos between technical and business teams, leading to: * Slow development cycles * Communication gaps between stakeholders * AI systems being shipped with “vibe checks” and not rigorous evals * Insufficient monitoring of production use cases * Integration challenges between different tools Vellum addresses these problems by providing a unified platform where all team members play a crucial role in the AI development process. How Does Vellum Work? --------------------- Vellum combines several key capabilities into a single, integrated platform: ### For Technical Teams * **Developer Tools**: Full IDE support and API integrations * **Version Control**: Built-in systems for managing prompts, orchestration logic and models * **Testing Framework**: Quantitative testing and performance monitoring. Clear visualization of inputs and outputs at each step of your graphs * **RAG Pipeline**: Document search and retrieval-augmented generation capabilities ### For Non-Technical Teams * **Low-Code Interface**: Visual tools for prompt engineering and testing * **Spreadsheet Integration**: Familiar interfaces for content management * **Collaborative Features**: Shared workspaces and feedback tools ### For Mixed Teams There are many ways teams can collaborate in Vellum. Here are just a few common patterns: * Subject matter experts write prompts, measure quality, and deploy updates * Engineers build integrations and test control flow * Changes sync between code and UI to easily share assets and feedback This flexible approach lets each team member contribute using their preferred tools while maintaining a single source of truth. Getting Started --------------- Choose your path to get started with Vellum: [Developer Guide\ \ Get started with our SDKs, APIs, and developer tools to build and deploy AI applications.](https://docs.vellum.ai/developers/getting-started/overview) [Subject Matter Expert Guide\ \ Learn how to build AI products using our low-code interface, visual tools, and collaborative features.](https://docs.vellum.ai/product/getting-started/overview) Need help? Contact our support team at [support@vellum.ai](mailto:support@vellum.ai) --- # Resources | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. [Product Documentation\ \ Learn about Vellum’s features, tools, and capabilities.](https://docs.vellum.ai/product/getting-started/overview) [Developer Documentation\ \ Interact with Vellum through our SDKs, APIs, and developer tools.](https://docs.vellum.ai/developers/getting-started/overview) [Example Gallery\ \ Explore real-world implementations and templates](https://docs.vellum.ai/product/workflows/examples/overview) [Blog\ \ Read the latest news and updates about Vellum.](https://vellum.ai/blog) [System Status\ \ Check Vellum service status and uptime](https://status.vellum.ai/) --- # Vellum's Help Center | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. No question, feedback, or issue is too small to share with us. We’re here to help you build better AI products. Email us -------- We respond quickly to emails, so don’t hesitate to reach out to us at [support@vellum.ai](mailto:support@vellum.ai) . When you do, please describe your use case and the specific problem you’re encountering so we can better assist you. Slack ----- Reach out in Slack if you’re an active Vellum customer with a shared Slack channel. That’s our primary channel of communication with customers and is monitored 24/7. In-app chat ----------- Chat with our support team using the “Get Help” button on the bottom left corner of your Vellum dashboard. Documentation ------------- If you prefer a to learn on your own, check out the articles here in our Documentation. We have plenty of articles with detailed explanations, screenshots, and videos to help you troubleshoot common issues. We’re constantly adding new resources, so be sure to check back often or suggest improvements. --- # Self-Hosted Vellum | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Vellum offers a self-hosted deployment option that allows you to run the complete Vellum platform within your own infrastructure. This enterprise-grade solution provides enhanced security, data sovereignty, and complete control over your AI development environment. [Request a Demo\ \ Contact our team to learn more about self-hosted Vellum and schedule a personalized demo.](https://www.vellum.ai/landing-pages/request-demo) Why Self-Host Vellum? --------------------- Self-hosted Vellum is ideal for organizations that require: * **Data Sovereignty**: Keep all your data within your own infrastructure and geographic boundaries * **Enhanced Security**: Maintain complete control over access, encryption, and security policies * **Compliance**: Meet strict regulatory requirements for data handling and processing * **Custom Infrastructure**: Integrate with existing enterprise systems and networking policies * **Air-Gapped Environments**: Deploy in environments without internet connectivity Get Started with Self-Hosted Vellum ----------------------------------- If you’re interested in deploying Vellum in your own infrastructure, our team is ready to help you get started. We provide comprehensive support throughout the deployment process, including: * Architecture planning and sizing recommendations * Installation and configuration assistance * Training and onboarding for your team * Ongoing support and maintenance Quick Links ----------- [Changelog\ \ View the latest updates and releases for self-hosted Vellum](https://docs.vellum.ai/self-hosting/changelog/v3) --- # Version 0.2 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. \[HOTFIX\] v0.2.20 - March 11, 2025 ----------------------------------- **Bugfixes** * Pins image version tags at correct SHAs so that images are not using latest which may break compatibility between upgrades * * * v0.2.19 - March 10, 2025 ------------------------ **New Additions** Added new models including: * **GPT-4.5 Preview** by OpenAI * **QwQ** by Qwen * Others you may find via our [official app changelog](https://docs.vellum.ai/changelog/2025/2025-02) **Helm chart changes** for finer tuned configurability for cluster operators **Bugfixes** * Various stability improvements * * * \[HOTFIX\] v0.2.13 - February 12, 2025 -------------------------------------- **Bugfixes** * Fixes `django.db.utils.OperationalError: could not translate host name "vellum-postgres-r.default.svc.cluster.local" to address: Name or service not known` error occurring in the clickhouse-backups cronjob * * * v0.2.11 - February 10, 2025 --------------------------- **New Additions** Added new models including: * **xAI model host** with Grok 2 * **DeepSeek V3** via Fireworks AI * Others you may find via our [official app changelog](https://docs.vellum.ai/changelog/2025/2025-02) **Backend changes** that result in a more responsive/performant UI **Bugfixes** * Various stability improvements * * * \[HOTFIX\] v0.2.9 - January 30, 2025 ------------------------------------ **Bugfixes** * Fixed `Failed to resolve ' | | $ | \# Then proceed with the upgrade in your kots admin console..... | **New Additions** * Added **DeepSeek R1** support * Added many new handlers to the cluster * Configuration can be found within **Vellum Message Handlers** subsection in the KOTS Admin Console **Removals** * Removed `job/vellum-ai-pubsub-init-hook` **Bugfixes** * Various stability improvements * * * v0.2.0 - December 20, 2024 -------------------------- **NOTE:** Due to [a known issue with the KEDA helm chart](https://github.com/kedacore/charts/issues/226) , we are not able to add it as a dependency. In order to deploy v0.2.x+, you must first [install KEDA in your cluster](https://keda.sh/docs/2.16/deploy/#install) | | | | --- | --- | | $ | \# Add kedacore repo to helm | | $ | ➜ helm repo add kedacore https://kedacore.github.io/charts | | $ | \# Update your helm repos | | $ | ➜ helm repo update | | $ | \# Install the KEDA chart to your cluster in the Vellum app namespace | | $ | ➜ helm install keda kedacore/keda --namespace | | $ | \# Then proceed with the upgrade in your kots admin console..... | **New Additions** * **First major architectural piece** — documents processing — now uses in-cluster RabbitMQ instead of Pub/Sub * We will continue to cut over pieces of our architecture to use RabbitMQ exclusively for event-driven workloads * Added **bitnami/rabbitmq** as a dependency * Added **various handlers for documents processing** * These handlers are configurable via a new section the KOTS config form labeled “Vellum Documents Handlers” * This replaces the ffs deployment * Added **[KEDA](https://keda.sh/) ScaledObjects and TriggerAuthentication** for autoscaling based on QueueLength in addition to typical CPU/Memory utilization metrics * Added **HPAs for the django and django-task-handlers deployments** — configurable via the KOTS admin console * These will likely be replaced in favor of KEDA HPAs in a future release * Added **annotation text area for Ingress object** for custom annotations **Bugfixes** * Cleaned up KOTS config form to only show cloud provider-specific options * Removed `/grafana` path from Ingress object * Fixed `iam.gke.io/gcp-service-account` missing annotation issue with the default ServiceAccount * Removed hardcoded read replica value for PostgreSQL --- # Version 0.3 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. v0.3.44 - February 6, 2026 -------------------------- * Addition of data deletion handler and cron job for async delete requests * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.14.3 **[Application Changelog](https://docs.vellum.ai/changelog/2026/2026-02#custom-interfaces-for-chat-agents) ** v0.3.43 - January 22, 2026 -------------------------- * User and organization management handler consolidation * Async migrations improvements * Addition of execution state write handler * Removal of deprecated handlers * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.13.5 **[Application Changelog](https://docs.vellum.ai/changelog/2026/2026-01#workflow-log-events) ** v0.3.42 - December 17, 2025 --------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.11.20 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-12#mistral-ai-model-provider) ** v0.3.41 - December 9, 2025 -------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.11.15 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-12#workflow-layout-dropdown) ** v0.3.40 - December 9, 2025 -------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.11.15 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-12#chat-history-operation-in-set-state-node) ** v0.3.39 - November 18, 2025 --------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.10.7 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-11#support-for-gemini-3-pro-preview) ** v0.3.38 - November 5, 2025 -------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.10.1 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-11#customizable-workflow-icons) ** v0.3.37 - October 21, 2025 -------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.9.6 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-10#settings-revamp) ** v0.3.36 - October 21, 2025 -------------------------- * Bug fixes and application updates **SDK and Python Workflow Server Latest Version:** 1.7.13 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-10#agent-builder-updates) ** v0.3.35 - October 15, 2025 -------------------------- * Bug fixes and application update **SDK and Python Workflow Server Latest Version:** 1.7.10 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-10#gpt-5-pro-models) ** v0.3.34 - October 7, 2025 ------------------------- * Bug fixes and application update **SDK and Python Workflow Server Latest Version:** 1.7.4 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-10#gpt-5-pro-models) ** v0.3.33 - October 5, 2025 ------------------------- * Configurable max workflow runtime * Removal of vellum-realtime-cross-encoder-stsb-roberta-large service * Bug fixes and Application Updates **SDK and Python Workflow Server Latest Version:** 1.7.1 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-10#agent-builder---generate-agent-nodes-with-integration-tools) ** v0.3.32 - September 29, 2025 ---------------------------- * Addition of vellum-monitoring service * Bug fixes and Application Updates **SDK and Python Workflow Server Latest Version:** 1.5.6 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-09#google-gemini-25-flash-models) ** v0.3.31 - September 17, 2025 ---------------------------- * Application Updates **SDK and Python Workflow Server Latest Version:** 1.4.1 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-09#custom-node-colors--icons) ** v0.3.30 - September 8, 2025 --------------------------- * Application Updates **SDK and Python Workflow Server Latest Version:** 1.3.5 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-09#create-custom-nodes-from-the-ui) ** v0.3.29 - September 4, 2025 --------------------------- * Application Updates **SDK and Python Workflow Server Latest Version:** 1.3.3 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-09#syntax-highlighting-in-prompt-jinja-blocks) ** v0.3.28 - August 26, 2025 ------------------------- * Service account name for Workflow servers can now be configured **SDK and Python Workflow Server Latest Version:** 1.2.5 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-08#api-keys-removed-from-side-nav) ** v0.3.27 - August 19, 2025 ------------------------- * Installs using the built in CloudNativePG Postgres can now resize the PVCs from KOTS admin **SDK and Python Workflow Server Latest Version:** 1.2.2 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-08#improved-variable-detection-when-pasting-prompts) ** v0.3.26 - August 14, 2025 ------------------------- **YOU MUST UPGRADE TO THIS RELEASE BY AUGUST 28th**: * [Bitnami recently announced](https://github.com/bitnami/charts/issues/35164) that they will be migrating all of their images to a new repository and no longer providing public support * This release addresses this issue by updating image references * You must upgrade to this release by August 28th otherwise subsequent upgrades or restarts can result in failed image pulls and downtime * Addressing Bitnami announcement * Upgraded migrations logic to improve upgrade speed and reliability * Bug fixes and improvements **SDK and Python Workflow Server Latest Version:** 1.2.1 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-08#models-page-revamp) ** v0.3.25 - August 5, 2025 ------------------------ * Minor changes to the Helm Chart **SDK and Python Workflow Server Latest Version:** 1.0.11 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-08) ** v0.3.24 - July 30, 2025 ----------------------- * Chart values default updates **SDK and Python Workflow Server Latest Version:** 1.0.8 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#prompt-node-unification) ** v0.3.23 - July 25, 2025 ----------------------- * Fix resourcequota conditional **SDK and Python Workflow Server Latest Version:** 1.0.6 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#model-picker-improvements) ** v0.3.22 - July 24, 2025 ----------------------- * Model Picker improvements **SDK and Python Workflow Server Latest Version:** 1.0.5 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#model-picker-improvements) ** v0.3.21 - July 23, 2025 ----------------------- * Removes unused environment variables in chart * Fixes weaviate backup issues **SDK and Python Workflow Server Latest Version:** 1.0.5 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#thinking-output-in-prompt-nodes-and-prompt-deployment-nodes) ** v0.3.20 - July 15, 2025 ----------------------- * Cluster wide resource configuration has been added * PVC sizes are no longer configurable and now must be manually resized **SDK and Python Workflow Server Latest Version:** 1.0.0 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#reasoning-outputs-in-prompt-sandboxes) ** **HOTFIX** v0.3.19 - July 9, 2025 --------------------------------- * Bug fix: A migration in version 0.3.18 resulted in new instances of self-hosted Vellum failing on creation * Default handler CPU requests have been decreased to reduce cluster size requirements **SDK and Python Workflow Server Latest Version:** 0.14.84 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#gemini-embedding-model-support) ** v0.3.18 - July 8, 2025 ---------------------- * Increased default memory configuration for Clickhouse * Permanently disabled Launch Darkly in VPC and added a manual feature flag override * Bug fixes and stability improvements **NOTES**: * With this release, the minimum supported Postgres version is 14 * A networking change in this release can result in issues. Customers may need to remove load balancer services for vellum-frontend as well as removing the ingress object before upgrading. If you encounter any issues or would like assistance, please reach out to Vellum. * If Clickhouse memory is manually configured, please increase to at least match default values * If Launch Darkly is enabled, please reach out to Vellum before upgrading **SDK and Python Workflow Server Latest Version:** 0.14.84 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#gemini-embedding-model-support) ** * * * v0.3.17 - July 1, 2025 ---------------------- * Task-Handlers has been removed * Support for new dedicated On-Prem and Embedded Cloud Providers * You can now specify any registry regardless of the cloud provider * Support for generic SMTP servers as an alternative to Mailgun **NOTES**: * If you are relying on the DNS record for `tasks.vellum.domain` please migrate to `api.vellum.domain` instead. * There are some large data migrations that will take place in this upgrade and subsequently resource requirements may need to be adjusted. **SDK and Python Workflow Server Latest Version:** 0.14.81 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-07#expression-input-output-type-improvements) ** * * * v0.3.16 - June 24, 2025 ----------------------- * Configurable default retention policies for new orgs * Added Pre-flight Hook Jobs configuration section in KOTS * Replaced LoadBalancer type services with ClusterIP * Bug fixes and stability improvements **NETWORKING CHANGE WARNING**: A networking change in this release can result in issues. Customers may need to remove load balancer services for vellum-django and vellum-predict as well as removing the ingress object before upgrading. If you encounter any issues or would like assistance, please reach out to Vellum. **SDK and Python Workflow Server Latest Version:** 0.14.73 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-06#support-for-gemini-25-pro-preview-06-05) ** * * * v0.3.15 - June 17, 2025 ----------------------- * Configurable ID for workspace to copy example prompt/workflows from on new org creation * Bug fixes and stability improvements **SDK and Python Workflow Server Latest Version:** 0.14.71 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-06#support-for-gemini-25-pro-preview-06-05) ** * * * v0.3.14 - June 10, 2025 ----------------------- * Bug fixes and stability improvements **SDK and Python Workflow Server Latest Version:** 0.14.70 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-06#gpt-4o-audio-preview-2025-06-03) ** * * * v0.3.13 - June 3, 2025 ---------------------- * Enable data retention policies at organization level * Bug fixes and stability improvements **SDK and Python Workflow Server Latest Version:** 0.14.66 **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-06#prompt--workflow-descriptions) ** * * * v0.3.12 - May 30, 2025 ---------------------- * AWS Bedrock models do not require extra role when deployed in AWS * Bug fixes and stability improvements **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-05#vertex-ai-claude-4-models-via-vellum) ** * * * v0.3.11 - May 21, 2025 ---------------------- * Configurable resource limits for code execution * Configure default staff access for new organizations * Bug fixes and stability improvements **[Application Changelog](https://docs.vellum.ai/changelog/2025/2025-05#ai-powered-prompt-improver-beta) ** * * * v0.3.10 - May 13, 2025 ---------------------- **NOTE:** In order to enable Vellum Workflows SDK functionality with this release, you must enable the flag **Install Knative** in the KOTS admin console. ![KOTS Admin Console showing Install Knative flag](https://vellum-ai.notion.site/image/attachment%3Ada2c90a5-9b25-4150-a222-cbd86962ee1f%3AScreenshot_2025-05-01_at_6.11.50_PM.png?table=block&id=1f29035c-57bd-80b6-badb-e0d7bfd3ae71&spaceId=71c05e3e-272b-4acf-9889-90a304d95d06&width=2000&userId=&cache=v2) KOTS Admin Console - Install Knative Flag **New App Additions** * Bug Fixes and Stability improvements **Model Additions** * New model additions for this release are listed here: [May 2025 Changelog](https://docs.vellum.ai/changelog/2025/2025-05) * * * v0.3.9 - May 6, 2025 -------------------- **NOTE:** In order to enable Vellum Workflows SDK functionality with this release, you must enable the flag **Install Knative** in the KOTS admin console. ![KOTS Admin Console showing Install Knative flag](https://vellum-ai.notion.site/image/attachment%3Ada2c90a5-9b25-4150-a222-cbd86962ee1f%3AScreenshot_2025-05-01_at_6.11.50_PM.png?table=block&id=1eb9035c-57bd-808b-98e9-faf5994abcc2&spaceId=71c05e3e-272b-4acf-9889-90a304d95d06&width=2000&userId=&cache=v2) KOTS Admin Console - Install Knative Flag \*\* New App Additions \*\* * Stability improvements **Model Additions** * New model additions for this release are listed here: [May 2025 Changelog](https://docs.vellum.ai/changelog/2025/2025-05) * * * v0.3.8 - April 30, 2025 ----------------------- **NOTE:** In order to enable Vellum Workflows SDK functionality with this release, you must enable the flag **Install Knative** in the KOTS admin console. ![KOTS Admin Console showing Install Knative flag](https://vellum-ai.notion.site/image/attachment%3Ada2c90a5-9b25-4150-a222-cbd86962ee1f%3AScreenshot_2025-05-01_at_6.11.50_PM.png?table=block&id=1e69035c-57bd-8076-a8bf-d9afec3163af&spaceId=71c05e3e-272b-4acf-9889-90a304d95d06&width=2000&userId=&cache=v2) KOTS Admin Console - Install Knative Flag **New App Additions** * **Grafana service** to enable Monitoring tab functionality for Prompt and Workflow Deployments * **Vembda and Codegen services** to enable custom code execution from within Vellum Workflows SDK * **Stability and configuration improvements** throughout * **Keycloak support** as auth provider **Model Additions** * New model additions for this release are listed here: [April 2025 Changelog](https://docs.vellum.ai/changelog/2025/2025-04) * * * v0.3.4 - April 10, 2025 ----------------------- **Bug Fixes** * Fix hook ignore configuration reads * Include both upper and lowercase http proxy env vars when all specified in admin console * Weuc handler postgres fix * * * v0.3.3 - April 9, 2025 ---------------------- **New Additions** * Enable HTTP proxy env var configuration * Make default storage class configurable * * * v0.3.2 - April 3, 2025 ---------------------- No infrastructure change * * * v0.3.1 - March 31, 2025 ----------------------- **New Additions** * Enable vellum workflows pull for self hosted customers --- # February 2026 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Custom Interfaces for Chat Agents --------------------------------- **February 4th, 2026** You can now generate custom interfaces for your Chat Agents. Access the new “Preview” mode from the Workflow Sandbox toolbar and build your interface by chatting with Vellum. The interface code is versioned and deployed alongside your Workflow, so publishing your Chat Agent also publishes its interface. You can update the interface independently of the Workflow. ![Workflow Sandbox showing the Preview mode tab highlighted in the toolbar](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/custom-interfaces-preview-mode-2026-02-04-e3ca732b.png) Preview mode in the Workflow Sandbox toolbar ![Preview mode showing the Generate UI button with interface suggestions](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/custom-interfaces-generate-ui-2026-02-04-ad39e76e.png) Generate UI to start building your custom interface ![A fully generated custom chat interface for an APUSH Tutor agent with welcome message and suggestion buttons](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/custom-interfaces-apush-tutor-2026-02-04-9f5530a1.png) A custom interface generated for a Chat Agent Microsoft Teams Integration --------------------------- **February 3rd, 2026** [Microsoft Teams](https://www.microsoft.com/) is now available as a native integration. Connect your Microsoft Teams workspace to Vellum and use it in your Workflows—send messages, post to channels, and more through Agent Nodes or Custom Nodes. Native Integrations Grouping ---------------------------- **February 3rd, 2026** The Integrations page now groups native integrations by Enabled and Disabled, making it easier to see which ones are ready to use with your Agents. ![Integrations page showing native integrations grouped into Enabled and Disabled sections](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/integrations-page-enabled-disabled-grouping-2026-02-03-7caa19bd.png) Native integrations grouped by Enabled and Disabled Message Queueing ---------------- **February 2nd, 2026** Agent Builder now supports message queueing during response generation. Send multiple instructions in advance, pause sending them, and let them run automatically once the current response finishes. ![Agent Builder showing message queue with queued instructions](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-message-queue-2026-02-02-v2-67dd4787.png) Message queue in Agent Builder --- # Changelog | February, 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Support for GPT-4.5 Preview via OpenAI -------------------------------------- **February 27th, 2025** We’ve added support for the [GPT-4.5 Preview and Snapshot 02/27/2025](https://platform.openai.com/docs/models/gpt-4-5#gpt-4-5) via OpenAI. Support for Llama 3.2 and Llama 3.3 models via AWS Bedrock ---------------------------------------------------------- **February 27th, 2025** We’ve added support for [Llama 3.2 and Llama 3.3 models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) via AWS Bedrock on the `us-west-2` region. We’ve added the following models: * Llama 3.2 1B Instruct * Llama 3.2 3B Instruct * Llama 3.2 11B Instruct * Llama 3.2 90B Instruct * Llama 3.3 70B Instruct Support for Claude 3.7 Sonnet via Anthropic ------------------------------------------- **February 24th, 2025** We’ve added support for Anthropic’s new Claude 3.7 Sonnet model (latest and 02/19/2025 snapshot variants). ![Claude 3.7 Sonnet](https://www.anthropic.com/news/claude-3-7-sonnet) Workflow SDK Code Preview ------------------------- **February 18th, 2025** For those participating in the Workflow SDK beta, you can now preview the SDK-code representation of your Workflows directly in the UI. To do so, click on the preview button: ![Workflow SDK Code Preview Button](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/preview-sdk-code-button.png) Doing so will open a side panel with a full representation of your Workflow in code form. ![Workflow SDK Code Preview Side Panel](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/preview-sdk-code-side-panel.png) This feature is still in beta, so please let us know if you encounter any issues or have feedback! You can learn more about how to opt into the Workflows SDK beta [here](https://docs.vellum.ai/changelog/2025/2025-01#beta-release-of-sdk-enabled-workflows) . Restore Button Moved to History Cards ------------------------------------- **February 16th, 2025** The Restore button has been moved from the top-right of Sandboxes pages to the History Cards themselves. This makes it easier to find and use the Restore button and revert back to a prior state of your Prompt or Workflow Sandbox. Workflow Node Layout Improvements --------------------------------- **February 16th, 2025** There has been a minor improvement to the layout of Workflow Nodes. Now, input variables are always displayed above output types, making it easier to skim and understand a Node from top to bottom. Code Execution, Templating, and Final Output Nodes are all affected by this change. Model Picker Improvements ------------------------- **February 16th, 2025** The Model Picker has been updated to make it easier to find the model you’re looking for. Now, you can sort models alphabetically by their name, the date they were introduced, and when they were last used by someone in your Workspace. ![Model Picker Improvements](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/model-picker-improvements.png) Additionally, filters and sort preferences are saved across sessions, so you don’t have to reapply them every time. Multiple Chat History Variables in Prompts ------------------------------------------ **February 16th, 2025** Up until now, we’ve restricted you to only have one Chat History variable in a Prompt. It’s quite uncommon to need more than one, but we understand that there are some cases where it can be useful. We’ve now lifted this restriction and you can now have multiple Chat History variables in a Prompt. Application-Wide Performance Improvements ----------------------------------------- **February 10th, 2025** We recently overhauled core pieces of our webserver to improve the performance of the Vellum web application across the board. You should generally notice snappier page load times and more responsive interactions throughout the app. Support for xAI as a Model Host and Grok Models ----------------------------------------------- **February 7th, 2025** We’ve added support for xAI as a Model Host and along with it, the following [Grok models](https://docs.x.ai/docs/models) have been implemented: * Grok Beta * Grok Vision Beta * Grok 2 1212 * Grok 2 Vision 1212 Support for Multiple Gemini Models via Vertex AI ------------------------------------------------ **February 7th, 2025** We’ve added support for several new Gemini models via Vertex AI: * [Gemini 2.0 Flash 001](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash) * [Gemini 2.0 Flash Lite](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash-lite) * [Gemini 2.0 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-pro) * [Gemini 2.0 Flash Experimental](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash) * [Gemini 1.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-1.5-flash) Support for DeepSeek V3 via Fireworks AI ---------------------------------------- **February 7th, 2025** We’ve added support for the [DeepSeek V3](https://fireworks.ai/models/fireworks/deepseek-v3) model via Fireworks AI. Support for Gemini 2.0 Flash 001 via Gemini ------------------------------------------- **February 7th, 2025** We’ve added support for the [Gemini 2.0 Flash 001](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-2.0-flash) model via Gemini. Support for OpenAI’s o3-mini 2025-01-31 Snapshot via Azure ---------------------------------------------------------- **February 3rd, 2025** We’ve added support for OpenAI’s [o3-mini 2025-01-31 snapshot](https://platform.openai.com/docs/models/#o3-mini) to be used as a self-hosted model via Azure. --- # December 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Sidebar Navigation ------------------ **December 31st, 2025** We’ve added a new sidebar navigation that puts all your primary navigation in one place, including Home, Workflows, Prompts, Documents, and Metrics. Settings and other app navigation options have moved from the dropdown menu to the Resources section at the bottom of the sidebar. The sidebar can collapse to show just icons, giving you more screen space when you need it. ![Vellum sidebar navigation showing Home, Workflows, Prompts, Documents, Metrics, and Resources section with Settings](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/sidebar-navigation-expanded-2025-12-31-c0cd6a26.png) Sidebar navigation showing main navigation options and Resources section Workflow Sandbox Multimode Layout --------------------------------- **December 30th, 2025** You can now switch between three modes in the Workflow Sandbox using tabs at the top of the screen: Edit, Run, and Code. Edit mode gives you all the tools to build and modify your Workflow, including adding Nodes and configuring inputs. Run mode keeps the execution console open and hides the editing controls, making it easier to view and debug your runs. Code mode opens a full-screen editor where you can view your Workflow as code. ![Workflow Sandbox in Run mode showing execution console with Workflow results](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-sandbox-run-mode-2025-12-30-970aebd7.png) Run mode with execution console open for debugging ![Workflow Sandbox in Code mode showing full-screen editor with workflow.py file](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-sandbox-code-mode-2025-12-30-ecbd5242.png) Code mode with full-screen editor for viewing Workflow code Vellum will automatically switch between Edit and Run modes as you interact with your Workflow, showing you exactly what you need, when you need it, and no more. Simplified Workspace Invite Flow -------------------------------- **December 28th, 2025** You can now invite teammates directly to your Workspace by entering their email addresses. Previously, you had to add teammates to your Organization first, then add them to your Workspace one at a time. Now just go to your Workspace settings, click “Invite”, and add as many teammates as you like. If you’re on our Business or Enterprise plans, you can also set their roles during the invitation. Referral Program ---------------- **December 28th, 2025** You can now earn free credits by sharing Vellum with others. Go to your profile or workflow dropdown and click “Earn free credits” to get your personal invite link. You’ll earn 5 credits every time someone signs up using your link. ![Share Vellum modal displaying Earn 5 credits badge, invite link, and referral program details](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/referral-program-share-vellum-modal-2025-12-28-0841591d.png) Share Vellum modal showing referral program details Gemini 3 Flash on Google Vertex AI ---------------------------------- **December 19th, 2025** We’ve added support for Google’s Gemini 3 Flash model on Google Vertex AI. View Changes After Agent Builder Edits -------------------------------------- **December 23rd, 2025** When Agent Builder edits your Workflow, you’ll now see a “Compare” button. Click it to open a modal showing a code diff of exactly what changed. You can undo those changes if you’re not happy with them. ![Vellum interface showing Compare button after Agent Builder makes edits to a Workflow](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-compare-changes-button-2025-12-23-91e49ce4.png) Compare button showing Workflow changes made by Agent Builder Gemini 3 Flash Model -------------------- **December 18th, 2025** We’ve added support for Google’s [Gemini 3 Flash](https://ai.google.dev/gemini-api/docs/models#gemini-3-flash) model (`gemini-3-flash-preview`). Mistral AI Model Provider ------------------------- **December 16th, 2025** Mistral AI is now available for use via our [Model Providers](https://app.vellum.ai/settings/model-providers/MISTRAL_AI) . Models that are now available for use is: * **Mistral Large 3** * **Mistral Medium 3.1** Full Screen Workflow Code Editor -------------------------------- **December 15th, 2025** The Workflow Sandbox code editor now opens in full screen instead of a right-side fly-out. This gives you more room to view and edit your Workflow’s code. Workflow Outputs Panel ---------------------- **December 15th, 2025** You can now define Workflow Outputs directly without needing to add a Final Output Node. Workflow Outputs can reference Node Outputs and State Values, and they resolve once at the end of your Workflow. If you have existing Final Output Nodes in your graph, they’ll stay in sync with their corresponding Workflow Outputs. ![Workflow Outputs Panel displaying output configuration with references to Node Outputs](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-outputs-panel-2025-12-15-ef13f752.png) Workflow Outputs Panel showing outputs configuration Workflow Deployment Release Comparison -------------------------------------- **December 15th, 2025** You can now compare two Workflow Deployment Releases to see what changed between them. Navigate to your Workflow’s Deployment, go to the Releases tab, and click the “Compare” button to open a modal showing a code diff between the two Releases. Workflow Sandbox History Comparison ----------------------------------- **December 15th, 2025** You can now compare your Workflow Sandbox against previous versions. Open your Workflow’s history, navigate to any prior version, and click the “Compare” button to see a code diff in a modal. GPT-5.2 Model ------------- **December 11th, 2025** We’ve added support for GPT-5.2 from OpenAI. Voice Input for Agent Builder ----------------------------- **December 10th, 2025** You can now use your voice to interact with Agent Builder instead of typing. Click the microphone icon in the input field to speak your instructions, context, or questions. This makes it much easier to braindump and share thorough context quickly about your use-case, anticipated edge cases, and more. Agent Builder will carefully make sense of the raw, unstructured details that you provide while it plans and executes its next moves. ![Agent Builder input field with microphone icon for voice input](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/voice-input-agent-builder-2025-12-28-a64afeb6.png) Voice input microphone icon in Agent Builder Workflow Layout Dropdown ------------------------ **December 10th, 2025** We’ve added a Workflow Layout Dropdown that consolidates navigation and global options. Sandbox, Evaluations, and Deployments now live in the dropdown, along with Edit Sandbox Details, Clone, Archive Sandbox, and the Environment picker. This gives you more space on the canvas while keeping all controls within reach. ![Workflow Layout Dropdown menu showing Sandbox, Evaluations, Deployments, and global options](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-layout-dropdown-2025-12-10-8bd6eca2.png) Workflow Layout Dropdown showing consolidated navigation options CSV Files as Workflow Inputs ---------------------------- **December 10th, 2025** You can now include CSV files as Workflow Inputs. Previously, you had to include them as attachments within Chat History messages or fetch them from external datasources. When you ask Agent Builder to build a Workflow with CSV inputs, it creates the Nodes you need to process your CSV—row by row, in batches, or passing the entire file to LLMs with CSV processing capabilities. CSV files can also be Workflow outputs. ![Workflow showing CSV input being processed row by row with nodes for reading, transforming, and creating output CSV](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/csv-workflow-inputs-processing-2025-12-10-3746a2b0.png) Workflow processing CSV input with row-by-row transformation Chat History Operation in Set State Node ---------------------------------------- **December 8th, 2025** When configuring state operations for a `chat_history` state variable in the Set State Node, you can now use the `Set` or `Append` operation with a specified role (User or Assistant) to initialize or update chat history in your Workflows. ![Set Initial User Message configuration showing chat history append operation](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-12/set-state-chat-history-operation.png) Set Initial User Message with chat history operation Revamped Workflow Deployment Overview Page ------------------------------------------ **December 4th, 2025** We’ve redesigned the Workflow Deployment Overview page. The new layout shows your options for running published Workflows: run it in an AI App, create a custom UI with the Lovable Integration, integrate via our APIs using code snippets, or use Workflow Triggers for scheduled execution. ![Revamped Workflow Deployment Overview page showing AI App, Lovable Integration, and Code Snippets options](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-deployment-overview-revamp-2025-12-04-1764876123.png) Updated Workflow Deployment Overview page --- # Changelog | April, 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Dynamic Model Select on Inline Prompt Nodes ------------------------------------------- **April 29th, 2025** Inline Prompt Nodes now accept expression inputs to represent the Model invoked in the Prompt. Previously, you could only statically choose a model from a list of available models. Now, you can also dynamically reference a model from an upstream Node or Workflow Input. Ports on Workflow Nodes ----------------------- **April 28th, 2025** All Workflow Nodes now support Ports. Previously you had to define a Conditional Node for control flow in a workflow. Now you can define Ports using Expression Inputs to define the control flow you want. Non-Streaming Prompt Nodes -------------------------- **April 28th, 2025** Prompt Nodes now support the option to run as streaming or non-streaming. The latter is considered more performant during parallelization or Workflows that feature Map Nodes. ![Non-Streaming Prompt Nodes](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-04/non-streaming-prompt-nodes.png) Test Suite Run Progress ----------------------- **April 25th, 2025** It’s now possible to see the progress of an evaluation report run while it’s actively running. Hovering over the progress indicator, a tooltip will display the number of test cases being run and the number of test cases completed. ![Test Suite Run Progress](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-04/test-suite-run-progress.png) Run From Node Re-enabled for SDK-Enabled Workflows -------------------------------------------------- **April 22nd, 2025** Back in March, we needed to temporarily disable our Run From Node feature while we reimagined how it should be designed in this new paradigm. Today, we are excited to announce that we are bringing _back_ this beloved feature for all SDK-Enabled Workflows. Just as before, to invoke the a Workflow starting from a given Node, click the mini play button that appears on the Node when a previous execution is loaded. This will use the “State” we saved from the previous execution in order to run the Workflow from that exact point in time going forward. ![SDK Run from Node](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-04/sdk-run-from-node.png) o4 Mini and o3 Models on Vellum ------------------------------- **April 16th, 2025** We’ve added [OpenAI’s newest Models](https://openai.com/index/introducing-o3-and-o4-mini/) to Vellum: * o3 * o4 Mini Metric Mapper Revamp -------------------- **April 15th, 2025** We’ve introduced a revamped UI that makes it easier to configure Metrics within the context of Evaluations. A Metric’s inputs are now automatically mapped to existing variables based on their name and type. We also now clearly indicate which inputs are required and simplify the process of using constant values. All in all, we hope that with these changes, it requires fewer clicks to set up a Metric and understand where it’s getting its inputs from! ![Revamped Metric Mapper](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/metric-mapper.png) Llama 4 Maverick via Groq on Vellum ----------------------------------- **April 14th, 2025** We’ve added support for Llama 4 Maverick 17B Instruct via Groq to Vellum GPT 4.1 Models on Vellum ------------------------ **April 14th, 2025** We’ve added the new GPT 4.1 Models via OpenAI to Vellum: * GPT-4.1 * GPT-4.1 (2025-04-14) Snapshot * GPT-4.1 Mini * GPT-4.1 Mini (2025-04-14) Snapshot * GPT-4.1 Nano * GPT-4.1 Nano (2025-04-14) Snapshot xAI Models on Vellum -------------------- **April 14th, 2025** We’ve added the new xAI models to Vellum: * Grok 3 Beta * Grok 3 Fast Beta * Grok 3 Mini Beta * Grok 3 Mini Fast Beta Test Case Error Filtering for Evaluation Reports ------------------------------------------------ **April 10th, 2025** We’ve introduced a new feature that allows you to filter and select failed test cases in evaluation reports. This enhancement enables you to re-run only the errored test cases, which is particularly beneficial when dealing with provider rate limits on extensive evaluation reports. ![Evaluation Report Error Filtering](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/error-filter.png) Sandbox Cost Tracking for Workflows ----------------------------------- **April 8th, 2025** We’ve now added cost tracking support for your Subworkflow and Deployment Subworkflow Nodes from inside our Workflow Sandbox. This is an aggregate sum of total costs for your workflow invoked by your Subworkflow Nodes, available immediately after running your workflow! ![Sandbox Workflow Cost](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/sandbox-workflow-costs.png) Llama 4 Models on Vellum ------------------------ **April 5th, 2025** We’ve added the new Llama 4 models to Vellum. These models are the latest and greatest from Meta. The new models are available from the following providers: * Fireworks AI * Llama 4 Scout * Llama 4 Maverick * Groq * Llama 4 Scout Add Node on Edge Drop --------------------- **April 3rd, 2025** Previously, the only way to add a Node was to drag and drop a new Node from the Nodes Panel. We’ve now introduced a faster way to add and connect Nodes: simply drag an edge from an existing Node and drop it where you want the new Node to be. A Node selection menu will appear at that location, allowing you to choose the type of Node you want to add. The new Node will be automatically placed and connected to the edge. Protected Release Tags ---------------------- **April 1st, 2025** You can now specify Release Tags that should be considered “protected.” This feature is meant to be used in conjunction with our recently released [Deployment Release Reviews](https://docs.vellum.ai/changelog/2025/2025-03#deployment-release-reviews) feature. ![Protected Release Tags Setting](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-04/protected-release-tags-setting.png) A protected Release Tag cannot be assigned to a Prompt or Workflow Deployment Release unless that Release has at least one approval from a Reviewer and no outstanding change requests. This feature is useful for ensuring that critical or sensitive Deployments are reviewed and approved before going live. Grouping a Selection into a New Subworkflow Node ------------------------------------------------ **April 1st, 2025** Previously, creating Subworkflows and organizing Nodes within them required manual copy-pasting of Nodes. We’ve now introduced a powerful new way to organize your Workflows by grouping Nodes into Subworkflows. You can now select multiple Nodes and automatically convert them into a new Subworkflow Node, which will maintain all the existing connections. Additionally, you can drag and drop selected Nodes onto existing Subworkflow Nodes to move them inside. This feature helps you better organize complex Workflows into logical components while reducing visual clutter in your Workflow canvas. When grouping Nodes, all existing connections and logic are automatically maintained, and you can easily modify Subworkflow contents through simple drag and drop interactions. --- # Changelog | December, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Reject on Error Toggle for Guardrail Nodes ------------------------------------------ **December 20th, 2024** You can now catch and handle errors in Guardrail Nodes just as you can most other Node types using the new “Reject on Error” toggle. Optional Input Variables for Prompts & Workflows ------------------------------------------------ **December 19th, 2024** You can now specify whether an input variable to a Prompt or Workflow is required or optional. This determines whether API calls made to invoke the Prompt or Workflow must include a value for that input variable. If the input variable is optional, the API call can omit its value and still be successful. If required, the API call must include a value for that input variable or else it will fail with a 400 status code response. Additionally, you may set explicit default values for optional input variables. If an API call omits a value for an optional input variable, the default value will be used instead. Newly created input variables going forward are optional by default with a `null` default value. This protects against breaking changes for existing API calls that do not include values for newly added variables. You can see a demo of it in action here: Support for JSON Files in Document Indexes ------------------------------------------ **December 18th, 2024** We now support uploading `.json` files to Document Indexes for indexing and searching. Map End-User Feedback to Ground Truth Variables in Evals -------------------------------------------------------- **December 18th, 2024** Capturing end-user feedback on a deployed AI system is a powerful way of understanding whether your system is performing as expected. However, even more valuable can be using this collected feedback in conjunction with evals. We’ve added the ability to map end-user feedback to ground truth variables in evals. This allows you to build up your eval datasets and the ground-truth data within, with real-world feedback from your end-users. Check out a demo of it in action below: Newly Added Models for Azure ---------------------------- * **December 14th, 2024**: We’ve added support for the newest [GPT-4o/GPT-4o Mini model](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure%2Cglobal-standard%2Cstandard-chat-completions#gpt-4o-and-gpt-4-turbo) on Azure: * [GPT-4o 08/06](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4o-0806) * [GPT-4o 11/20](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4o-1120) * [GPT-4o Mini 07/18](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4o-mini-0718) Link Search Results to Document and Expose Document Metadata ------------------------------------------------------------ **December 14th, 2024** When using the Search functionality within a Document Index, you can now see a link to the document that the search result is from. In addition to the link, you can also see the Document’s metadata in the search results. ![Search Results with Document Link and Metadata](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-12/document_link_and_metadata.png) Newly Added Model for Google ---------------------------- * **December 11th, 2024**: Added support for [Gemini 2.0 Flex](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-2.0-flash) via Google’s Vertex AI. Newly Added Support for Together AI Models ------------------------------------------ **December 11th, 2024** We’ve added support for Together AI’s [Together AI](https://docs.together.ai/docs/introduction) models. Along with the launch of the Together AI integration, we’ve added support for [Llama 3.3 70B Instruct Turbo](https://docs.together.ai/docs/serverless-models) . Newly Added Model for Groq -------------------------- * **December 8th, 2024**: We’ve added support for [Llama 3.3 70B Versatile](https://console.groq.com/docs/models#production-models) via Groq’s production models. Newly Added Model for Fireworks AI ---------------------------------- * **December 8th, 2024**: We’ve added support for [Llama 3.3 70B Instruct](https://fireworks.ai/models/fireworks/llama-v3p3-70b-instruct) via Fireworks AI. Newly Added Models for AWS Bedrock ---------------------------------- * **December 5th, 2024**: With the launch of [AWS Bedrock’s Nova Models](https://docs.aws.amazon.com/nova/latest/userguide/what-is-nova.html) , we’ve added support for Nova Micro v1, Nova Lite v1, and Nova Pro v1 in the `us-east-1` region. Newly Added Support for SambaNova AI Models ------------------------------------------- **December 5th, 2024** We’ve added support for [SambaNova AI](https://docs.sambanova.ai/home/latest/index.html) as one of our newest model hosts! Along with the launch of the SambaNova integration, we’ve added the following model: * [Llama 3.1 405B Instruct](https://community.sambanova.ai/docs?topic=193#p-256-llama-31-family-3) For more information, check out the [SambaNova API reference](https://docs.sambanova.ai/sambastudio/latest/api-reference.html) . Performance Improvements to the Workflow Editor ----------------------------------------------- **December 4th, 2024** The Workflow Editor UI could feel sluggish when working on sufficiently large Workflows consisting of many Nodes. We’ve made a number of broad sweeping optimizations so that editing Workflows in the UI should generally feel snappier and more responsive. Newly Added Models for OpenRouter --------------------------------- * **December 17th, 2024**: We’ve added support for [Llama 3 Lumimaid 8B](https://openrouter.ai/neversleep/llama-3-lumimaid-8b) via OpenRouter. * **December 17th, 2024**: We’ve added support for [Llama 3.3 70B Instruct](https://openrouter.ai/meta-llama/llama-3.3-70b-instruct) via OpenRouter. * **December 17th, 2024**: We’ve added support for [Eva Llama 3.33 70b](https://openrouter.ai/eva-unit-01/eva-llama-3.33-70b) via OpenRouter. * **December 17th, 2024**: We’ve added support for [Mistral Large 2411](https://openrouter.ai/mistralai/mistral-large-2411) via OpenRouter. * **December 5th, 2024**: We’ve added support for [Eva Qwen 2.5 72B](https://openrouter.ai/eva-unit-01/eva-qwen-2.5-72b) via OpenRouter. * **December 2nd, 2024**: We’ve added support for [Cohere’s Command R+ 08-2024](https://openrouter.ai/cohere/command-r-plus-08-2024) model through OpenRouter. --- # Changelog | May, 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Vertex AI Claude 4 Models via Vellum ------------------------------------ **May 27th, 2025** We now support both of Anthropic’s [newest Claude 4 Models](https://www.anthropic.com/news/claude-4) hosted on Vertex AI. AWS Bedrock Claude 4 Models via Vellum -------------------------------------- **May 27th, 2025** We now support both of Anthropic’s [newest Claude 4 Models](https://www.anthropic.com/news/claude-4) hosted on AWS Bedrock. Global Search Navigation ------------------------ **May 27th, 2025** You can now navigate Vellum using global search and keyboard shortcuts. Open global search by clicking the search bar or using cmd-K, then press `/` to display available navigation options and quickly jump to different sections of the platform without using your mouse. ![Global Search\ Modal](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ad91c2e0-27ad-4532-bf13-4cdac1af98ef-global-search-modal.png) Global search modal showing navigation options ![Global Search\ Navigation](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ad91c2e0-27ad-4532-bf13-4cdac1af98ef-global-search-navigation.png) Global search with navigation menu The global search supports: * **Search functionality**: Find any type of entity — Prompt, Workflow, Test Suite, or Document Index * **Quick navigation**: Jump to specific pages by typing ”/” followed by the page name * **Keyboard shortcuts**: Navigate entirely with your keyboard for improved efficiency Model Icons ----------- **May 22nd, 2025** We now show the logo of the LLM Provider alongside models throughout Vellum, making it easier to distinguish one model from another. ![Model Icons\ Feature](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/0580c395-e2c5-4075-8198-fb57d8b5454a-model-icons-feature.png) Model selection interface showing provider logos Anthropic’s Newest Claude 4 Models via Vellum --------------------------------------------- **May 22nd, 2025** We now support both of Anthropic’s [newest Claude 4 Models](https://www.anthropic.com/news/claude-4) released on May 22nd, 2025. * Claude Opus 4 * Claude Sonnet 4 Simplified Side Navigation -------------------------- **May 20th, 2025** We’ve simplified Vellum’s side navigation to remove a level of nesting and bring what matters most, front and center. Now, when working on a Prompt or Workflow, you can navigate between its Sandbox, Evaluation, and Deployments, all from within the main page, rather than rely on nested navigation within the side nav. ![Simplified Side\ Navigation](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/665f3311-cbaa-4019-9b26-dcb7dea07ca6-simplified-side-nav.png) Simplified navigation with Sandbox, Evaluations, and Deployments tabs With this update, we’ve also made broader sweeping changes to page layouts overall, making all pages consistent with their breadcrumbs, menus, and action buttons for a more intuitive user experience. AI-Powered Prompt Improver (Beta) --------------------------------- **May 19th, 2025** You can now use AI to generate improved prompts from within Prompt Sandboxes. This feature uses Anthropic under the hood and generates new prompts that follow prompting best practices. The Prompt Improver works particularly well for Anthropic models, but should apply to models from other providers, too. To use this feature: 1. Click the “Use AI to improve this prompt” button in your Prompt Sandbox 2. Wait for the improved prompt to be generated (up to 5 minutes) 3. Review the diff showing changes between your original prompt and the improved version 4. Click “Apply” to replace your existing prompt with the improved version ![Prompt Improver\ Button](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/43344a25-5761-4a54-8b2b-f9ec774fd76d-prompt-improver-button.png) The button to trigger the prompt improvement process ![Prompt Improver Diff\ View](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/43344a25-5761-4a54-8b2b-f9ec774fd76d-prompt-improver-diff.png) Diff view showing the improved prompt This feature is in beta and we welcome feedback. Note that this feature must be enabled by an organization admin and you must be comfortable with using Anthropic as a data subprocessor. Expression Inputs on Final Output Nodes --------------------------------------- **May 13th, 2025** Previously, Final Output Nodes only allowed you to reference values like Node Outputs, Inputs, etc. Now, you can do more with these values and write an Expression to define the Final Output you want. For example, you can use the accessor operator to reference an attribute in a json output from a Prompt Node. Smart Labels for Node Input Variables ------------------------------------- **May 9th, 2025** Previously, Node input labels remained static regardless of their content. Now, input labels automatically update to reflect the value or expression they contain, making your Workflows more intuitive and self-documenting. You can set an explicit value at any time, at which point it will no longer auto-update. Configurable Data Retention Policies ------------------------------------ **May 7th, 2025** Enterprise customers can now configure data retention policies for their organization. This new feature allows you to: * Set whether monitoring data is retained indefinitely (default) or for a specific time period * Choose from predefined retention periods (30, 60, 90, or 365 days) Data retention settings can be configured from the Organization Settings page under Advanced Settings. ![Data Retention Settings in Advanced\ Settings](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/44323b60-e474-4bc1-a2e5-abe070181f58-data-retention-settings.png) Data Retention Settings in Advanced Settings ![Data Retention\ Options](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/44323b60-e474-4bc1-a2e5-abe070181f58-data-retention-options.png) Data Retention Options DeepSeek v3 to Azure AI Foundry ------------------------------- **May 2nd, 2025** We now support the self-hosted DeepSeek v3 via Azure AI Foundry to Vellum. GPT 4.1 and 4.5 Model via Azure OpenAI -------------------------------------- **May 2nd, 2025** We now support the self-hosted GPT 4.1 and GPT 4.5 models via Azure OpenAI to Vellum. OpenAI Base64 PDF Files ----------------------- **May 2nd, 2025** We now support OpenAI Models newly added capability to use Base64 documents within API requests. Structured Outputs and Json Mode Support for x AI Models -------------------------------------------------------- **May 2nd, 2025** We have added the ability to have Grok models that are hosted on xAI to return responses in JSON format via an agnostic JSON Mode, or a formatted Structured Output. Microsoft OmniParser V2 Azure AI Foundry ---------------------------------------- **May 2nd, 2025** We’ve added support for Microsofts OmniParser V2 via Azure AI Foundry to Vellum Gemini Vertex AI Models Are Now Region Specific ----------------------------------------------- **May 2nd, 2025** Previously, Gemini Vertex AI Models would be added to your account regionally agnostic. In the configuration of the model you would have to specify which region you would like to utilize the model in, therefore limiting you to one instance of the model. We have pivoted to creating region specific instances for you to select. This allows you the option of enabling your model in different regions so you can set your workflows up for success with fallbacks to different regions. --- # October 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Settings Revamp --------------- **October 31st, 2025** We’ve reorganized the navigation to keep the focus on building. **Models** and **Integrations** now live under **Settings**, and your personal **Profile** and **Settings** have moved to the avatar menu in the top right. This update makes the top-level navigation cleaner and gives all configuration options a more unified home. ![Settings page displaying Models section with various model providers including AWS Bedrock, Anthropic, Azure AI Foundry, Azure OpenAI, and others](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/settings-revamp-models-page-1761921795-1761922161.png) Settings page showing Model providers Release Page Artifacts ---------------------- **October 28th, 2025** You can now toggle between the graph view and code artifact for each Release in your Workflow Deployment. The code artifact shown is what actually runs on Vellum’s servers and what gets pulled when you use the CLI. ![Workflow Preview showing the toggle between graph view and deployed code artifact, with workflow.py file displayed in the code view](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/release-page-artifacts-toggle-1761611099.png) Toggle between graph view and code artifact on the Release page Simplified Deployment URLs -------------------------- **October 24th, 2025** We’ve simplified the URL structure for Deployment pages. Previously, URLs required both the Sandbox ID and Deployment ID. Now, they only require the Deployment ID. For example: * Before: `/workflow-sandboxes//deployments/` * After: `/workflow-deployments/` This makes it easier to construct URLs in your external systems and link back to Vellum. You can also use a Deployment’s name in place of its ID for improved readability. Old URLs will continue to work and automatically redirect to the new format. New Native Integrations ----------------------- **October 24th, 2025** We’ve added support for six new native integrations: * [Asana](https://asana.com/) * [Box](https://www.box.com/) * [ClickUp](https://clickup.com/) * [Discord](https://discord.com/) * [Eventbrite](https://www.eventbrite.com/) * [Intercom](https://www.intercom.com/) You can now connect these integrations directly to your Workflows either in Agent Builder or in Agent Nodes or Custom Nodes. Agent Builder Updates --------------------- **October 20th, 2025** We’ve shipped several more improvements to Agent Builder: * **Selectable suggestions in chat.** You can now choose from multiple suggestions when Agent Builder offers options. ![Agent Builder conversational interface showing CRM selection between HubSpot and Salesforce for building a renewal risk monitoring agent](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-crm-selection-interface-1761003529-1761003685.png) Agent Builder helping build a Workflow with CRM integration selection * **Integration Context Awareness.** Previously, if you connected integrations like Notion, Slack, etc., Agent Builder could not access the context of the integrations. Now, it can invoke tools directly to create better Scenarios and Workflows. For example, it can look up a Slack Channel ID without you needing to manually find it in Slack. * **Improved Runtime Error Debugging.** Agent Builder is now much better at reading the results of the last Workflow run. This means Agent Builder is now much better at fixing certain runtime errors. * **Better relative date handling.** Agent Builder now uses more dynamic logic when building Agents that use relative dates, like “look at all tickets closed in Jira or Linear in the last two weeks.” Previously it had a tendency to hardcode dates, but now they will be relative to the moment the Agent / Workflow is executed. * **Native integration reliability improvements.** We’ve fixed several bugs and improved reliability when adding native integrations to Agent Nodes. Claude Haiku 4.5 Model ---------------------- **October 15th, 2025** We’ve added support for Anthropic’s latest model [Claude Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5) . Agent Builder Updates --------------------- **October 14th, 2025** We’ve shipped several improvements to Agent Builder: * **Threads are now generally available.** You can now leave and return to Agent Builder to continue a conversation without losing your progress. You can also create multiple conversation threads to organize different development sessions and ideas as you develop your Workflows. * **Ports support.** Agent Builder now supports Ports in Workflows. You can ask Agent Builder to create Workflows that use Ports to structure data flow between Nodes. * **Model selection and utilization.** We’ve made several improvements to model selection and utilization. Agent Builder will no longer select invalid models and has better awareness of features & configureation options available on different popular models such as Claude, Gemini, and OpenAI. * **Deployment Node support.** Agent Builder now supports Prompt Deployment Nodes and Workflow Deployment Nodes. You can use version-controlled, deployed Prompts and Workflows directly in the Workflows that Agent Builder helps you build. * **Documents support.** Improved how Agent Builder handles native document inputs to reduce errors and streamline setup. * **Document Index support.** Agent Builder can now create Document Indexes and view files within them. Agent Builder can set up and configure your document storage directly while building your Workflow. * **UX improvements.** The “Proceed with plan” flow is now smoother, setting up multiple integrations at once is clearer, and you’ll now see a button to open the inputs panel directly from Agent Builder chat when needed. * **Reliability & performance improvements.** Map Nodes now handle inputs from Prompt Nodes better and have improved output handling. Behind the scenes, we’ve made optimizations to improve latency and reduce crashes, especially when working with Workflows that contain Map Nodes. GPT-5 Pro Models ---------------- **October 6th, 2025** We’ve added support for OpenAI’s latest [GPT-5 Pro models](https://platform.openai.com/docs/models/gpt-5-pro) : * gpt-5-pro * gpt-5-pro-2025-10-06 Agent Builder - Generate Agent Nodes with Integration Tools ----------------------------------------------------------- **October 3rd, 2025** Agent Builder now automatically creates Agent Nodes with the right integration tools when you ask it to use integrations in your Workflows. It configures all the appropriate tools and API calls for you. If a tool is not supported as a native integration in Vellum, Agent Builder can create custom tools using Python, Agent handoffs, Inline Subworkflows, version-controlled Deployed Subworkflows, and more. ![Agent Node overview showing configured tools including Create linear issue, List linear projects, Fetch Notion Block Children, Fetch Notion Data, and Search Notion page](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-agent-node-tools-1759768633.png) Agent Node configured with integration tools for Notion and Linear ![Workflow execution view displaying Agent Node with tool call history and final output showing integration functionality](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-workflow-execution-tools-1759768633.png) Workflow execution showing tool call history for streaming progress and debugging Agent Builder - Configure Integrations -------------------------------------- **October 3rd, 2025** You can now configure third-party integrations directly from Agent Builder. Previously, you had to set them up manually in the Agent Node or Custom Node or manually configure API keys. Now, Agent Builder prompts you to connect integrations directly when it needs them for your Workflow. ![Agent Builder interface showing Connect Notion and Connect Linear buttons with example workflow plan](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-connect-integrations-1759768632.png) Agent Builder prompts you to connect required integrations directly from the interface Native Integrations ------------------- **October 2nd, 2025** You can now authenticate with third-party integrations directly in Vellum. Previously, you had to set them up through your own Composio account. You can manage integrations from the Integrations UI or directly in an Agent Node within a Workflow. Once configured, use them in Agent Nodes or Custom Nodes to invoke third-party tools. We currently support: Airtable, Calendly, Firecrawl, Gamma, Github, Hubspot, Linear, LinkedIn, Mailchimp, Notion, Perplexity, Reddit, Serp Api, Slack, Webflow, and Zendesk, with more coming soon. ![Integrations UI displaying available integrations including Airtable, Calendly, Firecrawl, Gamma, Github, Hubspot, Linear, LinkedIn, Mailchimp, Notion, Perplexity, Reddit, Serp Api, Slack, Webflow, and Zendesk](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/native-integrations-index-page-1761244109.png) Integrations UI showing 16 available integrations ![Slack integration detail page showing available tools including Set snooze duration, Add a custom emoji to a Slack team, Add an emoji alias, and Add a remote file](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/native-integrations-slack-tools-1761244109.png) Slack integration showing available tools for use in Workflows ![Integration selection modal showing available platforms with a Slack tool selected](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/select-integration-tools-dialog-1761246572.png) Available integration tools displayed in the integration selection modal with a Slack tool selected ![Workflow Sandbox showing Agent Node with configured Slack integration tool for setting snooze duration](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/native-integrations-agent-node-workflow-1761244109.png) Agent Node in Workflow Sandbox with native integration tools configured ![Custom Node SDK Preview showing Python code using client.integrations.execute_integration_tool with COMPOSIO integration provider for Slack](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/native-integrations-custom-node-code-1761244109.png) Custom Node using native integrations in code with SDK Preview Structured Outputs for Groq Models ---------------------------------- **October 2nd, 2025** We now support structured outputs for Groq models, making it easier to reliably parse responses that conform to your expected schemas. --- # January 2026 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Agent Builder Reconnection -------------------------- **January 22nd, 2026** Agent Builder will now automatically reconnect to your running Agent Builder execution if you navigate to a different page and then come back or if you experience a network interruption. Workflow Log Events ------------------- **January 20th, 2026** Workflows can now [emit log events](https://docs.vellum.ai/developers/workflows-sdk/core-concepts#emitting-log-events) . You can view these logs in the Workflow console or on the Execution Details page. ![Workflow console showing log events with INFO and ERROR levels](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-console-log-events-2026-01-20-221ff2d6.png) Log events in the Workflow console ![Execution Details page showing log events for a Node](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/execution-details-log-events-2026-01-20-59b1afe2.png) Log events on the Execution Details page Agent Code Downloads -------------------- **January 17th, 2026** You can now download your Agent’s code as a zip file directly from the UI. Click the “Download” button in the Agent Code panel. ![Agent Code panel showing Download button](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-code-download-button-2026-01-17-8e620acc.png) Download button in the Agent Code panel Chat Message Triggers --------------------- **January 14th, 2026** Users can now build Agents with first-class chat experiences by using the new Chat Message Trigger. When Chat Message Triggers are added to your agent, it automatically creates a State variable for `chat_history`, storing threads of the conversation on Vellum instead of having to do so yourself: ![Chat Trigger in Sandbox](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2026-01/chat-trigger.png) Chat Trigger in Sandbox When you invoke the Sandbox with a Chat Message Trigger, Vellum will pull up an interactive chat panel to interact with your Agent through a Chat experience: ![Chat Panel in Sandbox](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2026-01/chat-panel.png) Chat Panel in Sandbox Agents deployed with a Chat Message Trigger are available via the API. You can read the docs on how to do so [here](https://docs.vellum.ai/developers/client-sdk/workflows/deployments/execute-stream) . Custom Nodes over Code Execution Nodes -------------------------------------- **January 10th, 2026** Vellum will bias towards generated Custom Nodes over Code Execution Nodes. Custom Nodes have the benefit of being more performant and flexible. Over time, it’s likely that they’ll replace Code Execution Nodes entirely. Merge, Conditional, and Output Node Deprecations ------------------------------------------------ **January 6th, 2026** We’ve deprecated Merge, Conditional, and Output Nodes. You can no longer create new instances of these Nodes from the UI. They’ve been replaced by more flexible first-class concepts: * **Merge Node** → **Merge Strategy**: A setting available on all Node types. You no longer need a standalone Node to await multiple parallel branches. * **Conditional Node** → **Ports**: A setting found on all Nodes under the “Routing” tab. You can perform conditional logic to route down one path or another on any Node, instead of only on Conditional Nodes. * **Output Node** → **Workflow Outputs**: You no longer need a standalone Node to signify what your Workflow outputs. Instead, reference the Nodes whose outputs you care about directly as Workflow Outputs. Existing Workflows that contain these Node types will continue to work with no changes—you simply cannot create new instances going forward. Mocking Nodes & Integrations ---------------------------- **January 6th, 2026** Agent Builder can now create mock data for you when building Workflows. When it detects that your Workflow needs integrations like HubSpot, Slack, or Google Sheets, it’ll offer to build with mock data so you can test immediately without connecting the real integrations. This speeds up your testing cycle by making Prompts and Nodes run instantly, and prevents accidental writes to external APIs. ![Agent Builder showing Integration Requirements with options to build with mocks or connect integrations first](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-mock-integrations-2026-01-06-302a9b03.png) Agent Builder offering to build with mock data for integrations You can also save any previous run’s outputs as mock data with a single click. This makes it easy to capture real data and reuse it for faster testing. ![Workflow Sandbox showing tooltip to use span outputs as mock for this Node](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/save-run-outputs-as-mock-2026-01-06-e1d9760f.png) Save run outputs as mock data Workflow Favorites ------------------ **January 3rd, 2026** You can now mark Workflows as favorites by clicking the star icon. Favorited Workflows appear in your sidebar and home page, making them easier to find. Favorites are personal to you, so each user in a Workspace can have their own. ![Workflow header showing yellow star icon next to workflow name for favoriting](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-favorites-star-icon-2026-01-03-c8909c1e.png) Star icon for marking a Workflow as favorite ![Sidebar showing Favorites section with favorited Workflows listed](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-favorites-sidebar-list-2026-01-03-fd717c91.png) Favorited Workflows appear in the sidebar --- # August 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Workflows Console ----------------- **August 29th, 2025** You can now view detailed Workflow execution logs and results in the new console at the bottom of the Workflow Sandbox. The console provides a timeline view of all Workflow runs in chronological order, showing logs for each node including nested nodes in Subworkflows. ![Workflow Sandbox Console](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/workflow-sandbox-console.png) The timeline view lets you scrub through the execution chronologically once the Workflow is completed, while the right panel shows detailed results for each Node in the log. This makes it much easier to debug and monitor complex Workflow executions, providing an intuitive top-down hierarchy that showcases exactly what happened during each run, regardless of Workflow complexity. Workflow Canvas Collapse Toggle ------------------------------- **August 29th, 2025** You can now toggle the collapsed view for Workflow canvas nodes directly from the canvas interface. Previously, this toggle was buried in the Workflow Builder Settings where it was hard to find and rarely accessed. ![Collapse Node Toggle](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/collapse-node-toggle.png) The new toggle is prominently displayed in the canvas header, making it much more discoverable for users who want to quickly switch between expanded and collapsed node views. Additionally, the toggle state now persists to the URL, so when you share a Workflow with others, they’ll see the nodes in the same collapsed or expanded state that you configured. Workflow Builder Context Menu ----------------------------- **August 29th, 2025** You can now right-click in empty space within a Workflow to open a new context menu to get access to helpful actions. * **Create Node**: Quickly create a new Node at the location where you right-clicked * **Fit View**: Automatically adjust the viewport to fit your entire Workflow Node Side Panel Improvements ---------------------------- **August 27th, 2025** We’ve reorganized the Node Side Panel that opens when you click on a Node in a Workflow to make it more intuitive and easier to navigate: * **Renamed “Ports” tab to “Routing”**: The tab is functionally identical, but this rename makes it clearer how to set up conditional routing from one node to another. * **Streamlined Error Handling**: The “Adornments” tab has been removed and its contents moved to a new “Error Handling” section within the “Settings” tab. This change better communicates the intended purpose of these features. * **Simplified Node Panel**: Removed the ability to drag-and-drop Adornments from the Nodes panel onto individual nodes to reduce UI complexity. These changes maintain all existing functionality while making the interface more discoverable for both new and experienced users. MCP in Agent node ----------------- **August 21st, 2025** We now support MCP as a tool in Agent node. Select MCP server from `+ Tool` button within Agent node and connect to your remote MCP server. ![Add MCP Server](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/add-mcp-server.png) ![Configure MCP Server](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/configure-mcp-server.png) Agent node automatically handles tool discovery in the background. Once setup is complete, you’ll see all available tools displayed and ready to use. ![MCP Server Tools](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/mcp-tools.png) Workflow Sharing ---------------- **August 21st, 2025** We have added the ability to share your Workflows with others. To share your Workflow, click the `Share` button on the top right of the workflow. You can toggle if you want the workflow to be publicly available for anyone with a link or you can share internally by keeping the `Public Access` toggle off. ![Edit Code Preview Files Toggle](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/workflow-sharing.png) Editable Code Preview Files --------------------------- **August 21st, 2025** We now have the ability to edit files from the Code Preview. Before, you had to run `vellum workflows pull` to edit your code and then do a `vellum workflows push` to update the code of your workflow. Now you can click the toggle to enable Edit mode and modify the files straight from the Code Preview UI. ![Edit Code Preview Files Toggle](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/Editable%20Code%20Preview%20Files%20Toggle.png) API Keys Removed from Side Nav ------------------------------ **August 19th, 2025** We used to have an item in our side navigation called “API Keys.” Over time, this had become a dumping ground of miscellaneous api-related settings. We’ve since found new, more reasonable, homes for everything that used to be on that page: * LLM provider credentials -> [Model providers page](https://app.vellum.ai/model-providers) * Vellum API keys -> [Environments & API Keys tab in Workspace Settings](https://app.vellum.ai/organization?tab=workspaces&workspace-settings-tab=environments) * Secrets -> [Environments & API Keys tab in Workspace Settings](https://app.vellum.ai/organization?tab=workspaces&workspace-settings-tab=secrets) * HMAC token -> [Organization Settings](https://app.vellum.ai/organization?tab=settings) Improved Variable Detection When Pasting Prompts ------------------------------------------------ **August 15th, 2025** We’ve improved variable detection when pasting prompts into Rich Text inputs. Previously, when you pasted a prompt containing `{{ variable }}` placeholders that weren’t already defined in Vellum, these placeholders would be inserted as plain text rather than being recognized as variables. Now, undefined variables are automatically created, and `{{ variable }}` placeholders will render as variable chips as expected. This should make it easy to copy/paste existing prompts from your codebase or other tools into Vellum. Models Page Revamp ------------------ **August 12th, 2025** We’ve launched a major overhaul of our Models Page to make it easier to discover and manage the AI models available in your Workspace. Previously, we had one large grid that listed all models in Vellum, which could be overwhelming and made it difficult to find specific models of interest. But no more! ![Screenshot of the new Models Page showing a grid of model provider cards including Anthropic, AWS Bedrock, Azure AI Foundry, Azure OpenAI, BaseTen, and others with model counts and newest model information](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/88ea4c28-c67f-4ff9-994f-85153f654adf-models-page-provider-overview.png) New Models Page showing all model providers that Vellum integrates with **Key improvements include:** * **Provider-Based Navigation**: When you click on “Models” in the side nav, you’ll now see all model providers that Vellum integrates with, organized as easy-to-browse cards * **Provider-Level Configuration**: Each provider page now includes provider-level settings, most commonly where you’ll enter your LLM Provider API Keys * **Enhanced Model Discovery**: Click into any provider to see all models from that specific provider with improved search, sort, and filter capabilities ![Screenshot of the OpenAI provider page showing a list of GPT models with search functionality, status indicators, and provider configuration options including API key setup](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/88ea4c28-c67f-4ff9-994f-85153f654adf-models-page-provider-detail.png) Individual provider page showing OpenAI models with search, filtering, and provider configuration * **Improved Organization**: Models are now organized by provider, making it much easier to find models from specific providers like OpenAI, Anthropic, Google, and others * **Better Search & Filtering**: Enhanced search and filtering capabilities within each provider’s model list * **Cleaner Interface**: A more focused, less cluttered experience that scales better as we continue adding new model providers and models This update makes it significantly easier to navigate Vellum’s extensive model catalog, which now includes 23 different model providers and hundreds of individual models. Whether you’re looking for the latest GPT models from OpenAI or exploring options from Anthropic, Google, or other providers, we hope this update makes it easier to find them! Updated Node Handles UX ----------------------- **August 11th, 2025** We’ve updated the look and feel of Node Handles in the Workflow Builder to provide better visual feedback and improved usability: ![Screenshot showing the updated Node Handles with different visual states for nodes with 0, 1, and 2+ incoming edges, plus new plus icons on source handles](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/cb788aae-29a7-483f-894a-983d144f1339-node-handles-ux-update.png) Updated Node Handles showing different visual states for connection points **Key improvements include:** * **Visual States for Incoming Edges**: Node handles now display different visual indicators based on the number of incoming connections: * Nodes with 0 incoming edges * Nodes with 1 incoming edge * Nodes with 2+ incoming edges * **New Plus Icon**: Source handles now feature a plus icon for clearer identification * **Improved Edge Dropping**: Enhanced interactions when connecting and dropping edges between nodes These updates make it easier to understand your Workflow’s connection structure at a glance and provide a more intuitive experience when building complex workflows. Video Input Support ------------------- **August 8th, 2025** We’ve added support for video inputs to Prompts and Workflows when using supported models. Composio Tool support --------------------- **August 7th, 2025** We’ve added support to connect to over 250+ apps and 10K+ tools through our partner integration with [Composio](https://composio.dev/) . Composio makes it easy to connect with external data sources, APIs, and services, allowing you to extend the capabilities of your Vellum workflows. It’s a powerful way to integrate with tools like Google Sheets, Slack, Salesforce, and many more, enabling you to build complex workflows that interact with a wide range of external systems. It’s perfect for those who want to build internal tools or automate business processes. Once all set up, you can add Composio as a Tool in Agent Nodes (formerly known as Tool Calling Nodes) in your workflows. ![Add Composio Tool](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/add-composio-tool.png) You can then use Composio to call any of the supported tools in your Workflow. ![Composio Tool Selector](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-08/composio-tool-selector.png) To make use of Composio, you’ll need to: 1. Sign up for a Composio account 2. Configure the integrations you’re interested in 3. Create a Composio API Key 4. Add the Composio API Key to as an Environment Variable in your Vellum Workspace Insert Node Panel UX Improvements --------------------------------- **August 7, 2025** We’ve updated the Insert Node Panel in the Workflow Builder with several UX improvements to make it easier to find and add nodes to your workflows: ![Screenshot of the new Insert Node Panel showing the search bar, reorganized cards, and nested hierarchy with Basic Nodes and Execution Logic sections](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/6b9d41e1-e6b0-4334-a148-105799749e28-insert-node-panel-main-view.png) Updated Insert Node Panel with search functionality and improved organization **Key improvements include:** * **Search Bar**: Quickly find specific nodes by typing their name * **Updated Copy**: Clearer, more concise descriptions for each node type * **Smaller Cards**: More compact design that fits more nodes on screen * **Reorganized Layout**: Better categorization with logical groupings * **Nested Hierarchy**: Expandable sections for Control Flow and other categories ![Screenshot showing the expanded Control Flow section with various node types including Guardrail, Error, Conditional, Map, and Merge nodes](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/6b9d41e1-e6b0-4334-a148-105799749e28-insert-node-panel-control-flow.png) Control Flow section with nested hierarchy showing Guardrail, Error, Conditional, Map, and Merge nodes * **Split Subworkflow Options**: Subworkflows are now clearly separated into “Start from Scratch” and “Deployed Subworkflow” options for better clarity ![Screenshot of the Subworkflow section showing two distinct options: Start from Scratch and Deployed Subworkflow](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/6b9d41e1-e6b0-4334-a148-105799749e28-insert-node-panel-subworkflows.png) Subworkflow section showing the split between Start from Scratch and Deployed Subworkflow options We hope that these improvements make it easier to find the building blocks you need to create powerful AI workflows in Vellum. As always, we welcome your feedback on these changes! GPT 5 on OpenAI --------------- **August 7, 2025** We’ve added support for OpenAI’s latest [GPT-5 models](https://openai.com/gpt-5/) . You can find both API variants on our Models page for each GPT-5 Model: * **GPT-5 via Chat Completions API** * **GPT-5 via Responses API** * **GPT-5 Mini via Chat Completions API** * **GPT-5 Mini via Responses API** * **GPT-5 Nano via Chat Completions API** * **GPT-5 Nano via Responses API** GPT OSS 120B on Cerebras ------------------------ **August 5th, 2025** We’ve added support for GPT OSS 120B on Cerebras. GPT OSS Models on Groq ---------------------- **August 5th, 2025** We’ve added support for GPT OSS 120B and 20B on Groq. Anthropic’s Claude Opus 4.1 --------------------------- **August 5th, 2025** We’ve added support for Anthropic’s latest model [Claude Opus 4.1](https://www.anthropic.com/news/claude-opus-4-1) . OpenAI’s GPT OSS Models ----------------------- **August 5th, 2025** We’ve added support for OpenAI’s two latest models: * [**GPT OSS 120B**](https://platform.openai.com/docs/models/gpt-oss-120b) * [**GPT OSS 20B**](https://platform.openai.com/docs/models/gpt-oss-20b) Cerebras Qwen 3 480B Models --------------------------- **August 5th, 2025** We’ve added support for the three newest variants of the Qwen 3 480B models: * **Qwen 3 Coder 480B on Cerebras** * **Qwen 3 Thinking 480B on Cerebras** * **Qwen 3 Instruct 480B on Cerebras** Vertex AI Gemini 2.5 flash lite for Finetuning ---------------------------------------------- **August 4th, 2025** We’ve added support for [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash) finetuned models. You can add finetuned Vertex AI models to your Workspace by selecting “Fine-tuned Vertex AI Models” on the [models page](https://app.vellum.ai/models) . --- # September 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Workflow Sandbox Linter ----------------------- **September 30th, 2025** The Workflow Sandbox now includes a linter that actively scans your Workflow and highlights configuration-related warnings in real-time. Previously, it was easy to create invalid Workflows that looked fine, but would then fail when you tried to run them. You’ll now see warnings in three places: at the Node level, in the mini map, and in a list view within the console. Clicking a warning in the console automatically zooms to the relevant Node, making it quick to identify and fix issues. You can also use keyboard shortcuts to navigate between Nodes with warnings—Option + Shift + Up/Down on Mac or Alt + Shift + Up/Down on Windows. To view more details about a warning, simply hover over the warning card in the console or the warning icon on the Node. ![Screenshot showing the Workflow Sandbox with warnings displayed in the console list view, highlighting Entrypoint Node, Output Node, and Subworkflow Node warnings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/workflow-linting-console.png) Warnings are displayed in the console list view with details about which Nodes have issues ![Screenshot showing the Workflow canvas with warnings visible on Nodes and in the mini map view at the bottom left](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/workflow-linting-node-and-mini-map.png) Warnings also appear in the mini map, providing a high-level view of which areas of your Workflow need attention Google Gemini 2.5 Flash Models ------------------------------ **September 26th, 2025** We’ve added support for the following new models from Google: * `gemini-2.5-flash-preview-09-2025` * `gemini-2.5-flash-lite-preview-09-2025` OpenAI GPT-5 Codex Support -------------------------- **September 25th, 2025** We’ve added support for OpenAI’s GPT-5 Codex model, enabling you to leverage their latest code-focused capabilities directly within Vellum. Markdown Editor in Note Nodes ----------------------------- **September 23rd, 2025** Note Nodes can be an effective way to document your Workflow and communicate different aspects of it. Historically, you could change the font size of the note’s text, as well as its background color, but that was about it. Now, Note Nodes support markdown. You can add headers, style text, include code blocks, and even embed iframes. Coupled with the [recently added ability](https://docs.vellum.ai/changelog/2025/2025-09#positioning-nodes-in-front--behind-one-another) to move Note Nodes behind other nodes, you can really get creative with how you document your Workflows for others to follow along. ![Screenshot showing the Note Node Markdown Editor interface with formatting toolbar displaying options for headers, bold, italic, lists, code blocks, quotes, and iframe embedding, along with a preview showing formatted content including headers, paragraphs, lists, and embedded iframe](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/note-nodes-markdown-editor-interface-1758666989.png) Note Node Markdown Editor interface showing rich formatting options including headers, text styling, code blocks, lists, and iframe embedding capabilities AI App UI for Deployed Workflows -------------------------------- **September 22nd, 2025** We’ve introduced a dedicated AI App UI for all Deployed Workflows, allowing you and your end users to run and test Workflows directly in the Vellum web interface. This chat-based interface provides a first-class way to test out your Workflows without needing to integrate via API. The AI App UI uses the same inputs as the Workflow, and streams the ouputs back to the chat interface. Prior to this, testing deployed Workflows required developers to either implement API calls in their own applications, or use tools like Postman to manually hit the Workflow endpoints. Now, you have a dedicated interface for executing Workflows and previewing their results directly in Vellum. It’s accessible via the Share Workflow button on any deployed Workflow or from the Deployments overview page, and provides clean input forms that automatically match your Workflow’s parameters with real-time execution and streaming output support. ![Screenshot showing the AI App UI interface with input fields for text, number, and image upload, along with a Run button for Workflow execution](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ai-app-ui-workflow-interface-1758552189.png) The AI App UI provides an intuitive interface for testing deployed Workflows with clean input forms and real-time execution ![Screenshot displaying Workflow execution results with detailed output text and Edit Inputs/Run buttons for iterative testing](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ai-app-ui-execution-results-1758552189.png) Execution results display in real-time with detailed output and options to edit inputs for additional test runs ![Screenshot of the Share Workflow modal showing AI App and Read-Only Workflow options with URLs and configuration settings](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ai-app-ui-share-workflow-modal-1758552189.png) Access the AI App UI through the Share Workflow modal, alongside traditional read-only sharing options Agent Builder Interactive Components ------------------------------------ Released September 22nd, 2025 Agent Builder now supports rendering rich, interactive UI components in chat responses. Now, users can perform actions like creating Environment Variables, reviewing and accepting Workflow plans, and executing Workflows directly from chat responses. Dynamic Model Recommendations on Agent Builder ---------------------------------------------- **September 17th, 2025** Agent Builder can now intelligently recommend models for your workflows, selecting the best options based on their capabilities, context windows, performance, and cost based on your specific requirements and use case. Custom Node Colors & Icons -------------------------- **September 17th, 2025** You can now customize the color and icon for any Node in your Workflows! Simply click on a Node’s icon to open the customization interface, where you can choose from a variety of colors and search through a comprehensive icon library to find the perfect visual representation for your Node’s functionality. ![Screenshot showing the Node customization interface with color picker on the left showing various color options and icon selector on the right with a search bar and grid of available icons](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/custom-node-colors-and-icons-1758077876.png) Customize Node colors and icons by clicking on any Node's icon in your Workflow SDK Preview Code Search ----------------------- **September 17th, 2025** You can now use the search bar in the SDK Preview to match on content! Search results appear with highlighted search terms and enable quick navigation to specific files within your SDK codebase. The search functionality finds matches in both file names and file content, making it easier to locate and navigate to the code you need. ![Screenshot of SDK Preview interface showing search results for 'risk_score' with file matches (2) and content matches (13) displayed with highlighted search terms](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/sdk-preview-code-search-demo-1758135616.png) SDK Preview Code Search showing both file matches and content matches with highlighted search terms for quick navigation Custom inputs for Agent Node code tool -------------------------------------- **September 15th, 2025** You can now pass inputs to the code tool that will not be populated to the model. In the example below, `input_1` is passed as an input, and the model will only populate `greet`. During tool calling, both inputs will be passed into the function. ![Code Tool Input](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/Code%20Tool%20Input.png) Raw Data for Provider Errors ---------------------------- **September 15th, 2025** You can now see the raw response for Provider Errors that might show up when executing a Prompt. This is particularly useful when you want more context as to why a particular Prompt request to a model provider failed. ![Raw Data for Provider Errors](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/Raw%20Data%20Provider%20Error.png) Open Definition Button for Custom Nodes --------------------------------------- **September 11th, 2025** You can now open the Code Preview for your given Custom Node with this new button. Previously, you had to manually click the Code Preview button and navigate to the specific file for the Custom Node you are interested in. Now, you can save time clicking and searching with this button. ![Open Definition](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/Open%20Definition%20Button.png) Agent Builder (Beta) -------------------- **September 16th, 2025** Agent Builder is a conversational AI assistant that helps you build and optimize agent Workflows directly within Workflow Sandboxes. This new Beta feature transforms workflow creation from a manual process into an interactive conversation. When you create a new Workflow Sandbox, Agent Builder appears as a dedicated panel ready to help you describe what you want to build. Simply tell it your requirements—like “create a workflow that extracts key points from a document and finds relevant quotes”—and watch as it constructs your workflow step by step. ![Screenshot showing Agent Builder panel in a Workflow Sandbox with welcome message and prompt asking 'What do you want to build today?' with an 'Edit workflow' toggle](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-panel-welcome-1758065887.png) Agent Builder panel appears in new Workflow Sandboxes with a conversational interface ![Screenshot showing the Agent Builder button prominently displayed in the Workflow Sandbox toolbar alongside other workflow controls](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-toolbar-button-1758065887.png) Access Agent Builder easily through the toolbar button As Agent Builder works, it shows you real-time progress updates, indicating which Nodes it’s configuring (Map Nodes, Search Nodes, Custom Nodes, Prompt Nodes, and more). You can watch as it completes each step of the workflow construction process. ![Screenshot showing Agent Builder working on a complex workflow request with visible progress steps including 'Got Map Node details', 'Got Search Node details', and other completed tasks](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-working-progress-1758065903.png) Agent Builder shows detailed progress as it builds your workflow The workflow updates automatically appear in your Sandbox, complete with pre-filled sample data and a Run button for immediate testing. You can easily see the created workflow alongside the Agent Builder panel and start experimenting right away. ![Screenshot showing a completed workflow with multiple connected nodes in the main canvas area and the Agent Builder panel on the right side, ready for further interaction](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/agent-builder-completed-workflow-1758065903.png) Completed workflow appears in your Sandbox with the Agent Builder panel still available for modifications We’ve been using Agent Builder internally to create powerful automations for transcribing calls and documents, converting them into Linear tickets, and even reviewing GitHub PRs. We’re excited to see what the community will build with this conversational workflow creation experience. First-Class Image and Document Input Variables ---------------------------------------------- **September 10th, 2025** Previously if you wanted to pass an image or document into an LLM in Vellum, say to use the Vision capabilities similar to GPT-4 or GPT-5, you had to pass it as a Chat History variable type. This is unintuitive and difficult to set up and use. Often, you don’t want to pass a Chat History, just a file. Now, both Prompts and Workflows have first-class support for Image and Document variable types in their Input Variables. ![Screenshot showing Workflow inputs configuration with new Image and Document variable types available in the dropdown menu alongside existing String, Number, JSON, and Chat History options](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-image-document-inputs-1757541819.png) New Image and Document input variable types available in Workflow configuration ![Screenshot showing Prompt variable configuration interface with an image variable being configured, displaying file upload area for vellum-logo-black.png](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/prompt-image-document-inputs-1757541819.png) Image and Document input types are also available when configuring Prompt input variables Static Images & Documents in Prompts ------------------------------------ **September 10th, 2025** You can now attach static images and documents directly to your Prompts without needing to include them as part of a chat history input variable. This makes it much simpler to provide consistent reference materials that should be sent with every Prompt execution. To attach a static image or document, click the paperclip icon in the Prompt editor and choose “Insert Static Attachment”. The attached files will be included automatically with every execution of your Prompt. ![Screenshot showing the Prompt editor interface with a user attaching a static image via the paperclip icon menu, displaying options for Insert Variable Attachment and Insert Static Attachment](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/static-prompt-attachment-interface-1757539759.png) Adding static attachments to Prompts using the paperclip icon This feature is particularly useful when you have standard reference documents, product images, or other materials that need to be consistently included across all executions of a particular Prompt. Stack Trace for Workflow Execution Errors ----------------------------------------- **September 8th, 2025** You can now see stack traces for any errors that show up when you execute your Deployed Workflow. This is particularly useful when you want more context as to why a particular Execution failed by giving you the error message and the Stack Trace of the error that was surfaced. ![Stack Trace for Workflow Execution Errors](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/Stack%20Trace%20in%20Execution%20Details.png) Create Custom Nodes from the UI ------------------------------- **September 8th, 2025** You can now create Custom Nodes by drag-and-dropping them from the Node side panel. ![Custom Nodes in Side Panel](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/Custom%20Node%20in%20Side%20Panel.png) Custom Nodes allow you to define any custom behavior by specifying: * Input attributes * Outputs to return * And the underlying code to execute Previously, you could only create new Custom Nodes by pushing up the code representation of a Workflow that contained one via the `vellum workflows push` CLI command. Now, you can define the Custom Node and edit its underlying code straight from the UI. To do this, click the Code Preview button, toggle on edit mode, and edit the definition of for your Custom Node. Positioning Nodes In Front / Behind One Another ----------------------------------------------- **September 7th, 2025** You can now right-click on any Node in your Workflows to position it in front of or behind other Nodes. This feature is particularly useful when working with Note Nodes, allowing you to place them behind other Nodes to help document and organize different parts of your Workflow. ![Bring Node to Front and Back](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/bring-node-to-front-and-back.png) This layering system makes it easier to create visually organized Workflows where documentation and workflow logic can be cleanly separated, improving both readability and collaboration. Updated Data Retention Policy for New Organizations --------------------------------------------------- **September 4th, 2025** New Organizations now default to a 30-day retention period for monitoring data, helping optimize storage costs and performance while maintaining essential operational insights. This change provides a balanced approach to data management without impacting functionality. Existing Organizations maintain their current data retention settings, and Organizations on paid plans can still customize their retention periods to meet specific business requirements. If you need longer data retention for your use case, please reach out to our support team who can help configure the appropriate settings for your needs. Syntax Highlighting in Prompt Jinja Blocks ------------------------------------------ **September 1st, 2025** Prompt Jinja Blocks now include line numbers and syntax highlighting, making it significantly easier to write and validate Jinja code. ![Jinja Syntax Highlighting](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-09/prompt-jinja-blocks.png) Previously, Jinja blocks appeared as plain text without visual feedback, making it difficult to spot syntax errors or understand code structure. --- # Changelog | September, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Fireworks Llama 3.2 90B Vision Instruct --------------------------------------- **September 30th, 2024** Meta’s most recent open source vision model, [Llama 3.2 Vision Instruct](https://fireworks.ai/models/fireworks/llama-v3p2-90b-vision-instruct) , is now available in Vellum. This model excels in visual recognition, image reasoning, captioning, and answering diverse questions related to images and is a great open source option if you’re looking for a vision model. Private Models Cost Tracking ---------------------------- **September 26th, 2024** Models that are now created through the Custom Model Carousel on the [models page](https://app.vellum.ai/models) will have [cost tracking for prompt sandboxes](https://docs.vellum.ai/changelog/2024/2024-08#prompt-sandbox-cost-tracking) and [cost tracking for prompt deployments](https://docs.vellum.ai/changelog/2024/2024-09#cost-tracking-for-prompt-deployment-executions-table) . This means that you’ll be able to see the dollar cost of LLM calls to model providers even for your custom models. Google Gemini 1.5 002 Models ---------------------------- _September 24th, 2024_ Google Gemini’s newest [002 models](https://developers.googleblog.com/en/updated-production-ready-gemini-models-reduced-15-pro-pricing-increased-rate-limits-and-more/) `gemini-1.5-pro-002` & `gemini-1.5-flash-002` are now available in Vellum! They offer 50% reduced pricing, 2x higher rate limits, and 3x lower latency than the previous Gemini 1.5 models. Release Tag Column and Filter for Prompt Deployment Execution Table ------------------------------------------------------------------- _September 24th, 2024_ You can now view and filter on release tags attached to your prompt executions within the Prompt Deployment Execution Table! This addition allows for quick identification of the release version associated with each execution. You can enable this new column in the Columns dropdown. ![Prompt Deployment Executions Table with Release Tag Filter](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/prompt_deployment_release_tag_filter.png) New Prompt Caching Columns for Prompt Deployment Execution Table ---------------------------------------------------------------- _September 23rd, 2024_ A while back Anthropic added support for [Prompt Caching](https://docs.vellum.ai/changelog/2024/2024-08#prompt-caching-support-for-anthropic) . With this update, you’ll now see the number of Prompt Cache Read and Cache Creation Tokens used by a Prompt Deployment’s executions if it’s backed by an Anthropic model. This new monitoring data can be used to help analyze your cache hit rate with Anthropic and optimize your LLM spend. ![Prompt Executions with Cache Tokens](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/prompt_cache_tokens.png) Improved Latency Filter and Sorting for Workflow Executions ----------------------------------------------------------- _September 23rd, 2024_ You can now sort and filter by the Latency field in the Workflow Executions Table! This update allows for better prioritization and identification of executions with higher or lower latencies, as well as targeting executions within a range of latencies. We believe these improvements will greatly aid in monitoring and managing workflow executions and their performance and metrics! Improved Debugging for Map Nodes -------------------------------- _September 23rd, 2024_ It used to be difficult to debug problematic iterations when a Map Node failed. We now keep track of each iteration’s execution and make it easy to view them. You can page through a Map Node’s iterations one-by-one. ![Map Node Rejected Pagination](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/map-node-rejected-pagination.png) Each of these iterations, included the any that failed, are now also show in a Map Node’s full screen editor. ![Map Node Rejected Editor](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/map-node-rejected-editor.png) The full screen editor now also allows you to cycle through each of an executed Map Node’s iterations, making it easy to debug problematic iterations and iterate on the subworkflow used to produce that iteration’s execution. Resizable Node Editor Panel --------------------------- _September 20th, 2024_ For those of you using the new Workflow Builder, you’ll now be able to resize the Node Editor Panel. This update makes it much easier to edit complex Conditional Node rules, Chat History Messages, JSON values, and more. ![Resizable editor panel](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/ResizableSidePanel.gif) Evaluations Performance Improvements ------------------------------------ _September 17th, 2024_ While not as flashy as some of our other updates, we’ve undergone a major overhaul of our Evaluations backend resulting in significant performance improvements to the Evaluations page. Test Suites consisting of thousands of Test Cases used to feel sluggish and sometimes not load, but now load successfully and should feel much more responsive. Cost Tracking for Prompt Deployment Executions Table ---------------------------------------------------- _September 17th, 2024_ You can now see the cost of each Prompt Execution in the Prompt Executions Table. ![Cost tracking prompt executions](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/prompt_executions_cost_tracking.png) This is the next step of many we have planned for improving visibility into LLM costs in Vellum. You might use this to audit expensive calls and optimize your prompts to reduce costs. Optimized Prompt Deployment Executions Table -------------------------------------------- _September 13th, 2024_ This update brings a reduction in load times for filters and sorts; in some instances, dropping 2 minute load times to a few seconds. We’ve achieved this by switching to a more efficient data source, enabling more effective filtering and sorting capabilities. You’ll notice faster page load times across the board, resulting in a smoother, more responsive experience when working with Prompt Deployment Executions. This optimization sets the stage for exciting new features we have in the works. Stay tuned for more updates that will enhance your ability to analyze, and optimize your prompt executions. External ID Filtering for Workflow Deployment Executions -------------------------------------------------------- _September 13th, 2024_ Previously, when filtering workflow deployment executions by external IDs, you had to provide the exact string match to retrieve relevant results. Now, you can filter external IDs using a variety of string patterns. You can specify that the external ID should start with, end with, or contain certain substrings. This enhancement allows for more flexible filtering, making it easier to locate specific workflow deployment executions based on partial matches. ![new_external_id_filter_options](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/new_workflow_deployment_execution_external_id_filter_options.png) Workflow Execution Timeline View Revamp --------------------------------------- _September 13th, 2024_ We have given the Workflow Execution Timeline View a bit of a facelift. Along with a more modern look, we have added a couple quality of life improvements: * **Subworkflows**: Instead of needing to navigate to a separate page, you can now expand subworkflows to view their executions details within the same page. * **Node Pages**: Instead of cluttering the page with the details of all nodes at once, we now display the details for just one node at a time. Click on a node to view its inputs, outputs, and more. Each node has its own permalink so that you can share the url with others. ![Workflow Execution Timeline](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/workflow-execution-timeline.png) OpenAI Strawberry (o1) Models ----------------------------- _September 12th, 2024_ OpenAI’s newest [Strawberry (o1) models](https://openai.com/o1/) `o1-preview`, `o1-mini`, `o1-preview-2024-09-12`, & `o1-mini-2024-09-12` are now available in Vellum and have been added to all workspaces! Interactive Pages in Single Editor Mode --------------------------------------- _September 7th, 2024_ It used to be that when two people were on the same Prompt/Workflow Sandbox, only one person could edit and interact with the page. If you were a Viewer, you were unable to interact with the page at all and were blocked with a big page overlay. Now, the page overlay is gone and Viewers can interact with the page in a read-only mode and perform actions that don’t affect the state of the page. This includes things like scrolling, opening modals, copying text, etc. Expand Cost in Execute Prompt APIs ---------------------------------- _September 4th, 2024_ You can now opt in to receive the cost of a Prompt’s execution in the response of the [Execute Prompt](https://docs.vellum.ai/api-reference/prompts/execute-prompt#request.body.expand_meta.cost) and [Execute Prompt Stream](https://docs.vellum.ai/api-reference/prompts/execute-prompt-stream#request.body.expand_meta.cost) APIs. This is helpful if you want to capture the cost of executing a Prompt in your own system or if you want to provide cost transparency to your end users. To opt in, you can pass the `expand_meta` field in the request body with the `cost` key set to `true`. | | | | --- | --- | | 1 | { | | 2 | ..., | | 3 | "expand\_meta" : { | | 4 | "cost": true | | 5 | } | | 6 | } | You can expect a corresponding value to be included in the meta field on the response: | | | | --- | --- | | 1 | { | | 2 | ..., | | 3 | "meta": { | | 4 | "cost" : { | | 5 | "value" : 0.000450003, | | 6 | "unit" : "USD" | | 7 | } | | 8 | } | | 9 | } | This functionality is available in our SDKs beginning v0.8.9. Default Block Type Preference ----------------------------- _September 4th, 2024_ You can now set a default Block type to use when defining Prompts in Vellum. Whenever you see the “Add Block” or “Add Message” options in a Prompt Editor, your preferred Block type will be used. By default, the Block type is set to “Rich Text,” the newer option that supports Variable Chips. You can still switch between Block types for individual Blocks within the Prompt Editor. ![default block type toggle](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/default-block-type-toggle.png) New and Improved Code Editor ---------------------------- _September 3rd, 2024_ We now use [Monaco Editor](https://microsoft.github.io/monaco-editor/) for our code editor that is used by Workflow Code Nodes and custom Code Evaluation Metrics. Monaco is the same editor that Visual Studio Code uses under the hood. This offers a number of improvements including IntelliSense, semantic validation and syntax validation. Additionally we now inject Vellum Value types into the editor, so you can now have fully typed input values for things such as Chat History. Some of these improvements are currently only available for TypeScript and not Python. VPC Disable gVisor Option for Code Execution -------------------------------------------- _September 3rd, 2024_ VPC customers of Vellum can now disable gVisor sandboxing for code execution in self-hosted environments to significantly improve the performance of Code Nodes in Workflows. gVisor is needed for secure sandboxing in our Managed SASS platform, but in a self hosted environment where you’re the only organization, it’s not strictly required if you trust that users within your org won’t run malicious code. ![gVisor self hosted flag](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-09/gvisor-flag.png) Download Original Document from UI ---------------------------------- _September 2nd, 2024_ You can now download a file that was originally uploaded as a Document to a Document Index from the UI. You’ll find a new “Download Original” option in a Document’s ••• More Menu. --- # Changelog | January, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Prompt Deployment Usage Tracking -------------------------------- _January 29th, 2024_ Going forward, Vellum will now keep track of the token utilization of your Prompt Deployments. You can keep tabs on input, output, and total tokens used per request. ![Token Count Row Data](https://storage.googleapis.com/vellum-public/help-docs/token-count-row-data.png) You can also view this data in aggregate in the Monitoring tab. ![Token Count Time-Series Data](https://storage.googleapis.com/vellum-public/help-docs/token-count-time-series-data.png) This is a precursor to more advanced usage and billing features coming down the road. If there’s more you’d like to see here, please share your feedback with us! * * * Model Search Bar ---------------- _January 29th, 2024_ As the number of models available in Vellum grows, it’s become harder to find the model you’re looking for. To help with this, we’ve added a search bar to the model selection dropdown in the Prompt and Workflow editors. This will make it easier to find the model you’re looking for, especially as we continue to add more models to the platform. ![Model Search Bar](https://storage.googleapis.com/vellum-public/help-docs/model-search-bar.png) * * * Single Editor Mode ------------------ _January 25th, 2024_ We’ve collaborated with our friends at [velt.dev](https://velt.dev/) to deliver an all new “Single Editor Mode” in Prompt and Workflow Sandboxes. With this, only one person can edit a Prompt/Workflow at a time and you can hand off editing control to another collaborator. This is useful for avoiding conflicts when multiple people are trying to edit the same Prompt/Workflow at the same time. Check out the video below to see it in action! * * * API to Execute Workflow w/o Streaming ------------------------------------- _January 23rd, 2024_ We’ve added a new API endpoint for executing a Workflow Deployment without streaming back its incremental results. This is useful when you want to execute a Workflow and only care about its final result or if you’re invoking your Workflow via a service that doesn’t support HTTP Streaming like Zapier. * * * Workflow Deployment Execution Visualization Improvements -------------------------------------------------------- _January 22nd, 2024_ Now, when visiting the details page for a Workflow Deployment Execution, you’ll find an improved loading state as well as a simplified view for Conditional Nodes. * * * Upload/Download of Function Definitions --------------------------------------- _January 18th, 2024_ You can now easily import your existing function definition files (JSON or YAML) into Vellum function calling blocks as well as export functions you’ve already defined in Vellum to pass along to engineers for implementation. Check it out below! * * * Image Support for OpenAI Vision Models -------------------------------------- _January 18th, 2024_ Vellum now has API support for interacting with OpenAI’s vision models, such as `gpt-4-vision-preview`. You can learn more about OpenAI Vision models [here](https://platform.openai.com/docs/guides/vision) . Note that there is limited support for images in the Vellum UI at this time, but you can still use the API to interact with OpenAI Vision models. UI support coming soon! Here’s a quick example on how to send an image to the model, using our python sdk: | | | | --- | --- | | 1 | image\_link \= "https://storage.googleapis.com/vellum-public/help-docs/add\_prompt\_block\_button.png" | | 2 | response \= client.execute\_prompt( | | 3 | prompt\_deployment\_name\="github-loom-demo", | | 4 | inputs\=\[ |\ | 5 | PromptDeploymentInputRequest\_ChatHistory( |\ | 6 | name\="$chat\_history", |\ | 7 | value\=\[ |\ | 8 | ChatMessageRequest( |\ | 9 | role\=ChatMessageRole.USER, |\ | 10 | content\={ |\ | 11 | "type": "ARRAY", |\ | 12 | "value": \[ |\ | 13 | {"type": "STRING", "value": "What's in this image?"}, |\ | 14 | {"type": "IMAGE", "value": {"src": image\_link}}, |\ | 15 | \], |\ | 16 | }, |\ | 17 | ) |\ | 18 | \], |\ | 19 | type\=VellumVariableType.CHAT\_HISTORY, |\ | 20 | ), |\ | 21 | \], | | 22 | ) | | 23 | print(response.outputs\[0\].value) | * * * Folders ------- _January 12th, 2024_ You can now organize entities in Vellum via folders! You can nest folders, share them by url, and move entities between folders. * * * Support for Google Gemini Safety Settings ----------------------------------------- _January 12th, 2024_ There is now native support for setting the `safetySetting` parameters in Google Gemini prompts. You can learn more about how these parameters are used by Google in their docs [here](https://ai.google.dev/api/rest/v1beta/SafetySetting) . ![Gemini Custom Parameters](https://storage.googleapis.com/vellum-public/help-docs/gemini-custom-params.png) * * * Support for OpenAI JSON Mode, User ID, and Seed Params ------------------------------------------------------ _January 11th, 2024_ There is now native support for setting the `user` and `seed` parameters in OpenAI API requests, as well as specifying that the response format be of type JSON. You can learn more about how these parameters are used by OpenAI in their docs [here](https://platform.openai.com/docs/api-reference/chat/create) . ![OpenAI Custom Parameters](https://storage.googleapis.com/vellum-public/help-docs/open-ai-custom-params.png) * * * Cloning Workflow Scenarios -------------------------- _January 9th, 2024_ You can now clone a Workflow Scenario to create a new Scenario based on an existing one. This is useful when you want to create a new Scenario that is similar to an existing one, but with some changes. ![Clone Workflow Scenario](https://storage.googleapis.com/vellum-public/help-docs/clone-workflow-scenario.png) * * * API Key Metadata ---------------- _January 9th, 2024_ Now you can add and view metadata for your Vellum API keys. For example, you can see when an API key was created and by whom. You can also assign a label to an API key to help you keep track of its purpose and an environment tag so that you know where it’s used. ![API Key Metadata](https://storage.googleapis.com/vellum-public/help-docs/api-key-metadata.png) * * * Top-Level Workflow Execution Actions ------------------------------------ _January 4th, 2024_ You can now find the following actions at the top-level of the Workflow and Prompt Deployment Execution pages: * **Save as Scenario:** Useful for saving an edge case seen in production as a Scenario for qualitative eval. * **Save as Test Case:** Useful for saving an edge case seen in production to your bank of Test Cases for quantitative eval. * **View Details:** Drill in to see specifics about that specific Execution. ![Top-Level Workflow Execution Actions](https://storage.googleapis.com/vellum-public/help-docs/top-level-workflow-execution-actions.png) * * * Improved Error Messages in Code & API Nodes ------------------------------------------- _January 2nd, 2024_ API Nodes and Code Nodes within Workflows now have improved error messages. When an error occurs, the error message will now include the line number and column number where the error occurred. This will make it easier to debug errors in your Workflows. ![Improved Error Messages in Code & API Nodes](https://storage.googleapis.com/vellum-public/help-docs/workflow-api-node-code-node-error-messages.png) --- # Changelog | August, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Anthropic Google Vertex AI Support ---------------------------------- _August 30th, 2024_ We now support using Anthropic’s Claude 3.5 Sonnet, Claude 3 Opus and Claude 3 Haiku Models with [Google Vertex AI](https://cloud.google.com/vertex-ai) . You can add them to your workspace from the [models page](https://app.vellum.ai/models) . Anthropic Tool Use API for Function Calling ------------------------------------------- _August 30th, 2024_ We now support using Anthropic’s [Tool Use API](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) for function calling with Claude 3.5 Sonnet, Claude 3 Opus and Claude 3 Haiku Models. Previously Anthropic function calling had been supported by shimming function call XML into the prompt. Prompt Node Linked Deployments ------------------------------ _August 29th, 2024_ We have reworked the relationship of how Prompt Node’s interact with Deployments. Previously, there was: * No way to update a Prompt in one spot and have it update in multiple Workflows * Confusing UX around what it meant to import a Prompt Today we are releasing this new setup modal that appears when you create a Prompt Node: ![New Prompt Node Setup](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt-node-setup.png) The setup modal contains a new `Link to Deployment` option. This is a Prompt Node that references a Prompt Deployment _directly_ with a Release Tag. This allows for Workflows both in the Sandbox and as a Deployment to automatically pick up changes to the underlying Prompt without needing to update the Workflow by pointing to `LATEST`. To maintain a specific version of a Prompt Deployment, you can specify a user-defined Release Tag to keep the Prompt Node pinned to a specific version. In this way, they now work exactly as Subworkflow Nodes when you select `Link to Deployment` there: ![Prompt Node Linked Deployments](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt-node-linked-deployments.png) Workflow Executed By Filterable Column -------------------------------------- _August 29th, 2024_ Earlier this month, we restricted the Workflow Deployment Executions table to only show executions invoked via API requests. This helped to filter out all of the noise from other contexts in which a Workflow Deployment could be invoked, bringing focus to only data from production traffic. However, we’ve found that are still other contexts in which it’s useful to see Workflow Executions. You’ll now find a new `Executed By` column that shows what the immediate “parent” context was in which the Workflow was executed. This table is filtered down to just `API Request` by default, but you can opt in to include additional contexts, such invocation as a Subworkflow via a parent Workflow: ![Executed By Workflow Filter](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/executed_by_workflow_filter.png) Tool Choice Parameter Support for OpenAI ---------------------------------------- _August 28th, 2024_ We are excited to announce that you can now natively specify how prompts handle functions using OpenAI’s [Tool Choice](https://platform.openai.com/docs/api-reference/chat/create#chat-create-tool_choice) parameter. With the Tool Choice parameter, you can now dictate exactly when tools are used, allowing more precise and effective control of your prompt tools. This feature is now available across all OpenAI models that support functions. ![Tool Choice Enablement](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/tool_choice_parameter.png) Add Metadata to Workflow Executions ----------------------------------- _August 27th, 2024_ You can now add metadata to your Workflow Executions through the API. This is useful for tracking additional information about your executions, such as the source of the request or any other custom data you want to associate with the execution. This metadata is visible in the Workflow Execution Details page in the Vellum UI. You can view more information at the [API documentation](https://docs.vellum.ai/api-reference/workflows/execute-workflow#request.body.metadata) . ![Workflow Execution Details Metadata](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-executions-metadata.png) New Workflow Editor Beta Release -------------------------------- _August 26th, 2024_ Our new Workflow Editor is now available as an opt-in beta release. Next time you open the Workflow Editor, you’ll see an announcement with the option to turn on the new Editor experience. We’ve made a ton of improvements to the Editor UI, and more improvements are in the works. You should find that your Workflows are easier to navigate and edit, and more performant. The beta can also be toggled on or off in the workflow builder settings at any time. We’d love to get your feedback about the new experience, so please let us know what you think! ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-editor-ui-opt-in.png) ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-editor-ui-beta.png) View the Provider Payload on a Workflow’s Prompt Node ----------------------------------------------------- _August 26th, 2024_ You can now view the compiled provider payload on a Workflow’s Prompt Node. This is useful for debugging and understanding the exact data that was sent to the provider during a run, especially if you got some unexpected results. ![Workflow Provider Payload](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-provider-payload.png) Merging Two Adjacent Prompt Blocks ---------------------------------- _August 26th, 2024_ Merging two adjacent prompt blocks in the prompt editor is now possible! This feature is especially useful when you want to combine two prompt long prompt blocks into one. You can find this button in the top right drop down in the prompt editor. Only blocks of the same type can be merged. For example, you can merge two rich text blocks or two Jinja blocks, but you cannot merge a rich text block with a Jinja block. You can easily convert between the two, however, by clicking the three dots in the top right of the block and selecting “Convert to Jinja” or “Convert to Rich Text”. ![Merging Two Adjacent Blocks Dropdown](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/merge-two-blocks.png) Asynchronous Exports of Evaluation Reports ------------------------------------------ _August 26th, 2024_ Exports of evaluation reports are now asynchronous. You can export your evaluation report along with its results in CSV or JSON format, and an email will be sent to you once the export is done. This change is especially useful for large evaluation reports, where the export process and download can take some time. ![Evaluation Report Export](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/evaluation-report-export.png) JSON Schema Editor with $ref Support ------------------------------------ _August 26th, 2024_ Vellum let’s you define JSON Schemas in a few different places throughout the app to do things like define [Structured Outputs](https://docs.vellum.ai/changelog/2024/2024-08#openai-structured-outputs-support) and [Function Calls](https://docs.vellum.ai/product/prompts/prompt-engineering#function-calling) . Previously this UI was just a simple form that allowed you to define basic JSON schemas. This UI has been improved to support direct edits via a raw JSON editor. ![Raw Schema Button](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/raw_schema_button.png) From here, you can edit your JSON schema directly. This raw editor allows you to make use of all features supported by the [JSON Schema spec](https://json-schema.org/overview/what-is-jsonschema) , even if they may not yet be supported by our basic form UI. For example, you can now defined references (i.e. `$ref`) like this: as references: ![Raw Editor References](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/raw_editor_references.png) Support for Excel Files in Document Indexes ------------------------------------------- _August 23rd, 2024_ We now support uploading `.xls` and `.xlsx` files to Document Indexes for indexing and searching. Prompt Caching Support for Anthropic ------------------------------------ _August 22nd, 2024_ Anthropic recently released some exciting API changes that allow for [Prompt Caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching#how-prompt-caching-works) . This new feature allows for caching of frequently used portions of your Prompt for up to 5 minutes; which reduces the latency and cost of subsequent executions that include the same Prompt context. This powerful feature is now natively supported within Vellum! In order to use it, simply toggle the new cache options on a given Prompt Block for the supported Claude Sonnet 3.5 and Claude Haiku 3.0 models. ![Vellum Prompt Caching](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt_sandbox_cache.png) Prompt Execution Pages ---------------------- _August 22nd, 2024_ If you wanted to drill into a single Prompt Execution, previously you’d have to navigate to the Prompt Deployment’s Executions table and try to filter for the specific Execution ID you’re looking for. Now each row has a navigable link accessible from the table: ![Prompt Execution Link](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt-execution-link.png) This will navigate you to a dedicated page representing that specific Prompt Execution. From here, you can see details about the Execution like the raw HTTP data sent to and from the provider, any actuals recorded, the Vellum inputs and outputs to the prompt, and more! ![Prompt Execution Page](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt-execution-page.png) Historical Versions of Entities in Evaluation Reports ----------------------------------------------------- _August 21st, 2024_ Earlier this month, we introduced [Evaluation Report History](https://docs.vellum.ai/#evaluation-report-history) , which allows you to view a history of all Evaluation runs and revisit the results of any prior state. We’ve now enhanced this feature by adding the ability to preview or navigate directly to the version of the Workflow or Prompt as it existed during that specific run. ![Evaluation Report History](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/evaluation-report-history-entity-linking.png) GPT-4o Finetuning ----------------- _August 19th, 2024_ OpenAI’s newest GPT-4o models `gpt-4o-2024-08-06` and `gpt-4o-mini-2024-07-18` are now available as base models to add as OpenAI finetuned models. Workflow Execution Replay & Scrubbing ------------------------------------- _August 18th, 2024_ You can now replay and scrub through the execution of a Workflow in Workflow Sandbox and Deployment Execution Details pages. This feature is particularly useful for debugging and understanding the flow of your Workflow, especially if it contains loops where a single node might be run more than once. ![Workflow Execution Replay & Scrubbing](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-execution-replay-and-scrubbing.gif) OpenAI Structured Outputs Support --------------------------------- _August 15th, 2024_ OpenAI released some API changes that allow their newest models to support [Structured Outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) . This powerful new feature enables developers to strictly define the expected JSON object schemas from the model as part of the response through a model parameter, or through a function call. This new functionality is now natively integrated within Vellum! To use within the context of Function Calling, simply toggle on the `Strict` checkbox for any given Function Call: ![Function Call Strict](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/function-call-strict.png) To enable Structured Outputs as part of a general OpenAI response, configure the `JSON Schema` setting as part of model parameters: ![JSON Schema Strict](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/json-schema-strict.png) Both places come with upload/download functionality built into the form. Note that for function calling, this means we’ve reduced the scope of the upload/download to be _just_ the `Parameters` JSON schema field. This allows schemas to be cross-compatible between either location since we are working with an [open specification](https://json-schema.org/understanding-json-schema) . ![JSON Schema Strict](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/upload-download-schema.png) Native JSON Input Variable Support for Prompts ---------------------------------------------- _August 14th, 2024_ Vellum Prompts have historically been able to accept strings and chat histories as dynamic inputs to their template. If you wanted to operate on JSON, you’d have to pass it as a string and then parse it within the Prompt itself (i.e. perform `json.loads()` within a Jinja Block). Vellum Prompts now support native JSON as inputs! When you add an input variable to a Prompt, you can now select the new “JSON” type. ![JSON Variables Dropdown](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/json-variable-dropdown.png) JSON input values will render as prettified JSON objects when referenced in Rich Text Blocks and can be operated on directly without the need for `json.loads()` when referenced in Jinja Blocks. Workflow Deployment Executions Filtered to Just API Executions -------------------------------------------------------------- _August 12th, 2024_ Our Workflow Deployment Executions page used to list all executions of a Workflow Deployment, no matter where they were invoked from. However, this would often get confusing because you’d see a mix of results from both eval runs and production traffic in the same view. Our Workflow Deployment Executions page now filters down to just those executions that were invoked via the API. Executions from evaluations are still accessible from within the Evaluations UI by hovering over a row and clicking the “View Workflow Details” button: ![View Workflow Details](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/view-workflow-details.png) Add Specific Releases to Evaluation Reports ------------------------------------------- _August 12th, 2024_ We’ve updated Evaluation Reports to give you more control over the releases you evaluate. Previously, you could only add the latest release of a Deployment to your reports. Now, you can select specific releases by their tag, allowing you to compare different versions within your Evaluation Reports. ![Add Deployment](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/evaluation-report-add-by-release-tag.png) Workflow Sandbox Latency ------------------------ _August 9th, 2024_ You can now view the latency of Workflow Sandboxes and their Nodes. To enable viewing latency click the Workflow Sandbox settings gear icon in the top right and turn on the “View Latency” option. ![Workflow Sandbox Latency](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-latency-1.png) ![Workflow Sandbox Latency Settings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/workflow-latency-2.png) Prompt Sandbox Cost Tracking ---------------------------- _August 9th, 2024_ You can now see the dollar cost of a Prompt’s execution within both a Prompt Sandbox’s Prompt Editor and Comparison Mode views. These costs are calculated using model providers’ publicly available pricing data in conjunction with the number of input/output tokens used. ![Prompt Sandbox With Cost Tracking](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt_sandbox_with_cost_tracking.png) ![Prompt Sandbox Comparison With Cost Tracking](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/prompt_sandbox_comparison_with_cost_tracking.png) If you’re curious about a given model’s pricing, you can view details in the Model’s detail page. ![MLModel Detail Page with Billing Config](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/mlmodel_detail_page_with_cost_config.png) Most popular models already have pricing information populated, with support for even more models following in the coming days. Showing cost information in Prompt Sandboxes is just the first step! We’ll expose cost details throughout more of Vellum over time. GPT-4o 2024-08-06 ----------------- _August 6th, 2024_ OpenAI’s newest GPT-4o model `gpt-4o-2024-08-06` is now available in Vellum and has been added to all workspaces! Deployment Descriptions ----------------------- _August 2nd, 2024_ You can now update your Prompt and Workflow Deployments to include a human-readable description. This is useful for giving other members of your team a high-level summary of what the Prompt or Workflow does without needing to parse through the configuration or control flow. ![Update Deployment Description](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/update-deployment-description.png) Once set, the description will appear as part of the Deployment Details page within the Deployment Info section: ![Display Deployment Description](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/display-deployment-description.png) Evaluation Report History ------------------------- _August 1st, 2024_ It used to be that you could only view the latest set of Evaluation results for a given Prompt or Workflow. But now, you can view a history of all Evaluation runs and go back to view the results of any prior state. ![Evaluation Report History](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-08/evaluation-report-history.png) This is particularly helpful if you want to do things like compare the results of two different Evaluation runs, download the results of a past Evaluation run, or simply view the Test Cases that existed at that time. --- # Changelog | January, 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Support for OpenAI’s o3-mini ---------------------------- **January 31st, 2025** We’ve added support for OpenAI’s [o3-mini](https://platform.openai.com/docs/models/#o3-mini) model and the `o3-mini-2025-01-31` snapshot. Support for PowerPoint Files in Document Indexes ------------------------------------------------ **January 31st, 2025** We now support uploading `.pptx` files to Document Indexes for indexing and searching. Support for Perplexity Sonar Reasoning Model -------------------------------------------- **January 29th, 2025** We’ve added support for the [Perplexity Sonar Reasoning Model](https://docs.perplexity.ai/guides/model-cards) . Support for DeepSeek R1 Distill Llama 70b via Groq -------------------------------------------------- **January 27th, 2025** We’ve added support for [DeepSeek R1 Distill Llama 70b](https://console.groq.com/docs/models) via Groq. Support for pushing to a specific Workflow Sandbox or Workspace --------------------------------------------------------------- **January 28th, 2025** We’ve added two new options to the `vellum workflows push` command: * `--workflow-sandbox-id` - A specific Workflow Sandbox ID to use when pushing. This provides an alternative to the module name for identifying a Workflow to push. * `--workspace` - A specific Workspace config to use when pushing. This provides an alternative to the `VELLUM_API_KEY` environment variable for identifying a Workspace to push a Workflow to. These changes are now available in version `0.13.15` of the Vellum SDK. Domain-Based Organization Join Policies --------------------------------------- **January 28th, 2025** We’ve completely redesigned the Organization Settings page to give organization administrators more control. Admins can now configure organization join policies and manage how new users are added to their organization. With this, org admins can opt in to allow for new users with pre-approved email domains to automatically join the organization upon signup without needing to be manually invited. ![Organization Settings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/organization-settings.png) As an admin, you can add new email domains to the list of verified domains by selecting amongst email domains used by existing users in the organization. Self-serve Organization Setup ----------------------------- **January 28th, 2025** We’re excited to announce that organization setup is now fully self-service during user onboarding. New users can either create their own organization or automatically join an existing one based on their email domain if the organization allows for it. ![Organization Setup](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/organization-setup.png) Beta Release of SDK-Enabled Workflows ------------------------------------- **January 27th, 2025** Our engineering team has been hard at work on a new [SDK](https://github.com/vellum-ai/vellum-python-sdks/tree/main/src/vellum/workflows#--vellum-workflows-sdk--) for building AI-powered Workflows. Today, we’re releasing our initial support for Workflows SDK within Vellum itself. You can invoke the [Vellum CLI](https://docs.vellum.ai/developers/workflows-sdk/api-reference/cli) to `pull` Workflows defined in the UI down as SDK code, make changes, and then `push` it back up to Vellum! The SDK brings with it a full suite of new features to Vellum Workflows, including the ability to define your own Custom Nodes. In order to enable the infrastructure that power these new features on Vellum, simply check the “SDK Compatible” checkbox while creating a new Workflow: ![SDK Compatible Workflow](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/sdk-compatible-workflow.png) You can also convert existing Workflows to be SDK compatible by opting in via Workflow Sandbox Settings: ![SDK Compatible Workflow](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/toggle-sdk-compatibility.png) Workflows SDK and the infrastructure that powers it is still in active development. We would love to hear your feedback! We are giving active Vellum customers initial access to both while we gear up for a full public release in the coming weeks. Support for Text Search on Documents List Endpoint -------------------------------------------------- **January 27th, 2025** We’ve added support for text search when listing Documents using the (search query parameter)\[[https://docs.vellum.ai/developers/client-sdk/documents/list#request.query.search](https://docs.vellum.ai/developers/client-sdk/documents/list#request.query.search)\ \]. This allows you to filter for Documents that contain the search query in their `label` or `external_id`. This parameter is available starting in version `0.13.14` of the Vellum SDK. Support for o1-mini (2024-09-12) on Self-Managed OpenAI on Azure ---------------------------------------------------------------- **January 25th, 2025** We’ve added support for the [o1-mini (2024-09-12)](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#o1-and-o1-mini-models) model on Vellum’s Self-Managed OpenAI on Azure integration. Support for DeepSeek R1 via Together AI --------------------------------------- **January 24th, 2025** We’ve added support for [DeepSeek R1](https://docs.together.ai/docs/serverless-models) via Together AI. Support for DeepSeek R1 via Fireworks AI ---------------------------------------- **January 24th, 2025** We’ve added support for [DeepSeek R1](https://fireworks.ai/models/fireworks/deepseek-r1) via Fireworks AI. Support for Newest Perplexity Models ------------------------------------ **January 23rd, 2025** We’ve added support for the newest Perplexity models, [Sonar and Sonar Pro](https://www.perplexity.ai/hub/blog/introducing-the-sonar-pro-api) . Support for Gemini Exp 1206 --------------------------- **January 23rd, 2025** We’ve added support for Google’s [gemini-exp-1206](https://blog.google/feed/gemini-exp-1206/) model. Support for DeepSeek Reasoning Model ------------------------------------ **January 22nd, 2025** We’ve added support for [DeepSeek’s new reasoning model](https://api-docs.deepseek.com/guides/reasoning_model) . Prompt Node Cost and Model Name in Workflows -------------------------------------------- You can now see token cost and a model name in Prompt Node results when invoking a _Workflow Deployment_ via the Execute Workflow Stream API, by passing in `True` to the `expand_meta.cost` or the `expand_meta.model_name` | | | | --- | --- | | 1 | stream \= client.execute\_workflow\_stream( | | 2 | workflow\_deployment\_name\="demo", | | 3 | inputs\=\[ |\ | 4 | WorkflowRequestInputRequest\_String( |\ | 5 | type\="STRING", |\ | 6 | name\="foo", |\ | 7 | value\="bar", |\ | 8 | ), |\ | 9 | \], | | 10 | event\_types\=\["WORKFLOW", "NODE"\], | | 11 | expand\_meta\=WorkflowExpandMetaRequest( | | 12 | cost\=True, | | 13 | model\_name\=True, | | 14 | ) | | 15 | ) | | 16 | | | 17 | for event in stream: | | 18 | if event.type == "NODE" and event.data.state == "FULFILLED": | | 19 | node\_result\_data \= event.data.data | | 20 | if node\_result\_data and node\_result\_data.type == "PROMPT": | | 21 | print(node\_result\_data.data.execution\_meta.cost) | | 22 | print(node\_result\_data.data.execution\_meta.model\_name) | Support for Gemini 2.0 Flash Thinking Mode ------------------------------------------ **January 15th, 2025** We’ve added support for the [Gemini 2.0 Flash Thinking Mode](https://ai.google.dev/gemini-api/docs/thinking-mode) model. Workflow Outputs Panel ---------------------- **January 10th, 2024** You’ll now see a new “Outputs” button that opens up the new “Workflows Outputs” panel in a Workflow Sandbox. ![Workflow Outputs Panel](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/workflow-outputs-panel-2.png) From here, you can see all the outputs that your Workflow produces, and easily navigate to the Nodes that produce them. In the coming weeks, you’ll also be able to directly edit your Workflow’s outputs from this panel. Newly Added Support for Gemini 1.5 Flash ---------------------------------------- **January 11th, 2025** We’ve added support for [Gemini 1.5 Flash](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash) model that points to Latest Stable. Newly Added Support for DeepSeek Models --------------------------------------- **January 2nd, 2024** We’ve added support for [DeepSeek AI](https://www.deepseek.com/) models. Along with the launch of the DeepSeek integration, we’ve added support for [DeepSeek V3 Chat](https://api-docs.deepseek.com/quick_start/pricing) . Function Call Inputs in Chat Messages ------------------------------------- **January 2nd, 2025** There is now first-class support for Function Call inputs to Chat Messages. This allows you to simulate the behavior of a Function Call output from a model in Vellum as part of a message in Chat History. To see this feature in action, check out the video below: --- # June 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Default Release Tags -------------------- **June 29th, 2025** By default, when creating a Prompt or Workflow Deployment Node in a Workflow, Vellum defaults to using the `LATEST` Release Tag of that Deployment. However, Organizations can now configure a global default Release Tag that will be used as the default Release Tag instead. This setting can be found in Organization Settings under the Advanced Settings section: ![Default Tag Settings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-06/default-tag-settings.png) This feature allows teams to standardize their deployment practices by setting a custom default release tag (such as “production” or “staging”) that will be automatically applied to new Prompt or Workflow Deployment Nodes: ![Default Tag Usage](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-06/default-tag-usage.png) AI-Generated Workflow Descriptions ---------------------------------- **June 27th, 2025** When editing a Workflow’s details, Vellum will now automatically generate a draft description for Workflows that don’t already have one. This AI-powered feature analyzes your Workflow’s structure, nodes, and metadata to create a helpful starting point for documentation. ![Edit Details modal showing AI-generated Workflow description](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/bf536508-e3ed-44c5-9abf-e3782fce7860-ai-generated-workflow-descriptions-edit-details.png) AI-generated description appears when editing Workflow details The feature works by sending your Workflow definition and metadata to an LLM using Vellum’s own infrastructure, ensuring the generated descriptions are contextually relevant and accurate. You can edit, accept, or replace the generated description as needed. You can enable or disable this feature from your Organization Settings under the “AI Features” section. This gives you full control over whether AI-generated descriptions are offered to users in your organization. ![Organization Settings showing AI-Generated Workflow Descriptions toggle](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/bf536508-e3ed-44c5-9abf-e3782fce7860-ai-generated-workflow-descriptions-organization-settings.png) AI Features settings in Organization Settings OpenAI Chat Completions Web Search Tool Support ----------------------------------------------- **June 24th, 2025** We now support two of OpenAI’s Search Preview models, `GPT 4o Search Preview` and `GPT 4o Mini Search Preview`, within Vellum to make use of two new Web Search tools that OpenAI has released for their Chat Completions Web Search models: * Search Context Size: Giving you the ability to set whether you would like to use `low`, `medium`, or `high` search context. * User Location: Giving you the ability to restrict the models search scope to a specific users location. Draggable Custom Nodes from Side Panel -------------------------------------- You can now drag and drop Custom Nodes directly from the side panel into your Workflow graph on any SDK-enabled Workflows that use a Custom Docker Image with Custom Nodes. To enable this feature, ensure your Custom Docker Image includes a `vellum_custom_nodes` folder containing your Node. Once you build and push the image to Vellum, your Custom Nodes will automatically appear in the UI side panel. This makes it significantly easier for non-engineers in your Workspace to build Workflows that feature reusable logic as its own Node! Custom Images for Running Workflows ----------------------------------- Workflows that are SDK enabled can now be run in a Docker Image of your choice! By default, all Workflows run in our base image, `vellumai/python-workflow-runtime:latest`. You can now extend that base image to define a _new_ Docker image, push it up to Vellum, then associate it with your Workflow: ![Custom Docker Image](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-06/custom-docker-image.png) Pushing up your own Docker Image is most valuable for those who want to give their Workflows access to custom packages or libraries. For more information on how to set this up for your Workspace, check out our [docs](https://docs.vellum.ai/developers/workflows-sdk/custom-container-images) . Custom Nodes Run in Vellum -------------------------- **June 18th, 2025** The Workflows SDK has support for defining [Custom Nodes](https://docs.vellum.ai/developers/workflows-sdk/tutorials/custom-nodes) . You can now run `vellum workflows push [module]` on any Workflow that features a Custom Node, and that Node will be runnable in the UI! ![Custom Node Chain](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-06/custom-node-chain.png) You can use Custom Nodes to execute any logic you want, allowing Sandbox users a simple interface for configuring what goes into the node. They also support all of the other features now common to Workflow Nodes: * Ports * Adornments * Trigger behavior * Expressive Attributes * and more! Custom Nodes in the UI have the following limitations that we are hoping to alleviate soon: * `run` method is not yet editable from the UI * Initiating a new Custom Node from the graph Support for Codex Mini (Latest) ------------------------------- **June 17th 2025** We’ve added support for OpenAI’s codex-mini-latest model. Support for new Gemini 2.5 models --------------------------------- **June 17th 2025** We’ve added support for the following new Gemini 2.5 models: * Gemini 2.5 Pro (Stable) * Gemini 2.5 Flash (Stable) * Gemini 2.5 Flash Lite (06/17 Preview) Support for Gemini 2.5 Pro Preview 06-05 ---------------------------------------- **June 13th 2025** We’ve added support for Gemini API’s newest version of Gemini 2.5 Pro, the the June 5th Gemini 2.5 Pro Preview. Support for OpenAI’s Responses API released with O3 Pro ------------------------------------------------------- **June 11th 2025** We’ve added support for OpenAI’s new [Responses API endpoint](https://platform.openai.com/docs/api-reference/responses) , unlocking the ability to integrate with OpenAI models that are invokable via the Responses API. Along with support for Responses API, the first model we’ve chosen to integrate is the newly released [O3 Pro](https://platform.openai.com/docs/models/o3-pro) model. GPT-4o Audio Preview (2025-06-03) --------------------------------- **June 4th, 2025** We’ve added the new gpt-4o-audio-preview-2025-06-03 model from OpenAI to Vellum. Duplicate Test Cases in Evaluation Reports ------------------------------------------ You can now **duplicate Test Cases** directly from the Evaluation Report UI — making it much faster to create slight variations without starting from scratch. After selecting one or more Test Cases in the table, a **Duplicate** button will appear in the toolbar. Clicking it will instantly create copies of the selected Test Cases with all values duplicated. ![Evaluation Report showing Duplicate button in toolbar with selected Test Cases](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ea2be77b-478a-4459-8282-e076be56cd7b-test-case-duplication-feature.png) Duplicate button appears in toolbar when Test Cases are selected This is especially useful when you’re adding a few new Test Cases that are mostly the same as existing ones — no need to manually recreate them or switch to CSV import just to save time. Bulk Apply Value to Test Cases ------------------------------ You can now efficiently apply a cell’s value to multiple Test Cases at once. A new icon button appears in each Test Case cell that, when clicked, opens a confirmation modal asking whether you want to apply the current cell’s value to: * **All Test Cases** if none are selected * **Only selected Test Cases** if one or more are selected ![Test Suite interface showing bulk apply functionality with icon button](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/f19ed5ba-6b7c-4248-9d26-4e14d434ad25-bulk-apply-test-cases-feature.png) Bulk apply value feature showing the new icon button This feature significantly reduces repetitive manual edits when you need to set the same value across multiple Test Cases. Helpful tooltip and modal messages clearly indicate whether the action will affect all Test Cases or just the selected ones, making it easy to understand the scope of your changes before applying them. Edit Test Cases in a Resizable Side Drawer ------------------------------------------ **June 4th, 2025** We’ve updated the **Evaluation Report table** to make editing Test Cases smoother and more intuitive. ### Side Drawer for Editing Click on any editable cell to open a right-side drawer where you can: * Edit variable values and row details * Switch between **“Values”** and **“Details”** tabs * Save or reset your changes * Navigate between rows with Next/Previous buttons ![Test Case editing side drawer with Values tab showing variable editing interface](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ea2be77b-478a-4459-8282-e076be56cd7b-test-case-side-drawer-values-tab.png) Test Case editing side drawer showing Values tab ### Cleaner, Focused UI Editing is now more spacious and organized with resizable drawer and inputs—no more cramped inline inputs. ### Better Keyboard & Focus Handling Fields auto-focus when selected, and scroll into view with enhanced keyboard shortcuts: * `⌘ + S` (or `Ctrl + S`) – Save current edits * `⌘ + ↑` / `⌘ + ↓` (or `Ctrl + arrow`) – Move to previous/next row This update makes it easier to view and edit full Test Cases—especially helpful when working with complex inputs or many variables. Updated Deployment Flow ----------------------- We’ve redesigned the Deployment Flow for Prompt and Workflow Sandboxes to provide a cleaner, more intuitive experience. The Create and Update Deployment forms have been streamlined with improved organization and clearer visual hierarchy. ![New Create Deployment form showing cleaner layout with Create New and Update Existing tabs](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/43eb0a59-34dd-449a-817a-fff0e2ce21f5-deployment-flow-create-form.png) Redesigned Create Deployment form with improved layout Additionally, the Releases tab in the Deployment Details page has been redesigned for better usability and visual clarity, making it easier to manage Release Tags and track Deployment history. ![Redesigned Releases tab showing cleaner deployment information and release management interface](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/43eb0a59-34dd-449a-817a-fff0e2ce21f5-deployment-flow-releases-tab.png) Improved Releases tab in Deployment Details Condensed Node View ------------------- **June 4th, 2025** You can now enable Condensed Node View from the Navigation Settings to streamline your workflow interface. This feature displays all Nodes in a more compact format, significantly reducing visual clutter and making it easier to navigate complex workflows with many Nodes. This is particularly useful when working with large, multi-step workflows where the standard Node view can become overwhelming. The condensed view helps you maintain a clear overview of your entire workflow structure while still preserving full functionality—simply click on any Node to expand it and make changes as needed. Deployment Release Descriptions ------------------------------- You can now add descriptions to your Prompt and Workflow Deployments during the Deployment process. This new feature allows you to document the purpose, changes, or context for each Deployment directly within the platform. When creating a new Deployment, you’ll see an optional description field where you can provide details about what this specific Deployment includes, why it was created, or any other relevant information for your team. ![Deploy Prompt modal showing Release Description field](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ccba5e40-da79-4e6d-800d-e43d7123991f-deployment-release-description-modal.png) Release Description field in the Deploy Prompt modal Once deployed, the Release Description will be visible in the Deployment Details page, providing valuable context about what changed in each release. ![Deployment Details page showing Release Description](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/ccba5e40-da79-4e6d-800d-e43d7123991f-deployment-release-description-display.png) Release Description displayed in Deployment Details This feature helps teams maintain better documentation of their Deployment history and understand the evolution of their Prompts and Workflows. **Coming soon**: You will also be able to add and edit descriptions after a Deployment has been created. Prompt & Workflow Descriptions ------------------------------ **June 3rd, 2025** You can now specify a description for a Prompt or Workflow. To add a description, click on “Edit Details” from within the More Menu. ![Edit Details option in the More Menu](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/11b3d35a-cd57-40c0-81e7-928a247644d9-workflow-edit-details-menu.png) Edit Details option in the More Menu This will open a modal where you can edit the Prompt/Workflow’s label as well as description. ![Edit Details modal showing label and description fields](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/11b3d35a-cd57-40c0-81e7-928a247644d9-workflow-edit-details-modal.png) Edit Details modal for adding descriptions Upon specifying a description, an info icon will appear where you and others can hover over to see a tooltip explaining what the Prompt/Workflow is intended to do. Descriptions are also shown in the List View of Prompt/Workflow index pages. Specifying descriptions can serve as a helpful reminder for what the Prompt or Workflow is meant to do, as well as provide documentation for others on your team. --- # November 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. New Native Integrations ----------------------- **November 26th, 2025** We’ve added support for 5 new Native Integrations: * [Miro](https://miro.com/) * [Monday](https://monday.com/) * [ProductBoard](https://www.productboard.com/) * [Spotify](https://spotify.com/) * [Todoist](https://www.todoist.com/) Multi-Modal Workflow Outputs ---------------------------- **November 24th, 2025** Workflows can now output documents, images, videos, and audio. The simplest way to build Workflows with multi-modal outputs is to use Agent Builder—describe what you want to create, and Agent Builder will generate Custom Nodes that integrate with the relevant generation APIs. Multi-modal outputs are displayed in the console, monitoring layers, and AI Apps. ![Multi-modal Workflow output showing generated outfit image in AI App](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/multimodal-workflow-output-ai-app-2025-11-24-1764028511.png) Multi-modal Workflow output in AI App ![Workflow Sandbox showing image output in console](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/multimodal-workflow-output-console-2025-11-24-1764028512.png) Multi-modal output in Workflow Sandbox console New Workflow Sandbox Layout --------------------------- **November 24th, 2025** We’ve updated the Workflow Sandbox layout. The Agent Builder panel is now top-level, we’ve removed breadcrumbs, and simplified the navigation tabs to Sandbox/Evaluations/Deployments in the center, giving you more space for the canvas and console. ![Updated Workflow Sandbox layout with Agent Builder at top level](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-sandbox-layout-nov-2025-1763997410.png) Updated Workflow Sandbox layout with Agent Builder at top level Workflow Comparison in Publishing Flow -------------------------------------- **November 22nd, 2025** You can now click the “Compare” button when publishing a Workflow to see a code diff between your Sandbox and the latest deployed Release. ![Compare button in Publish Workflow dialog](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-compare-button-1763838499.png) Compare button in Publish Workflow dialog ![Comparison modal showing code diff](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-compare-modal-1763838500.png) Comparison modal showing code diff New Native Integrations ----------------------- **November 20th, 2025** We’ve added support for 27 new Native Integrations: * [AccuLynx](https://acculynx.com/) * [Affinity](https://www.affinity.co/) * [AgencyZoom](https://www.agencyzoom.com/) * [Ahrefs](https://ahrefs.com/) * [Brevo](https://www.brevo.com/) * [Canvas](https://www.instructure.com/canvas) * [Coda](https://coda.io/) * [Coinbase](https://www.coinbase.com/en-in) * [Facebook](https://www.facebook.com/) * [Fireflies](https://www.fireflies.ai/) * [Google Analytics](https://analytics.google.com/) * [Google Docs](https://docs.google.com/) * [Google Slides](https://docs.google.com/presentation) * [Google Tasks](https://tasks.google.com/tasks/) * [Google Photos](https://photos.google.com/) * [Google Search Console](https://search.google.com/search-console) * [HeyGen](https://www.heygen.com/) * [Instagram](https://www.instagram.com/) * [JungleScout](https://www.junglescout.com/) * [Klaviyo](https://www.klaviyo.com/) * [Linkup](https://www.linkup.com/) * [Listen Notes](https://www.listennotes.com/) * [LMNT](https://drinklmnt.com/) * [Semantic Scholar](https://www.semanticscholar.org/) * [Shortcut](https://www.shortcut.com/) * [YouSearch](https://you.com/) * [ZenRows](https://www.zenrows.com/) Support for Kimi K2 Thinking on Fireworks AI -------------------------------------------- **November 19th, 2025** We’ve added support for Kimi K2 Thinking via Fireworks AI. This model provides advanced reasoning capabilities for complex problem-solving tasks. Set State Node -------------- **November 19th, 2025** You can now add a Set State Node to your Workflows to update state variables during execution. This allows you to maintain and modify state across your Workflow runs, such as tracking counters, accumulating chat history, or storing intermediate values. To use the Set State Node, add it to your Workflow and configure one or more state operations. Each operation specifies: * **State to update**: Select the state variable you want to modify * **Value**: Define the new value using expressions, variables, or operations like addition or concatenation For example, you can increment a counter (`counter = counter + 1`) or append to chat history (`chat_history = chat_history.concat(Agent.ChatHistory)`). Check Workflow Execution Status ------------------------------- **November 19th, 2025** You can now check the status of a Workflow execution using the [Check Workflow Execution Status endpoint](https://docs.vellum.ai/developers/client-sdk/workflows/check-execution-status) . The endpoint returns the current execution state (`PENDING`, `FULFILLED`, `REJECTED`, etc.), along with outputs and an execution detail URL once the Workflow completes. This makes it easy to poll for completion when using async execution patterns. Async Workflow Execution ------------------------ **November 19th, 2025** We’ve added support for asynchronous Workflow execution, allowing you to initiate long-running Workflows without waiting for completion. Use the new [Execute Workflow Async endpoint](https://docs.vellum.ai/developers/client-sdk/workflows/execute-workflow-async) to start a Workflow execution and receive an `execution_id` immediately. Async executions automatically queue when you exceed your concurrency limit, making this endpoint ideal for batch jobs where you don’t need everything to complete at once. You can initiate many executions quickly and they’ll process as capacity becomes available. Track execution completion and access outputs using the [Check Workflow Execution Status endpoint](https://docs.vellum.ai/developers/client-sdk/workflows/check-execution-status) . See our [Batching Executions guide](https://docs.vellum.ai/product/workflows/advanced/batching-executions) for polling patterns and webhook alternatives. Workflow Triggers ----------------- **November 19th, 2025** You can now configure Workflows to execute automatically based on schedules or external events. Previously, Workflows were only ever executed when explicitly invoked (e.g. via API or through an AI App). Now you can configure Workflow Triggers directly in the Workflow Sandbox by adding either a Scheduled Trigger or an Integration Trigger. You’ll find Triggers as new node types at the bottom of the “Add” panel. ![Adding a Workflow Trigger](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/Add%20Workflow%20Trigger%20.png) Adding a Workflow Trigger Scheduled Triggers run Workflows on a recurring schedule using cron expressions. You can describe your schedule in plain English (like “every day at 9am”) and Vellum will convert to a cron expression, or provide the cron expression directly. ![Scheduled Trigger Configuration](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/Scheduled%20Trigger%20Configuration.png) Scheduled Trigger Configuration Integration Triggers invoke your Workflow automatically in response to webhook events that come from Native Integrations (e.g., Slack, Gmail, GitHub, etc). After authenticating with the Native Integration, you can configure the specific event type and related settings. For example, with a Linear Integration Trigger, you can specify a team ID to only trigger on issues created within that team. ![Linear Integration Trigger Configuration](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/linear_integration_config-1763597740.png) Linear Integration Trigger Configuration When testing your Workflow with Triggers, add a new Scenario from the Inputs panel and select the option that corresponds to your Trigger. This automatically populates the Scenario with the event payload attributes from the integration, letting you specify custom values and test with simulated Trigger data. ![Add new Workflow Trigger Scenario](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/Add%20new%20Workflow%20Trigger%20Scenario.png) Add new Workflow Trigger Scenario When you deploy a Workflow Sandbox containing a Trigger, the Workflow Deployment executes automatically based on that configuration. Scheduled Triggers run according to the cron schedule, and Integration Triggers fire when the specified event occurs. You can view and enable/disable Triggers on the Workflow Deployment overview page. ![Workflow Trigger Deployment](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/Workflow%20Trigger%20Deployment.png) Workflow Trigger Deployment Triggers mark a big step towards using Vellum workflows to build Agents that operate autonomously in response to real-world events. We can’t wait to see what you build with them! Personal API Keys Now Respect User Permissions ---------------------------------------------- **November 19th, 2025** Personal API Keys now respect your user permissions and roles. If you don’t have the necessary permissions for an action, API requests will return a 403 error. For example, you’ll need the Deployment Editor role to deploy Workflows via API using your Personal API Key. Support for Gemini 3 Pro Preview -------------------------------- **November 18th, 2025** We’ve added support for Gemini 3 Pro Preview via the Gemini and Vertex AI APIs, along with the new `thinkingLevel` and `mediaResolution` API features. Agent Builder Plan Diagrams --------------------------- **November 18, 2025** Agent Builder will now create visual diagrams to help convey its plan while you build. ![Agent Builder Plan Diagrams](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/agent_builder_plan_diagram3.png) Agent Builder Plan Diagrams You can click on the diagram to expand it: ![Agent Builder Plan Diagrams - Expanded View](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/agent_builder_plan_diagram3_expanded.png) Agent Builder Plan Diagrams - Expanded View Agent Builder CSV, TXT, MD File Support --------------------------------------- **November 18, 2025** You can now upload CSV, TXT, and MD files to Agent Builder to streamline building new Workflows and Agents. Below are just a few ways you could use this: * Upload a CSV of mock customer data to build a data extraction workflow * Upload a markdown document with product specifications to build a product comparison agent * Upload a text file with your company’s guidelines to build a compliance checking agent ![Agent Builder File Upload UI](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-11/agent_builder_csv_txt_support.png) Agent Builder File Upload Agent Builder PDF and Image Inputs ---------------------------------- **November 16, 2025** You can now upload PDF and Image files to Agent Builder to streamline building new Workflows and Agents. Below are just a few ways you could use this: * Upload a sketch or diagram of a Workflow architecture, Agent Builder will build it for you * Upload example images or documents for Agent Builder to automate prompt generation for classification or data extraction tasks * Upload brand guidelines to convert into prompts for style adherence ![Agent Builder File Upload](https://storage.googleapis.com/vellum-public/help-docs/agent-builder-file-upload.gif) Agent Builder File Upload Building an entire Workflow from a Diagram Support for OpenAI’s GPT 5.1 via Vellum --------------------------------------- **November 13th, 2025** We now support OpenAI’s [GPT 5.1 model](https://platform.openai.com/docs/models/gpt-5.1) via Chat Completions and Responses endpoints. Updated Condensed Nodes ----------------------- **November 13th, 2025** We’ve redesigned Nodes in Workflows with a cleaner, more condensed look that helps you see more of your Workflow at a glance. * Simplified Node appearance with less visual bulk * Node results now appear in the console organized by timeline, rather than underneath each Node * The console automatically highlights the current streaming Node as your Workflow executes left to right * You can now see your entire Workflow even when the console is open * Improved visual states for running Nodes, outputs, and errors * Cleaned up the appearance of connections between Nodes Generic Private Code Package Repositories ----------------------------------------- **November 13th, 2025** You can now configure generic private code package repositories for Python that support basic authentication. This allows you to pull dependencies from custom PyPI-compatible repositories. You can use these private repositories when using Code Execution Nodes in Workflows and also in Code Metrics. To add a private repository you can navigate to the [Private Package Repository page here.](https://app.vellum.ai/organization/private-repositories) You can also add a repository by clicking the “Add Private Repository button” in the dropdown of the new Repository field when adding a package. Web Search Tool support for Anthropic Claude Models --------------------------------------------------- **November 12th, 2025** You can now enable the use of [Web Search Tools](https://docs.claude.com/en/docs/agents-and-tools/tool-use/web-search-tool) for Anthropic Claude models. Environment Variables as Metric Inputs -------------------------------------- **November 11th, 2025** Evaluation Metrics can now reference Environment Variables as inputs. This is useful for custom Metrics that need secret values like API Keys. Previously, you’d reference workspace-level secrets directly. Web Search Node --------------- **November 10th, 2025** You can now add Web Search Node to your Workflows from the Workflow Sandbox UI. Pass in a query and it’ll return the top 5 search results from the internet, including the page title and URL for each result. ![Web Search Node outputs showing search results with titles and URLs](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/web-search-node-example-2025-11-11-1762827082.png) Web Search Node showing search results This node used the Serp API Integration under-the-hood to actually perform the web search. Console Improvements -------------------- **November 10th, 2025** We’ve made several improvements to the console for viewing and debugging Workflow runs. * **Auto-select streaming Nodes.** The console now automatically selects the latest streaming Node as your Workflow executes, making it easier to follow along in real-time. * **Better timeline view.** The timeline view has improved spacing, colors, icons, and loading states for a cleaner debugging experience. * **Enhanced Subworkflow and Map Node timelines.** We’ve improved the timeline display for Subworkflows and Map Node runs to make debugging these complex execution patterns easier. * **Console in Subworkflows.** The console is now available inside Subworkflows, giving you a snapshot of what happened in the last run without leaving the Subworkflow context. * **Faster Map Node event viewing.** Viewing events inside the Map Node after a run is now much faster and smoother. Agent Builder Audio Notifications --------------------------------- **November 8th, 2025** Agent Builder now plays an audio chime when it’s done building your Workflow if you switched to another browser tab or application while waiting. Evaluating Workflows That Use Integrations ------------------------------------------ **November 8th, 2025** You can now run evaluations for Workflows that use third-party integrations like Notion or Slack. Previously, these evaluations would fail when they tried to invoke an integration. Now they’ll succeed as long as you’ve authenticated with the integrations used in the Workflow. Environment Variables & Secrets Simplification ---------------------------------------------- **November 7th, 2025** We’ve simplified how you manage Environment Variables, Secrets, and API Keys in Workspace Settings. * **Environment Variables and Secrets are now unified**: There’s no longer a distinction between the two. Just create Environment Variables and mark them as secret with a toggle—we’ll handle the secure storage under-the-hood. * **Consolidated management page**: Environments and Workspaces are now managed on a single “Manage” page. * **Dedicated API Keys page**: API Keys now have their own page within Workspace Settings for easier access. ![New Environment Variable creation modal showing Name and Value fields with a Secret toggle](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/environment-variable-creation-modal-2025-11-07-1762487550.png) Create Environment Variables with optional secret toggle Customizable Workflow Icons --------------------------- **November 5th, 2025** You can now customize the color and icon of your Workflows. Just click the more menu, then Edit Details, and click on the icon shown. We’ll then show the icon on the Workflows homepage, in your AI App, and more. ![Edit Workflow Details modal showing color palette and icon selector for customizing Workflow appearance](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2q2HPCBfINPu2XGmdXdc8rjpkpW/workflow-icon-customization-modal-2025-11-05-1762360465.png) Customize Workflow colors and icons New Native Integrations ----------------------- **November 1st, 2025** We’ve added support for 14 new native integrations: * [Apollo](https://www.apollo.io/) * [Atlassian](https://www.atlassian.com/) * [Bitbucket](https://bitbucket.org/) * [BrowserBase](https://www.browserbase.com/) * [Cal](https://cal.com/) * [ElevenLabs](https://www.elevenlabs.io/) * [Exa](https://exa.ai/) * [Mem0](https://mem0.ai/) * [Neon](https://neon.tech/) * [Parsera](https://parsera.org/) * [People Data Labs](https://www.peopledatalabs.com/) * [PostHog](https://posthog.com/) * [Tavily](https://www.tavily.com/) * [Semrush](https://www.semrush.com/) You can now connect these integrations directly to your Workflows in Agent Builder, Agent Nodes, or Custom Nodes. --- # Changelog | May, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Context Menu for Workflow Edges and Nodes ----------------------------------------- _May 31th, 2024_ You can now right-click on Workflow Edges to open a context menu to allow you to delete them without having to hunt down the trash icon. You can also now right-click on Workflow Nodes to delete them as well. ![Workflow Context Menu](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/workflow-context-menus.png) Breadcrumbs and Page Header Improvements ---------------------------------------- _May 31th, 2024_ We’ve significantly improved folder and page breadcrumbs throughout the app. Prompts, Test Suites, Workflows, and Documents now display the entire folder path of your current page, making it much easier to navigate through your folder structure. We’ve also updated the overflow styling for breadcrumbs: instead of an ellipsis, you’ll now see a count of hidden breadcrumbs, which can be accessed via a dropdown menu. Additionally, the pages mentioned above, along with Workflow/Prompt Evaluations and Deployments, now feature the same updated header design. ![Breadcrumbs and header updates](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/breadcrumbs-update.png) Subworkflow Node Navigation --------------------------- _May 31st, 2024_ When viewing the execution details of a Workflow, Subworkflow nodes executed as part of that run will now have a link to _its_ execution page. ![Subworkflow Navigation](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/subworkflow-navigation.png) Prompt Deployment Actuals Metadata ---------------------------------- _May 29th, 2024_ When submitting execution Actuals for Prompts, you can now optionally include a metadata field. This field can contain arbitrary data, and will be saved and shown in the Executions tab of your Prompt Deployment. ![Prompt Actuals Metadata](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/prompt-execution-actuals-metata.png) This is particularly helpful if you want to capture feedback/quality across multiple custom dimensions. Learn more in our [API docs here](https://docs.vellum.ai/api-reference/api-reference/submit-completion-actuals) . Replay Workflow from Node ------------------------- _May 29th, 2024_ One of the biggest burdens when developing Workflows in Vellum is having to rerun your _entire_ Workflow whenever you want to make a change to just a single node and want to see its downstream effects. You can now re-run a Workflow from a specific Node! After running a Workflow for the first time, you’ll see this new play icon above each Node. ![Replay From Node Icon](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/replay-from-node-icon.png) Doing so will re-use results from the previous execution for all upstream nodes and only actually execute the target node and all nodes downstream of it. ![Replay From Node Execution](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/replay-from-node-execution.png) We hope this helps you decrease iteration cycles and save on LLM costs! Improvements to Saving Executions as Scenarios & Test Cases ----------------------------------------------------------- _May 29th, 2024_ Saving Prompt/Workflow Deployment Executions from production API calls to an Evaluation dataset as Test Cases is a great way to close the feedback loop between monitoring and experimentation. However, this process has historically been time-consuming when you have many Executions to save. We’ve made a number of improvements to this process: 1. You can now multi-select to bulk save Executions as Test Cases 2. We now default to the correct Sandbox/Test Suite when saving Executions as Scenarios/Test Cases 3. You’ll now see warnings if the Sandbox/Test Suite you’re saving to has required variables that are missing from the Execution Check out a full demo here: Prompt Sandbox History Update ----------------------------- _May 28th, 2024_ Previously, editing past versions of a Prompt Sandbox could be confusing, with unclear indications of which version you were modifying and how it was being saved. Now, the history view for a Prompt Sandbox is read-only. To edit a previous version, simply click the Restore button, and a new editable version will be created from that specific version. Workflow Deployment Actuals Metadata ------------------------------------ _May 28th, 2024_ When submitting execution Actuals for Workflows, you can now optionally include a metadata field. This field can contain arbitrary data, and will be saved and shown in the Executions tab of your Workflow Deployment. This is particularly helpful if you want to capture feedback/quality across multiple custom dimensions. | | | --- | | curl -X POST https://predict.vellum.ai/v1/submit-workflow-execution-actuals \\ | | -H "X\_API\_KEY: "$VELLUM\_API\_KEY"" \\ | | -H "Content-Type: application/json" \\ | | -d '{ | | "execution\_id": "be975a69-33c7-4ff0-b6ac-d8008198db1e", | | "actuals": \[ |\ | { |\ | "output\_type": "STRING", |\ | "output\_key": "final-output", |\ | "quality": 0.8, |\ | "metadata": { |\ | "user\_score": 1.0, |\ | "internal\_score": 1.0, |\ | "internal\_notes": "The output was not factually correct." |\ | } |\ | } |\ | \] | | }' | Guardrail Workflow Nodes ------------------------ _May 23rd, 2024_ You can now use Metrics inside of Workflows with the new Guardrail Node! Guardrail Nodes let you run pre-defined evaluation criteria at runtime as part of a Workflow execution so that you can drive downstream behavior based on that Metric’s score. For example, if building a RAG application, you might determine whether the generated response passes some threshold for [Ragas Faithfulness](https://docs.ragas.io/en/latest/concepts/metrics/faithfulness.html) and if not, loop around to try again. ![Guardrail Nodes](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/guardrail-nodes.png) Chat Mode Revamp ---------------- _May 22th, 2024_ Chat Mode in Prompt Sandboxes has received a major facelift! The left side of the new interface will be familiar to anyone using the Prompt Editor, while the rest of the interface retains its functionality with a fresh new look. We’ve also fixed some UX wonk and minor bugs during the restyling process. ![Chat Mode Styling Update](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/chat-mode-styling-update.png) Double-Click to Resize Rows & Columns in Prompt Sandboxes --------------------------------------------------------- _May 22th, 2024_ You can now double-click on resizable row and column edges in both Comparison and Chat modes to auto-expand that row/column to its maximum size. If already at maximum size, double-clicking will reset them to their default size. Additionally, in Comparison mode, double-clicking on cell corners will auto-resize both dimensions simultaneously. Improved Image Support in Chat History Fields --------------------------------------------- _May 22th, 2024_ We’ve made several changes to enhance the UX of working with images. Chat History messages now include an explicit content-type selector, making it easier to work with image content using supported models. You can now add publicly-hosted images in multiple ways: by pasting an image URL, pasting a copied image, or dragging and dropping an image from another window. Additionally, we’ve added limited support for embedded images. You can embed an image directly into the prompt by copy/pasting or dragging/dropping an image file from your computer’s file browser. This method has a 1MB size limit and is an interim solution as we continue to explore image upload and hosting options. ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/improved-image-support.png) ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/content-type-select.png) ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/image-config-modal.png) Gemini 1.5 Flash ---------------- _May 20th, 2024_ Google’s [Gemini 1.5 Flash model](https://deepmind.google/technologies/gemini/flash/) is now available in Vellum. You can add it to your workspace from the [models page](https://app.vellum.ai/models) . Llama 3 Models on Bedrock ------------------------- _May 14th, 2024_ We now support both of the Llama 3 models on AWS Bedrock. You can add them to your workspace from the [models page](https://app.vellum.ai/models) . GPT-4o Models ------------- _May 13th, 2024_ OpenAI’s newest [GPT-4o models](https://openai.com/index/hello-gpt-4o/) `gpt-4o` & `gpt-4o-2024-05-13` are now available in Vellum and have been added to all workspaces! ![GPT 4o](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/gpt-4o.png) Organization and Workspace Names in Side Nav -------------------------------------------- _May 13th, 2024_ You can now view the active Organization’s name and the active Workspace’s name in the left sidebar navigation. ![Workspace and Org Name Nav](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/org-name-sidebar.png) Run All Button on Evaluation Reports ------------------------------------ _May 10th, 2024_ There’s now a “Run All” button on evaluation reports that runs a test suite for all variants. Instead of running each variant individually, you can now run them all with one click. ![Prompt Node Execution](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/run-all-button.png) Prompt Node Monitoring ---------------------- _May 9th, 2024_ Vellum is now capturing monitoring data for deployed Prompt Nodes. Whenever a deployed Workflow invokes a Prompt Node, it will now show a link displaying the Prompt Deployment label: ![Prompt Node Monitoring](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/prompt-node-monitoring.png) Clicking on the link will take you to the _Prompt’s executions_ page, where you can then see all metadata captured for the execution, including the raw request data sent to the model: ![Prompt Node Execution](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/prompt-node-execution.png) Groq Support ------------ _May 9th, 2024_ Vellum now has a native integration with the LPU Inference Engine, [Groq](https://groq.com/) . All public models on Groq are now available to add to your workspace. Be sure to add your API key as a Secret named `GROQ_API_KEY` on the “Secrets” tab of your [Workspace Settings](https://app.vellum.ai/organization?tab=workspaces&workspace-settings-tab=secrets) . Groq is an LLM hosting provider that offers incredible inference speed for open source LLMs, including the recently released (and very hyped!) [Llama 3](https://llama.meta.com/llama3/) model. ![Groq Support](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/groq-support.png) Function Calling in Prompt Evaluation ------------------------------------- _May 8th, 2024_ Prompts that output function calls can now be evaluated via Test Suites. This allows you to define Test Cases consisting of the inputs to the prompt, and the expected function call, then assert that there’s a match. For more, check out our [docs](https://docs.vellum.ai/product/evaluation/quantitative-evaluation#function-calling) . ![Function Call Prompts](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/function-tests-edit.png) Out-of-Box Ragas Metrics ------------------------ _May 7th, 2024_ Test-driven development for your RAG-based LLM pipelines is now easier than ever within Vellum! Three new [Ragas Metrics](https://docs.ragas.io/en/latest/index.html) – [Context Revelancy](https://docs.ragas.io/en/v0.1.5/concepts/metrics/context_relevancy.html) , [Answer Relevance](https://docs.ragas.io/en/latest/concepts/metrics/answer_relevance.html) and [Faithfulness](https://docs.ragas.io/en/latest/concepts/metrics/faithfulness.html) – are now available out-of-box in Vellum. These can be used within Workflow Evaluations to measure the quality of a RAG system. For more info, check out our new help center article on [Evaluating RAG Pipelines](https://docs.vellum.ai/product/evaluation/evaluating-rag-pipelines) . ![Ragas Metrics](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/ragas-metrics.png) Subworkflow Node Streaming -------------------------- _May 7th, 2024_ Subworkflow Nodes can now stream their output(s) to parent workflows. This allows you to compose workflows using modular subworkflows without sacrificing the ability to delivery incremental results to your end user. Note that only nodes immediately prior to Final Output Nodes can have their output(s) streamed. ![Subworkflow Streaming](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/subworkflow-streaming.gif) Default Test Case Concurrency in Evaluations -------------------------------------------- _May 4th, 2024_ You can now configure how many Test Cases should be run in parallel during an Evaluation. You might lower this value if you’re running into rate limits from the LLM provider, or might increase this value if your rate limits are high. ![Test Case Concurrency](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-05/default-test-case-concurrency.png) --- # Changelog | February, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Add Entity to Folder API ------------------------ _February 29th, 2024_ We’ve exposed a new API endpoint to add an existing entity to an existing folder. This is useful if you want to programmatically organize your entities in Vellum. You can find the new endpoint and details on how to invoke it in our [API documentation](https://docs.vellum.ai/api-reference/api-reference/folder-entities/add-entity-to-folder) . Vellum is SOC 2 Type 2 Compliant -------------------------------- _February 28th, 2024_ Vellum is now SOC 2 Type 2 compliant! This means that an independent auditor has verified that Vellum’s information security practices, policies, and procedures meet the SOC 2 standards for security, availability, processing integrity, confidentiality, and privacy. If you’d like to learn more about Vellum’s security practices or request a copy of our SOC 2 report, please reach out to us at [security@vellum.ai](mailto:security@vellum.ai) . Save Workflow Execution from Details Page ----------------------------------------- _February 23rd, 2024_ Previously you were able to save your workflow execution to a test suite or sandbox scenario from the executions table. Now you can do the same from each execution’s details page! Both the “Save As Test Case” and “Save As Scenario” buttons should now appear on the top right of the execution: ![Save Workflow Execution Details](https://storage.googleapis.com/vellum-public/help-docs/save_workflow_execution_detail.png) Workflow Builder UI Settings ---------------------------- _February 21st, 2024_ Have you ever wanted to pan around workflows using the W-A-S-D keys? Looking for more control over your screen real estate? Good news! You can now adjust these settings and more in the new workflow UI settings! Access the settings by clicking the new gear icon in the top right of your workflow builder. ![Workflow Builder Settings](https://storage.googleapis.com/vellum-public/help-docs/workflow-settings.png) Custom Release Tags ------------------- _February 21st, 2024_ You can now manage your Prompt and Workflow release process with greater flexibility and control using Custom Release Tags! Pin your Vellum API requests to tags you define for a given Prompt/Workflow Deployment. These tags can be easily re-assigned within the Vellum app so you can update your production, staging or other custom environment to point to a new version of a prompt or workflow — all without making any code changes! Going forward, new customers of Vellum will no longer see the legacy “Environment” tags in Vellum’s UI. Custom Release Tags are the new, first-class mechanism for managing different releases of the same prompt/workflow in Vellum. We will slowly be deprecating and removing the legacy “Environment” tags for existing customers. Learn more about Managing Releases in our [Help Center article](http://docs.vellum.ai/product/deployments/managing-releases) or watch the video walkthrough below: Better Function Call Display ---------------------------- _February 15th, 2024_ We’ve beautified the display of model function calls in both prompt sandboxes and workflow prompt nodes! Say goodbye to the hard to read and mundane JSON strings. ![Fireworks Function Call Model](https://storage.googleapis.com/vellum-public/help-docs/function-call-display.png) Evaluation Reports ------------------ _February 12th, 2024_ Test Suite Runs have received a big upgrade, and now live in its own tab - Evaluations. You are now able to compare a Prompt or Workflow Variant against a Deployment, and view aggregate Metrics like Median or P90. See a demo of the complete set of updates here: Fireworks Function Calling Model -------------------------------- _February 5nd, 2024_ OpenAI’s GPT models have traditionally led the way in supporting structured data generation through function calling. But late last year Fireworks AI splashed in with their own [function calling model](https://fireworks.ai/blog/fireworks-raises-the-quality-bar-with-function-calling-model-and-api-release) ! This model is now available in Vellum for those interested in an open source alternative to GPT. ![Fireworks Function Call Model](https://storage.googleapis.com/vellum-public/help-docs/fireworks-function-call.png) * * * Cloning Workflow Nodes ---------------------- _February 2nd, 2024_ When you hover over any node in your Workflow editor, you will see a new `Duplicate Node` icon. Clicking on this will create a new copy of a node! Never again will you need to start a node from scratch when you want to just tweak a field or two. ![Clone Nodes](https://storage.googleapis.com/vellum-public/help-docs/clone-nodes.png) * * * Prompt Node Retries ------------------- _February 1st, 2024_ You can now detect when a Prompt Node within a Workflow errors by using a Conditional Node. Using this, you can now build out retry logic around Prompt Nodes within your Workflow! This is useful if you want to catch retryable errors (like rate limit errors from an LLM provider) and try making the call to the LLM again. See a demo of it in action here: --- # Changelog | March, 2025 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Gemini Function Calling Support ------------------------------- **March 31st, 2025** We’ve added support for function calling using Google’s Gemini models. Prompt Comparison/Diffing ------------------------- **March 28th, 2025** You can now compare two different Prompt Variants side-by-side and see a diff of the changes between them. This is useful for understanding the differences made between one version of a Prompt and another. You can access this feature by clicking on the “View Diff” button in the top right of a Prompt Sandbox’s Comparison Mode. ![Prompt Comparison View Diff Button](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/prompt-comparison-view-diff-button.png) This will open a modal with a side-by-side comparison of the two Prompt Variants. ![Prompt Comparison Modal](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/prompt-comparison-view-diff.png) New Workflow Deployment Executions APIs --------------------------------------- **March 26th, 2025** We now have two new apis [List Workflow Deployment Executions](https://docs.vellum.ai/api-reference/workflows/deployments/list-workflow-deployment-event-executions) and [Retrieve Workflow Deployment Execution](https://docs.vellum.ai/api-reference/workflows/deployments/workflow-deployment-event-execution) for listing your Executions for a specific Workflow Deployment, and retrieving any specific Workflow Deployment Execution. Support for Gemini 2.5 Pro Model -------------------------------- **March 25th, 2025** We’ve added support for Gemini 2.5 Pro Experiemental (Version 03-25) with a whopping 1M input token context window and 64k output tokens via [Google’s Gemini API](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-pro) Select all rows in Evaluation Reports ------------------------------------- **March 25th, 2025** You can now select all rows on a page in the Evaluation Report by clicking the checkbox in the header. This selection persists across pages, with the checkbox applying only to the current page. ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/evaluation-table-select-all-rows.png) Workflow Execution Detail Cost Breakdown ---------------------------------------- **March 24th, 2025** We are now displaying the aggregated cost for a Workflow Execution’s inside of our Workflow Execution’s Detail Page. This aggregation is available for all Workflow spans within your Workflow Execution, giving a holistic view of your total execution cost. ![Cost Breakdown](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/workflow_details_cost.png) Deployment Release Reviews -------------------------- Inspired by Github PR reviews, you can now provide reviews on a Prompt or Workflow Deployment Release after it has been deployed. This is helpful if your company mandates a review process for all changes that make it to production. To see Release Reviews in action, check out the demo video below: Support for Llama 3.3 70B via Cerebras -------------------------------------- **March 21st, 2025** We’ve added support for Llama 3.3 70B via [Cerebras AI](https://inference-docs.cerebras.ai/introduction) New Webhook and Datadog Events ------------------------------ **March 20th, 2025** We are releasing 2 new events for Webhooks and Datadog integrations: * `workflow.execution.initiated` * `workflow.execution.fulfilled` These events can be used for better visibility in the execution of a Workflow, such as retrieving the input values used to kick it off, as well as the output values that it produced. You might also use these events to track latency, data drift, and more. Retry and Try Node adornments ----------------------------- **March 22nd, 2025** Error handling and retry logic have both been traditionally quite cumbersome in Vellum. Often, you want to catch errors around a single node and “adorn” it with different error handling behavior. Today we are releasing Retry and Try _Node Adornments_. ![SDK Node Adornments](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/sdk-node-adornments.png) Adornments are accessible in the Node sidepanel after clicking on a Node. They act as Nodes _themselves_, that reference the node it wraps as a single node Subworkflow: * Retry Node Adornments repeatedly invokes the Node until it either succeeds or the max number of attempts are hit. * Try Node Adornments attempts to invoke the Node, and continues with an `Error` output if it fails. Monitoring views for Node Adornment invocations show as if the targeted Node was invoked as a single node Subworkflow: ![Node Adornment Monitoring](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/node-adornment-monitoring.png) Monitoring View Overhaul ------------------------ **March 22nd, 2025** We’re excited to introduce completely revamped Monitoring interfaces for Prompts and Workflows Deployments that bring significant improvements to how you visualize and analyze your AI system performance. ![Revamped Monitoring View](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/grafana-monotoring-revamp.png) Our new Grafana-based implementation loads data faster and handles metric processing more efficiently, making it easier than ever to monitor your AI applications at scale. Key improvements include: * Faster dashboard load times * Improved date range and Release Tag selection * Zoom in on specific time ranges Copy/Paste Prompt Variants -------------------------- **March 20th, 2025** It is now possible to copy a JSON representation of a Vellum prompt to your clipboard and then paste over another Prompt Variant. This is especially useful if you want to go back to a prior history state, copy just one Prompt, and then paste it in your current live draft. ![Copy/Paste Prompt Variants](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/copy-paste-prompt-variants.png) Updated Workflow Node Inputs ---------------------------- **March 20th, 2025** Workflow Nodes used to use a chunky UI for configuring their inputs. We’ve now updated this UI to be simpler by making use of what we call “Expression Inputs.” These Node inputs are functionally the same as before, but tease soon-to-come functionality where you’ll be able to define more complex expressions for what gets passed to a Node. ![Workflow Node Expression Inputs](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/workflow-node-expression-inputs.png) Metric Setup Improvements ------------------------- **March 20th, 2025** We’ve significantly simplified the process of configuring and editing Evaluation Metrics in Vellum. You can now manage your evaluation metrics directly within an evaluation report – adding, editing, and removing metrics – all without leaving the Prompt/Workflow that you’re evaluating. ![Add a Metric](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/add-metric.png) You’ll also find a new Metric Settings button that opens a modal where you can configure your metrics directly ![Metric Settings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/metric-settings.png) Datadog Integration ------------------- **March 20th, 2025** It’s now possible to receive real-time updates about actions taking place in Vellum using Datadog. ![Datadog](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/datadog.png) From the organization settings page, you can configure a Datadog integration with a custom list of events you care about. This is useful if your organization already uses Datadog, or you’d like to leverage Datadog’s monitoring, alerting, and BI capabilities using your Vellum data. All New Workflows are SDK-Enabled --------------------------------- **March 20th, 2025** About two months ago, we began the Beta period of SDK-enabled Workflows. These Workflows use the new [Workflows SDK](https://github.com/vellum-ai/vellum-python-sdks/tree/main/src/vellum/workflows#--vellum-workflows-sdk--) as the underlying engine and run these Workflows in a, secure, more performant, isolated environment. The Workflows SDK enables exciting new functionality including custom nodes, custom docker runtimes, new expression inputs, Node adornments, and so much more. As of today, _all new Workflows_ going forward will be SDK-enabled by default. We expect that all features from the old Workflows engine are supported, with the exception of “Run from Node”, which we hope to reenable later this month. To revert a Workflow that is SDK-enabled back to the legacy Workflows engine, simply toggle off this checkbox from the Workflow settings: ![SDK Compatible Workflow](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-01/toggle-sdk-compatibility.png) If you have an existing Workflow and would like to make it SDK-enabled, you can use the same toggle. JSON Outputs for Prompt Nodes ----------------------------- **March 19th, 2025** Until now in SDK enabled workflows, you only had string or array outputs for Prompt Nodes. With this update, you will now be able to reference a Prompt Node’s JSON output if the `json_mode` or `json_schema` parameter is enabled. This is useful as it enables you to not have to do any additional casting in a Templating Node or Code Execution Node. ![prompt_node_json_output](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/prompt_node_json_output.png) PDF Support for Gemini 2.0 Flash Models --------------------------------------- **March 19th, 2025** The following Gemini 2.0 Flash models now allow for drag-and-drop PDF documents to be used within your Prompts. * Gemini 2.0 Flash Experimental * Gemini 2.0 Flash Experimental Thinking Mode * Gemini 2.0 Flash Automatic Evaluations Setup --------------------------- **March 14th, 2025** Now, when navigating to the Evaluations tab of a Prompt/Workflow for the first time, Vellum will auto-generate an initial Test Suite for you. We’ll automatically create one Test Case per Scenario found in the Sandbox and prepare everything needed for you to add Metrics and Ground Truth data. ![Auto Evaluation Report Initialization](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/auto-eval-report-initialization.png) File Hosting for Images and PDFs -------------------------------- **March 11th, 2025** Until now, when you provided an image to a Prompt in Vellum, it’d either have to be a public URL or the base64 encoded representation of the image. With this update, we now support secure file hosting, such that when you drag-and-drop an image (and also now PDFs!) into a Prompt, Vellum securely hosts the file on your behalf. The end result is that you can now provide much larger images and PDFs to LLMs within Vellum without worrying about a decrease in performance or page load times. PDFs as a Prompt Input ---------------------- **March 11th, 2025** Recently, certain model providers like Anthropic have begun supporting PDFs as native LLM inputs using a special content type called `document` (check out their docs for details [here](https://docs.anthropic.com/en/docs/build-with-claude/pdf-support) . This is similar to how you might provide a multi-modal model with an image as an input, but now you can provide a PDF as well. Vellum now also supports passing PDFs as inputs to a Prompt for models that support it. You can do this by drag-and-dropping a PDF file into a Chat History variable in a Prompt. The mechanics are very similar to how you might work with images (see details [here](https://docs.vellum.ai/product/prompts/images) ). This is particularly useful for data extraction tasks, where you might want to extract structured data from a PDF and then use that data to power some downstream process. Support for Qwen QwQ Models via Groq ------------------------------------ **March 11th, 2025** We’ve added support for a variety of Qwen’s newest [QwQ 32B](https://console.groq.com/docs/models#preview-models) models via Groq’s preview models. We’ve added the following models: * QwQ 32B * QwQ 2.5 Coder 32B * QWQ 2.5 32B Support for Qwen QwQ 32B via Fireworks AI ----------------------------------------- **March 11th, 2025** We’ve added support for Qwen’s newest [QwQ 32B](https://fireworks.ai/models/fireworks/qwq-32b) model via Fireworks AI. Webhooks -------- **March 10th, 2025** It’s now possible to receive real-time updates about actions taking place in Vellum using Webhooks. ![Webhooks](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/webhooks.png) From the organization settings page, you can configure a webhook endpoint with a custom list of events you care about. You can further customize it with your own auth configurations. This is useful if you’d like to store Vellum monitoring data in your own external data stores. For example, you might save events to a Data Warehouse to power a custom health dashboard. Keep an eye out, as more event types will be added soon! Workflow Deployment Executions - Cost Column -------------------------------------------- **March 7th, 2025** You can now see the total cost per Workflow Execution for a given Workflow Deployment in its Executions table. This toggle can be shown/hidden via the “columns” menu. ![Cost Column](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-03/cost_column.png) This is useful for getting a sense of how much a given AI use-case costs to support. This column will be populated for new Workflow Executions going forward and sums the costs associated with all Prompt invocations within the Workflow’s execution (included nested invocations within Subworkflow Nodes, Map Nodes, etc.). We’ll be exposing more cost metrics throughout Vellum in the coming weeks. Stay tuned! Prompt Sandbox Pagination ------------------------- **March 5th, 2025** We’ve added pagination to the Prompt Sandbox page. Now, when you have a large number of Scenarios in a Prompt Sandbox, they’ll be split across multiple pages. You can navigate between pages using the pagination controls at the bottom ![Prompt Sandbox Pagination](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/prompt-sandbox-scenario-pagination.png) This should result in performance improvements for those with large Prompt Sandboxes. Global Search ------------- **March 3rd, 2025** We’ve added an eagerly-awaited-for and long-overdue feature to Vellum – Global Search 🎉 You can now search across all your Prompts, Workflows, Document Indexes, and more using the new Search bar in the Vellum side nav. ![Global Search Side Nav](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/global-search-side-nav.png) Doing so will pull up a search bar where you can search for any resource in your Workspace. You can directly navigate to the resource from the search results. ![Global Search Omnibox](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2025-02/global-search-omni-bar.png) You can also access Global Search from any page through the keyboard shortcut `Cmd/Ctrl + K`. Give it a try and let us know what you think! --- # Changelog | April, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Support for Gemini 1.5 Pro -------------------------- _April 30th, 2024_ Gemini 1.5 Pro is now available in Vellum. You can add it to your workspace through the [models page](https://app.vellum.ai/models) . Improved Monitoring on Workflow Deployments ------------------------------------------- _April 30th, 2024_ We’ve added new functionality to the monitoring tab on workflow deployments. It’s now possible to see a breakdown of executions by the release tag used, and further filter down based on a specific release tag. ![Release Tag Monitoring](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/release-tag-monitoring.png) Reusable Metrics ---------------- _April 30th, 2024_ Introducing Reusable Metrics! Metrics can now be shared across your Test Suites making it easier for you to consistently test and evaluate your Prompt / Workflow quality. Define a suite of Custom Metrics tailored to your business logic and use-case to save time and ensure standardized evaluation criteria. ![Metric Definition](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/Metric-Definition-Changelog.png) Prompt Blocks ------------- _April 30th, 2024_ Prompts can now be be broken down into multiple sections and organized using “blocks.” Prompt blocks can be reordered, and toggled on or off. Splitting your Prompt into multiple blocks can make it easier to navigate complex Prompts and help you focus on iterating on specific sections. Check out the demo below to see how it works! Filtering Executions on Release Tags ------------------------------------ _April 29th, 2024_ It’s now possible to filter workflow deployment executions by the release tag used when executing the workflow. This can be very useful for monitoring differences between releases of a deployment. Are you still using an older release in production? Are executions of your new release behaving as expected? ![Execution Release Tags](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-04/execution-release-tags.png) Faster Queries on Workflow Deployment Executions ------------------------------------------------ _April 26th, 2024_ The executions tab of the workflow deployments page now fetches historical executions much faster. This tab is a great way to see how your customers are actually using your deployments. In our test for deployments with over 200k executions, data now loads in under 4 seconds instead of the previous 15+ seconds - a 4x speed improvement. Support for Evaluating External Functions ----------------------------------------- _April 25th, 2024_ Vellum’s Evaluation framework can now be used to test arbitrary functions defined in your codebase – not just Prompts and Workflows managed by Vellum. For example, you might test a prompt chain that lives in your codebase and that’s defined using another third party library. This can be particularly useful if you want to incrementally migrate to Vellum Prompts/Workflows, but ensure that the outputs remain consistent. For a detailed example of how to use Vellum’s evaluation framework to test external functions, see the [python example here](https://github.com/vellum-ai/vellum-python-sdks/blob/main/examples/Running%20a%20Test%20Suite%20on%20an%20External%20Function.ipynb) . Fireworks Finetuned Models -------------------------- _April 24th, 2024_ Vellum now supports models that you’ve fine-tuned on [Fireworks AI](https://fireworks.ai/) . You can add your fine-tuned Fireworks model by navigating to the [Models page](https://app.vellum.ai/models) and clicking on the featured model template at the top. ![Fireworks Model Template](https://storage.googleapis.com/vellum-public/help-docs/fireworks_model_template.png) Note that only the Mistral family of models are supported currently. If there are other base models that you would like to see supported, please reach out to us! Updated Prompt UI ----------------- _April 23rd, 2024_ We’ve updated the prompt editing UI throughout Vellum. You’ll see the new look in the Prompt Editor, Comparison Mode, Chat Mode, Prompt Nodes in Workflows, and Deployment Overviews. This is the first in a series of exciting improvements to the prompt editing experience that will be rolling out over the coming weeks and months. ![New Prompt Block UI](https://storage.googleapis.com/vellum-public/help-docs/new_prompt_block_ui.png) New Upsert Prompt Sandbox Scenario API -------------------------------------- _April 23rd, 2024_ The API for upserting a Prompt Sandbox Scenario now requests and responds with schemas that are more consistent with other Vellum APIs, using discriminated unions for improved type safety. This API is available on version `0.4.0` of our SDKs. You can find the API documentation for it [here](https://docs.vellum.ai/api-reference/api-reference/sandboxes/upsert-sandbox-scenario) . Function Call Input in Test Cases --------------------------------- _April 23rd, 2024_ Workflows support Function Call values as a valid output type. Because these function calls often come from models, it is valuable to have evaluations on these workflows that ensure that the function call output is what we expect. Test suites in Vellum now support specifying test case input and evaluation values. ![Test Case Function Call](https://storage.googleapis.com/vellum-public/help-docs/function-call-test-cases.png) Support for Additional Models ----------------------------- _April 19th, 2024_ The following models are now available in Vellum: * Llama-3-70B-Instruct * Llama-3-8B-Instruct * Mixtral-8x22B-Instruct-v0.1 They can be added to your workspace through the [models page](https://app.vellum.ai/models) . Claude 3 Opus Prompt Generators ------------------------------- _April 18th, 2024_ If you’ve been using GPT models, you’ve likely relied on prompt engineering tips that worked well for those models. But when you apply the same prompts to Claude 3 Opus, you might notice they don’t perform as expected. This happens because Claude 3 Opus is trained using different methods and data, so the way you prompt it differs from how you would prompt GPT-4. We have some helpful [tips in our guide](https://www.vellum.ai/blog/prompt-engineering-tips-for-claude) , but as of today, you can convert your prompts even faster… ### GPT-4 to Claude 3 Opus Prompts We’ve released a free tool for that allows you to paste your GPT-4 prompt and get an adapted Claude 3 Opus prompt with suggestions for dynamic variables. You can [try the tool here](https://tools.vellum.ai/gpt4-to-claude-opus) . ![GPT-4 to Claude 3 Opus](https://storage.googleapis.com/vellum-public/help-docs/gpt-4-to-claude-3-opus-prompt-converter-tool.png) ### Claude 3 Opus Prompt Generator If you don’t have a working GPT-4 prompt but need to create a prompt for Claude 3 Opus from scratch, you can use our second new free tool – “Claude Prompt Generator.” This generator lets you input your `'prompt objective'` and creates a suitable prompt for Claude 3 Opus, with suggestions for dynamic variables that you should include. You can [try the tool here](https://tools.vellum.ai/opus-prompt-generator) . Max Tokens Warning ------------------ _April 10th, 2024_ When iterating on a Prompt in Vellum’s Prompt Sandbox, you may find that its output stops mid-sentence. This is often because the “Max Tokens” parameter is set too low, or the prompt itself is too long. To help you identify when this is the case, we’ve added a warning that will appear when this max is hit. ![Max Tokens Warning](https://storage.googleapis.com/vellum-public/help-docs/max-tokens-warning.png) GPT-4 Turbo 04/09/2024 Model ---------------------------- _April 9th, 2024_ OpenAI’s newest GPT-4 Turbo model `gpt-4-turbo-2024-04-09` is now available in Vellum! Usage Tracking in Prompt Sandbox and Prompt API ----------------------------------------------- _April 9th, 2024_ We have added the ability for you to track model host usage from the `execute-prompt` [API](https://docs.vellum.ai/api-reference/api-reference/execute-prompt#request.body.expand_meta.usage) . This API update is available on version `0.3.21` of our SDKs. You can also now view model host usage in the Prompt Sandbox by enabling the “Track Usage” toggle in your Prompt Sandbox’s settings. ![Usage Tracking Sandbox](https://storage.googleapis.com/vellum-public/help-docs/usage-tracking-sandbox.png) New API for Listing a Test Suite’s Test Cases --------------------------------------------- _April 8th, 2024_ We have a new [API](https://docs.vellum.ai/api-reference/api-reference/test-suites/list-test-suite-test-cases) available in beta for listing the Test Cases belonging to a Test Suite at `GET /v1/test-suites/{id}/test-cases`. This API is available on version `0.3.20` of our SDKs. Prompt Editor ------------- _April 5th, 2024_ Prompt Sandboxes have an entirely new view mode: Prompt Editor. It’s a dedicated space for iterating on a single Variant and Scenario. All of the features you need to work quickly are easily accessible, and collapsible sections make it simple to free up screen space. There are even more improved experiences and exciting coming down the pike for Prompt Editor, and many of those improvements will make their way into Comparison and Chat Modes, as well. Copy and Paste Logit Bias ------------------------- _April 5th, 2024_ You can now copy logit bias parameters from one Prompt Variant and paste them into another Prompt. This works in both Prompt Sandboxes and Prompt Nodes within Workflows. ![Logit Bias Copy](https://storage.googleapis.com/vellum-public/help-docs/logit-bias-copy.png) Test Suite Improvements ----------------------- _April 4th, 2024_ We’ve made some changes to our Test Suite UX. Here’s what’s new: * **Simplified Creation Process**: We’ve broken down the test suite creation into clear, manageable steps, ensuring a more guided and less overwhelming setup. * **In-Context Editing**: You can now edit test suites directly from the Prompt or Workflow evaluations page via a new, sleek modal. * **Enhanced Error Messaging**: We’ve revamped our error messages to be clearer and more actionable. You’ll now receive specific feedback that pinpoints exactly where things went wrong. ![Test Suite Improvements](https://storage.googleapis.com/vellum-public/help-docs/test-suite-error.png) New APIs for Accessing Test Suite Runs -------------------------------------- _April 3rd, 2024_ We have two new [APIs](https://docs.vellum.ai/api-reference/api-reference/test-suite-runs/retrieve) available in beta for accessing your Test Suite Runs: * A Retrieve endpoint to fetch metadata about the test suite run like it’s current state at `GET /v1/test_suite_runs/{id}` * A List executions endpoint to fetch the results of the test suite run at `GET /v1/test_suite_runs/{id}/executions` These APIs are available on version `0.3.15` of our SDKs. --- # Changelog | March, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Configurable Chunk Settings for Document Indexes ------------------------------------------------ _March 26th, 2024_ We’ve added the ability to configure the chunk size and the overlap between consecutive chunks for Document Indexes. You can find it under the “Advanced” section when creating or cloning a Document Index. ![Document Index Chunk Settings](https://storage.googleapis.com/vellum-public/help-docs/document-index-chunk-settings.png) Workflow Template Node Debugging -------------------------------- _March 26th, 2024_ There’s a new debugging feature for iterating on Workflow Template Nodes. You can click the new “Test” button in the full-screen editor and test your template without having to run the whole Workflow. Then you can further iterate on your template by modifying your test data in the “Test Data” tab. ![Workflow Template Debugger](https://storage.googleapis.com/vellum-public/help-docs/template-node-debugger-arrows.png) Workflow Node Search -------------------- _March 26th, 2024_ We have added a new Workflow node search feature to help you find your way in large and complex Workflows. Click the new search icon in the top right to quickly find the node you are looking for, or use the `⌘ + shift + F` shortcut (`ctrl + shift + F` on windows). ![Workflow Node Search](https://storage.googleapis.com/vellum-public/help-docs/workflow-search-for-node.png) Workflow Node Mocking --------------------- _March 26th, 2024_ While iteratively developing a Workflow in Vellum, you often want to focus on improving a specific branch or node. It can be cumbersome to re-run the entire Workflow just to test the part you’re iterating on, especially if you already know what the upstream nodes are going to output. To help address this, Vellum now supports _node mocking_. You can now mock out a Prompt or Subworkflow Node such that its execution is skipped and a hard-coded value is returned. This can help you dramatically speed up your Workflow development since you no longer have to wait for early Prompt Nodes to complete. This has the added benefit of saving the expense of tokens with LLM providers! For more information on Workflow Node Mocking, visit our new [help center page](https://docs.vellum.ai/product/workflows/experimentation#node-mocking) . ![Workflow Node Mocking](https://storage.googleapis.com/vellum-public/help-docs/workflow-node-mocking.png) Claude 3 and Mistral on Bedrock ------------------------------- _March 23rd, 2024_ We now support both of the Claude 3 and both of the Mistral models on AWS Bedrock. Add these models to your workspace by heading to the [models](https://app.vellum.ai/models) page and searching for the one you need from the search bar. ![Bedrock Models](https://storage.googleapis.com/vellum-public/help-docs/claude3-mistral-bedrock.png) Navigation Updates ------------------ _March 22nd, 2024_ We’ve made some significant changes to Vellum’s navigation UI. The app sidebar has been reorganized with the goal of making it easier to navigate between a Prompt/Workflow’s Sandbox, Evaluations, and Deployments. You’ll find that after you’ve clicked on a “Prompt” or “Workflow,” there’s an integrated submenu within the navigation sidebar that shows “Sandbox,” “Evaluations,” and “Deployments.” Additionally, you’ll find that some nav items, such as “Deployments”, “Models,” “API Keys,” “Organization,” and “Profile” have been grouped into the new “More” and “Settings” nav items. ![Navigation Updates](https://storage.googleapis.com/vellum-public/help-docs/nav-updates.png) Read-only Workflow Diagrams --------------------------- _March 22th, 2024_ You can now see a read-only view of workflow diagrams for Workflow Deployment Executions, Workflow Test Case Executions, and Workflow Releases. You can access the diagram by clicking the “Graph View” icon tab on the top right. This is particularly helpful if you want to visualize what your Workflow looked like at that time, as well as visualize the execution path your Workflow took. ![Workflow Deployment Execution](https://storage.googleapis.com/vellum-public/help-docs/workflow-deployment-execution-diagram.png) Test Suite Table Updates ------------------------ _March 21th, 2024_ The Test Cases table on the Test Suites page has been updated to use the same new styling and functionality as the Test Cases table that you’ll find when viewing a Prompt/Workflow Evaluation Report. With this, adding, editing, and deleting Test Cases is generally more reliable. Additionally, special variables types, like Chat History, have an improved display are are no longer displayed as raw JSON. ![Test Suite Table Updates](https://storage.googleapis.com/vellum-public/help-docs/test_suite_table.png) Additional Headers on API Nodes ------------------------------- _March 20th, 2024_ Previously, API Nodes only accepted one configurable header, defined on the `Authorization` section on the node. You can now configure additional headers in the new advanced `Settings` section. Header values could be regular `STRING` values or Secrets, and any headers defined here would override the Authorization header. ![API Node Headers](https://storage.googleapis.com/vellum-public/help-docs/api-node-headers.png) Indicators for Deployed Prompt/Workflow Sandboxes ------------------------------------------------- _March 19th, 2024_ You can now tell at a glance whether a given Prompt/Workflow Sandbox has been deployed. You can also hover over the tag to see when it was last deployed. ![Sandbox Deployment Tag](https://storage.googleapis.com/vellum-public/help-docs/sandbox-deployment-tag.png) Cancellable Workflow Deployment Executions ------------------------------------------ _March 18th, 2024_ You can now cancel running Workflow Deployment Executions. Simply click the cancel button on the Workflow Execution details page. ![Cancellable Workflows](https://storage.googleapis.com/vellum-public/help-docs/cancel-workflow-deployments.png) Code Execution Metric Debugging ------------------------------- _March 18th, 2024_ There’s a new debugging feature for iterating on custom Code Metrics. You can click the new “Test” button and test your code without having to run the whole test suite. You can update the example data that’s passed into your Code Metric by going to the “Test Data” tab. ![Workflow Code Execution Debugger](https://storage.googleapis.com/vellum-public/help-docs/code-eval-metric-debugger.png) Workflow Details for Workflow Evaluations ----------------------------------------- _March 18th, 2024_ You can now view Workflow Execution details from the Workflow Evaluations table! To view the details, click on the new “View Workflow Details” button located within a test case’s value cell. ![Workflow Executions2](https://storage.googleapis.com/vellum-public/help-docs/workflow-evaluation-details2.png) ![Workflow Executions1](https://storage.googleapis.com/vellum-public/help-docs/workflow-evaluation-details.png) Subworkflow Nodes ----------------- _March 14th, 2024_ Are your Workflows becoming giant and unwieldy? Wish you could define composable groups of nodes to be used across Workflows? We’re excited to introduce the latest node type in the Workflows node picker - Subworkflow Nodes! With Subworkflow Nodes, you can now link directly to deployed Workflows to reuse commonly grouped nodes and execution logic. Subworkflow Nodes also supports release tagging, giving users the flexibility to either pin to a specific version (say, `production`) or always automatically update with `LATEST`. ![Subworkflow Nodes](https://storage.googleapis.com/vellum-public/help-docs/subworkflow-nodes-ga.png) For more details, check out our [new help center doc](https://docs.vellum.ai/product/workflows/nodes/subworkflow-node) . Image Support in the UI ----------------------- _March 13th, 2024_ Image support is LIVE in the Vellum UI for OpenAI’s GPT-4 Turbo with Vision! Vellum API’s have had image support for a while and now you can add images in your Prompt and Workflow Sandbox scenarios! ![Image Support in Vellum UI](https://storage.googleapis.com/vellum-public/help-docs/images-in-prompts-walkthrough.gif) For more details on supported image formats and working with OpenAI’s vision models in Vellum, check out our [new help center doc](https://docs.vellum.ai/product/prompts/images) . Workflow Node Input Value Display --------------------------------- _March 11th, 2024_ You can now view a Node’s input values directly from the Workflow Editor! This makes it easier to understand what data is being passed into a Node and to debug issues. ![Workflow Node Input Value Display](https://storage.googleapis.com/vellum-public/help-docs/workflow-node-inputs.gif) Inline Editing for Evaluations ------------------------------ _March 11th, 2024_ You can now edit test cases directly from the “Evaluations” tab in Workflows and Prompts! The new editing interface makes it easier than ever to make changes to test cases with long variable values, allows you to edit Chat History values with the same drag-and-drop editor you use elsewhere in the app, and adds support for formatted editing of JSON. We’re continuing to add support for more variable types and will soon be applying this new edit flow to other tables throughout the app. Workflow Node ‘Reject on Error’ Toggle -------------------------------------- _March 9th, 2024_ Previously, if a Node in a Workflow errored, the Workflow would proceed to execute until another downstream Node tried to use the output of the Node that errored and would only then terminate. This made Workflows hard to debug and put the onus on you to implement error handling. Going forward, by default, Workflows will immediately terminate if a Node errors. There are still cases in which you might want to continue despite a Node error (e.g. implementing your own error handling or retry logic). In this cases, you can disable the new “Reject on Error” toggle. ![Workflow Node Reject on Error](https://storage.googleapis.com/vellum-public/help-docs/node-error-toggle.png) Historical Workflow Nodes have this toggle disabled so that there’s no change in behavior. New Nodes going forward will have this toggle enabled by default. Workflow Code Execution Node Debugging -------------------------------------- _March 8th, 2024_ We have introduced a new debugging feature for workflow code execution nodes! You can click the new “Test” button in the full-screen editor and test your code without having to run the whole workflow! Then you can further iterate on your code by modifying your test data in the “Test Data” tab. ![Workflow Code Execution Debugger](https://storage.googleapis.com/vellum-public/help-docs/code-node-debugger.png) List Document Indexes API ------------------------- _March 7th, 2024_ We’ve exposed a new API endpoint to list all the Document Indexes in a Workspace. You can find the details of the API [here](https://docs.vellum.ai/api-reference/api-reference/document-indexes/list) . In-Progress Workflows Executions Now Available in Monitoring ------------------------------------------------------------ _March 6th, 2024_ You previously had to wait for a workflow to fully resolve before seeing it in the Workflow Executions table. We now start publishing executions as soon as Workflows are initiated! This allows users building complex Workflows to see any that are still in progress: ![In Progress Workflow Executions Table](https://storage.googleapis.com/vellum-public/help-docs/inprogress-executions-table.png) We also updated the Workflow Execution Details page to also reflect in progress workflows: ![In Progress Workflow Execution Details](https://storage.googleapis.com/vellum-public/help-docs/inprogress-execution-details.png) Expand Scenario in Prompt Sandbox --------------------------------- _March 6th, 2024_ Looking for more room to edit your scenarios in the prompt sandbox? We’ve just added an expand scenario modal! You can now easily make changes to scenarios with longer inputs. ![Expand Scenario Modal](https://storage.googleapis.com/vellum-public/help-docs/expand-scenario-modal.png) Code Execution Logs ------------------- _March 6th, 2024_ You can now use `print` or `console.log` statements in code execution nodes and view the logs by looking at a node’s result and clicking the logs tab. ![Code Logs Exec Nodes](https://storage.googleapis.com/vellum-public/help-docs/code-exec-logs.png) We’ve also added logs for Metrics. You can view them by enabling the logs column in the table columns settings. ![Code Logs Eval](https://storage.googleapis.com/vellum-public/help-docs/code-exec-metric-logs.png) Code Execution Improvements --------------------------- _March 6th, 2024_ We’ve made some huge improvements to code execution! You can now include custom packages for Code Execution Workflow nodes and Code Execution Metrics. On top of this, we have added support for TypeScript. You can select the programming language you want from the new “Runtime” dropdown. We have also introduced a few smaller improvements: * The maximum size for code input values has been increased to 10mb, a significant leap from the previous cap of 128k characters * The layout of the workflow code execution node editor has been revamped with a new side by side layout * All Vellum input types are now supported for code execution node input variables * Line numbers in the code editor will no longer be squished together ![Code Execution Improvements](https://storage.googleapis.com/vellum-public/help-docs/code-execution-improvements.png) Claude 3 -------- _March 5th, 2024_ Anthropic’s two newest models, Claude 3 Opus and Claude 3 Sonnet, are now both available in Vellum! These models have been added to all workspaces so they should be selectable from prompt sandboxes upon refresh. ![Claude 3](https://storage.googleapis.com/vellum-public/help-docs/claude-3-models.png) In-App Support Now Accessed via “Get Help” Button ------------------------------------------------- _March 5th, 2024_ It used to be that the In-App Support Widget we showed in the bottom right corner of the screen would get in the way of other actions like Save buttons. Now, that widget is hidden by default and you can open it by clicking the “Get Help” button in the side navigation. Once opened, we also now display bookmarked links to useful resources like the Vellum Help Docs. ![Get Help Button Opens Chat Widget](https://storage.googleapis.com/vellum-public/help-docs/hide-chat-widget-by-default.gif) Workflow Error Nodes -------------------- _March 4th, 2024_ It’s now possible to terminal a Workflow and raise an error through the use of Error Nodes. You can either re-raise an error from an upstream node, or construct and raise a custom error message. * * * Retrieve Workflow Deployment API -------------------------------- _March 1st, 2024_ We’ve exposed a new API endpoint to retrieve details of a Workflow Deployment. This is useful if you want to do things like programmatically detect if a Workflow Deployment with a specific name exists, or has the inputs/outputs you expect. You can find the details of the API [here](https://docs.vellum.ai/api-reference/api-reference/workflow-deployments/retrieve) . --- # Changelog | October, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. AWS Bedrock Support for Anthropic’s Claude 3.5 Sonnet ----------------------------------------------------- **October 31st, 2024** We’ve added support for Anthropic’s Claude-3-5-sonnet-20241022-v2:0 model on AWS Bedrock. [AWS Bedrock Support](https://docs.anthropic.com/en/api/claude-on-amazon-bedrock) Google Cloud Vertex AI Support for Anthropic’s Claude 3.5 Sonnet ---------------------------------------------------------------- **October 31st, 2024** We’ve added support for Anthropic’s Claude-3-5-sonnet-20241022-v2:0 model on Google Cloud Vertex AI. [Google Cloud Vertex AI Support](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-5-sonnet-v2?project=vocify-prod) Retrieve Workspace Secret or Update Workspace Secret ---------------------------------------------------- **October 31st, 2024** We’ve added two new API endpoints for retrieving a Workspace Secret and updating a Workspace Secret. * For retrieving a Workspace Secret, check out our [GET API here](https://docs.vellum.ai/changelog/2024/api-reference/secrets/retrieve-workspace-secret) . * For updating a Workspace Secret, check out our [PATCH API here](https://docs.vellum.ai/changelog/2024/api-reference/secrets/update-workspace-secret) . This API is available in our SDKs beginning with version 0.8.30. Prompt Timeout Enabled for Prompt Deployments --------------------------------------------- **October 29th, 2024** You can now set timeouts for Prompt Deployments. With this, you can ensure that any Prompt Execution will timeout if it lasts longer than the specified amount of time. To set a timeout for a Prompt Deployment, navigate to the “Parameters” section within the Prompt Sandbox and scroll down to toggle the “Timeout” setting on. You can then set the timeout duration in seconds. After deploying your Prompt, your Prompt Deployment will respect your configured timeout. ![Visit Prompt Parameters to set Timeout](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/step1-prompt-timeout.png) ![Set Timeout Duration](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/step2-prompt-deployment-timeout.png) Reorder Test Suite Variables ---------------------------- **October 24th, 2024** You can now reorder Input and Evaluation Variables within a Test Suite’s settings page. Drag and drop the variables into the order you prefer. This new order will automatically be reflected in your Evaluation Reports. ![Reorder Variables in Evaluation Report](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/reorder-variables-input.png) Support for Perplexity AI’s Online Models ----------------------------------------- **October 24th, 2024** We’ve added support for Perplexity AI’s newest Sonar Online models, which provide real-time web search capabilities integrated directly into the language model. Check out Perplexity AI’s [online models](https://docs.perplexity.ai/guides/model-cards#perplexity-sonar-models) here: The supported models are: * LLama 3.1 Sonar Small 128k Online * LLama 3.1 Sonar Large 128k Online * LLama 3.1 Sonar Huge 128k Online These models offer several key features: 1. **Real-time web search**: The models can perform live internet searches to retrieve up-to-date information. 2. **Contextual understanding**: They can interpret search results in the context of the user’s query. 3. **Source citation**: The models provide citations for information sourced from the web. 4. **Multilingual support**: They can understand and generate content in multiple languages. 5. **Long-context understanding**: The models can handle extended conversations and complex queries. To use these models in Vellum, simply select the appropriate Sonar Online model when configuring your Prompt or Workflow. The model will automatically perform web searches when needed to supplement its knowledge and provide the most current and relevant information. Note: Using these models may result in slightly longer processing times due to the real-time web search functionality, but they offer significantly enhanced capabilities for tasks requiring up-to-date information. Support for Perplexity AI Models -------------------------------- **October 24th, 2024** We’ve added support for [Perplexity AI](https://docs.perplexity.ai/api-reference/chat-completions) as one of our newest model hosts! Along with the launch of the Perplexity AI integration, we’ve added the following models: * [Perplexity AI: LLama 3.1 Sonar Small 128k Chat](https://docs.perplexity.ai/guides/model-cards#perplexity-chat-models) * [Perplexity AI: LLama 3.1 Sonar Large 128k Chat](https://docs.perplexity.ai/guides/model-cards#perplexity-chat-models) * [Perplexity AI: LLama 3.1 8B Instruct](https://docs.perplexity.ai/guides/model-cards#open-source-models) * [Perplexity AI: LLama 3.1 70B Instruct](https://docs.perplexity.ai/guides/model-cards#open-source-models) ![Perplexity AI Models](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/perplexity_chat_models.png) Support for LLama 3.1 Lumimaid 70B and Magnum v4 72B models on OpenRouter ------------------------------------------------------------------------- **October 24th, 2024** We’ve added support for the [LLama 3.1 Lumimaid 70B](https://openrouter.ai/neversleep/llama-3.1-lumimaid-70b) and [Magnum v4 72B](https://openrouter.ai/anthracite-org/magnum-v4-72b) models on OpenRouter! Support for Gemini 1.5 Flash 8B Model ------------------------------------- **October 23nd, 2024** In addition to the existing support for the Gemini 1.5 models, we’ve added support for the Gemini 1.5 Flash 8B model to Vellum! * [Gemini 1.5 Flash 8B](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash-8b) Claude 3.5 Sonnet 2024-10-22 Live! ---------------------------------- **October 22nd, 2024** We’ve added support for Anthropic’s latest 10/22/2024 snapshot of [Claude 3.5 Sonnet](https://docs.anthropic.com/en/docs/about-claude/models) to Vellum! ![Claude 3.5 Sonnet](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/claude-3-5-20241022-support.png) Support for Cerebras-AI Models ------------------------------ **October 22nd, 2024** We’ve added support for [Cerebras-AI](https://inference-docs.cerebras.ai/introduction) to Vellum! Along with the launch of the Cerebras-AI API, we’ve added the following models: * [Cerebras-AI: llama3.1-8b](https://inference-docs.cerebras.ai/api-reference/models#models) * [Cerebras-AI: llama3.1-70b](https://inference-docs.cerebras.ai/api-reference/models#models) ![Cerebras-AI Models](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/cerebras-support.png) Configurable Prompt Node Timeouts --------------------------------- **October 22nd, 2024** You can now set a max timeout for Prompt Nodes within Workflows. With this, you can ensure that no one LLM invocation will run for too long and slow down the Workflow overall and instead, fail early if it does. To set a timeout for a Prompt Node, simply navigate to the new “Settings” section and toggle the “Timeout” setting on. You can then set the timeout duration in seconds. ![Prompt Settings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/prompt-settings-timeout.png) New API for Listing Entities in a Folder ---------------------------------------- **October 20th, 2024** We now have a new API endpoint for listing all entities in a folder. This endpoint allows you to retrieve all entities in a folder, including subfolders, with a single API call. You can use this endpoint to quickly get a list of all entities in a folder with high-level metadata about them. For details, check out our [API Reference here](https://docs.vellum.ai/api-reference/folders/list-folder-entities) . This API is available in our SDKs beginning version 0.8.25. Datadog and Webhook Logging Beta Integrations --------------------------------------------- **October 15th, 2024** Logs for your Prompts, Workflows and Documents can now be streamed to Datadog and external Webhooks. This is useful if you want deeper insight into key events that happen in Vellum in your external systems. For example, you might set up a Datadog alert that fires when there are multiple subsequent failures when executing a Workflow Deployment. These integrations are currently in beta. If you’d like to participate in the beta period and want help setting up the integration, please contact Vellum Support. Eva Qwen and Rocinante Added to OpenRouter Integration ------------------------------------------------------ **October 13th, 2024** We’ve added 2 additional new models to Vellum via our OpenRouter integration! 1. [Eva Qwen 2.5 14B](https://openrouter.ai/eva-unit-01/eva-qwen-2.5-14b) - A powerful model based on the Qwen architecture. 2. [Rocinante 12B](https://openrouter.ai/thedrummer/rocinante-12b) - A versatile 12 billion parameter model. Vertex AI Embedding Model Support --------------------------------- **October 15th, 2024** We’re excited to announce the addition of the Vertex AI embedding models `text-embedding-004` and `text-multilingual-embedding-002` to Vellum! These models can be selected when creating a Document Index. ![Vertex AI Embeddings](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/vertex-ai-embeddings.png) New Models Added to OpenRouter Integration ------------------------------------------ **October 11th, 2024** We now have the addition of 8 new models integrated into Vellum via our OpenRouter integration: 1. [Magnum v2 72B](https://openrouter.ai/anthracite-org/magnum-v2-72b) - A powerful model designed to achieve prose quality similar to Claude 3 models. 2. [Nous: Hermes 3 405B Instruct](https://openrouter.ai/nousresearch/hermes-3-llama-3.1-405b) - A frontier-level, full-parameter finetune of the Llama-3.1 405B foundation model. 3. [NousResearch: Hermes 2 Pro - Llama-3 8B](https://openrouter.ai/nousresearch/hermes-2-pro-llama-3-8b) - An upgraded version of Nous Hermes 2 with improved capabilities. 4. [Nous: Hermes 3 405B Instruct (extended)](https://openrouter.ai/nousresearch/hermes-3-llama-3.1-405b:extended) - An extended context version of Hermes 3 405B Instruct. 5. [Goliath 120B](https://openrouter.ai/alpindale/goliath-120b/api) - A large LLM created by combining two fine-tuned Llama 70B models. 6. [Dolphin 2.9.2 Mixtral 8x22B](https://openrouter.ai/cognitivecomputations/dolphin-mixtral-8x22b/api) - An uncensored model designed for instruction following, conversation, and coding. 7. [Anthropic: Claude 3.5 Sonnet (self-moderated)](https://openrouter.ai/anthropic/claude-3.5-sonnet:beta/api) - A faster, self-moderated endpoint of Claude 3.5 Sonnet. 8. [Liquid: LFM 40B MoE](https://openrouter.ai/liquid/lfm-40b/api) - A 40.3B Mixture of Experts (MoE) model for general-purpose AI tasks. These new models offer a wide range of capabilities, from improved prose quality and instruction following to extended context lengths and specialized tasks like coding. Users can now leverage these models in their Vellum projects, expanding the possibilities for AI-powered applications. Workflow Edge Type Improvements ------------------------------- **October 10th, 2024** In the past, it could be quite difficult to achieve a perfectly straight line between two Nodes in a Workflow with the “smooth-step” edge type, but those days are behind us, friends. You’ll now see that your edges will automagically snap into straight-line connectors whenever they’re close-to-horizontal. AutoLayout and AutoConnect for Workflows ---------------------------------------- **October 10th, 2024** Two exciting new features have been added to Workflows — AutoLayout and AutoConnect. AutoLayout allows you to instantly organize your workflow via algorithm, making it easier than ever to tame even the most-unruly of Workflows. AutoConnect will automatically connect any unconnected Nodes in your Workflow by creating edges from left to right (more-or-less). Both of these features are accessible via new buttons in the bottom left toolbar in your Workflow Sandboxes. In the event that you only want to use AutoConnect or AutoLayout on a specific subset of Nodes, simply drag to select and you’ll see a new temporary toolbar that allows you to do just that. Reorder Entities in Evaluation Reports -------------------------------------- **October 9th, 2024** You can now reorder entities in the Evaluation Report table. Simply select the “Reorder” option in the entity column’s menu to adjust the order to your preference. ![Evaluation Report Entity Reorder](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/evaluation-report-entity-reorder.png) Online Evaluations for Workflow and Prompt Deployments ------------------------------------------------------ **October 3rd, 2024** We’re excited to announce the launch of [Online Evaluations for Workflow and Prompt Deployments](https://docs.vellum.ai/product/evaluation/online-evaluations) ! This new feature allows you to configure Metrics for your Deployments to be evaluated in real-time as they’re executed. Key highlights include: * **Continuous Assessment**: Automatically evaluate the quality of your deployed LLM applications as they handle live requests. * **Flexible Configuration**: Set up multiple Metrics to assess different aspects of your Deployment’s performance. * **Easy Access to Results**: View evaluation results directly in the execution details of your Deployments. It works by configuring Metrics for your Workflow or Prompt Deployment in the new “Metrics” tab. ![Configure Metrics for use in Online Evals](https://storage.googleapis.com/vellum-public/help-docs/online-evals/online-evals-metric-config.png) Once configured, every execution of your Deployment will be evaluated against these Metrics. You can then view the results alongside the execution details. ![See results of Metrics alongside Execution details](https://storage.googleapis.com/vellum-public/help-docs/online-evals/online-evals-execution-details.png) For more details on how to get started with Online Evaluations, check out our [help documentation](https://docs.vellum.ai/product/evaluation/online-evaluations) . OpenRouter Model Hosting + WizardLM-2 8x22B ------------------------------------------- **October 2nd, 2024** We’ve added OpenRouter as a new model host in Vellum! OpenRouter provides access to a wide range of AI models through a single API, expanding the options of models available to our users. As part of our new OpenRouter integration, we’re pleased to introduce the [WizardLM-2 8x22B](https://openrouter.ai/models/microsoft/wizardlm-2-8x22b) model to our platform. WizardLM-2 8x22B is known for its strong performance across various natural language processing tasks and is now available for use in your Vellum projects. ![OpenRouter Model Host](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/open-router-model-host.png) Prompt Caching Support for OpenAI --------------------------------- **October 2nd, 2024** Today OpenAI introduced [Prompt Caching](https://openai.com/index/api-prompt-caching/) for GPT-4o and o1 models. Subsequent invocations of the same prompt will produce outputs with lower latency and up to 50% reduced costs. To follow this, we’ve begun capturing cache tokens in Vellum’s monitoring layer. With this update, you’ll now see the number of Prompt Cache Tokens used by a Prompt Deployment’s executions if it’s backed by an OpenAI model. This new monitoring data can be used to help analyze your cache hit rate with OpenAI and optimize your LLM spend. Filter and Sort on Metric Scores -------------------------------- **October 1st, 2024** You can now filter and sort on a Metric’s score within Evaluation Reports. This makes it easy to find all Test Cases that failed below a given threshold for a given Metric. ![](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-10/evaluation-report-metric-sort-filter.png) --- # Changelog | November, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Workflows SDK Beta Release -------------------------- **November 24th, 2024** We’re excited to announce the beta release of the Vellum Workflows SDK! The Workflows SDK allows you to define the control flow of your AI systems using Python and push/pull the definition of your Workflows to/from Vellum’s UI. The Workflows SDK is currently in closed beta. If you’d like to participate and provide feedback, please contact us. For more information, check out the [Vellum Workflows SDK docs](https://docs.vellum.ai/developers/workflows-sdk/introduction) . We’ll be sharing more on the Workflows SDK in the coming weeks, so stay tuned! Prompt/Workflow Deployment History APIs --------------------------------------- **November 23rd, 2024** We’ve added two new APIs for retrieving details about a Prompt/Workflow Deployment at a specific history point. You can key off of either a `history_id` or the name of a Release Tag to retrieve the state of a Deployment at that point in time. For more info, check out our API reference: * [Prompt Deployment History](https://docs.vellum.ai/api-reference/prompts/deployments/retrieve-history) * [Workflow Deployment History](https://docs.vellum.ai/api-reference/workflows/deployments/retrieve-history) These APIs are available in our SDKs beginning version 0.9.15. Prompt Sandbox UI Improvements ------------------------------ **November 21st, 2024** Prompt Sandboxes have received a major facelift, making it easier to reason about a Prompt’s input variables and values, as well as making it possible to more easily resize panels to suit your needs. ### Prompt Editor The Prompt Editor tab now consolidates the “Input Variables” and “Scenarios” panels, so that you now define variables and provide values in the same place. All panels are resizeable. ![Prompt Editor](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-11/prompt-editor-improvements.png) ### Comparison Mode Comparison Mode makes use of the same variable input elements, making it possible to rename input variables right next to where you’d provide their values. ![Comparison Mode](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-11/comparison-mode-improvements.png) ### Chat Mode Lastly, Chat Mode has gotten major updates, such that you can now resize any panel and all UI elements look consistent across all modes. ![Chat Mode](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-11/chat-mode-improvements.png) Support for GPT-4o (2024-11-20) ------------------------------- **November 20th, 2024** We’ve added support for the latest snapshot of GPT-4o (2024-11-20) to Vellum. New API for Listing Release Tags -------------------------------- **November 16th, 2024** We now have new API endpoints for listing all Release Tags associated with a Prompt/Workflow Deployment. For details, check out our API Reference: * [Prompt Deployment Release Tags](https://docs.vellum.ai/api-reference/prompts/deployments/list-release-tags) * [Workflow Deployment Release Tags](https://docs.vellum.ai/api-reference/workflows/deployments/list-release-tags) These APIs are available in our SDKs beginning version 0.9.5. Quality of Life Improvements ---------------------------- **November 15th, 2024** We’ve made a number of small quality of life improvements throughout Vellum. ### Persisted Sort Order for Index Pages We now remember the sort order you set on index pages. If you sort by a column, leave the page, and come back, the sort order will be preserved. ### Optional Heads for Test Case CSV Upload We used to require that all CSV files uploaded for Test Cases have a header row with all expected columns. Now, you can choose to upload a CSV with a header row containing only a subset of columns and Vellum will set the values of unspecified columns to `null`. ![Optional Headers for Test Case CSV Upload](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-11/evals-optional-csv-heads.png) ### Semicolon Support in CSV Uploads We now support semicolons as delimiters in CSV files uploaded for use as Test Cases. ### Increased Max Concurrency for Evaluations We used to limit the max number of concurrent Test Case runs to 12. You can now crank up the concurrency to 36. ### Use Full Height for Displaying Subworkflows We now use the full height of the screen when displaying subworkflows in the workflow editor. ![Full-Height Display of Subworkflows](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-11/subworkflow-node-full-height.png) ### Remove “Copy of” When Cloning Nodes When copy/pasting Nodes in a Workflow, we used to prefix the label of each cloned Node with “Copy of.” This is no longer the case. ### Live Updates for Workflow Execution Views When on the detail page of a Workflow Execution, the information displayed was static. If the Workflow was still running, you had to refresh the page to see changes. Now, the page will update as the Workflow progresses. Audio Input Support via API for GPT 4o Audio Models --------------------------------------------------- **November 5th, 2024** We’ve added audio input capabilities via our API for the following models: * gpt-4o-audio-preview * gpt-4o-audio-preview-2024-10-01 You can now pass audio input to these models using a base64 encoded data URL. You can read more about audio input capabilities for these models in [OpenAI’s documentation](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in) . To learn more about how to use audio input with Vellum, read our [documentation](https://docs.vellum.ai/api-reference/prompts/execute-prompt#request.body.inputs.CHAT_HISTORY.value.content.AUDIO.type) . Support for Anthropic’s Updated Haiku 3.5 Model ----------------------------------------------- **November 5th, 2024** We’ve added support for Anthropic’s Haiku 3.5 10-22-2024 model. [Anthropic’s Claude Haiku 3.5](https://docs.anthropic.com/en/docs/about-claude/models) /api-reference/prompts/execute-prompt#request.body.inputs.CHAT\_HISTORY.value.content.AUDIO.type --- # Changelog | July, 2024 | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. Metadata Filtering in Search Nodes ---------------------------------- _July 31st, 2024_ For a while now you’ve been able to supply structured JSON metadata alongside Documents and then filtering on that metadata when making an [API call](https://docs.vellum.ai/api-reference/document-indexes/search) to search across Documents in a Document index (see [here](https://docs.vellum.ai/product/documents/metadata-filtering) for more info). However, Search Nodes within Workflows didn’t offer this same functionality through the UI. The workaround has been to use a Code Node or API Node and invoke Vellum’s Search API manually. We’re happy to share that the UI has reached parity with the API and you can now filter on metadata natively in Search Nodes. You’ll be able to construct arbitrarily complex boolean logic using the new Metadata Filters section of the Search Node’s Advanced settings. ![Search Node Metadata Filtering](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/search-node-metadata-filtering.png) Test Suite Test Case External IDs --------------------------------- _July 30th, 2024_ We’ve added a new feature to Test Suites that allows you to optionally assign an external ID to each Test Case. This is useful if you track your Test Cases in an external system and you want to periodically sync them with Vellum. You assign an external ID to each Test Case upon creation and then later upsert Test Cases to that Test Suite, keying off of the external ID. ![Upload Test Suite Test Cases Modal](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/test-case-upsert-external-id.png) Index Page Sorting ------------------ _July 30th, 2024_ We’ve added another quality-of-life improvement for the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. You’ll now see a “Sort by” dropdown next to the other page-level controls. You can now sort both folders’ and files’ by created date, modified date, and label. If there are other sort fields that you’d find useful, please let us know! ![Index page sort dropdown](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/index-page-sort-closed.png) ![Index page sort options](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/index-page-sort-open.png) Auto-Conversion to Variable Chips on Paste ------------------------------------------ _July 30, 2024_ Building on our recent update that introduced [Prompt Variable Chips](https://docs.vellum.ai/changelog/2024/july#prompt-variable-chips) , we’ve improved the experience by adding support for copy/pasting variables across blocks of different types. Now, when you copy text that includes a `{{ my_var }}` variable reference from a Jinja block and paste it into a Rich Text block, it’s seamlessly converted into a variable chip. Google Vertex AI Support ------------------------ _July 29th, 2024_ We now support [Google Vertex AI](https://cloud.google.com/vertex-ai) models. Previously you could only use Google AI Studio for using Google’s models. You can add them to your workspace from the [models page](https://app.vellum.ai/models) . ![Vertex AI Usage](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/vertex-ai.png) Expandable Meta Params in Retrieve Provider Payload Endpoint ------------------------------------------------------------ _July 26th, 2024_ For a while now we’ve had [an API](https://docs.vellum.ai/api-reference/prompts/retrieve-provider-payload) for compiling a Prompt and retrieving the exact payload that Vellum would send to a model provider on your behalf. We now support a new parameter in this API – `expand_meta`. With `expand_meta`, you can opt-in to return additional metadata relating to the compiled prompt payload. Learn more about which fields are expandable in our [API docs here](https://docs.vellum.ai/api-reference/prompts/retrieve-provider-payload#request.body.expand_meta) . This new field is available in our SDKs starting v0.7.3. Prompt Node Usage in Workflows ------------------------------ _July 25th, 2024_ You can now see token counts and other usage metrics appear in Prompt Node results when invoking Workflows in the Workflow Sandbox: ![Prompt Node Usage](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/prompt-node-usage.png) This setting is now on by default, but can be toggled off in the Workflow Builder Settings. You can also now return usage data when invoking a _Workflow Deployment_ via API, by passing in `True` to the `expand_meta.usage` parameter on either Execute Workflow endpoints. | | | | --- | --- | | 1 | stream \= client.execute\_workflow\_stream( | | 2 | workflow\_deployment\_name\="demo", | | 3 | inputs\=\[ |\ | 4 | WorkflowRequestInputRequest\_String( |\ | 5 | type\="STRING", |\ | 6 | name\="foo", |\ | 7 | value\="bar", |\ | 8 | ), |\ | 9 | \], | | 10 | event\_types\=\["WORKFLOW", "NODE"\], | | 11 | expand\_meta\=WorkflowExpandMetaRequest( | | 12 | usage\=True | | 13 | ) | | 14 | ) | | 15 | | | 16 | for event in stream: | | 17 | if event.type == "NODE" and event.data.state == "FULFILLED": | | 18 | node\_result\_data \= event.data.data | | 19 | if node\_result\_data and node\_result\_data.type == "PROMPT": | | 20 | print(node\_result\_data.data.execution\_meta.usage) | Enable/Disable All Workflow Node Mocks -------------------------------------- _July 25th, 2024_ Mocking Prompt Nodes helps to save token usage and time when developing the later stages of your Workflow. However, once the Workflow is in a good state, it’s often useful to run the full Workflow end-to-end without mocks to make sure it all comes together. Previously, you had to enable/disable each mock individually. Now, beneath the scenario inputs there is a toggle that allows you to enable/disable all mocks in a workflow at once. ![Enable Disable All Mocks](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/enable-disable-all-mocks.png) Support for Bulk Upserting Test Suite Test Cases via API -------------------------------------------------------- _July 24th, 2024_ For a while now we’ve had [an API](https://docs.vellum.ai/api-reference/test-suites/test-cases/bulk-update) for creating, replacing, and deleting Test Cases in a Test Suite in bulk. We now support a fourth operation in this API – upsert. With upsert, you can provide an `external_id` and a Test Case payload. If there is already a Test Case with that `external_id`, it’ll be replaced. Otherwise, it’ll be created. This new operation is available in our SDKs starting v0.6.12. Llama 3.1 on Groq ----------------- _July 23rd, 2024_ Meta’s newest [Llama 3.1 models](https://ai.meta.com/blog/meta-llama-3-1/) are now available in Vellum through our [Groq](https://wow.groq.com/now-available-on-groq-the-largest-and-most-capable-openly-available-foundation-model-to-date-llama-3-1-405b/) integration! Deployed Prompt Variant Display ------------------------------- _July 19th, 2024_ When on the Prompt Deployment Overview page, you can now see the name of the Prompt Variant that’s been deployed. This is useful if your Prompt Sandbox has multiple Prompt Variants that you were comparing against one another and you’re not sure which one is currently deployed. ![Deployed Prompt Variant Display](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/deployed-prompt-variant-display.png) Improvements to Prompt Chat History Variables --------------------------------------------- _July 18th, 2024_ It used to be that Prompts that accepted a dynamic Chat History required an input variable whose name was specifically `$chat_history`. This nomenclature caused frequent confusion and was a bit cumbersome to work with. Now, you can name Chat History input variables whatever you want and even rename them after-the-fact. As part of this, we’ve also centralized input variable definitions so that whether you want to create a String variable or a Chat History variable, you can do so via the “Add” button in the “Input Variables” section of the Prompt Editor. ![Add Prompt Input Variable Button](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/add-prompt-input-variable-button.png) Copyable Text to Clipboard -------------------------- _July 18th, 2024_ We’ve introduced the ability to copy Prompt Variant IDs, Document Indexes, Models, Workflow Deployment Names and IDs, Document Keys, and Prompt Deployment Names and IDs to clipboard. This feature comes with an enhanced UI with intuitive indicators and tooltips for copyable fields. ![Copy Text to Clipboard](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/copy-text-to-clipboard-1.png) ![Copy Text to Clipboard](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/copy-text-to-clipboard-2.png) GPT-4o Mini ----------- _July 18th, 2024_ OpenAI’s newest [GPT-4o Mini models](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/) `gpt-4o-mini` & `gpt-4o-mini-2024-07-18` are now available in Vellum and have been added to all workspaces! Prompt Variable Chips --------------------- _July 18th, 2024_ It used to be that any time you wanted to reference a variable in a Prompt, you did so using `{{ myVariable }}` syntax. While powerful if you need to use more complex Jinja templating syntax, using double-curlies for simple variable substitution can be a bit cumbersome. 1. They are harder to visually parse from the rest of your Prompt 2. They can get confusing when dealing with json, which also uses double-curly brackets 3. Whenever you rename a variable, you need to hunt down its usages. To make this easier, we’ve introduced a new way to reference variables in Prompts: Variable Chips. Variable Chips are small, clickable chips that you can reference in your Prompt text. You can add them by beginning to type `{{` or by typing `/`. Renaming a variable automatically renames all of its references. Variable chips can be used in the new “Rich Text” block type. New Prompt blocks will default to Rich Text, but you can change existing blocks to Rich Text by clicking the block type dropdown in the block’s toolbar and converting from Jinja to Rich Text and vice versa. Check out a full video demo here: New Layout for Sandbox Evaluations ---------------------------------- _July 17th, 2024_ Previously, when a Prompt/Workflow had multiple Test Suites associated with it, we’d shown them all on the page at once. This made navigation difficult (you had to scroll up and down to see each) and could also lead to performance issues. We’ve addressed these issues updating the page layout to display just one Test Suite at a time with a searchable select input that allows you to easily load and view each table one at a time. ![Sandbox Evaluation Select](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/eval-select-closed.png) New “Add Document to Document Index” API ---------------------------------------- _July 16th, 2024_ We’ve introduced a new API for adding previously uploaded Documents to a Document Index. This API is useful when you have a Document that had previously been added to one Document Index and you want to add it to another without having to re-upload its contents altogether. It’s available in our SDKs beginning version 0.6.10. You can find docs for this new API [here](https://docs.vellum.ai/api-reference/document-indexes/add-document) . Prompt Deployment Executions Table Improvements ----------------------------------------------- _July 12th, 2024_ We’ve made several quality-of-life enhancements to the Prompt Deployment Executions table, simplifying the process of adding and editing ‘Desired Output’ values. The entire table has been updated to align with the design of our other tables, such as Evaluations, ensuring a familiar editing experience. Additionally, it is now easier than ever to expand/collapse and copy values. Moreover, we’ve significantly improved the consistency and usability of the ‘Quality’ column. You can now edit quality ratings with a single click, and the ‘Desired Output’ column will automatically update to reflect your rating where applicable. ![Prompt Deployment Executions Table](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/executions-quality-input.png) Constant Values in Workflow Node Inputs --------------------------------------- _July 11th, 2024_ It’s often the case that you might want to specify a constant value as a Workflow Node Input, either as the input’s primary value or as its fallback value. The solution up until now was to specify a Templating Node, have it output a constant value, and then feed its output to the downstream Node. Today, we are releasing the ability to inline constant values directly within Workflow Node inputs! First, start typing in the Node Input until the no options modal shows: ![New Constant Link](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/new-constant-link.png) A modal will appear to specify your value: ![New Constant Modal](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/new-constant-modal.png) Upon confirming, Vellum will use an icon to denote that the input value represents a constant. As part of this work, we also added icons for all other Node Input types: ![Constant Value Display](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/constant-value-display.png) Note that constant values will always drop to the last fallback option of a given Node input, and there can only be maximum one constant defined per input. This is due to the nature fallback values – fallbacks are only used if other values aren’t available (i.e. the node that produced the value hadn’t executed yet). In the case of constants, their values are always present. Test Case CSV Upload in Evaluation Reports ------------------------------------------ _July 9th, 2024_ We’ve introduced the ability to upload Test Cases to a Test Suite directly from within the Evaluations tab of a Prompt or Workflow. Now, you’ll find an “Upload Test Cases” button in the table header of every Evaluations table, for both Workflows and Prompt Sandboxes whereas previously, you needed to first navigate to the Test Suite itself and upload from there. ![Test Case Upload in Evaluation Reports](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/eval-page-test-case-upload.png) Index Page List View -------------------- _July 3rd, 2024_ We’ve introduced a list-view toggle to the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. Your preferred view will be saved automatically by entity type, allowing you to, for instance, default to list view for Prompts and grid view for Documents. ![Index Page List View](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/index-page-list-view.jpg) Collapsible Index Page Sections ------------------------------- _July 2nd, 2024_ You can now collapse sections on the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. Simply click the heading of any section to toggle the visibility of all folders and items within that section. ![Collapsible Index Page Sections](https://storage.googleapis.com/vellum-public/help-docs/changelogs/2024-07/index-page-collapse.png) --- # Push Feature Status | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. The `push` feature in Vellum Workflows SDK allows you to upload locally defined Vellum resources to the Vellum application. This feature is in active development, with different components and use cases at varying levels of support. Support Stages -------------- The support for the `push` feature can be in one of the following stages for different components and workflows: 1. **Blocked on `push`**: The feature is not yet supported for this component or workflow type. Attempting to push will result in an error. 2. **Supported but hidden from UI**: The feature works via the CLI or SDK, but there is no corresponding UI element in the Vellum application to indicate or manage this capability. 3. **Supported but read-only in UI**: The feature works and is visible in the UI, but you cannot modify the pushed components through the UI interface. 4. **Fully supported and editable**: The feature is fully supported, visible in the UI, and can be edited through both the UI and code interfaces. Current Status -------------- The support level for different components is determined on a case-by-case basis. If you encounter issues with pushing specific components or workflows, please contact Vellum support for the most up-to-date information on support status. ### Generally Supported Components The following components typically have good support for the `push` feature: * Basic workflow structures * Standard node types (Prompt, Search, API, Templating) * Simple control flow patterns ### Components with Potential Limitations The following components may have limited support or be in earlier stages of the support lifecycle: * Complex nested workflows * Custom node types * Advanced control flow patterns (complex loops, conditional branching) * Workflows with external integrations Using the Push Command ---------------------- To push your workflow to Vellum, use the following command: | | | | --- | --- | | 1 | vellum workflows push --workflow-sandbox-id= --include-sandbox | If you encounter an error indicating that push is not supported for a particular component, you may need to: 1. Check if there’s an alternative approach to achieve the same result 2. Wait for support to be added in a future release 3. Contact Vellum support for guidance Checking Push Support Status ---------------------------- To verify if your workflow can be pushed successfully, you can use the `--dry-run` option: | | | | --- | --- | | 1 | vellum workflows push --workflow-sandbox-id= --dry-run --include-sandbox | This will validate your workflow without actually pushing it to Vellum, allowing you to identify any unsupported components. Troubleshooting --------------- If you encounter issues with the `push` feature: 1. Ensure you’re using the latest version of the Vellum Workflows SDK 2. Check the error message for specific information about unsupported components 3. Try simplifying your workflow to identify which component is causing the issue 4. Use the `--strict` flag to get more detailed error information: For specific guidance on your use case, contact Vellum support at [support@vellum.ai](mailto:support@vellum.ai) . --- # Datasets Overview | Vellum | Documentation For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page. `vellum.workflows.inputs.DatasetRow` Datasets allow you to define test scenarios and sample data for local workflow development. They are stored in the `sandbox.py` file within your workflow directory and are included when you push or pull your workflow artifact using the Vellum CLI. DatasetRow ---------- The `DatasetRow` class represents a single test scenario with a label, inputs, optional trigger, and optional mocks. ### Attributes label strRequired A descriptive label for the test scenario. This helps identify the scenario in the Vellum UI and logs. inputs Union\[BaseInputs, Dict\[str, Any\]\]Required The input data for the workflow. Can be either a `BaseInputs` instance or a dictionary of input values. workflow\_trigger Optional\[BaseTrigger\] Optional trigger instance for this scenario. Use this to test workflows that are triggered by schedules or integrations. mocks Optional\[Sequence\[Union\[BaseOutputs, MockNodeExecution\]\]\] Optional sequence of node output mocks for testing scenarios. Allows you to override node outputs during local execution. Basic Usage ----------- The `sandbox.py` file defines your dataset and creates a `WorkflowSandboxRunner` to execute your workflow locally. | | | | --- | --- | | 1 | from vellum.workflows.inputs import DatasetRow | | 2 | from vellum.workflows.sandbox import WorkflowSandboxRunner | | 3 | | | 4 | from .inputs import Inputs | | 5 | from .workflow import Workflow | | 6 | | | 7 | dataset \= \[ |\ | 8 | DatasetRow(label\="Scenario 1", inputs\=Inputs(user\_message\="Hello")), |\ | 9 | DatasetRow(label\="Scenario 2", inputs\=Inputs(user\_message\="How are you?")), |\ | 10 | \] | | 11 | | | 12 | runner \= WorkflowSandboxRunner(workflow\=Workflow(), dataset\=dataset) | | 13 | | | 14 | if \_\_name\_\_ == "\_\_main\_\_": | | 15 | runner.run() | You can run a specific scenario by passing an index to the `run()` method: | | | | --- | --- | | 1 | runner.run(index\=1) # Runs "Scenario 2" | Inputs ------ Inputs can be provided as either a typed `BaseInputs` instance or a dictionary. Using typed inputs provides better IDE support and validation. ###### Using Typed Inputs The `Inputs` class is defined in your workflow’s `./inputs.py` file: | | | | --- | --- | | 1 | \# ./inputs.py | | 2 | from vellum.workflows.inputs import BaseInputs | | 3 | | | 4 | class Inputs(BaseInputs): | | 5 | user\_message: str | | 6 | temperature: float = 0.7 | Then reference it in your `sandbox.py`: | | | | --- | --- | | 1 | \# ./sandbox.py | | 2 | from vellum.workflows.inputs import DatasetRow | | 3 | | | 4 | from .inputs import Inputs | | 5 | | | 6 | dataset \= \[ |\ | 7 | DatasetRow( |\ | 8 | label\="With typed inputs", |\ | 9 | inputs\=Inputs(user\_message\="Hello", temperature\=0.5), |\ | 10 | ), |\ | 11 | \] | ###### Using Dictionary Inputs | | | | --- | --- | | 1 | from vellum.workflows.inputs import DatasetRow | | 2 | | | 3 | dataset \= \[ |\ | 4 | DatasetRow( |\ | 5 | label\="With dict inputs", |\ | 6 | inputs\={"user\_message": "Hello", "temperature": 0.5}, |\ | 7 | ), |\ | 8 | \] | Triggers -------- Triggers allow you to test workflows that are activated by schedules or external integrations. The `workflow_trigger` attribute accepts any trigger type that extends `BaseTrigger`. When using `workflow_trigger`, you should not define `inputs` as the trigger provides its own input context. ### Available Trigger Types | Trigger | Description | | --- | --- | | `ScheduleTrigger` | For workflows triggered on a schedule (cron-based) | | `IntegrationTrigger` | For workflows triggered by external integrations | | `ManualTrigger` | For workflows triggered manually | ###### Using Schedule Triggers The trigger class is defined in your workflow’s `./triggers/scheduled.py` file: | | | | --- | --- | | 1 | \# ./triggers/scheduled.py | | 2 | from vellum.workflows.triggers import ScheduleTrigger | | 3 | | | 4 | class MySchedule(ScheduleTrigger): | | 5 | pass | Then reference it in your `sandbox.py`: | | | | --- | --- | | 1 | \# ./sandbox.py | | 2 | from datetime import datetime | | 3 | from vellum.workflows.inputs import DatasetRow | | 4 | | | 5 | from .triggers.scheduled import MySchedule | | 6 | | | 7 | dataset \= \[ |\ | 8 | DatasetRow( |\ | 9 | label\="Scheduled execution", |\ | 10 | workflow\_trigger\=MySchedule( |\ | 11 | current\_run\_at\=datetime.now(), |\ | 12 | next\_run\_at\=datetime.now(), |\ | 13 | ), |\ | 14 | ), |\ | 15 | \] | ---