# Table of Contents
- [What is Portkey? - Portkey Docs](#what-is-portkey-portkey-docs)
- [Unknown](#unknown)
- [Introduction - Portkey Docs](#introduction-portkey-docs)
- [Upgrade to Model Catalog - Portkey Docs](#upgrade-to-model-catalog-portkey-docs)
- [Common Errors & Resolutions - Portkey Docs](#common-errors-resolutions-portkey-docs)
- [Developer Forum - Portkey Docs](#developer-forum-portkey-docs)
- [How to Contribute - Portkey Docs](#how-to-contribute-portkey-docs)
- [Contact Us - Portkey Docs](#contact-us-portkey-docs)
- [Feature Comparison - Portkey Docs](#feature-comparison-portkey-docs)
- [Open Source - Portkey Docs](#open-source-portkey-docs)
- [Prometheus Metrics - Portkey Docs](#prometheus-metrics-portkey-docs)
- [December '23 Migration - Portkey Docs](#december-23-migration-portkey-docs)
- [Make Your First Request - Portkey Docs](#make-your-first-request-portkey-docs)
- [September - Portkey Docs](#september-portkey-docs)
- [Errors - Portkey Docs](#errors-portkey-docs)
- [August - Portkey Docs](#august-portkey-docs)
- [July - Portkey Docs](#july-portkey-docs)
- [Supported Providers - Portkey Docs](#supported-providers-portkey-docs)
- [Prompts - Portkey Docs](#prompts-portkey-docs)
- [Authentication - Portkey Docs](#authentication-portkey-docs)
- [Building an LLM-as-a-Judge System for AI (Customer Support) Agent - Portkey Docs](#building-an-llm-as-a-judge-system-for-ai-customer-support-agent-portkey-docs)
- [Headers - Portkey Docs](#headers-portkey-docs)
- [AWS SageMaker - Portkey Docs](#aws-sagemaker-portkey-docs)
- [Google Gemini - Portkey Docs](#google-gemini-portkey-docs)
- [Overview - Portkey Docs](#overview-portkey-docs)
- [June - Portkey Docs](#june-portkey-docs)
- [Universal API - Portkey Docs](#universal-api-portkey-docs)
- [Enterprise Architecture - Portkey Docs](#enterprise-architecture-portkey-docs)
- [Observability (OpenTelemetry) - Portkey Docs](#observability-opentelemetry-portkey-docs)
- [OpenAI - Portkey Docs](#openai-portkey-docs)
- [Ultimate AI SDR - Portkey Docs](#ultimate-ai-sdr-portkey-docs)
---
# What is Portkey? - Portkey Docs
[Skip to main content](https://portkey.ai/docs/introduction/what-is-portkey#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
Ctrl K
Search...
Navigation
Introduction
What is Portkey?
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, and more accurate at the same time. Here’s a product walkthrough (3 mins):
###
[](https://portkey.ai/docs/introduction/what-is-portkey#integrate-in-3-lines-of-code)
Integrate in 3 Lines of Code
Python
NodeJS
REST API
OpenAI Python SDK
OpenAI NodeJS SDK
Copy
Ask AI
from portkey_ai import Portkey
portkey = Portkey(
api_key="YOUR_PORTKEY_API_KEY",
provider="@YOUR_PROVIDER"
)
chat_complete = portkey.chat.completions.create(
model="gpt-3.5-turbo",
messages=[\
{"role": "system", "content": "You are a helpful assistant."},\
{"role": "user", "content": "Hello!"}\
]
)
print(chat_complete.choices[0].message.content)
While you’re here, why not [give us a star](https://git.new/ai-gateway-docs)
? It helps us a lot!
###
[](https://portkey.ai/docs/introduction/what-is-portkey#faqs)
FAQs
Will Portkey increase the latency of my API requests?
Portkey is hosted on edge workers throughout the world, ensuring minimal latency. Our benchmarks estimate a total latency addition between 20-40ms compared to direct API calls. This slight increase is often offset by the benefits of our caching and routing optimizations.Our edge worker locations:
Is my data secure?
Portkey AI is ISO:27001 and SOC 2 certified, and GDPR & HIPAA compliant. We maintain best practices for service security, data storage, and retrieval. All data is encrypted in transit and at rest using industry-standard AES-256 encryption. For enhanced security, we offer:
1. On request, we can enable a feature that does NOT store any of your request and response body objects in Portkey datastores or our logs.
2. For enterprises, we offer managed hosting to deploy Portkey inside private clouds.
For more information on these options, contact us at [support@portkey.ai](mailto:support@portkey.ai)
.
Will Portkey scale if my app explodes?
Portkey is built on scalable infrastructure and can handle millions of requests per minute with very high concurrency. We currently serve over 25M requests daily with a 99.99% uptime. Our edge architecture & scaling capabilities ensure we can accommodate sudden spikes in traffic without performance degradation.[View our Status Page](https://status.portkey.ai/)
Does Portkey impose timeouts on requests?
We _DO NOT_ impose any explicit timeout for our free OR paid plans currently. While we don’t time out requests on our end, we recommend implementing client-side timeouts appropriate for your use case to handle potential network issues or upstream API delays.
Do you support SSO?
Yes! We support SSO with any custom OIDC provider.
What are the pricing options? Is there a free trial?
Portkey’s Gateway is open source and free to use. On managed version, Portkey offers a free plan with 10k requests per month. We also offer paid plans with more requests and additional features.
Where can I reach you?
We’re available all the time on [Discord](https://discord.gg/DD7vgKK299)
, or on our support email - [](mailto:support@portkey.ai)
[support@portkey.ai](mailto:support@portkey.ai)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/introduction/what-is-portkey.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/introduction/what-is-portkey)
[Make Your First Request](https://portkey.ai/docs/introduction/make-your-first-request)
Ctrl+I
Assistant
Responses are generated using AI and may contain mistakes.
---
# Unknown
\# Portkey Docs ## Docs - \[How to Contribute\](https://docs.portkey.ai/docs/README.md) - \[Get cache hit latency data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-cache-hit-latency-data.md) - \[Get cache hit rate data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-cache-hit-rate-data.md) - \[Get cost data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-cost-data.md) - \[Get error rate data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-error-rate-data.md) - \[Get errors data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-errors-data.md) - \[Get feedback data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-feedback-data.md) - \[Get feedback per ai models data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-feedback-per-ai-models-data.md) - \[Get feedback score distribution data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-feedback-score-distribution-data.md) - \[Get latency data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-latency-data.md) - \[Get requests data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-requests-data.md) - \[Get requests per user data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-requests-per-user-data.md) - \[Get rescued requests data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-rescued-requests-data.md) - \[Get status code data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-status-code-data.md) - \[Get tokens data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-tokens-data.md) - \[Get unique status code data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-unique-status-code-data.md) - \[Get users data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-users-data.md) - \[Get weighted feedback data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-weighted-feedback-data.md) - \[Get Metadata Grouped Data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/groups-paginated-data/get-metadata-grouped-data.md) - \[Get Model Grouped Data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/groups-paginated-data/get-model-grouped-data.md) - \[Get User Grouped Data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/groups-paginated-data/get-user-grouped-data.md) - \[Get User Grouped Data\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/analytics/summary/get-all-cache-data.md) - \[Create API Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/create-api-key.md) - \[Delete an API Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/delete-an-api-key.md) - \[List API Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/list-api-keys.md) - \[Retrieve an API Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/retrieve-an-api-key.md) - \[Update API Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/update-api-key.md) - \[List Audit Logs\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/audit-logs/list-audit-logs.md) - \[Create Config\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/create-config.md) - \[Delete Config\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/delete-config.md) - \[List Config Versions\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/list-config-versions.md) - \[List Configs\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/list-configs.md) - \[Retrieve Config\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/retrieve-config.md) - \[Update Config\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/configs/update-config.md) - \[Create Guardrail\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/create-guardrail.md): Creates a new guardrail with specified checks and actions - \[Delete Guardrail\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/delete-guardrail.md): Deletes an existing guardrail - \[List Guardrails\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/list-guardrails.md): Retrieves a paginated list of guardrails for the specified workspace or organisation - \[Retrieve Guardrail\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/retrieve-guardrail.md): Retrieves details of a specific guardrail by ID or slug - \[Update Guardrail\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/update-guardrail.md): Updates an existing guardrail's name, checks, or actions - \[Create Integration\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/create-integration.md) - \[Delete Integration\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/delete-integration.md) - \[List Integrations\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/list-integrations.md) - \[Delete Custom Model\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/models/delete-custom-model.md): Removes multiple custom models from an integration by their slugs. - \[List Model Access\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/models/list-model-access.md): Retrieves all model access for a specific integration with their configuration and pricing details. - \[Update Model Access\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/models/update-model-access.md): Updates model access, pricing configuration, and settings for multiple models in an integration. Can enable/disable models and configure custom pricing. - \[Retrieve Integration\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/retrieve-integration.md) - \[Update Integration\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/update-integration.md) - \[List Workspace Access\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/workspaces/list-workspace-access.md): Retrieves workspace access configuration for an integration, including usage limits and rate limits. - \[Update Workspace Access\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/integrations/workspaces/update-workspace-access.md): Updates workspace access permissions, usage limits, and rate limits for an integration. Can configure global workspace access or per-workspace settings. - \[Create Prompt Collection\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/collections/create-collection.md): Creates a new collection in the specified workspace - \[Delete Prompt Collection\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/collections/delete-collection.md): Deletes a collection - \[List Prompt Collections\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/collections/list-collections.md): Lists all collections in the specified workspace - \[Retrieve Prompt Collection\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/collections/retrieve-collection.md): Retrieves details of a specific collection - \[Update Prompt Collection\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/collections/update-collection.md): Updates a collection's details - \[Create Prompt\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/create-prompt.md) - \[Delete Prompt\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/delete-prompt.md) - \[Create Label\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/labels/create-label.md): Creates a new label in the system - \[Delete Label\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/labels/delete-label.md): Deletes a label - \[List Labels\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/labels/list-labels.md): Returns a list of labels based on filters - \[Retrieve Label\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/labels/retrieve-label.md): Returns a specific label by its ID - \[Update Label\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/labels/update-label.md): Updates an existing label - \[List Prompt Versions\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/list-prompt-versions.md) - \[List Prompts\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/list-prompts.md) - \[Create Partial\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/create-partial.md) - \[Delete Partial\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/delete-partial.md) - \[List Partial Versions\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/list-partial-versions.md) - \[List Partials\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/list-partials.md) - \[Publish Partial\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/publish-partial.md) - \[Retrieve Partial\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/retrieve-partial.md) - \[Update Partial\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/partials/update-partial.md) - \[Publish Prompt\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/publish-prompt.md) - \[Retrieve Prompt\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/retrieve-prompt.md) - \[Retrieve Prompt Version\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/retrieve-prompt-version.md) - \[Update Prompt\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/update-prompt.md) - \[Update Prompt Version\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/prompts/update-prompt-version.md) - \[Create Provider\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/providers/create-provider.md) - \[Delete Provider\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/providers/delete-provider.md) - \[List Providers\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/providers/list-providers.md) - \[Retrieve Provider\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/providers/retrieve-provider.md) - \[Update Provider\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/providers/update-provider.md) - \[Delete a user invite\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/user-invites/delete-a-user-invite.md) - \[Invite a User\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/user-invites/invite-a-user.md): Send an invite to user for your organization - \[Resend a user invite\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/user-invites/resend-a-user-invite.md): Resend an invite to user for your organization - \[Retrieve all user invite\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/user-invites/retrieve-all-user-invites.md) - \[Retrieve an user invite\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/user-invites/retrieve-an-invite.md) - \[Remove a user\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/users/remove-a-user.md) - \[Retrieve a user\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/users/retrieve-a-user.md) - \[Retrieve all users\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/users/retrieve-all-users.md) - \[Update a user\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/users/update-a-user.md) - \[Create Virtual Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/create-virtual-key.md) - \[Delete Virtual Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/delete-virtual-key.md) - \[List Virtual Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/list-virtual-keys.md) - \[Retrieve Virtual Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/retrieve-virtual-key.md) - \[Update Virtual Key\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/update-virtual-key.md) - \[Add a Workspace Member\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspace-members/add-a-workspace-member.md) - \[Remove Workspace Member\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspace-members/remove-workspace-member.md) - \[Retrieve a Workspace Member\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspace-members/retrieve-a-workspace-member.md) - \[Retrieve all Workspace Member\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspace-members/retrieve-all-workspace-members.md) - \[Update Workspace Member\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspace-members/update-workspace-member.md) - \[Create Workspace\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/create-workspace.md) - \[Delete a Workspace\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/delete-a-workspace.md) - \[Retrieve a Workspace\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/retrieve-a-workspace.md) - \[Retrieve all Workspaces\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/retrieve-all-workspaces.md) - \[Update Workspace\](https://docs.portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/update-workspace.md) - \[Create Feedback\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/feedback/create-feedback.md): This endpoint allows users to submit feedback for a particular interaction or response. - \[Update Feedback\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/feedback/update-feedback.md): This endpoint allows users to update existing feedback. - \[Insert a Log\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/insert-a-log.md): Submit one or more log entries - \[Cancel a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/cancel-a-log-export.md) - \[Create a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/create-a-log-export.md) - \[Download a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/download-a-log-export.md) - \[List a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/list-log-exports.md) - \[Retrieve a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/retrieve-a-log-export.md) - \[Start a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/start-a-log-export.md) - \[Update a Log Export\](https://docs.portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/update-a-log-export.md) - \[Errors\](https://docs.portkey.ai/docs/api-reference/admin-api/error.md): Error codes returned by the Admin API and how to resolve them. - \[Introduction\](https://docs.portkey.ai/docs/api-reference/admin-api/introduction.md): Manage your Portkey organization and workspaces programmatically - \[OpenAPI Specification\](https://docs.portkey.ai/docs/api-reference/admin-api/open-api-specification.md) - \[Create Assistant\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/assistants/create-assistant.md) - \[Delete Assistant\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/assistants/delete-assistant.md) - \[List Assistant\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/assistants/list-assistants.md) - \[Modify Assistant\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/assistants/modify-assistant.md) - \[Retrieve Assistant\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/assistants/retrieve-assistant.md) - \[Create Message\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/messages/create-message.md) - \[Delete Message\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/messages/delete-message.md) - \[List Message\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/messages/list-messages.md) - \[Modify Message\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/messages/modify-message.md) - \[Retrieve Message\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/messages/retrieve-message.md) - \[List Run Steps\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/run-steps/list-run-steps.md) - \[Retrieve Run Steps\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/run-steps/retrieve-run-steps.md) - \[Cancel Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/cancel-run.md) - \[Create Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/create-run.md) - \[Create thread and Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/create-thread-and-run.md) - \[list Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/list-runs.md) - \[Modify Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/modify-run.md) - \[Retrieve Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/retrieve-run.md) - \[Submit Tool Outputs to Run\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/runs/submit-tool-outputs-to-run.md) - \[Create Thread\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/threads/create-thread.md) - \[Delete Thread\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/threads/delete-thread.md) - \[Modify Thread\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/threads/modify-thread.md) - \[Retrieve Thread\](https://docs.portkey.ai/docs/api-reference/inference-api/assistants-api/threads/retrieve-thread.md) - \[Create Speech\](https://docs.portkey.ai/docs/api-reference/inference-api/audio/create-speech.md) - \[Create Transcription\](https://docs.portkey.ai/docs/api-reference/inference-api/audio/create-transcription.md) - \[Create Translation\](https://docs.portkey.ai/docs/api-reference/inference-api/audio/create-translation.md) - \[Authentication\](https://docs.portkey.ai/docs/api-reference/inference-api/authentication.md) - \[Cancel Batch\](https://docs.portkey.ai/docs/api-reference/inference-api/batch/cancel-batch.md) - \[Create Batch\](https://docs.portkey.ai/docs/api-reference/inference-api/batch/create-batch.md) - \[List Batch\](https://docs.portkey.ai/docs/api-reference/inference-api/batch/list-batch.md) - \[Retrieve Batch\](https://docs.portkey.ai/docs/api-reference/inference-api/batch/retrieve-batch.md) - \[Chat\](https://docs.portkey.ai/docs/api-reference/inference-api/chat.md) - \[Completions\](https://docs.portkey.ai/docs/api-reference/inference-api/completions.md) - \[Gateway Config Object\](https://docs.portkey.ai/docs/api-reference/inference-api/config-object.md) - \[Embeddings\](https://docs.portkey.ai/docs/api-reference/inference-api/embeddings.md) - \[Errors\](https://docs.portkey.ai/docs/api-reference/inference-api/error-codes.md) - \[Delete File\](https://docs.portkey.ai/docs/api-reference/inference-api/files/delete-file.md) - \[List Files\](https://docs.portkey.ai/docs/api-reference/inference-api/files/list-files.md) - \[Retrieve File\](https://docs.portkey.ai/docs/api-reference/inference-api/files/retrieve-file.md) - \[Retrieve File Content\](https://docs.portkey.ai/docs/api-reference/inference-api/files/retrieve-file-content.md) - \[Upload File\](https://docs.portkey.ai/docs/api-reference/inference-api/files/upload-file.md) - \[Cancel Fine-tuning\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/cancel-fine-tuning.md) - \[Create Fine-tuning Job\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/create-fine-tuning-job.md): Finetune a provider model - \[List Fine-tuning Checkpoints\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/list-fine-tuning-checkpoints.md) - \[List Fine-tuning Events\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/list-fine-tuning-events.md) - \[List Fine-tuning Jobs\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/list-fine-tuning-jobs.md) - \[Retrieve Fine-tuning Job\](https://docs.portkey.ai/docs/api-reference/inference-api/fine-tuning/retrieve-fine-tuning-job.md) - \[Gateway to Other APIs\](https://docs.portkey.ai/docs/api-reference/inference-api/gateway-for-other-apis.md): Access any custom provider endpoint through Portkey API - \[Headers\](https://docs.portkey.ai/docs/api-reference/inference-api/headers.md): Header requirements and options for the Portkey API - \[Create Image\](https://docs.portkey.ai/docs/api-reference/inference-api/images/create-image.md) - \[Create Image Edit\](https://docs.portkey.ai/docs/api-reference/inference-api/images/create-image-edit.md) - \[Create Image Variation\](https://docs.portkey.ai/docs/api-reference/inference-api/images/create-image-variation.md) - \[Introduction\](https://docs.portkey.ai/docs/api-reference/inference-api/introduction.md): This documentation provides detailed information about the various ways you can access and interact with Portkey - \*\*a robust AI gateway\*\* designed to simplify and enhance your experience with Large Language Models (LLMs) like OpenAI's GPT models. - \[Models\](https://docs.portkey.ai/docs/api-reference/inference-api/models/models.md): Lists the currently available models that can be used through Portkey, and provides basic information about each one. - \[Moderations\](https://docs.portkey.ai/docs/api-reference/inference-api/moderations.md) - \[OpenAPI Specification\](https://docs.portkey.ai/docs/api-reference/inference-api/open-api-specification.md) - \[Prompt Completions\](https://docs.portkey.ai/docs/api-reference/inference-api/prompts/prompt-completion.md): Execute your saved prompt templates on Portkey - \[Prompt Render\](https://docs.portkey.ai/docs/api-reference/inference-api/prompts/render.md): Renders a prompt template with its variable values filled in - \[Response Schema\](https://docs.portkey.ai/docs/api-reference/inference-api/response-schema.md) - \[Delete a Response\](https://docs.portkey.ai/docs/api-reference/inference-api/responses/delete-response.md) - \[Create a Response\](https://docs.portkey.ai/docs/api-reference/inference-api/responses/responses.md) - \[List Input Items\](https://docs.portkey.ai/docs/api-reference/inference-api/responses/retrieve-inputs.md) - \[Get a Response\](https://docs.portkey.ai/docs/api-reference/inference-api/responses/retrieve-response.md) - \[Supported Providers\](https://docs.portkey.ai/docs/api-reference/inference-api/supported-providers.md) - \[C# (.NET)\](https://docs.portkey.ai/docs/api-reference/sdk/c-sharp.md): Integrate Portkey in your \`.NET\` app easily using the OpenAI library and get advanced monitoring, routing, and enterprise features. - \[Supported SDKs\](https://docs.portkey.ai/docs/api-reference/sdk/list.md): Find the best way to use Portkey in your preferred language. - \[Node.js\](https://docs.portkey.ai/docs/api-reference/sdk/node.md): Official Portkey Node.js SDK – robust, modern, and fully typed integration for JavaScript and TypeScript developers. - \[Python\](https://docs.portkey.ai/docs/api-reference/sdk/python.md): Official Portkey Python SDK to help take your AI apps to production - \[December\](https://docs.portkey.ai/docs/changelog/2024/dec.md) - \[November\](https://docs.portkey.ai/docs/changelog/2024/nov.md) - \[October\](https://docs.portkey.ai/docs/changelog/2024/oct.md) - \[April\](https://docs.portkey.ai/docs/changelog/2025/apr.md) - \[August\](https://docs.portkey.ai/docs/changelog/2025/august.md) - \[February\](https://docs.portkey.ai/docs/changelog/2025/feb.md) - \[January\](https://docs.portkey.ai/docs/changelog/2025/jan.md) - \[July\](https://docs.portkey.ai/docs/changelog/2025/july.md) - \[June\](https://docs.portkey.ai/docs/changelog/2025/june.md) - \[March\](https://docs.portkey.ai/docs/changelog/2025/mar.md) - \[May\](https://docs.portkey.ai/docs/changelog/2025/may.md) - \[September\](https://docs.portkey.ai/docs/changelog/2025/september.md) - \[Backend\](https://docs.portkey.ai/docs/changelog/backend.md) - \[Data Service\](https://docs.portkey.ai/docs/changelog/data-service.md) - \[Enterprise Gateway\](https://docs.portkey.ai/docs/changelog/enterprise.md) - \[Frontend\](https://docs.portkey.ai/docs/changelog/frontend.md) - \[Helm Chart\](https://docs.portkey.ai/docs/changelog/helm-chart.md) - \[Node.js\](https://docs.portkey.ai/docs/changelog/node-sdk-changelog.md) - \[Latest Updates\](https://docs.portkey.ai/docs/changelog/product.md) - \[Python\](https://docs.portkey.ai/docs/changelog/python-sdk-changelog.md) - \[Overview\](https://docs.portkey.ai/docs/guides/getting-started.md) - \[101 on Portkey's Gateway Configs\](https://docs.portkey.ai/docs/guides/getting-started/101-on-portkey-s-gateway-configs.md): You are likely familiar with how to make an API call to GPT4 for chat completions. - \[A/B Test Prompts and Models\](https://docs.portkey.ai/docs/guides/getting-started/a-b-test-prompts-and-models.md): A/B testing with large language models in production is crucial for driving optimal performance and user satisfaction. - \[Function Calling\](https://docs.portkey.ai/docs/guides/getting-started/function-calling.md): Get the LLM to interact with external APIs! - \[Getting started with AI Gateway\](https://docs.portkey.ai/docs/guides/getting-started/getting-started-with-ai-gateway.md) - \[Image Generation\](https://docs.portkey.ai/docs/guides/getting-started/image-generation.md) - \[Llama 3 on Groq\](https://docs.portkey.ai/docs/guides/getting-started/llama-3-on-groq.md) - \[Return Repeat Requests from Cache\](https://docs.portkey.ai/docs/guides/getting-started/return-repeat-requests-from-cache.md): If you have multiple users of your GenAI app triggering the same or similar queries to your models, fetching LLM response from the models can be slow and expensive. - \[Tackling Rate Limiting\](https://docs.portkey.ai/docs/guides/getting-started/tackling-rate-limiting.md): LLMs are \*\*costly\*\* to run. As their usage increases, the providers have to balance serving user requests v/s straining their GPU resources too thin. They generally deal with this by putting \_rate limits\_ on how many requests a user can send in a minute or in a day. - \[Trigger Automatic Retries on LLM Failures\](https://docs.portkey.ai/docs/guides/getting-started/trigger-automatic-retries-on-llm-failures.md) - \[Overview\](https://docs.portkey.ai/docs/guides/integrations.md) - \[Anyscale\](https://docs.portkey.ai/docs/guides/integrations/anyscale.md): Portkey helps bring Anyscale APIs to production with its abstractions for observability, fallbacks, caching, and more. Use the Anyscale API \*\*through\*\* Portkey for. - \[Deepinfra\](https://docs.portkey.ai/docs/guides/integrations/deepinfra.md) - \[Groq\](https://docs.portkey.ai/docs/guides/integrations/groq.md) - \[Introduction to GPT-4o\](https://docs.portkey.ai/docs/guides/integrations/introduction-to-gpt-4o.md) - \[Langchain\](https://docs.portkey.ai/docs/guides/integrations/langchain.md) - \[Llama 3 on Portkey + Together AI\](https://docs.portkey.ai/docs/guides/integrations/llama-3-on-portkey-+-together-ai.md): Try out the new Llama 3 model directly using the OpenAI SDK - \[Mistral\](https://docs.portkey.ai/docs/guides/integrations/mistral.md): Portkey helps bring Mistral's APIs to production with its observability suite & AI Gateway. - \[Mixtral 8x22b\](https://docs.portkey.ai/docs/guides/integrations/mixtral-8x22b.md) - \[Sync Open WebUI Feedback → Portkey\](https://docs.portkey.ai/docs/guides/integrations/openwebui-to-portkey.md): How to export thumbs-up/down from Open WebUI and ingest into Portkey using a one-file Python or Node script. - \[Segmind\](https://docs.portkey.ai/docs/guides/integrations/segmind.md) - \[Vercel AI\](https://docs.portkey.ai/docs/guides/integrations/vercel-ai.md): Portkey is a control panel for your Vercel AI app. It makes your LLM integrations prod-ready, reliable, fast, and cost-efficient. - \[null\](https://docs.portkey.ai/docs/guides/prompts.md) - \[Build a chatbot using Portkey's Prompt Templates\](https://docs.portkey.ai/docs/guides/prompts/build-a-chatbot-using-portkeys-prompt-templates.md): Portkey's prompt templates offer a powerful solution for testing and building chatbots. - \[Optimizing Prompts for Customer Support using Portkey | LLama Prompt Ops Integration\](https://docs.portkey.ai/docs/guides/prompts/llama-prompts.md) - \[Building an LLM-as-a-Judge System for AI (Customer Support) Agent\](https://docs.portkey.ai/docs/guides/prompts/llm-as-a-judge.md) - \[Ultimate AI SDR\](https://docs.portkey.ai/docs/guides/prompts/ultimate-ai-sdr.md): Building a sophisticated AI SDR agent leveraging internet search and evals to draft personalized outreach emails in 15 seconds - \[Overview\](https://docs.portkey.ai/docs/guides/use-cases.md) - \[Build an article suggestion app with Supabase pgvector, and Portkey\](https://docs.portkey.ai/docs/guides/use-cases/build-an-article-suggestion-app-with-supabase-pgvector-and-portkey.md): Consider that you have list of support articles that you want to suggest it to users when users search for it. You want to suggest as best fit as possible. With the availability of tools like Large Language Model (LLMs) and Vector Databases, the approach towards suggestions & recommendation systems has significantly evolved. - \[Comparing Top10 LMSYS Models with Portkey\](https://docs.portkey.ai/docs/guides/use-cases/comparing-top10-lmsys-models-with-portkey.md) - \[Comparing DeepSeek Models Against OpenAI, Anthropic & More Using Portkey\](https://docs.portkey.ai/docs/guides/use-cases/deepseek-r1.md) - \[Detecting Emotions with GPT-4o\](https://docs.portkey.ai/docs/guides/use-cases/emotions-with-gpt-4o.md) - \[Enforcing JSON Schema with Anyscale & Together\](https://docs.portkey.ai/docs/guides/use-cases/enforcing-json-schema-with-anyscale-and-together.md): Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes - \[Unified LLM API with Automatic Failover & Error Handling\](https://docs.portkey.ai/docs/guides/use-cases/enterprise-ready-unified-api.md): Build a single API interface with automatic failover and unified error handling across OpenAI, Anthropic, and AWS Bedrock - \[Fallback from SDXL to Dall-e-3\](https://docs.portkey.ai/docs/guides/use-cases/fallback-from-sdxl-to-dall-e-3.md): Generative AI models have revolutionized text generation and opened up new possibilities for developers. - \[Testing Application Resilience with Fallbacks\](https://docs.portkey.ai/docs/guides/use-cases/fallbacks-test.md) - \[Few-Shot Prompting\](https://docs.portkey.ai/docs/guides/use-cases/few-shot-prompting.md): LLMs are highly capable of following a given structure. By providing a few examples of how the assistant should respond to a given prompt, the LLM can generate responses that closely follow the format of these examples. - \[How to use Portkey Guardrails for PII Protection\](https://docs.portkey.ai/docs/guides/use-cases/guardrail-pii-use-case.md): Portkey Guardrails is a powerful tool for protecting Personally Identifiable Information (PII) in your AI applications. This guide provides a comprehensive walkthrough of implementing PII protection using Portkey's guardrail capabilities. - \[How to use OpenAI SDK with Portkey Prompt Templates\](https://docs.portkey.ai/docs/guides/use-cases/how-to-use-openai-sdk-with-portkey-prompt-templates.md): Portkeys Prompt Playground allows you to test and tinker with various hyperparameters without any external dependencies and deploy them to production seamlessly. Moreover, all team members can use the same prompt template, ensuring that everyone works from the same source of truth. - \[Web Search for Any LLM in LibreChat\](https://docs.portkey.ai/docs/guides/use-cases/librechat-web-search.md) - \[Multi-Tenant LLM Access with Enterprise Control for Customer-Facing Apps\](https://docs.portkey.ai/docs/guides/use-cases/multi-tenant-ai-feature.md) - \[Portkey with OpenAI Computer Use\](https://docs.portkey.ai/docs/guides/use-cases/openai-computer-use.md): Leverage Portkey with OpenAI's Computer Use tool for automated browser interactions with enterprise-grade observability and controls - \[How to Run Structured Output Evals at Scale\](https://docs.portkey.ai/docs/guides/use-cases/run-batch-evals.md) - \[Run Portkey on Prompts from Langchain Hub\](https://docs.portkey.ai/docs/guides/use-cases/run-portkey-on-prompts-from-langchain-hub.md): Writing the right prompt is often hard to get a quality LLM response. You want the prompt to be specialized and exhaustive enough for your problem. There is a high chance someone else might’ve stumbled across a similar situation and written the prompt you’ve been figuring out all this while. - \[Setting up resilient Load balancers with failure-mitigating Fallbacks\](https://docs.portkey.ai/docs/guides/use-cases/setting-up-resilient-load-balancers-with-failure-mitigating-fallbacks.md): Companies often face challenges of scaling their services efficiently as the traffic to their applications grow - when you’re consuming APIs, the first point of failure is that if you hit the API too much, you can get rate limited. Loadbalancing is a proven way to scale usage horizontally without overburdening any one provider and thus staying within rate limits. - \[Setup OpenAI -> Azure OpenAI Fallback\](https://docs.portkey.ai/docs/guides/use-cases/setup-openai-greater-than-azure-openai-fallback.md): Portkey Fallbacks can automatically switch your app's requests from one LLM provider to another, ensuring reliability by allowing you to fallback among multiple LLMs. - \[Smart Fallback with Model-Optimized Prompts\](https://docs.portkey.ai/docs/guides/use-cases/smart-fallback-with-model-optimized-prompts.md): Portkey can help you easily create fallbacks from one LLM to another, making your application more reliable. While Fallback ensures reliability, it also means that you'll be running a prompt optimized for one LLM on another, which can often lead to significant differences in the final output. - \[Tracking LLM Costs Per User with Portkey\](https://docs.portkey.ai/docs/guides/use-cases/track-costs-using-metadata.md): Monitor and analyze user-level LLM costs across 1600+ models using Portkey's metadata and analytics API. - \[4. Advanced Strategies for Performance Improvement\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/advanced-strategies.md) - \[5. Architectural Considerations\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/architectural-considerations.md) - \[10. Conclusion and Key Takeaways\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/conclusion-and-key-takeaways.md): Summarizing the key strategies for LLM cost optimization and performance improvement - \[7. Cost Effective Development Practices\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/cost-effective-development.md) - \[Executive Summary\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/executive-summary.md): Overview of LLM cost optimization and performance improvement strategies - \[3. FrugalGPT Techniques for Cost Optimization\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/frugalgpt-techniques.md) - \[9. Future Trends in LLM Cost Optimization\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/future-trends.md) - \[1. Introduction\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/introduction.md): An overview of the challenges and opportunities in LLM cost optimization - \[2. Understanding LLM Cost Drivers\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/llm-cost-drivers.md): An overview of the factors that influence costs in Large Language Model applications - \[6. Operational Best Pracitces\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/operational-best-practices.md) - \[8. User Education and Change Management\](https://docs.portkey.ai/docs/guides/whitepapers/optimizing-llm-costs/user-education.md) - \[Overview\](https://docs.portkey.ai/docs/integrations/agents.md): Portkey helps bring your agents to production - \[Agno AI\](https://docs.portkey.ai/docs/integrations/agents/agno-ai.md): Use Portkey with Agno to build production-ready autonomous AI agents - \[Autogen\](https://docs.portkey.ai/docs/integrations/agents/autogen.md): Use Portkey with Autogen to take your AI Agents to production - \[Bring Your own Agents\](https://docs.portkey.ai/docs/integrations/agents/bring-your-own-agents.md): You can also use Portkey if you are doing custom agent orchestration! - \[Control Flow\](https://docs.portkey.ai/docs/integrations/agents/control-flow.md): Use Portkey with Control Flow to take your AI Agents to production - \[CrewAI\](https://docs.portkey.ai/docs/integrations/agents/crewai.md): Use Portkey with CrewAI to take your AI Agents to production - \[Langchain Agents\](https://docs.portkey.ai/docs/integrations/agents/langchain-agents.md) - \[LangGraph\](https://docs.portkey.ai/docs/integrations/agents/langgraph.md): Use Portkey with LangGraph to take your AI agent workflows to production - \[Langroid\](https://docs.portkey.ai/docs/integrations/agents/langroid.md) - \[LiveKit\](https://docs.portkey.ai/docs/integrations/agents/livekit.md): Build production-ready voice AI agents with Portkey's enterprise features - \[Llama Agents by Llamaindex\](https://docs.portkey.ai/docs/integrations/agents/llama-agents.md): Use Portkey with Llama Agents to take your AI Agents to production - \[OpenAI Agents SDK (Python)\](https://docs.portkey.ai/docs/integrations/agents/openai-agents.md): Use Portkey with OpenAI Agents SDK to take your AI Agents to production - \[OpenAI Agents SDK (TypeScript)\](https://docs.portkey.ai/docs/integrations/agents/openai-agents-ts.md): Use Portkey with OpenAI Agents SDK to take your AI Agents to production - \[OpenAI Swarm\](https://docs.portkey.ai/docs/integrations/agents/openai-swarm.md): The Portkey x Swarm integration brings advanced AI gateway capabilities, full-stack observability, and reliability features to build production-ready AI agents. - \[Phidata\](https://docs.portkey.ai/docs/integrations/agents/phidata.md): Use Portkey with Phidata to take your AI Agents to production - \[Pydantic AI\](https://docs.portkey.ai/docs/integrations/agents/pydantic-ai.md): Use Portkey with PydanticAI to take your AI Agents to production - \[Strands Agents\](https://docs.portkey.ai/docs/integrations/agents/strands.md): Use Portkey with AWS's Strands Agents to take your AI Agents to production - \[Overview\](https://docs.portkey.ai/docs/integrations/ai-apps.md) - \[Microsoft Azure\](https://docs.portkey.ai/docs/integrations/cloud/azure.md): Discover how you can build your Gen AI platform on Azure using Portkey - \[Integrations\](https://docs.portkey.ai/docs/integrations/ecosystem.md) - \[Acuvity\](https://docs.portkey.ai/docs/integrations/guardrails/acuvity.md): Acuvity is model agnostic GenAI security solution. It is built to secure existing and future GenAI models, apps, services, tools, plugins and more. - \[Aporia\](https://docs.portkey.ai/docs/integrations/guardrails/aporia.md) - \[Azure Guardrails\](https://docs.portkey.ai/docs/integrations/guardrails/azure-guardrails.md): Integrate Microsoft Azure's powerful content moderation services & PII guardrails with Portkey - \[AWS Bedrock Guardrails\](https://docs.portkey.ai/docs/integrations/guardrails/bedrock-guardrails.md): Secure your AI applications with AWS Bedrock's guardrail capabilities through Portkey. - \[Bring Your Own Guardrails\](https://docs.portkey.ai/docs/integrations/guardrails/bring-your-own-guardrails.md): Integrate your custom guardrails with Portkey using webhooks - \[Javelin\](https://docs.portkey.ai/docs/integrations/guardrails/javelin.md): Javelin provides comprehensive AI security guardrails including Trust & Safety, Prompt Injection Detection, and Language Detection for enterprise AI agents & applications. - \[Lasso Security\](https://docs.portkey.ai/docs/integrations/guardrails/lasso.md): Lasso Security protects your GenAI apps from data leaks, prompt injections, and other potential risks, keeping your systems safe and secure. - \[Mistral\](https://docs.portkey.ai/docs/integrations/guardrails/mistral.md): Mistral moderation service helps detect and filter harmful content across multiple policy dimensions to secure your AI applications. - \[Palo Alto Networks Prisma AIRS\](https://docs.portkey.ai/docs/integrations/guardrails/palo-alto-panw-prisma.md): Comprehensive AI security platform providing runtime protection against prompt injections, data leakage, and AI-specific threats - \[Pangea\](https://docs.portkey.ai/docs/integrations/guardrails/pangea.md): Pangea AI Guard helps analyze and redact text to prevent model manipulation and malicious content. - \[Patronus AI\](https://docs.portkey.ai/docs/integrations/guardrails/patronus-ai.md): Patronus excels in industry-specific guardrails for RAG workflows. - \[Pillar\](https://docs.portkey.ai/docs/integrations/guardrails/pillar.md) - \[Prompt Security\](https://docs.portkey.ai/docs/integrations/guardrails/prompt-security.md): Prompt Security detects and protects against prompt injection, sensitive data exposure, and other AI security threats. - \[Replace Custom Regex Patterns\](https://docs.portkey.ai/docs/integrations/guardrails/regex.md): Redact custom patterns with Regex in Portkey. - \[Overview\](https://docs.portkey.ai/docs/integrations/libraries.md) - \[Anthropic Computer Use\](https://docs.portkey.ai/docs/integrations/libraries/anthropic-computer-use.md) - \[AnythingLLM\](https://docs.portkey.ai/docs/integrations/libraries/anythingllm.md): Add usage tracking, cost controls, and security guardrails to your AnythingLLM deployment - \[Autogen (DEPRECATED)\](https://docs.portkey.ai/docs/integrations/libraries/autogen.md): AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. - \[Claude Code\](https://docs.portkey.ai/docs/integrations/libraries/claude-code.md): Integrate Portkey with Claude Code for enterprise-grade AI coding assistance with observability and governance - \[Cline\](https://docs.portkey.ai/docs/integrations/libraries/cline.md): Add enterprise-grade observability, cost tracking, and governance to your Cline AI coding assistant - \[OpenAI Codex CLI\](https://docs.portkey.ai/docs/integrations/libraries/codex.md): Add usage tracking, cost controls, and security guardrails to your Codex CLI deployment - \[Cursor\](https://docs.portkey.ai/docs/integrations/libraries/cursor.md): Learn how to integrate Portkey's enterprise features with Cursor for enhanced observability, reliability and governance. - \[DSPy\](https://docs.portkey.ai/docs/integrations/libraries/dspy.md): Integrate DSPy with Portkey for production-ready LLM pipelines - \[GitHub Copilot\](https://docs.portkey.ai/docs/integrations/libraries/github-copilot.md): Use Portkey with GitHub Copilot for centralized governance, observability, and cost controls. - \[Goose\](https://docs.portkey.ai/docs/integrations/libraries/goose.md): Add usage tracking, cost controls, and security guardrails to your Goose AI agent - \[Instructor\](https://docs.portkey.ai/docs/integrations/libraries/instructor.md): With Portkey, you can confidently take your Instructor pipelines to production and get complete observability over all of your calls + make them reliable - all with a 2 LOC change! - \[Jan\](https://docs.portkey.ai/docs/integrations/libraries/janhq.md): Add usage tracking, cost controls, and security guardrails to your Jan deployment - \[Langchain (JS/TS)\](https://docs.portkey.ai/docs/integrations/libraries/langchain-js.md): Portkey adds core production capabilities to any Langchain app. - \[Langchain (Python)\](https://docs.portkey.ai/docs/integrations/libraries/langchain-python.md): Supercharge Langchain apps with Portkey: Multi-LLM, observability, caching, reliability, and prompt management. - \[Langflow\](https://docs.portkey.ai/docs/integrations/libraries/langflow.md): Add enterprise-grade features to your Langflow AI workflows with Portkey - \[LibreChat\](https://docs.portkey.ai/docs/integrations/libraries/librechat.md): Cost tracking, observability, and more for LibreChat - \[LlamaIndex (Python)\](https://docs.portkey.ai/docs/integrations/libraries/llama-index-python.md): The \*\*Portkey x LlamaIndex\*\* integration brings advanced \*\*AI gateway\*\* capabilities, full-stack \*\*observability\*\*, and \*\*prompt management\*\* to apps built on LlamaIndex. - \[Microsoft Semantic Kernel\](https://docs.portkey.ai/docs/integrations/libraries/microsoft-semantic-kernel.md) - \[MindsDb\](https://docs.portkey.ai/docs/integrations/libraries/mindsdb.md): Integrate MindsDB with Portkey to build enterprise-grade AI use-cases - \[MongoDB\](https://docs.portkey.ai/docs/integrations/libraries/mongodb.md) - \[n8n\](https://docs.portkey.ai/docs/integrations/libraries/n8n.md): Add usage tracking, cost controls, and security guardrails to your n8n workflows - \[OpenAI Agent Builder (TS)\](https://docs.portkey.ai/docs/integrations/libraries/openai-agent-builder.md): Use Portkey with OpenAI Agent Builder to add observability, reliability, and production features to your visual agent workflows - \[OpenAI Agent Builder (Python)\](https://docs.portkey.ai/docs/integrations/libraries/openai-agent-builder-python.md): Use Portkey with OpenAI Agent Builder to add observability, reliability, and production features to your visual agent workflows - \[Portkey with Any OpenAI Compatible Project\](https://docs.portkey.ai/docs/integrations/libraries/openai-compatible.md): Learn how to integrate Portkey's enterprise features with any OpenAI Compliant project for enhanced observability, reliability and governance. - \[Open WebUI\](https://docs.portkey.ai/docs/integrations/libraries/openwebui.md): Enterprise-grade cost tracking, observability, and more for Open WebUI - \[Image Genration with OpenWebUI\](https://docs.portkey.ai/docs/integrations/libraries/openwebui/image-generation-with-openwebui.md) - \[Promptfoo\](https://docs.portkey.ai/docs/integrations/libraries/promptfoo.md): Portkey brings advanced \*\*AI gateway\*\* capabilities, full-stack \*\*observability\*\*, and \*\*prompt management\*\* + \*\*versioning\*\* to your \*\*Promptfoo\*\* projects. This document provides an overview of how to leverage the strengths of both the platforms to streamline your AI development workflow. - \[Roo Code\](https://docs.portkey.ai/docs/integrations/libraries/roo-code.md): Add enterprise-grade observability, cost tracking, and governance to your Roo AI coding assistant - \[Supabase\](https://docs.portkey.ai/docs/integrations/libraries/supabase.md) - \[ToolJet\](https://docs.portkey.ai/docs/integrations/libraries/tooljet.md): ToolJet is a low-code platform that lets you build apps by connecting APIs and data sources, with Portkey integration adding AI features like chat interfaces and automation. - \[Vercel\](https://docs.portkey.ai/docs/integrations/libraries/vercel.md): Integrate Portkey with Vercel AI SDK for production-ready and reliable AI apps - \[Zed\](https://docs.portkey.ai/docs/integrations/libraries/zed.md): Learn how to integrate Portkey's enterprise features with Zed for enhanced observability, reliability and governance. - \[Overview\](https://docs.portkey.ai/docs/integrations/llms.md): Portkey connects with all major LLM providers and orchestration frameworks. - \[AI21\](https://docs.portkey.ai/docs/integrations/llms/ai21.md) - \[Anthropic\](https://docs.portkey.ai/docs/integrations/llms/anthropic.md) - \[Computer use tool\](https://docs.portkey.ai/docs/integrations/llms/anthropic/computer-use.md) - \[Count Tokens\](https://docs.portkey.ai/docs/integrations/llms/anthropic/count-tokens.md) - \[Prompt Caching\](https://docs.portkey.ai/docs/integrations/llms/anthropic/prompt-caching.md) - \[Remote MCP Support\](https://docs.portkey.ai/docs/integrations/llms/anthropic/remote-mcp.md) - \[Anyscale\](https://docs.portkey.ai/docs/integrations/llms/anyscale-llama2-mistral-zephyr.md): Integrate Anyscale endpoints with Portkey seamlessly and make your OSS models production-ready - \[AWS SageMaker\](https://docs.portkey.ai/docs/integrations/llms/aws-sagemaker.md): Route to your AWS Sagemaker models through Portkey - \[Azure AI Foundry\](https://docs.portkey.ai/docs/integrations/llms/azure-foundry.md): Learn how to integrate Azure AI Foundry with Portkey to access a wide range of AI models with enhanced observability and reliability features. - \[Azure OpenAI\](https://docs.portkey.ai/docs/integrations/llms/azure-openai/azure-openai.md): Azure OpenAI is a great alternative to accessing the best models including GPT-4 and more in your private environments. Portkey provides complete support for Azure OpenAI. - \[Batches\](https://docs.portkey.ai/docs/integrations/llms/azure-openai/batches.md): Perform batch inference with Azure OpenAI - \[Files\](https://docs.portkey.ai/docs/integrations/llms/azure-openai/files.md): Upload files to Azure OpenAI - \[Fine-tune\](https://docs.portkey.ai/docs/integrations/llms/azure-openai/fine-tuning.md): Fine-tune your models with Azure OpenAI - \[AWS Bedrock\](https://docs.portkey.ai/docs/integrations/llms/bedrock/aws-bedrock.md) - \[Batches\](https://docs.portkey.ai/docs/integrations/llms/bedrock/batches.md): Perform batch inference with Bedrock - \[AWS Bedrock Knowledge Bases\](https://docs.portkey.ai/docs/integrations/llms/bedrock/bedrock-knowledgebase.md): Create, manage, and connect your LLMs to organizational data using AWS Bedrock Knowledge Bases through Portkey. - \[Embeddings\](https://docs.portkey.ai/docs/integrations/llms/bedrock/embeddings.md): Get embeddings from Bedrock - \[Files\](https://docs.portkey.ai/docs/integrations/llms/bedrock/files.md): Upload files to S3 for Bedrock batch inference - \[Fine-tune\](https://docs.portkey.ai/docs/integrations/llms/bedrock/fine-tuning.md): Fine-tune your models with Bedrock - \[Prompt Caching on Bedrock\](https://docs.portkey.ai/docs/integrations/llms/bedrock/prompt-caching.md) - \[Bring Your Own LLM\](https://docs.portkey.ai/docs/integrations/llms/byollm.md): Integrate your privately hosted LLMs with Portkey for unified management, observability, and reliability. - \[Cerebras\](https://docs.portkey.ai/docs/integrations/llms/cerebras.md) - \[Cohere\](https://docs.portkey.ai/docs/integrations/llms/cohere.md) - \[Dashscope\](https://docs.portkey.ai/docs/integrations/llms/dashscope.md): Integrate Dashscope with Portkey for seamless completions, prompt management, and advanced features like streaming, function calling, and fine-tuning. - \[Deepbricks\](https://docs.portkey.ai/docs/integrations/llms/deepbricks.md): Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including \[Deepbricks\](https://deepbricks.ai/). - \[Deepgram\](https://docs.portkey.ai/docs/integrations/llms/deepgram.md): Portkey provides a robust and secure gateway to use and observe Deepgrm's Speech-to-Text API. - \[Deepinfra\](https://docs.portkey.ai/docs/integrations/llms/deepinfra.md) - \[DeepSeek\](https://docs.portkey.ai/docs/integrations/llms/deepseek.md): Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including DeepSeek models. - \[Featherless AI\](https://docs.portkey.ai/docs/integrations/llms/featherless.md) - \[Fireworks\](https://docs.portkey.ai/docs/integrations/llms/fireworks.md) - \[Files\](https://docs.portkey.ai/docs/integrations/llms/fireworks/files.md): Upload files to Fireworks - \[Fine-tune\](https://docs.portkey.ai/docs/integrations/llms/fireworks/fine-tuning.md): Fine-tune your models with Bedrock - \[Google Gemini\](https://docs.portkey.ai/docs/integrations/llms/gemini.md) - \[Github\](https://docs.portkey.ai/docs/integrations/llms/github.md) - \[Groq\](https://docs.portkey.ai/docs/integrations/llms/groq.md) - \[Hugging Face\](https://docs.portkey.ai/docs/integrations/llms/huggingface.md) - \[Inference.net\](https://docs.portkey.ai/docs/integrations/llms/inference.net.md): Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including the models hosted on \[Inference.net\](https://www.inference.net/). - \[Jina AI\](https://docs.portkey.ai/docs/integrations/llms/jina-ai.md) - \[Lambda Labs\](https://docs.portkey.ai/docs/integrations/llms/lambda.md): Integrate Lambda with Portkey AI for seamless completions, prompt management, and advanced features like streaming and function calling. - \[Lemonfox-AI\](https://docs.portkey.ai/docs/integrations/llms/lemon-fox.md): Integrate LemonFox with Portkey for seamless completions, prompt management, and advanced features like streaming, function calling, and fine-tuning. - \[Lepton AI\](https://docs.portkey.ai/docs/integrations/llms/lepton.md) - \[Lingyi (01.ai)\](https://docs.portkey.ai/docs/integrations/llms/lingyi-01.ai.md) - \[LocalAI\](https://docs.portkey.ai/docs/integrations/llms/local-ai.md) - \[Mistral AI\](https://docs.portkey.ai/docs/integrations/llms/mistral-ai.md) - \[Monster API\](https://docs.portkey.ai/docs/integrations/llms/monster-api.md): MonsterAPIs provides access to generative AI model APIs at 80% lower costs. Connect to MonsterAPI LLM APIs seamlessly through Portkey's AI gateway. - \[Moonshot\](https://docs.portkey.ai/docs/integrations/llms/moonshot.md) - \[Ncompass\](https://docs.portkey.ai/docs/integrations/llms/ncompass.md) - \[Nebius\](https://docs.portkey.ai/docs/integrations/llms/nebius.md) - \[Nomic\](https://docs.portkey.ai/docs/integrations/llms/nomic.md) - \[Novita AI\](https://docs.portkey.ai/docs/integrations/llms/novita-ai.md) - \[Nscale (EU Sovereign)\](https://docs.portkey.ai/docs/integrations/llms/nscale.md) - \[Ollama\](https://docs.portkey.ai/docs/integrations/llms/ollama.md) - \[OpenAI\](https://docs.portkey.ai/docs/integrations/llms/openai.md): Learn to integrate OpenAI with Portkey, enabling seamless completions, prompt management, and advanced functionalities like streaming, function calling and fine-tuning. - \[Batches\](https://docs.portkey.ai/docs/integrations/llms/openai/batches.md): Perform batch inference with OpenAI - \[Files\](https://docs.portkey.ai/docs/integrations/llms/openai/files.md): Upload files to OpenAI - \[Fine-tune\](https://docs.portkey.ai/docs/integrations/llms/openai/fine-tuning.md): Fine-tune your models with OpenAI - \[Prompt Caching\](https://docs.portkey.ai/docs/integrations/llms/openai/prompt-caching-openai.md) - \[Remote MCP Support\](https://docs.portkey.ai/docs/integrations/llms/openai/remote-mcp.md) - \[Structured Outputs\](https://docs.portkey.ai/docs/integrations/llms/openai/structured-outputs.md): Structured Outputs ensure that the model always follows your supplied \[JSON schema\](https://json-schema.org/overview/what-is-jsonschema). Portkey supports OpenAI's Structured Outputs feature out of the box with our SDKs & APIs. - \[OpenRouter\](https://docs.portkey.ai/docs/integrations/llms/openrouter.md) - \[Perplexity AI\](https://docs.portkey.ai/docs/integrations/llms/perplexity-ai.md) - \[Predibase\](https://docs.portkey.ai/docs/integrations/llms/predibase.md) - \[Recraft AI\](https://docs.portkey.ai/docs/integrations/llms/recraft-ai.md) - \[Reka AI\](https://docs.portkey.ai/docs/integrations/llms/reka-ai.md) - \[Replicate\](https://docs.portkey.ai/docs/integrations/llms/replicate.md) - \[SambaNova\](https://docs.portkey.ai/docs/integrations/llms/sambanova.md): Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including \[SambaNova AI\](https://sambanova.ai/). - \[Segmind\](https://docs.portkey.ai/docs/integrations/llms/segmind.md) - \[SiliconFlow\](https://docs.portkey.ai/docs/integrations/llms/siliconflow.md) - \[Snowflake Cortex\](https://docs.portkey.ai/docs/integrations/llms/snowflake-cortex.md) - \[Stability AI\](https://docs.portkey.ai/docs/integrations/llms/stability-ai.md) - \[Suggest a new integration!\](https://docs.portkey.ai/docs/integrations/llms/suggest-a-new-integration.md) - \[Together AI\](https://docs.portkey.ai/docs/integrations/llms/together-ai.md) - \[Triton\](https://docs.portkey.ai/docs/integrations/llms/triton.md): Integrate Trtiton-hosted custom models with Portkey and take them to production - \[Upstage AI\](https://docs.portkey.ai/docs/integrations/llms/upstage.md): Integrate Upstage with Portkey AI for seamless completions, prompt management, and advanced features like streaming and embedding. - \[Google Vertex AI\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai.md) - \[Batches\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai/batches.md): Perform batch inference with Vertex AI - \[Controlled Generations\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai/controlled-generations.md): Controlled Generations ensure that the model always follows your supplied \[JSON schema\](https://json-schema.org/overview/what-is-jsonschema). Portkey supports Vertex AI's Controlled Generations feature out of the box with our SDKs & APIs. - \[Embeddings\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai/embeddings.md): Get embeddings from Vertex AI - \[Files\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai/files.md): Upload files to Google Cloud Storage for Vertex AI fine-tuning and batch inference - \[Fine-tune\](https://docs.portkey.ai/docs/integrations/llms/vertex-ai/fine-tuning.md): Fine-tune your models with Vertex AI - \[vLLM\](https://docs.portkey.ai/docs/integrations/llms/vllm.md): Integrate vLLM-hosted custom models with Portkey and take them to production - \[Voyage AI\](https://docs.portkey.ai/docs/integrations/llms/voyage-ai.md): Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including Voyage AI's embedding and Re-rank endpoints. - \[Workers AI\](https://docs.portkey.ai/docs/integrations/llms/workers-ai.md) - \[xAI (Grok)\](https://docs.portkey.ai/docs/integrations/llms/x-ai.md): Portkey supports xAI's chat completions, completions, and embeddings APIs. - \[ZhipuAI / ChatGLM / BigModel\](https://docs.portkey.ai/docs/integrations/llms/zhipu.md) - \[Claude Desktop\](https://docs.portkey.ai/docs/integrations/mcp-clients/claude.md): Connect your MCP servers to Claude Desktop through Portkey's secure MCP gateway - \[Claude Code\](https://docs.portkey.ai/docs/integrations/mcp-clients/claude-code.md): Connect your MCP servers to Claude Code CLI through Portkey's secure MCP gateway - \[Cursor\](https://docs.portkey.ai/docs/integrations/mcp-clients/cursor.md): Connect your MCP servers to Cursor IDE through Portkey's secure MCP gateway - \[LibreChat\](https://docs.portkey.ai/docs/integrations/mcp-clients/librechat.md): Connect your MCP servers to LibreChat through Portkey's secure MCP gateway - \[VS Code\](https://docs.portkey.ai/docs/integrations/mcp-clients/vs-code.md): Connect your MCP servers to Visual Studio Code through Portkey's secure MCP gateway - \[Atlassian Remote MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/atlassian-mcp-server.md): The Atlassian Remote MCP Server is a cloud-based bridge between your Atlassian Cloud site and MCP-compatible clients, enabling secure interaction with Jira, Confluence, and Compass in real time. - \[Cerebras Code MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/cerebras-code-mcp-server.md): The \*\*Cerebras Code MCP Server (v1.3.3)\*\* integrates high-speed code generation and editing capabilities into AI-assisted IDEs such as Claude Code, Cline, and Cursor. It allows you to use your preferred AI (Claude, Qwen, etc.) for planning and strategy, and delegate the actual code-writing and modification tasks to Cerebras' optimized models — maximizing speed while avoiding API rate limits. - \[Datadog MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/datadog-mcp-server.md): The Datadog MCP server enables AI agents to interact with Datadog monitoring, dashboards, metrics, logs, and alerts through MCP. Built for conversational and automated observability workflows. - \[Figma MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/figma-mcp-server.md): The Figma MCP server connects Figma's design environment to MCP clients, enabling agents to translate designs into code, extract variables, and align output with your design system. - \[Firebase MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/firebase-mcp-server.md): The Firebase MCP server gives AI agents programmatic access to Firebase projects, including Auth user management, Firestore data access, Storage rules, and GraphQL features via the Firebase CLI. - \[Firecrawl MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/firecrawl-mcp-server.md): The Firecrawl MCP server enables AI agents to perform web scraping, crawling, search, extraction, and research through MCP. Built for content discovery and information retrieval workflows. - \[GitHub MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/github-mcp-server.md): The GitHub MCP server enables AI agents to safely interact with repositories, issues, pull requests, commits, and code search through MCP. Built for conversational and automated engineering workflows. - \[Gitlab MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/gitlab-mcp-server.md): The GitLab MCP server enables AI agents to interact with GitLab—the DevOps platform for software development, security, and operations. - \[Grafana MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/grafana-mcp-server.md): The Grafana MCP server enables AI agents to interact with Grafana dashboards, metrics, logs, incidents, alerts, and monitoring tools through MCP. Built for observability and monitoring workflows. - \[Kubernetes MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/kubernetes-mcp-server.md): The Kubernetes MCP server provides a flexible Model Context Protocol (MCP) interface for managing and interacting with Kubernetes and OpenShift clusters. - \[Linear MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/linear-mcp-server.md): The Linear MCP server provides a standardized interface for AI models and agents to securely access and manage data from your Linear workspace. - \[Notion MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/notion-mcp-server.md): The Notion MCP server enables AI agents to interact with Notion, a versatile workspace that combines note-taking, project management, knowledge bases, and relational databases. Through MCP, assistants can programmatically retrieve, search, and update pages, blocks, and databases—turning Notion into a live context source or task sink for AI workflows. - \[Open Library MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/open-library-mcp-server.md): The Open Library MCP server provides AI agents with access to Open Library's book and author catalog through a simple read-only API with no authentication required. - \[Playwright MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/playwright-mcp-server.md): The Playwright MCP server provides browser automation capabilities using Playwright through the Model Context Protocol (MCP), enabling deterministic web interaction via accessibility trees rather than screenshots. - \[Postman MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/postman-mcp-server.md): The Postman MCP Server connects Postman to AI tools, enabling assistants and agents to securely access and interact with Postman workspaces, collections, environments, and APIs through natural language. It bridges Postman's API platform with AI-driven automation, allowing developers to query, evaluate, and modify Postman resources directly from their MCP-compatible clients. - \[Qdrant MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/qdrant-mcp-server.md): The Qdrant MCP server enables AI agents to store and retrieve semantic information using vector search through MCP. Built for persistent memory, contextual knowledge, and efficient semantic retrieval workflows. - \[Stripe MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/stripe-mcp-server.md): The Stripe MCP server exposes Stripe's payments and billing operations to AI agents through the Model Context Protocol, providing secure access to customers, products, pricing, invoices, and more. - \[Vectara MCP server\](https://docs.portkey.ai/docs/integrations/mcp-servers/vectara-mcp-server.md): The Vectara MCP server provides AI agents with access to Vectara's semantic search and retrieval-augmented generation (RAG) APIs. - \[Submit an Integration\](https://docs.portkey.ai/docs/integrations/partner.md) - \[Exa Online Search\](https://docs.portkey.ai/docs/integrations/plugins/exa.md): Transform offline LLMs into online models with real-time internet search capabilities. - \[Arize Phoenix\](https://docs.portkey.ai/docs/integrations/tracing-providers/arize.md): Extend Portkey’s powerful AI Gateway with Arize Phoenix for unified LLM observability, tracing, and analytics across your ML stack. - \[FutureAGI\](https://docs.portkey.ai/docs/integrations/tracing-providers/future-agi.md): Integrate FutureAGI with Portkey for automated LLM evaluation and comprehensive observability - \[HoneyHive\](https://docs.portkey.ai/docs/integrations/tracing-providers/honeyhive.md): Integrate HoneyHive observability with Portkey's AI gateway for comprehensive LLM monitoring and advanced routing capabilities - \[Langfuse\](https://docs.portkey.ai/docs/integrations/tracing-providers/langfuse.md): Integrate Langfuse observability with Portkey's AI gateway for comprehensive LLM monitoring and advanced routing capabilities - \[Langsmith\](https://docs.portkey.ai/docs/integrations/tracing-providers/langsmith.md): Integrate LangSmith observability with Portkey's AI gateway for comprehensive LLM monitoring and advanced routing capabilities - \[Pydantic Logfire\](https://docs.portkey.ai/docs/integrations/tracing-providers/logfire.md): Modern Python observability with automatic OpenAI instrumentation and intelligent gateway routing - \[MLflow Tracing\](https://docs.portkey.ai/docs/integrations/tracing-providers/ml-flow.md): Enhance LLM observability with automatic tracing and intelligent gateway routing - \[OpenLIT\](https://docs.portkey.ai/docs/integrations/tracing-providers/openlit.md): Simplify AI development with OpenTelemetry-native observability and intelligent gateway routing - \[OpenTelemetry Python SDK\](https://docs.portkey.ai/docs/integrations/tracing-providers/opentelemetry-python-sdk.md): Direct OpenTelemetry instrumentation with full control over traces and intelligent gateway routing - \[Phoenix(Arize) Open-Telemetry\](https://docs.portkey.ai/docs/integrations/tracing-providers/phoenix.md): AI observability and debugging platform with OpenInference instrumentation and intelligent gateway routing - \[Traceloop (OpenLLMetry)\](https://docs.portkey.ai/docs/integrations/tracing-providers/traceloop.md) - \[Milvus\](https://docs.portkey.ai/docs/integrations/vector-databases/milvus.md) - \[Qdrant\](https://docs.portkey.ai/docs/integrations/vector-databases/qdrant.md) - \[Portkey Features\](https://docs.portkey.ai/docs/introduction/feature-overview.md): Explore the powerful features of Portkey - \[Make Your First Request\](https://docs.portkey.ai/docs/introduction/make-your-first-request.md): Integrate Portkey and analyze your first LLM call in 2 minutes! - \[What is Portkey?\](https://docs.portkey.ai/docs/introduction/what-is-portkey.md): Portkey AI is a comprehensive platform designed to streamline and enhance AI integration for developers and organizations. It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in your Generative AI apps. - \[Configure API Key Access Permissions for Workspaces\](https://docs.portkey.ai/docs/product/administration/configure-api-key-access-permissions.md) - \[Configure Logs Access Permissions for Workspace\](https://docs.portkey.ai/docs/product/administration/configure-logs-access-permissions-in-workspace.md) - \[Configure Model Access Permissioning for AI Integrations\](https://docs.portkey.ai/docs/product/administration/configure-model-access-permissioning-for-ai-integratinon.md) - \[Configure Virtual Key Access Permission for Workspaces\](https://docs.portkey.ai/docs/product/administration/configure-virtual-key-access-permissions.md) - \[Configure Workspace Provisioning for Your AI Integrations\](https://docs.portkey.ai/docs/product/administration/configure-workspace-provisioning-for-your-ai-integrations.md) - \[Configure Request Logging\](https://docs.portkey.ai/docs/product/administration/configuring-request-logging.md): Control what data is stored for all LLM requests in your organization - \[Enforce Budget Limits and Rate Limits for Your API Keys\](https://docs.portkey.ai/docs/product/administration/enforce-budget-and-rate-limit.md): Configure budget and rate limits on API keys to effectively manage AI spending and usage across your organization - \[Enforce Budget Limits on Your AI Provider\](https://docs.portkey.ai/docs/product/administration/enforce-budget-limits-on-your-ai-provider.md) - \[Enforcing Default Configs on API Keys\](https://docs.portkey.ai/docs/product/administration/enforce-default-config.md): Learn how to attach default configs to API keys for enforcing governance controls across your organization - \[Enforcing Org Level Guardrails\](https://docs.portkey.ai/docs/product/administration/enforce-orgnization-level-guardrails.md) - \[Enforce Rate Limits on Your AI Provider\](https://docs.portkey.ai/docs/product/administration/enforce-rate-limits-on-your-ai-provider.md) - \[Enforce Workspace Budget and Rate Limits\](https://docs.portkey.ai/docs/product/administration/enforce-workspace-budget-limts-and-rate-limits.md): Configure budget and rate limits at the workspace level to effectively manage AI spending and resource allocation - \[Enforcing Workspace Level Guardrails\](https://docs.portkey.ai/docs/product/administration/enforce-workspace-level-guardials.md) - \[Enforcing Request Metadata\](https://docs.portkey.ai/docs/product/administration/enforcing-request-metadata.md) - \[AI Gateway\](https://docs.portkey.ai/docs/product/ai-gateway.md): The world's fastest AI Gateway with advanced routing & integrated Guardrails. - \[Automatic Retries\](https://docs.portkey.ai/docs/product/ai-gateway/automatic-retries.md): LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature. - \[Unified Batch Inference\](https://docs.portkey.ai/docs/product/ai-gateway/batches.md): Run large‑scale inference jobs through one consistent endpoint - \[Cache (Simple & Semantic)\](https://docs.portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic.md) - \[Canary Testing\](https://docs.portkey.ai/docs/product/ai-gateway/canary-testing.md): You can use Portkey's AI gateway to also canary test new models or prompts in different environments. - \[Circuit Breaker\](https://docs.portkey.ai/docs/product/ai-gateway/circuit-breaker.md): Configure per-strategy circuit protection and failure handling - \[Conditional Routing\](https://docs.portkey.ai/docs/product/ai-gateway/conditional-routing.md) - \[Configs\](https://docs.portkey.ai/docs/product/ai-gateway/configs.md): This feature is available on all Portkey plans. - \[Fallbacks\](https://docs.portkey.ai/docs/product/ai-gateway/fallbacks.md) - \[Files\](https://docs.portkey.ai/docs/product/ai-gateway/files.md): Upload files to Portkey and reuse the content in your requests - \[Fine-tuning\](https://docs.portkey.ai/docs/product/ai-gateway/fine-tuning.md): Run your fine-tuning jobs with Portkey Gateway - \[Load Balancing\](https://docs.portkey.ai/docs/product/ai-gateway/load-balancing.md): Load Balancing feature efficiently distributes network traffic across multiple LLMs. - \[Multimodal Capabilities\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities.md) - \[Function Calling\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/function-calling.md): Portkey's AI Gateway supports function calling capabilities that many foundational model providers offer. In the API call you can describe functions and the model can choose to output text or this function name with parameters. - \[Image Generation\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/image-generation.md): Portkey's AI gateway supports image generation capabilities that many foundational model providers offer. - \[Speech-to-Text\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/speech-to-text.md): Portkey's AI gateway supports STT models like Whisper by OpenAI. - \[Text-to-Speech\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/text-to-speech.md): Portkey's AI gateway currently supports text-to-speech models on \`OpenAI\` and \`Azure OpenAI\`. - \[Thinking Mode\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/thinking-mode.md) - \[Vision\](https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/vision.md): Portkey's AI gateway supports vision models like GPT-4V by OpenAI, Gemini by Google and more. - \[Realtime API\](https://docs.portkey.ai/docs/product/ai-gateway/realtime-api.md): Use OpenAI's Realtime API with logs, cost tracking, and more! - \[Remote MCP\](https://docs.portkey.ai/docs/product/ai-gateway/remote-mcp.md): Portkey's AI gateway has MCP server support that many foundational model providers offer. - \[Request Timeouts\](https://docs.portkey.ai/docs/product/ai-gateway/request-timeouts.md): Manage unpredictable LLM latencies effectively with Portkey's \*\*Request Timeouts\*\*. - \[Strict OpenAI Compliance\](https://docs.portkey.ai/docs/product/ai-gateway/strict-open-ai-compliance.md) - \[Universal API\](https://docs.portkey.ai/docs/product/ai-gateway/universal-api.md): Portkey's Universal API provides a consistent interface to integrate a wide range of modalities (text, vision, audio) and LLMs (hosted OR local) into your apps. - \[Virtual Keys\](https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys.md): Portkey's virtual key system allows you to securely store your LLM API keys in our vault, utilizing a unique virtual identifier to streamline API key management. - \[Connect Bedrock with Amazon Assumed Role\](https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.md): How to create a new integration for Bedrock using Amazon Assumed Role Authentication - \[Budget Limits\](https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits.md): Budget Limits lets you set cost limits on providers/integrations - \[Rate Limits\](https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/rate-limits.md): Set Rate Limts to your Integrations/Providers - \[Enterprise Offering\](https://docs.portkey.ai/docs/product/enterprise-offering.md) - \[Access Control Management\](https://docs.portkey.ai/docs/product/enterprise-offering/access-control-management.md): With customizable user roles, API key management, and comprehensive audit logs, Portkey provides the flexibility and control needed to ensure secure collaboration & maintain a strong security posture - \[OTel Integration (Analytics Data)\](https://docs.portkey.ai/docs/product/enterprise-offering/analytics-logs-export.md) - \[Audit Logs\](https://docs.portkey.ai/docs/product/enterprise-offering/audit-logs.md): Track and monitor all administrative activities across your Portkey organization with comprehensive audit logging. - \[Budget Limits\](https://docs.portkey.ai/docs/product/enterprise-offering/budget-limits.md) - \[Enterprise Components\](https://docs.portkey.ai/docs/product/enterprise-offering/components.md) - \[KMS Integration\](https://docs.portkey.ai/docs/product/enterprise-offering/kms.md): Customers can bring their own encryption keys to Portkey AI to encrypt data at storage. - \[Logs Export\](https://docs.portkey.ai/docs/product/enterprise-offering/logs-export.md) - \[Org Management\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management.md): A high-level introduction to Portkey's organization management structure and key concepts. - \[API Keys (AuthN and AuthZ)\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/api-keys-authn-and-authz.md): Discover how Admin and Workspace API Keys are used to manage access and operations in Portkey. - \[JWT Authentication\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/jwt.md): Configure JWT-based authentication for your organization in Portkey - \[Organizations\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/organizations.md): Understand the role and features of Organizations, the highest level of abstraction in Portkey's structure. - \[Azure Entra\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/scim/azure-ad.md): Setup Azure Entra for SCIM provisioning with Portkey. - \[Okta\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/scim/okta.md): Set up Okta for SCIM provisioning with Portkey. - \[Overview\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/scim/scim.md): SCIM integration with Portkey. - \[SSO\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/sso.md): SSO support for enterprises - \[User Roles & Permissions\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/user-roles-and-permissions.md): Learn about Portkey's comprehensive role-based access control system across Organizations and Workspaces. - \[Workspaces\](https://docs.portkey.ai/docs/product/enterprise-offering/org-management/workspaces.md): Explore Workspaces, the sub-organizational units that enable granular project and team management. - \[Security @ Portkey\](https://docs.portkey.ai/docs/product/enterprise-offering/security-portkey.md): Portkey AI provides a secure, reliable AI gateway for the seamless integration and management of large language models (LLMs). - \[Guardrails\](https://docs.portkey.ai/docs/product/guardrails.md): Ship to production confidently with Portkey Guardrails on your requests & responses - \[Creating Raw Guardrails (in JSON)\](https://docs.portkey.ai/docs/product/guardrails/creating-raw-guardrails-in-json.md): With the raw Guardrails mode, we let you define your Guardrail checks & actions however you want, directly in code. - \[Guardrails for Embedding Requests\](https://docs.portkey.ai/docs/product/guardrails/embedding-guardrails.md): Apply security and data validation measures to vector embedding requests to protect sensitive information and ensure data quality. - \[List of Guardrail Checks\](https://docs.portkey.ai/docs/product/guardrails/list-of-guardrail-checks.md) - \[PII Redaction\](https://docs.portkey.ai/docs/product/guardrails/pii-redaction.md): Replace any sensitive data in requests with standard identifiers - \[MCP\](https://docs.portkey.ai/docs/product/mcp.md) - \[MCP Gateway\](https://docs.portkey.ai/docs/product/mcp-gateway.md): The universal adapter for Model Context Protocol—connect once, access every MCP tool . - \[Architecture\](https://docs.portkey.ai/docs/product/mcp-gateway/architecture.md) - \[Deployment\](https://docs.portkey.ai/docs/product/mcp-gateway/deployment.md) - \[MCP Clients\](https://docs.portkey.ai/docs/product/mcp-gateway/mcp-clients.md) - \[MCP Hub\](https://docs.portkey.ai/docs/product/mcp-gateway/mcp-hub.md): Portkey's MCP Hub to connect 1600+ LLMs with any Remote MCP Server - \[Quickstart\](https://docs.portkey.ai/docs/product/mcp-gateway/quickstart.md) - \[Model Catalog\](https://docs.portkey.ai/docs/product/model-catalog.md): A single pane to view and manage every AI provider and model in your organization. It provides centralized governance, discovery, and usage controls for all your AI resources. - \[Budget Limits\](https://docs.portkey.ai/docs/product/model-catalog/budget-limits.md) - \[Connect Bedrock with Amazon Assumed Role\](https://docs.portkey.ai/docs/product/model-catalog/connect-bedrock-with-amazon-assumed-role.md): How to create an integrate Bedrock using Amazon Assumed Role Authentication on Portkey - \[Adding Custom Models\](https://docs.portkey.ai/docs/product/model-catalog/custom-models.md): Learn how to add custom and fine-tuned models to your Portkey Model Catalog for seamless integration. - \[Integrations\](https://docs.portkey.ai/docs/product/model-catalog/integrations.md): Securely store and manage AI provider credentials across your organization with centralized governance controls - \[Overriding Model Details\](https://docs.portkey.ai/docs/product/model-catalog/model-overrides.md): Learn how to set custom pricing for any base model in your Portkey Model Catalog to reflect your specific costs. - \[AI Model Provisioning\](https://docs.portkey.ai/docs/product/model-catalog/model-provisioning.md) - \[Rate Limits\](https://docs.portkey.ai/docs/product/model-catalog/rate-limits.md) - \[Workspace Provisioning\](https://docs.portkey.ai/docs/product/model-catalog/workspace-provisioning.md) - \[Observability (OpenTelemetry)\](https://docs.portkey.ai/docs/product/observability.md): Gain real-time insights, track key metrics, and streamline debugging with our comprehensive observability suite. - \[Analytics\](https://docs.portkey.ai/docs/product/observability/analytics.md) - \[Auto-Instrumentation \[BETA\]\](https://docs.portkey.ai/docs/product/observability/auto-instrumentation.md): Portkey's auto-instrumentation allows you to instrument tracing and logging for multiple LLM/Agent frameworks and view the logs, traces, and metrics in a single place. - \[Budget Limits\](https://docs.portkey.ai/docs/product/observability/budget-limits.md) - \[Model Pricing and Cost Management\](https://docs.portkey.ai/docs/product/observability/cost-management.md): Learn how Portkey handles pricing data, cost calculations, and pricing updates across different deployment modes. - \[Feedback\](https://docs.portkey.ai/docs/product/observability/feedback.md): Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. - \[Filters\](https://docs.portkey.ai/docs/product/observability/filters.md) - \[Logs\](https://docs.portkey.ai/docs/product/observability/logs.md): The Logs section presents a chronological list of all the requests processed through Portkey. - \[Logs Export\](https://docs.portkey.ai/docs/product/observability/logs-export.md): Easily access your Portkey logs data for further analysis and reporting - \[Metadata\](https://docs.portkey.ai/docs/product/observability/metadata.md): Add custom context to your AI requests for better observability and analytics - \[OpenTelemetry for LLM Observability\](https://docs.portkey.ai/docs/product/observability/opentelemetry.md): Leverage OpenTelemetry with Portkey for comprehensive LLM application observability, combining gateway insights with full-stack telemetry. - \[Supported OTel Libraries\](https://docs.portkey.ai/docs/product/observability/opentelemetry/list-of-supported-otel-instrumenters.md): Portkey works with any OpenTelemetry-compatible instrumentation. Here are some popular options. - \[Tracing\](https://docs.portkey.ai/docs/product/observability/traces.md): The \*\*Tracing\*\* capabilities in Portkey empowers you to monitor the lifecycle of your LLM requests in a unified, chronological view. - \[Open Source\](https://docs.portkey.ai/docs/product/open-source.md) - \[Feature Comparison\](https://docs.portkey.ai/docs/product/product-feature-comparison.md): Comparing Portkey's Open-source version and Dev, Pro, Enterprise plans. - \[Prompt Engineering Studio\](https://docs.portkey.ai/docs/product/prompt-engineering-studio.md) - \[Prompt API\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-api.md): Learn how to integrate Portkey's prompt templates directly into your applications using the Prompt API - \[Guides\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-guides.md): Learn how to get the most out of Portkey Prompts with these practical guides - \[Integrations\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-integration.md) - \[Prompt Library\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-library.md) - \[Prompt Observability\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-observability.md) - \[Prompt Partials\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-partial.md): With Prompt Partials, you can save your commonly used templates (which could be your instruction set, data structure explanation, examples etc.) separately from your prompts and flexibly incorporate them wherever required. - \[Prompt Playground\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-playground.md) - \[Prompt Versioning & Labels\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/prompt-versioning.md) - \[Tool Library\](https://docs.portkey.ai/docs/product/prompt-engineering-studio/tool-library.md) - \[Enterprise Architecture\](https://docs.portkey.ai/docs/self-hosting/hybrid-deployments/architecture.md): Comprehensive guide to Portkey's hybrid deployment architecture for enterprises - \[EKS\](https://docs.portkey.ai/docs/self-hosting/hybrid-deployments/aws/eks.md): This enterprise-focused document provides comprehensive instructions for deploying the Portkey software in a hybrid mode on Amazon EKS clusters, designed to meet the needs of large-scale, mission-critical applications. It includes specific recommendations for component sizing, high availability, and integration with monitoring systems. - \[AWS Marketplace\](https://docs.portkey.ai/docs/self-hosting/hybrid-deployments/aws/marketplace.md): This enterprise-focused document provides comprehensive instructions for deploying the Portkey software using AWS Marketplace. - \[Azure\](https://docs.portkey.ai/docs/self-hosting/hybrid-deployments/azure.md): This enterprise-focused document provides comprehensive instructions for deploying the Portkey software on Azure Kubernetes Service (AKS), tailored to meet the needs of large-scale, mission-critical applications. It includes specific recommendations for component sizing, high availability, and integration with monitoring systems. - \[GCP\](https://docs.portkey.ai/docs/self-hosting/hybrid-deployments/gcp.md): This enterprise-focused document provides comprehensive instructions for deploying the Portkey software on Google Cloud Platform (GCP), tailored to meet the needs of large-scale, mission-critical applications. - \[Prometheus Metrics\](https://docs.portkey.ai/docs/self-hosting/prometheus-metrics.md): Comprehensive monitoring and observability for Portkey Enterprise Gateway through Prometheus metrics - \[Common Errors & Resolutions\](https://docs.portkey.ai/docs/support/common-errors-and-resolutions.md): Since Portkey functions as a gateway - you may encounter Portkey-related, as well as non-Portkey related errors while using our services. - \[Contact Us\](https://docs.portkey.ai/docs/support/contact-us.md) - \[Developer Forum\](https://docs.portkey.ai/docs/support/developer-forum.md): Are you navigating the challenging journey of transitioning LLMs from prototype stages to full-scale production? You're not alone. As this frontier of technology continues to expand, the roadmap isn't always clear. Best practices, guidelines, and efficient methodologies are still on the horizon. - \[December '23 Migration\](https://docs.portkey.ai/docs/support/portkeys-december-migration.md) - \[Upgrade to Model Catalog\](https://docs.portkey.ai/docs/support/upgrade-to-model-catalog.md): Learn how to upgrade to Model Catalog to replace Virtual Keys
---
# Introduction - Portkey Docs
[Skip to main content](https://portkey.ai/docs/api-reference/admin-api/introduction#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
API Reference
Introduction
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey Admin API](https://portkey.ai/docs/api-reference/admin-api/introduction#portkey-admin-api)
* [Understanding the Admin API Ecosystem](https://portkey.ai/docs/api-reference/admin-api/introduction#understanding-the-admin-api-ecosystem)
* [Resource Management](https://portkey.ai/docs/api-reference/admin-api/introduction#resource-management)
* [Analytics and Monitoring](https://portkey.ai/docs/api-reference/admin-api/introduction#analytics-and-monitoring)
* [User and Workspace Administration](https://portkey.ai/docs/api-reference/admin-api/introduction#user-and-workspace-administration)
* [Authentication Strategy](https://portkey.ai/docs/api-reference/admin-api/introduction#authentication-strategy)
* [Access Control and Permissions Model](https://portkey.ai/docs/api-reference/admin-api/introduction#access-control-and-permissions-model)
* [Creating and Managing API Keys](https://portkey.ai/docs/api-reference/admin-api/introduction#creating-and-managing-api-keys)
* [Through the Portkey Dashboard](https://portkey.ai/docs/api-reference/admin-api/introduction#through-the-portkey-dashboard)
* [Through the API](https://portkey.ai/docs/api-reference/admin-api/introduction#through-the-api)
* [Understanding API Key Capabilities](https://portkey.ai/docs/api-reference/admin-api/introduction#understanding-api-key-capabilities)
* [Security and Compliance: Audit Logs](https://portkey.ai/docs/api-reference/admin-api/introduction#security-and-compliance%3A-audit-logs)
* [Getting Started with the Admin API](https://portkey.ai/docs/api-reference/admin-api/introduction#getting-started-with-the-admin-api)
* [Need Support?](https://portkey.ai/docs/api-reference/admin-api/introduction#need-support%3F)
[](https://portkey.ai/docs/api-reference/admin-api/introduction#portkey-admin-api)
Portkey Admin API
========================================================================================================
The Portkey Admin API provides programmatic access to manage your organization, workspaces, and resources. Whether you’re automating routine administration tasks, integrating Portkey with your existing systems, or customizing your deployment at scale, this API gives you the tools to control every aspect of your Portkey implementation.
[](https://portkey.ai/docs/api-reference/admin-api/introduction#understanding-the-admin-api-ecosystem)
Understanding the Admin API Ecosystem
------------------------------------------------------------------------------------------------------------------------------------------------
The Admin API is organized around key capabilities that let you manage different aspects of your Portkey environment. Let’s explore what you can build and automate:
###
[](https://portkey.ai/docs/api-reference/admin-api/introduction#resource-management)
Resource Management
At the foundation of Portkey are the resources that define how your AI implementation works. These can all be managed programmatically:
[Configs\
-------\
\
Create and manage configuration profiles that define routing rules, model settings, and more.](https://portkey.ai/docs/api-reference/admin-api/control-plane/configs/create-config)
[Virtual Keys\
------------\
\
Manage virtual API keys that provide customized access to specific configurations.](https://portkey.ai/docs/api-reference/admin-api/control-plane/virtual-keys/create-virtual-key)
[API Keys\
--------\
\
Create and manage API keys for accessing Portkey services.](https://portkey.ai/docs/api-reference/admin-api/control-plane/api-keys/create-api-key)
###
[](https://portkey.ai/docs/api-reference/admin-api/introduction#analytics-and-monitoring)
Analytics and Monitoring
Once your resources are configured, you’ll want to measure performance and usage. The Admin API gives you powerful tools to access analytics data:
[Summary Data\
------------\
\
Retrieve aggregated usage statistics and performance metrics.](https://portkey.ai/docs/api-reference/admin-api/control-plane/analytics/summary/get-all-cache-data)
[Grouped Data\
------------\
\
Access detailed analytics organized by metadata, model, or user.](https://portkey.ai/docs/api-reference/admin-api/control-plane/analytics/groups-paginated-data/get-model-grouped-data)
[Time Series Data\
----------------\
\
Monitor performance trends, costs, errors, feedback, and usage patterns over time.](https://portkey.ai/docs/api-reference/admin-api/control-plane/analytics/graphs-time-series-data/get-cost-data)
###
[](https://portkey.ai/docs/api-reference/admin-api/introduction#user-and-workspace-administration)
User and Workspace Administration
Beyond resources and analytics, you’ll need to manage who has access to your Portkey environment:
[Users & Invites\
---------------\
\
Manage user accounts, permissions, and access. Send and manage user invitations.](https://portkey.ai/docs/api-reference/admin-api/control-plane/users/retrieve-a-user)
[Workspaces & Members\
--------------------\
\
Create workspaces and manage team membership and permissions within workspaces.](https://portkey.ai/docs/api-reference/admin-api/control-plane/workspaces/create-workspace)
[](https://portkey.ai/docs/api-reference/admin-api/introduction#authentication-strategy)
Authentication Strategy
--------------------------------------------------------------------------------------------------------------------
Now that you understand what the Admin API can do, let’s explore how to authenticate your requests. Portkey uses a sophisticated access control system with two types of API keys, each designed for different use cases:
Admin API Key
-------------
**Organization-wide access**These keys grant access to administrative operations across your entire organization.
Only Organization Owners and Admins can create and manage Admin API keys.
Workspace API Key
-----------------
**Workspace-specific access**These keys provide targeted access to resources within a single workspace.
Workspace Managers can create and manage Workspace API keys.
The key you use determines which operations you can perform. For organization-wide administrative tasks, you’ll need an Admin API key. For workspace-specific operations, you can use a Workspace API key.
[](https://portkey.ai/docs/api-reference/admin-api/introduction#access-control-and-permissions-model)
Access Control and Permissions Model
----------------------------------------------------------------------------------------------------------------------------------------------
Portkey’s hierarchical access control system governs who can use which APIs. Let’s examine how roles, API keys, and permissions interact: This access model follows a clear hierarchy:
| Role | Can Create Admin API Key | Can Create Workspace API Key | Access Scope |
| --- | --- | --- | --- |
| Organization Owner | ✅ | ✅ (any workspace) | All organization resources |
| Organization Admin | ✅ | ✅ (any workspace) | All organization resources |
| Workspace Manager | ❌ | ✅ (managed workspace only) | Single workspace resources |
| Workspace Member | ❌ | ❌ | Limited workspace access |
[](https://portkey.ai/docs/api-reference/admin-api/introduction#creating-and-managing-api-keys)
Creating and Managing API Keys
----------------------------------------------------------------------------------------------------------------------------------
Now that you understand the permission model, let’s look at how to create the API keys you’ll need:
###
[](https://portkey.ai/docs/api-reference/admin-api/introduction#through-the-portkey-dashboard)
Through the Portkey Dashboard
The simplest way to create an API key is through the Portkey dashboard:
Creating an Admin API Key in the Portkey Dashboard
###
[](https://portkey.ai/docs/api-reference/admin-api/introduction#through-the-api)
Through the API
You can also create keys programmatically:
Creating Admin API Key
Creating Workspace API Key
Copy
Ask AI
curl -X POST https://api.portkey.ai/v1/api-keys/organisation/service
-H "x-portkey-api-key: YOUR_EXISTING_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name":"API_KEY_NAME_0809",
"scopes":[\
"logs.export",\
"logs.list",\
"logs.view"\
]
}'
[](https://portkey.ai/docs/api-reference/admin-api/introduction#understanding-api-key-capabilities)
Understanding API Key Capabilities
------------------------------------------------------------------------------------------------------------------------------------------
Both key types have different capabilities. This table clarifies which operations each key type can perform:
| Operation | Admin API Key | Workspace API Key |
| --- | --- | --- |
| Manage organization settings | ✅ | ❌ |
| Create/manage workspaces | ✅ | ❌ |
| Manage users and permissions | ✅ | ❌ |
| Create/manage configs | ✅ (All workspaces) | ✅ (Single workspace) |
| Create/manage virtual keys | ✅ (All workspaces) | ✅ (Single workspace) |
| Access Analytics | ✅ (All workspaces) | ✅ (Single workspace) |
| Create/update feedback | ❌ | ✅ |
[](https://portkey.ai/docs/api-reference/admin-api/introduction#security-and-compliance%3A-audit-logs)
Security and Compliance: Audit Logs
----------------------------------------------------------------------------------------------------------------------------------------------
For security-conscious organizations, Portkey provides comprehensive audit logging of all Admin API operations. These logs give you complete visibility into administrative actions:
Audit Logs in the Portkey Dashboard
Every administrative action is recorded with:
* User identity
* Action type and target resource
* Timestamp
* IP address
* Request details
This audit trail helps maintain compliance and provides accountability for all administrative changes.[Audit Logs Documentation\
------------------------\
\
Learn more about Portkey’s audit logging capabilities](https://portkey.ai/docs/product/enterprise-offering/access-control-management#audit-logs)
[](https://portkey.ai/docs/api-reference/admin-api/introduction#getting-started-with-the-admin-api)
Getting Started with the Admin API
------------------------------------------------------------------------------------------------------------------------------------------
Now that you understand the Admin API ecosystem, authentication, and permissions model, you’re ready to start making requests. Here’s what you’ll need:
1. **Appropriate role**: Ensure you have the right permissions (Org Owner/Admin for Admin API, Workspace Manager for Workspace API)
2. **API key**: Generate the appropriate key from the Portkey dashboard
3. **Make your first request**: Use your key in the request header
For developers looking to integrate with the Admin API, we provide a complete OpenAPI specification that you can use with your API development tools:[OpenAPI Specification\
---------------------\
\
Download the OpenAPI spec for the Admin API](https://portkey.ai/docs/api-reference/admin-api/open-api-specification)
[](https://portkey.ai/docs/api-reference/admin-api/introduction#need-support%3F)
Need Support?
--------------------------------------------------------------------------------------------------
If you need help setting up or using the Admin API, our team is ready to assist:[Book a Support Call\
-------------------\
\
Schedule time with our team to get personalized help with the Admin API](https://portkey.sh/demo-1)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/api-reference/admin-api/introduction.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/api-reference/admin-api/introduction)
[Errors](https://portkey.ai/docs/api-reference/admin-api/error)
⌘I
---
# Upgrade to Model Catalog - Portkey Docs
[Skip to main content](https://portkey.ai/docs/support/upgrade-to-model-catalog#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
Upgrade to Model Catalog
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Why Upgrade to the Model Catalog?](https://portkey.ai/docs/support/upgrade-to-model-catalog#why-upgrade-to-the-model-catalog%3F)
* [First, What is the Model Catalog?](https://portkey.ai/docs/support/upgrade-to-model-catalog#first%2C-what-is-the-model-catalog%3F)
* [How It Enhances the Virtual Key Experience](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-it-enhances-the-virtual-key-experience)
* [Is It Backwards Compatible? (The Important Question)](https://portkey.ai/docs/support/upgrade-to-model-catalog#is-it-backwards-compatible%3F-the-important-question)
* [So, What Has Changed?](https://portkey.ai/docs/support/upgrade-to-model-catalog#so%2C-what-has-changed%3F)
* [The New Experience by Role](https://portkey.ai/docs/support/upgrade-to-model-catalog#the-new-experience-by-role)
* [For Organization Admins: Central Command & Control](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-organization-admins%3A-central-command-%26-control)
* [For Workspace Admins: Inherit and Customize](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-workspace-admins%3A-inherit-and-customize)
* [For Workspace Members (Developers): A Simpler, Better DX](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-workspace-members-developers-%3A-a-simpler%2C-better-dx)
* [How Your Workflows Change: A Practical Guide](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-your-workflows-change%3A-a-practical-guide)
* [1\. From “Creating Virtual Keys” to “Creating AI Providers”](https://portkey.ai/docs/support/upgrade-to-model-catalog#1-from-%E2%80%9Ccreating-virtual-keys%E2%80%9D-to-%E2%80%9Ccreating-ai-providers%E2%80%9D)
* [2\. From “Sending Virtual Keys in Requests” to “Using the Model Parameter”](https://portkey.ai/docs/support/upgrade-to-model-catalog#2-from-%E2%80%9Csending-virtual-keys-in-requests%E2%80%9D-to-%E2%80%9Cusing-the-model-parameter%E2%80%9D)
* [3\. From “Using Virtual Keys in Configs” to “Using Models in Configs”](https://portkey.ai/docs/support/upgrade-to-model-catalog#3-from-%E2%80%9Cusing-virtual-keys-in-configs%E2%80%9D-to-%E2%80%9Cusing-models-in-configs%E2%80%9D)
* [4\. From “Using Virtual Keys in Prompts” to… Nothing!](https://portkey.ai/docs/support/upgrade-to-model-catalog#4-from-%E2%80%9Cusing-virtual-keys-in-prompts%E2%80%9D-to%E2%80%A6-nothing)
* [How to Get Started](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-to-get-started)
> We are thrilled to introduce the **Model Catalog**, a powerful new feature designed to give your organization end-to-end control, governance, and visibility over your AI models. The Model Catalog is the evolution of our Virtual Keys feature, built to handle the complexities of enterprise-scale AI operations. This guide will walk you through what the Model Catalog is, how it enhances your current workflow, and what the experience looks like for every user in your organization.
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#why-upgrade-to-the-model-catalog%3F)
Why Upgrade to the Model Catalog?
--------------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#first%2C-what-is-the-model-catalog%3F)
First, What is the Model Catalog?
The Model Catalog fundamentally upgrades the Virtual Key experience by introducing a centralized, organization-level management layer. It addresses key governance pain points you have faced, allowing for:
1. **Centralized Provider Management:** Create a single (or multiple) provider integration (e.g., for Azure, OpenAI, Bedrock) at the organization level, and securely provision it to multiple workspaces without re-entering credentials.
2. **Workspace Provisioning:** Easily control which workspaces have access to which provider integrations, and set distinct budgets and rate limits for each workspace from a single screen.
3. **Granular Model Provisioning:** For any given provider integration, you can specify exactly which models (e.g., `gpt-4o`, `claude-4-sonnet`) are available, preventing users from accessing unapproved, expensive, or deprecated models.
4. **Simplified Model Calling:** Your developers can now switch between providers and models directly in their API call using a simple `model = "@provider/model_name"` format, without needing to manage different virtual keys or complex configs for simple requests.
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-it-enhances-the-virtual-key-experience)
How It Enhances the Virtual Key Experience
The Model Catalog is designed to be a seamless upgrade. Here’s a quick comparison:
| Feature | **Old Way (Virtual Keys)** | **New Way (Model Catalog)** |
| --- | --- | --- |
| **Creation** | Manually create a virtual key in _each_ workspace, even with the same credentials. | Create one **Integration** at the Org level and **provision** it to many workspaces. |
| **Budgeting** | Budgets are set per virtual key. Managing shared provider costs across teams is difficult. | At the org level directly, assign specific budgets/limits **per workspace**. |
| **Model Access** | A virtual key grants access to all models available under that provider key. Control requires complex configs or guardrails. | Define an explicit allow-list of models for each **Integration**. Workspaces only see what you’ve enabled. |
| **Making Calls** | Use the `virtual_key` header or bind a virtual key to a config, then bind that to an API key. | Simply pass `model: "@provider_slug/model_slug"` in the request body. The old way still works perfectly. |
| **Visibility** | Org Admins have no central view of all provider credentials being used across all workspaces. | Org Admins have a central **Integrations** dashboard to see all connected providers, including those created by workspaces. |
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#is-it-backwards-compatible%3F-the-important-question)
Is It Backwards Compatible? (The Important Question)
**Yes, the Model Catalog is 100% backwards compatible.**
* Your existing Virtual Keys will be automatically migrated and will appear as “AI Providers” in the Model Catalog, scoped to their original workspace.
* **No code will break.** All your existing applications and scripts that use the `virtual_key` header in requests or configs will continue to work exactly as they do today.
You can adopt the new features at your own pace without any disruption to your production workloads.
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#so%2C-what-has-changed%3F)
So, What Has Changed?
----------------------------------------------------------------------------------------------------------------
While the Model Catalog is an important upgrade, the transition is designed to be completely seamless. Here’s the most important thing you need to know:
* Your existing Virtual Keys have been automatically migrated. They are now called “AI Providers”, and they keep their original slugs. All your existing code, configs, and prompts will continue to work without any changes.
With that key point in mind, let’s explore the new capabilities this unlocks for every role in your organization and how your daily workflows are enhanced.
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#the-new-experience-by-role)
The New Experience by Role
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-organization-admins%3A-central-command-%26-control)
For Organization Admins: Central Command & Control
As an Org Admin, you will see a new **Integrations** tab in your organization settings. This is your central hub for managing all LLM providers.
1. **Connect an Integration:** Instead of creating a virtual key in a workspace, you now “Connect” a provider here. The process of adding credentials is the same. You can create multiple integrations for the same provider (e.g., “Azure-Prod” and “Azure-Dev”).
2. **Provision to Workspaces:** In the “Workspace Provisioning” step, you’ll see a list of all your workspaces. You can toggle access for each one and set specific budgets and rate limits. You can also choose to automatically provision this integration for any new workspaces created in the future.
3. **Provision Models:** In the “Model Provisioning” step, you can specify exactly which models are accessible through this integration. You can choose to enable all models or select them individually.
There is also a “Workspace-Created” tab to view and manage any integrations created by Workspace Admins themselves, giving you full visibility.
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-workspace-admins%3A-inherit-and-customize)
For Workspace Admins: Inherit and Customize
As a Workspace Admin, you will now see **Model Catalog** on your sidebar. The old “Virtual Keys” will be marked as deprecated.
* **Inherited Providers:** In the “AI Providers” tab, you will automatically see all the integrations your Org Admin has provisioned for your workspace.
* **Creating New Providers:** You still have flexibility:
1. **Inherit from Org:** You can create a new provider that inherits from an Org-level integration. This is useful for creating multiple providers with more restrictive budgets than the one assigned to your workspace.
2. **Create a New Integration (The Old Way):** If enabled by your Org Admin, you can still create a brand new integration exclusively for your workspace, just like you created Virtual Keys before.
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#for-workspace-members-developers-%3A-a-simpler%2C-better-dx)
For Workspace Members (Developers): A Simpler, Better DX
As a developer, your experience is significantly streamlined.
* **The Model Garden:** A new “Models” tab provides a complete gallery of every single model you have access to across all providers in your workspace. You can click on any model to get ready-to-use code snippets.
* **Simplified API Calls:** You no longer need to worry about which virtual key to use for which model. With a single Portkey API key, you can call any model you have access to by specifying the provider and model in the `model` parameter:
Copy
Ask AI
# Switch models and providers on the fly!
client.chat.completions.create(
model="@openai-prod/gpt-4o",
messages=[...]
)
client.chat.completions.create(
model="@bedrock-us/claude-3-sonnet-v1",
messages=[...]
)
* * *
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-your-workflows-change%3A-a-practical-guide)
How Your Workflows Change: A Practical Guide
The Model Catalog enhances every area where you previously used Virtual Keys. Let’s break down how each of your existing workflows translates to the new, more powerful system.
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#1-from-%E2%80%9Ccreating-virtual-keys%E2%80%9D-to-%E2%80%9Ccreating-ai-providers%E2%80%9D)
1\. From “Creating Virtual Keys” to “Creating AI Providers”
The Old Way
-----------
You created a Virtual Key inside a specific workspace. This had to be repeated for every workspace needing access to the same provider credentials.
The New Way
-----------
You now create a central **Integration** at the Org level and provision it to many workspaces. You can also still create workspace-only integrations, or “inherit” from an Org integration to create providers with stricter limits.
**What’s Better?** This “create once, provision many” model saves significant time, reduces the risk of configuration errors, and gives you a single place to manage provider credentials and model access for your entire organization.
* * *
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#2-from-%E2%80%9Csending-virtual-keys-in-requests%E2%80%9D-to-%E2%80%9Cusing-the-model-parameter%E2%80%9D)
2\. From “Sending Virtual Keys in Requests” to “Using the Model Parameter”
The Old Way
-----------
You sent the Virtual Key slug in the `virtual_key` header of your request.
The New Way
-----------
The `virtual_key` header is fully backward compatible and will continue to work. However, the recommended and more powerful method is to specify the provider and model directly in the `model` parameter of your request body.
We’ve renamed “Virtual Keys” to **AI Providers** at the workspace level. To reference one in a request, you simply prefix its slug with an `@` symbol in your inference requests. **What’s Better?** This method is more explicit and keeps all model-related information in one place—the `model` parameter. It eliminates the need for a separate header and makes switching between models and providers incredibly simple.
Portkey & OpenAI Python SDK
Portkey & OpenAI Node.js SDK
REST API (cURL)
Copy
Ask AI
# Note: Your AI Provider slug is "my-azure-prod"
# Note: The model you want to use is "gpt-4o"
response = client.chat.completions.create(
model="@my-azure-prod/gpt-4o", # The new, recommended format
messages=[{"role": "user", "content": "Hello!"}]
)
* * *
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#3-from-%E2%80%9Cusing-virtual-keys-in-configs%E2%80%9D-to-%E2%80%9Cusing-models-in-configs%E2%80%9D)
3\. From “Using Virtual Keys in Configs” to “Using Models in Configs”
The Old Way
-----------
You added a `virtual_key` field to your Portkey Config, either at the root level or inside a strategy target (like fallback or load balance).
The New Way
-----------
This continues to work perfectly. For new configs, you can now specify the `model` directly in a target’s `override_params`. This unlocks powerful, multi-provider strategies.
**What’s Better?** You are no longer limited to falling back between virtual keys of the same provider. You can now create a fallback strategy from an Azure OpenAI deployment to a direct OpenAI integration, or from Bedrock to Google AI, all within a single, elegant config. **Example: Multi-Provider Fallback Config** Here’s a config that tries a model on Azure first. If that fails, it automatically falls back to the same model on AWS Bedrock.
Copy
Ask AI
{
"strategy": {
"mode": "fallback",
"on_status_codes": [429, 500, 503]
},
"targets": [\
{\
// Target 1: Try Azure first\
"override_params": {\
"model": "@azure-us-east/gpt-4o"\
}\
},\
{\
// Target 2: Fallback to Bedrock if Azure fails\
"override_params": {\
"model": "@bedrock-main/anthropic.claude-3-sonnet-20240229-v1:0"\
}\
}\
]
}
When you make a request using this config, you don’t need to specify a model at all. Portkey handles the complex routing for you.
* * *
####
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#4-from-%E2%80%9Cusing-virtual-keys-in-prompts%E2%80%9D-to%E2%80%A6-nothing)
4\. From “Using Virtual Keys in Prompts” to… Nothing!
The Old Way
-----------
When creating a prompt template, you selected a Virtual Key from a dropdown to power the prompt.
The New Way
-----------
**No action is required from you.** We’ve handled this automatically.
**What’s Better?** A seamless, zero-effort migration. Your existing prompt templates have been automatically updated behind the scenes to use the new AI Provider system. They will continue to work exactly as before without any changes on your end. When creating new prompts, you’ll now select from a list of AI Providers.
* * *
###
[](https://portkey.ai/docs/support/upgrade-to-model-catalog#how-to-get-started)
How to Get Started
The Model Catalog is currently available as a feature-flagged release.
1. **Contact Us:** Reach out to the Portkey team [here](https://portkey.wiki/community)
, and we will enable the Model Catalog for your organization.
2. **Test in a Dev Org:** We recommend enabling it for a non-production or sandbox organization first. This allows you to explore the new workflows, set up your first Org-level integrations, and prepare your teams.
3. **Plan Your Rollout:** Once you’re comfortable, we can enable it for your production organization. You can then begin to migrate your workspace-level virtual keys to centrally managed integrations.
We are incredibly excited for you to experience the next level of control and efficiency with the Portkey Model Catalog.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/support/upgrade-to-model-catalog.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/support/upgrade-to-model-catalog)
[Prometheus Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics)
[How to Contribute](https://portkey.ai/docs/README)
⌘I
---
# Common Errors & Resolutions - Portkey Docs
[Skip to main content](https://portkey.ai/docs/support/common-errors-and-resolutions#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
Common Errors & Resolutions
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Identifying the error source](https://portkey.ai/docs/support/common-errors-and-resolutions#identifying-the-error-source)
* [How to verify if it’s Portkey error](https://portkey.ai/docs/support/common-errors-and-resolutions#how-to-verify-if-it%E2%80%99s-portkey-error)
* [Common Portkey Errors](https://portkey.ai/docs/support/common-errors-and-resolutions#common-portkey-errors)
[](https://portkey.ai/docs/support/common-errors-and-resolutions#identifying-the-error-source)
Identifying the error source
-------------------------------------------------------------------------------------------------------------------------------
1. Errors exclusively originating from Portkey are **prefixed with “Portkey Error”.**
2. While errors originating from your LLM providers (or frameworks) are **returned as they are**, without any transformations.
[](https://portkey.ai/docs/support/common-errors-and-resolutions#how-to-verify-if-it%E2%80%99s-portkey-error)
How to verify if it’s Portkey error
-----------------------------------------------------------------------------------------------------------------------------------------------------
You can quickly verify if the problem is originating from Portkey by **running the same request without Portkey**. If it executes successfully, then it’s likely that the error is with Portkey or its integration.
[](https://portkey.ai/docs/support/common-errors-and-resolutions#common-portkey-errors)
Common Portkey Errors
-----------------------------------------------------------------------------------------------------------------
1. **Errors related to Missing Mandatory Headers**: This is a common error where certain mandatory headers might be missing from the request. Make sure that all the necessary headers as specified in the respective feature documentation are included in your requests.
2. **Errors related to Invalid Header Values**: At times, an incorrect or unsupported value might be passed in a header, causing this error. Cross-check the values provided against the allowed ones mentioned in our documentation.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/support/common-errors-and-resolutions.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/support/common-errors-and-resolutions)
[Developer Forum](https://portkey.ai/docs/support/developer-forum)
[December '23 Migration](https://portkey.ai/docs/support/portkeys-december-migration)
⌘I
---
# Developer Forum - Portkey Docs
[Skip to main content](https://portkey.ai/docs/support/developer-forum#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
Developer Forum
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Enter the LLMs in Prod Community:](https://portkey.ai/docs/support/developer-forum#enter-the-llms-in-prod-community%3A)
* [Join the Community on Discord.](https://portkey.ai/docs/support/developer-forum#join-the-community-on-discord)
[](https://portkey.ai/docs/support/developer-forum#enter-the-llms-in-prod-community%3A)
Enter the LLMs in Prod Community:
-----------------------------------------------------------------------------------------------------------------------------
1. **Collaborate with Industry Practitioners:** Dive deep into discussions, share experiences, and gain valuable insights from professionals who are on the same journey of deploying LLMs in production. Whether you’re a novice or a seasoned expert, there’s always something new to learn and someone to learn from.
2. **Official Portkey Support:** Have a query? Facing a challenge? All of the Portkey team is on Discord to assist you. Get your questions answered swiftly and reliably.
[](https://portkey.ai/docs/support/developer-forum#join-the-community-on-discord)
Join the [**Community on Discord**](https://t.co/lZEFk6kbbb)
**.**
--------------------------------------------------------------------------------------------------------------------------------------------------------
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/support/developer-forum.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/support/developer-forum)
[Contact Us](https://portkey.ai/docs/support/contact-us)
[Common Errors & Resolutions](https://portkey.ai/docs/support/common-errors-and-resolutions)
⌘I
---
# How to Contribute - Portkey Docs
[Skip to main content](https://portkey.ai/docs/README#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
How to Contribute
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey AI Docs](https://portkey.ai/docs/README#portkey-ai-docs)
* [Getting Started](https://portkey.ai/docs/README#getting-started)
* [1\. Clone the Repository](https://portkey.ai/docs/README#1-clone-the-repository)
* [2\. Install Mintlify CLI](https://portkey.ai/docs/README#2-install-mintlify-cli)
* [3\. Run the Local Server](https://portkey.ai/docs/README#3-run-the-local-server)
* [How to Contribute](https://portkey.ai/docs/README#how-to-contribute)
* [What You Can Contribute](https://portkey.ai/docs/README#what-you-can-contribute)
* [Need Help?](https://portkey.ai/docs/README#need-help%3F)
* [Community](https://portkey.ai/docs/README#community)
[](https://deepwiki.com/Portkey-AI/gateway)
[](https://portkey.ai/docs/README#portkey-ai-docs)
Portkey AI Docs
======================================================================
This repository contains the source code for [Portkey AI](https://portkey.ai/)
’s documentation, built using [Mintlify](https://mintlify.com/)
.
We welcome contributions to improve the accuracy, clarity, and coverage of our docs.
* * *
[](https://portkey.ai/docs/README#getting-started)
Getting Started
----------------------------------------------------------------------
To contribute or preview the documentation locally:
###
[](https://portkey.ai/docs/README#1-clone-the-repository)
1\. Clone the Repository
Copy
Ask AI
git clone https://github.com/portkey-ai/docs-core.git
cd docs-core
###
[](https://portkey.ai/docs/README#2-install-mintlify-cli)
2\. Install Mintlify CLI
The documentation is built using Mintlify. Install the `mint` CLI globally with any of the following:
Copy
Ask AI
npm install -g mint
# OR
pnpm add -g mint
# OR
yarn global add mint
Refer to the [Mintlify Docs](https://mintlify.com/docs)
if you run into issues.
###
[](https://portkey.ai/docs/README#3-run-the-local-server)
3\. Run the Local Server
Copy
Ask AI
mint dev
This will start a local server so you can preview your changes as you edit.
* * *
[](https://portkey.ai/docs/README#how-to-contribute)
How to Contribute
--------------------------------------------------------------------------
We appreciate all contributions—small or large. Here’s how you can help:
1. **Fork** the repository.
2. **Create a branch** for your changes:
Copy
Ask AI
git checkout -b your-branch-name
3. **Make edits** and test them locally using `mint dev`.
4. **Commit and push** your changes.
5. **Open a pull request** to the `main` branch with a clear description.
* * *
[](https://portkey.ai/docs/README#what-you-can-contribute)
What You Can Contribute
--------------------------------------------------------------------------------------
* Fix typos or broken links
* Improve explanations or formatting
* Add missing documentation
* Suggest structural improvements
* Maintain existing integrations with Portkey
* * *
[](https://portkey.ai/docs/README#need-help%3F)
Need Help?
--------------------------------------------------------------
Join the `#portkey-docs` channel in our [Discord](https://portkey.wiki/community)
to ask questions or share suggestions. You can also open an issue here on GitHub.
[](https://portkey.ai/docs/README#community)
Community
----------------------------------------------------------
Join our growing community around the world, for help, ideas, and discussions on AI.
* View our official [Blog](https://portkey.wiki/gh-78)
* Chat with us on [Discord](https://portkey.wiki/community)
* Follow us on [Twitter](https://portkey.wiki/gh-79)
* Connect with us on [LinkedIn](https://portkey.wiki/gh-80)
* Visit us on [YouTube](https://portkey.wiki/gh-103)
* Join our [Dev community](https://portkey.wiki/gh-82)
* * *
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/README.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/README)
[Upgrade to Model Catalog](https://portkey.ai/docs/support/upgrade-to-model-catalog)
[Contact Us](https://portkey.ai/docs/support/contact-us)
⌘I
---
# Contact Us - Portkey Docs
[Skip to main content](https://portkey.ai/docs/support/contact-us#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
Contact Us
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
Despite your best troubleshooting efforts and our own testing efforts, there may be times when you come across an issue that isn’t resolved. Reach out to the Portkey team below and we should get back to you as soon as possible:
[](https://portkey.ai/cdn-cgi/l/email-protection#2f5c5a5f5f405d5b6f5f405d5b444a56014e46)
[](https://portkey.ai/cdn-cgi/l/email-protection#2f5c5a5f5f405d5b6f5f405d5b444a56014e46)
[Email\
-----](https://portkey.ai/cdn-cgi/l/email-protection#2f5c5a5f5f405d5b6f5f405d5b444a56014e46)
[](https://portkey.ai/cdn-cgi/l/email-protection#2f5c5a5f5f405d5b6f5f405d5b444a56014e46)
[\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#b7c4c2c7c7d8c5c3f7c7d8c5c3dcd2ce99d6de)
[](https://discord.gg/vGv94Ht77p)
[](https://discord.gg/vGv94Ht77p)
[Discord Server\
--------------](https://discord.gg/vGv94Ht77p)
[](https://discord.gg/vGv94Ht77p)
[#support channel](https://discord.gg/vGv94Ht77p)
[](https://github.com/portkey-ai/rubeus)
[](https://github.com/portkey-ai/rubeus)
[Open Source Gateway\
-------------------](https://github.com/portkey-ai/rubeus)
[](https://github.com/portkey-ai/rubeus)
[Issue board](https://github.com/Portkey-AI/gateway)
Enterprise Customers
--------------------
Directly reachout via Slack Connect
To help address your issue swiftly and accurately, please gather as much of the following information as possible:
* **Description of the issue**: Include any error messages you are seeing and describe the behavior you’re experiencing and how it differs from your expectations.
* **Steps to reproduce the issue**: Provide clear steps on how we can reproduce the issue on our end.
* **Code Samples**: If possible, share code snippets that are causing the error. Ensure you’ve removed any sensitive data before sharing.
* **Request and Response Data**: If applicable, include the request you’re making and the response you’re receiving. Ensure any sensitive data is redacted.
* **Screenshots or Screen Recordings**: Visuals can help us understand and diagnose issues faster. If possible, include screenshots or screen recordings.
* **Environment Details**: Share details about your environment. For example, are you using a specific programming language or library? What’s the version? Are you seeing the issue in all environments (development, production, etc.)?
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/support/contact-us.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/support/contact-us)
[How to Contribute](https://portkey.ai/docs/README)
[Developer Forum](https://portkey.ai/docs/support/developer-forum)
⌘I
---
# Feature Comparison - Portkey Docs
[Skip to main content](https://portkey.ai/docs/product/product-feature-comparison#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Product
Feature Comparison
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
Portkey has a generous free tier (10k requests/month) on our **Dev** plan — but, as you move to production-scale, you may benefit from Portkey’s **Pro** or **Enterprise** plans.
650+ Global Orgs
----------------
Trust Portkey for democratizing and productionizing Gen AI
2.5 Trillion+ LLM Tokens
------------------------
Processed on Portkey so far
99.9% Uptime SLA
----------------
Ensuring your AI services are always available
7,400 Stars
-----------
The most popular and performant AI Gateway in the market
[](https://portkey.ai/docs/product/product-feature-comparison#why-enterprises-choose-portkey)
Why Enterprises Choose Portkey
--------------------------------------------------------------------------------------------------------------------------------
Enterprise customers leverage Portkey to **observe**, **govern**, and **optimize** their Gen AI services at scale across the entire org. Our Enterprise plan offers advanced security configurations, dedicated support, and customized infrastructure designed for high-volume production deployments.
[For Scale & Performance\
-----------------------\
\
▪️ Processing millions of requests▪️ Need Five 9s reliability▪️ Routing to private LLMs▪️ Building for multiple partner teams and departments](https://portkey.sh/demo-20)
[For Security & Compliance\
-------------------------\
\
▪️ SOC2, ISO27001, GDPR, HIPAA▪️ VPC/Airgapped deployment▪️ PII anonymization▪️ Advanced access controls](https://portkey.sh/demo-20)
Here’s a detailed comparison of our plans to help you choose the right solution for your needs:
tl;dr Summary for each plan
[Open Source\
-----------\
\
For developers needing complete control over AI infrastructure on their own servers.**Ideal for:**\
\
* Technical teams with privacy requirements\
* Self-hosted AI applications\
* Local development & experimentation](https://github.com/portkey-ai/gateway)
[Dev Plan\
--------\
\
Free starter plan with basic observability and key management features.**Ideal for:**\
\
* Solo developers building POCs\
* Startups in early development\
* Testing Portkey capabilities](https://app.portkey.ai/signup)
[Pro Plan\
--------\
\
For growing teams with advanced caching, alerts, and access control needs.**Ideal for:**\
\
* Startups scaling AI applications\
* SaaS products integrating AI\
* Teams needing collaboration features](https://app.portkey.ai/signup)
[Enterprise Plan\
---------------\
\
Enterprise-grade security, compliance, and scalability with flexible deployment.**Ideal for:**\
\
* Organizations with compliance needs\
* Handling sensitive data\
* Custom infrastructure requirements\
* Multiple deployment options (SaaS, Hybrid, Airgapped)](https://portkey.sh/demo-20)
| ### Product / Plan | Open Source
----------- | Dev (Free Forever)
------------------ | Pro ($49/Month)
--------------- | Enterprise (Custom)
------------------- |
| --- | --- | --- | --- | --- |
| Get Started | [Run Local
---------](https://github.com/portkey-ai/gateway) | [Sign Up
-------](https://app.portkey.ai/signup) | [Upgrade
-------](https://app.portkey.ai/signup) | [Book Call
---------](https://portkey.sh/demo-20) |
| Requests per Month | No Limit | 10K | 100K | Custom |
| Overage | \- | No Overage Allowed | $9/Month
for Every 100K
Up to 3M Requests | Custom Pricing |
| #### Observability | | | | |
| Logs | | | | |
| Traces | | | | |
| Feedback | | | | |
| Custom Metadata | | | | |
| Filters | | | | |
| Alerts | | | | |
| FinOps + Executive Dashboard | | | | |
| Retention Period | | 3 Days | 30 Days | Custom |
| #### AI Gateway | | | | |
| Universal API | | | | |
| Automatic Fallbacks | | | | |
| Loadbalancing | | | | |
| Conditional Routing | | | | |
| Automatic Retries | | | | |
| Request Timeouts | | | | |
| Config Management | | | | |
| LLM Key Management | | | | (with Budgeting, Rate Limiting support) |
| Simple Caching | | 1 Day TTL SUport | Unlimited TTL Stream from Cache | Unlimited TTL Stream from Cache |
| Semantic Caching | | | Unlimited TTL Stream from Cache | Unlimited TTL Stream from Cache |
| Unified Construct for Fine-Tuning, Files, Batche APIs | | | | |
| Support for AWS, GCP, Azure Private LLM Deployments | | | | |
| #### Prompt Management | | | | |
| Prompt Templates | | Upto 3 Templates | Unlimited | Unlimited |
| Playground | | | | |
| API Deployment | | | | |
| Versioning | | | | |
| Variable Management | | | | |
| Prompt Partials | | | | |
| Side-by-Side Comparison | | | | |
| User Access Control | | | | |
| #### Guardrails | | | | |
| Deterministic Guardrails | | | | |
| Partner Guardrails (LLM or Non LLM based) | | | | |
| Portkey LLM Guardrails with PII / PHI Redaction | | | | |
| #### Autonomous Fine-Tuning | | | | |
| Continuous Improvement | | | | |
| #### Security & Compliance | | | | |
| Role Based Access Control | | | | (Advanced) |
| Team Management | | | | (Advanced) |
| Audit Logs | | | | |
| Admin APIs (Control Plane & Data Plane) | | | | |
| SCIM Provisioning | | | | |
| JWT-based Authentication | | | | |
| Bring Your Own Key for Encryption | | | | |
| Enforce Org-level Metadata Reporting | | | | |
| Enforce Org-level LLM Guardrails | | | | |
| SSO with Okta Auth | | | | |
| SOC2, ISO27001, GDPR, HIPAA Compliance Certificates | | | | |
| BAA Signing for Compliances | | | | |
| VPC Managed Hosting | | | | |
| Private Tenancy | | | | |
| Configurable Retention Periods | | | | |
| Configurable exports to datalakes | | | | |
| Org Management | | | | |
[](https://portkey.ai/docs/product/product-feature-comparison#enterprise-deployment-options)
Enterprise Deployment Options
------------------------------------------------------------------------------------------------------------------------------
Portkey offers a range of deployment options designed to meet the diverse security, compliance, and operational requirements of enterprise organizations.
* Portkey-Managed SaaS
* Hybrid Deployment
* Airgapped Deployment
###
[](https://portkey.ai/docs/product/product-feature-comparison#portkey-managed-enterprise-saas)
Portkey-Managed Enterprise SaaS
A fully-managed solution on Portkey’s secure cloud infrastructure with enterprise-grade features and dedicated resources.**Ideal for:** Organizations seeking enterprise capabilities without the operational overhead of self-hosting.**Key Features:**
* Isolated cluster exclusively for your organization’s data
* Dedicated infrastructure for optimal performance
* Complete suite of enterprise features (RBAC, SSO, audit logs)
* Guaranteed SLAs with priority support
* SOC2, ISO27001, GDPR, and HIPAA compliant
**Benefits:**
* Faster implementation timeline (1-2 weeks)
* No infrastructure management required
* Automatic updates and scaling
* Reduced operational costs
* Predictable pricing model
* * *
All enterprise deployment options include comprehensive security features such as SOC2, ISO27001, GDPR, and HIPAA compliance certifications, PII anonymization, custom data retention policies, and encryption at rest and in transit.
Ready to explore Enterprise options?
------------------------------------
**Schedule a 30-minute consultation** with our solutions team to discuss your specific requirements and see a live demo.[Book Your Consultation](https://calendly.com/portkey-ai/quick-meeting)
[](https://portkey.ai/docs/product/product-feature-comparison#enterprise-implementation-journey)
Enterprise Implementation Journey
--------------------------------------------------------------------------------------------------------------------------------------
1
Discovery Call
30-minute discussion to understand your use case and requirements
2
Technical Deep Dive
Call with the Portkey engineering team to demo exact requirements & understand deployment options
3
Solution Design | Parallel Purchasing/Legal Discussions
Our team configures the optimal deployment model for your needs
4
Onboarding | Parallel Contract Close
Most enterprise customers are fully operational within 3-4 weeks
[Not sure which deployment model is right for you?\
-------------------------------------------------\
\
Our enterprise team can help assess your requirements and recommend the optimal deployment strategy for your organization’s specific needs.Book a Consultation here.](https://portkey.sh/demo-20)
[](https://portkey.ai/docs/product/product-feature-comparison#interested%3F-schedule-a-call-below)
Interested? Schedule a Call Below
----------------------------------------------------------------------------------------------------------------------------------------
[](https://portkey.ai/docs/product/product-feature-comparison#frequently-asked-questions)
Frequently Asked Questions
------------------------------------------------------------------------------------------------------------------------
What are the different deployment options available for Portkey Enterprise?
Portkey Enterprise offers three deployment options:
1. **Portkey-Managed SaaS**: A fully-managed solution where your data is hosted on Portkey’s secure infrastructure with an isolated cluster exclusively for your organization.
2. **Hybrid Deployment**: The AI Gateway and data plane run in your own environment while Portkey manages the control plane, ensuring all sensitive LLM data stays within your infrastructure.
3. **Fully Airgapped Deployment**: All components (Data plane, Control plane, and AI Gateway) are deployed within your infrastructure with zero data leaving your network.
Each option is designed to meet different security, compliance, and operational requirements. Our enterprise team can help you determine which option is best for your specific needs.
What is the typical implementation timeline for Enterprise customers?
Most Enterprise customers can be fully onboarded within 1-2 weeks, depending on specific requirements. Our dedicated implementation team will work with you to ensure a smooth transition. For complex deployments or those requiring special compliance measures, the timeline may extend to 3-4 weeks.
How does Portkey handle migration from other AI infrastructure solutions?
Portkey offers a comprehensive migration service for Enterprise customers, including API mapping, configuration setup, and verification testing. Our team will work with you to ensure minimal disruption during the transition. We provide detailed documentation and support throughout the migration process.
What integration capabilities are available for Enterprise customers?
Enterprise customers have access to all Portkey integration capabilities, including custom API integrations, webhooks, SDKs for major programming languages, and specialized connectors for enterprise systems. We also offer custom integration development for specific needs.
What level of support do Enterprise customers receive?
Enterprise customers receive priority 24/7 support with guaranteed response times (typically under 1 hour), a dedicated customer success manager, regular performance reviews, and direct access to our engineering team when needed. We also provide customized SLAs based on your specific requirements.
How does Portkey ensure data security and compliance?
Portkey’s Enterprise plan includes comprehensive security features such as SOC2, ISO27001, GDPR, and HIPAA compliance certifications. We offer PII anonymization, custom data retention policies, VPC deployment options, and BAA signing for healthcare organizations. All data is encrypted in transit and at rest.
Can we run Portkey in our own cloud environment?
Yes, Enterprise customers can deploy Portkey in their own AWS, GCP, or Azure environments, with options for VPC peering, private network connectivity, and dedicated infrastructure. Our team will work with your cloud operations staff to ensure optimal deployment.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/product/product-feature-comparison.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/product/product-feature-comparison)
[Open Source](https://portkey.ai/docs/product/open-source)
[Architecture](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture)
⌘I
---
# Open Source - Portkey Docs
[Skip to main content](https://portkey.ai/docs/product/open-source#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Product
Open Source
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey AI Gateway](https://portkey.ai/docs/product/open-source#portkey-ai-gateway)
* [Contribute here.](https://portkey.ai/docs/product/open-source#contribute-here)
* [AI Grants Finder](https://portkey.ai/docs/product/open-source#ai-grants-finder)
* [Gateway Reports](https://portkey.ai/docs/product/open-source#gateway-reports)
* [2025 AI Infrastructure Benchmark Report](https://portkey.ai/docs/product/open-source#2025-ai-infrastructure-benchmark-report)
* [GPT-4 is getting faster](https://portkey.ai/docs/product/open-source#gpt-4-is-getting-faster)
* [Collaborations](https://portkey.ai/docs/product/open-source#collaborations)
[](https://portkey.ai/docs/product/open-source#portkey-ai-gateway)
[Portkey AI Gateway](https://github.com/portkey-ai/rubeus)
----------------------------------------------------------------------------------------------------------------------------------
We have open sourced our battle-tested AI Gateway to the community - it connects to 250+ LLMs with a unified interface and a single endpoint, and lets you effortlessly setup fallbacks, load balancing, retries, and more. This gateway is in production at Portkey processing billions of tokens every day.
####
[](https://portkey.ai/docs/product/open-source#contribute-here)
[Contribute here](https://github.com/portkey-ai/rubeus)
.
* * *
[](https://portkey.ai/docs/product/open-source#ai-grants-finder)
[AI Grants Finder](https://grantsfinder.portkey.ai/)
--------------------------------------------------------------------------------------------------------------------------
Community resource for AI builders to find `GPU credits`, `grants`, `AI accelerators`, or `investments` - all in a single place. Continuously updated, and sometimes also featuring [exclusive deals](https://twitter.com/PortkeyAI/status/1692463628514156859)
. Access the data [here](https://airtable.com/appUjtBcdLQIgusqW/shrAU1e4M5twTmRal)
.
* * *
[](https://portkey.ai/docs/product/open-source#gateway-reports)
[Gateway Reports](https://portkey.ai/blog/tag/benchmarks/)
-------------------------------------------------------------------------------------------------------------------------------
We collaborate with the community to dive deep into how the LLMs & their inference providers are performing at scale, and publish gateway reports. We track latencies, uptime, cost changes, fluctuations across various modalitites like time-of-day, regions, token-lengths, and more.
####
[](https://portkey.ai/docs/product/open-source#2025-ai-infrastructure-benchmark-report)
[2025 AI Infrastructure Benchmark Report](https://portkey.ai/llms-in-prod-25)
Insights from analyzing 2 trillion+ tokens, across 90+ regions and 650+ teams in production. The report contains:
* Trends shaping AI adoption and LLM provider growth.
* Benchmarks to optimize speed, cost and reliability.
* Strategies to scale production-grade AI systems.
[The report is available here\
----------------------------](https://portkey.ai/llms-in-prod-25)
* * *
####
[](https://portkey.ai/docs/product/open-source#gpt-4-is-getting-faster)
[GPT-4 is getting faster](https://portkey.ai/blog/gpt-4-is-getting-faster/)
* * *
[](https://portkey.ai/docs/product/open-source#collaborations)
Collaborations
---------------------------------------------------------------------------------
Portkey supports various open source projects with additional production capabilities through its custom integrations.[Check out our expanding ecosystem of integrations\
-------------------------------------------------](https://portkey.ai/docs/integrations/ecosystem)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/product/open-source.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/product/open-source)
[KMS Integration](https://portkey.ai/docs/product/enterprise-offering/kms)
[Feature Comparison](https://portkey.ai/docs/product/product-feature-comparison)
⌘I
---
# Prometheus Metrics - Portkey Docs
[Skip to main content](https://portkey.ai/docs/self-hosting/prometheus-metrics#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Self-Hosting
Prometheus Metrics
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Overview](https://portkey.ai/docs/self-hosting/prometheus-metrics#overview)
* [Metrics Endpoint](https://portkey.ai/docs/self-hosting/prometheus-metrics#metrics-endpoint)
* [Global Configuration](https://portkey.ai/docs/self-hosting/prometheus-metrics#global-configuration)
* [Default Labels](https://portkey.ai/docs/self-hosting/prometheus-metrics#default-labels)
* [Standard Node.js Runtime Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#standard-node-js-runtime-metrics)
* [Process Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#process-metrics)
* [Event Loop Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#event-loop-metrics)
* [Garbage Collection Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#garbage-collection-metrics)
* [Custom Application Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#custom-application-metrics)
* [Universal Label Schema](https://portkey.ai/docs/self-hosting/prometheus-metrics#universal-label-schema)
* [1\. Gateway Request Counter](https://portkey.ai/docs/self-hosting/prometheus-metrics#1-gateway-request-counter)
* [2\. HTTP Request Duration Distribution](https://portkey.ai/docs/self-hosting/prometheus-metrics#2-http-request-duration-distribution)
* [3\. LLM Provider Request Duration](https://portkey.ai/docs/self-hosting/prometheus-metrics#3-llm-provider-request-duration)
* [4\. Portkey Processing Time (Excluding Streaming Latency)](https://portkey.ai/docs/self-hosting/prometheus-metrics#4-portkey-processing-time-excluding-streaming-latency)
* [5\. LLM Last Byte Latency Analysis](https://portkey.ai/docs/self-hosting/prometheus-metrics#5-llm-last-byte-latency-analysis)
* [6\. Total Portkey Request Duration](https://portkey.ai/docs/self-hosting/prometheus-metrics#6-total-portkey-request-duration)
* [7\. LLM Cost Accumulator](https://portkey.ai/docs/self-hosting/prometheus-metrics#7-llm-cost-accumulator)
* [8\. Authentication Performance Analysis](https://portkey.ai/docs/self-hosting/prometheus-metrics#8-authentication-performance-analysis)
* [9\. Rate Limiting Performance Metrics](https://portkey.ai/docs/self-hosting/prometheus-metrics#9-rate-limiting-performance-metrics)
* [10\. Pre-Request Middleware Pipeline Performance](https://portkey.ai/docs/self-hosting/prometheus-metrics#10-pre-request-middleware-pipeline-performance)
* [11\. Post-Request Processing Pipeline Performance](https://portkey.ai/docs/self-hosting/prometheus-metrics#11-post-request-processing-pipeline-performance)
* [12\. Cache System Performance Analysis](https://portkey.ai/docs/self-hosting/prometheus-metrics#12-cache-system-performance-analysis)
* [13\. gRPC Request Conversion Performance](https://portkey.ai/docs/self-hosting/prometheus-metrics#13-grpc-request-conversion-performance)
* [Configuration](https://portkey.ai/docs/self-hosting/prometheus-metrics#configuration)
* [Dynamic Metadata Label System](https://portkey.ai/docs/self-hosting/prometheus-metrics#dynamic-metadata-label-system)
* [Comprehensive Monitoring Examples](https://portkey.ai/docs/self-hosting/prometheus-metrics#comprehensive-monitoring-examples)
* [Traffic Analysis & Capacity Planning](https://portkey.ai/docs/self-hosting/prometheus-metrics#traffic-analysis-%26-capacity-planning)
* [Performance Monitoring & SLA Tracking](https://portkey.ai/docs/self-hosting/prometheus-metrics#performance-monitoring-%26-sla-tracking)
* [Cache Performance Optimization](https://portkey.ai/docs/self-hosting/prometheus-metrics#cache-performance-optimization)
* [Cost Management & Business Intelligence](https://portkey.ai/docs/self-hosting/prometheus-metrics#cost-management-%26-business-intelligence)
* [Security & Authentication Performance](https://portkey.ai/docs/self-hosting/prometheus-metrics#security-%26-authentication-performance)
* [Advanced Performance Analysis](https://portkey.ai/docs/self-hosting/prometheus-metrics#advanced-performance-analysis)
* [gRPC Gateway Performance Analysis](https://portkey.ai/docs/self-hosting/prometheus-metrics#grpc-gateway-performance-analysis)
* [Alerting Rules Examples](https://portkey.ai/docs/self-hosting/prometheus-metrics#alerting-rules-examples)
* [Critical Performance Alerts](https://portkey.ai/docs/self-hosting/prometheus-metrics#critical-performance-alerts)
* [Cost Management Alerts](https://portkey.ai/docs/self-hosting/prometheus-metrics#cost-management-alerts)
* [Security Considerations](https://portkey.ai/docs/self-hosting/prometheus-metrics#security-considerations)
* [Troubleshooting](https://portkey.ai/docs/self-hosting/prometheus-metrics#troubleshooting)
* [Common Issues](https://portkey.ai/docs/self-hosting/prometheus-metrics#common-issues)
* [Related Documentation](https://portkey.ai/docs/self-hosting/prometheus-metrics#related-documentation)
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#overview)
Overview
---------------------------------------------------------------------------------
The Portkey Enterprise Gateway exposes detailed telemetry data through Prometheus metrics, enabling comprehensive observability for LLM gateway operations. These metrics cover the entire request lifecycle from authentication through response delivery, including cost tracking, performance monitoring, and cache analytics. This monitoring capability is essential for:
* **Performance Optimization**: Identify bottlenecks and optimize gateway performance
* **Cost Management**: Track and analyze LLM usage costs in real-time
* **Capacity Planning**: Understanding traffic patterns and scaling requirements
* **SLA Monitoring**: Ensure service level agreements are met
* **Security Monitoring**: Track authentication and authorization performance
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#metrics-endpoint)
Metrics Endpoint
-------------------------------------------------------------------------------------------------
**Endpoint**: `/metrics`
**Method**: GET
**Content-Type**: `text/plain; version=0.0.4; charset=utf-8`
**Authentication**: Typically open (check your deployment configuration)
Ensure your Prometheus server can access the `/metrics` endpoint. In production deployments, consider securing this endpoint or restricting access to monitoring infrastructure only.
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#global-configuration)
Global Configuration
---------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#default-labels)
Default Labels
All custom metrics are automatically labeled with:
* `app`: Service identifier (from `SERVICE_NAME` environment variable)
* `env`: Deployment environment (from `NODE_ENV` environment variable)
These labels enable multi-environment and multi-service monitoring in shared Prometheus deployments.
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#standard-node-js-runtime-metrics)
Standard Node.js Runtime Metrics
---------------------------------------------------------------------------------------------------------------------------------
The gateway automatically exposes Node.js runtime metrics with the `node_` prefix using the `prom-client` default collectors:
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#process-metrics)
Process Metrics
* **CPU Usage**: `node_process_cpu_user_seconds_total`, `node_process_cpu_system_seconds_total`
* **Memory**: `node_process_resident_memory_bytes`, `node_process_heap_bytes`
* **File Descriptors**: `node_process_open_fds`, `node_process_max_fds`
* **Process Uptime**: `node_process_start_time_seconds`
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#event-loop-metrics)
Event Loop Metrics
* **Event Loop Lag**: `node_eventloop_lag_seconds` (critical for detecting Node.js performance issues)
* **Event Loop Utilization**: Tracks how busy the event loop is
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#garbage-collection-metrics)
Garbage Collection Metrics
* **GC Duration**: `node_gc_duration_seconds` with custom buckets optimized for LLM gateway workloads
* **Custom GC Buckets**: `[0.001, 0.01, 0.1, 1, 1.5, 2, 3, 5, 7, 10, 15, 20, 30, 45, 60, 90, 120, 240, 500, 1000, 6000]` (seconds)
These buckets are specifically tuned for applications handling variable-length LLM processing workloads.
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#custom-application-metrics)
Custom Application Metrics
---------------------------------------------------------------------------------------------------------------------
The Portkey Enterprise Gateway exposes 14 custom metrics designed to provide deep visibility into LLM gateway operations, performance characteristics, and business metrics.
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#universal-label-schema)
Universal Label Schema
All custom metrics share a common labeling schema enabling multi-dimensional analysis:
* `method`: HTTP verb (GET, POST, PUT, DELETE, etc.)
* `endpoint`: Normalized API endpoint path (e.g., `/v1/chat/completions`, `/v1/completions`)
* `code`: HTTP response status code (200, 400, 500, etc.)
* `provider`: LLM provider identifier (openai, anthropic, azure-openai, etc.)
* `model`: Specific model name (gpt-4, claude-3, etc.)
* `source`: Request origination source or client identifier
* `stream`: Boolean indicator for streaming responses (“true”/“false”)
* `cacheStatus`: Cache interaction result (“hit”, “miss”, “disabled”, “error”)
* `metadata_*`: Dynamic labels from request metadata (see configuration section)
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#1-gateway-request-counter)
1\. Gateway Request Counter
**Metric Name**: `request_count`
**Type**: Counter
**Unit**: Requests **Technical Description**: Monotonic counter tracking every HTTP request processed by the gateway. This is the primary metric for understanding traffic volume, request patterns, and success/failure rates across different dimensions. **Use Cases**:
* **Traffic Analysis**: Monitor request volume trends and identify peak usage periods
* **Error Rate Monitoring**: Calculate error rates by dividing 4xx/5xx responses by total requests
* **Provider Distribution**: Understand which LLM providers are most heavily utilized
* **Model Popularity**: Track adoption of different AI models across your organization
* **Cache Effectiveness**: Monitor cache hit rates to optimize performance and costs
**Key Monitoring Patterns**:
Copy
Ask AI
# Request rate by provider
rate(request_count[5m]) by (provider)
# Error rate percentage
sum(rate(request_count{code=~"4..|5.."}[5m])) / sum(rate(request_count[5m])) * 100
# Requests by model
sum(rate(request_count[5m])) by (model, provider)
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#2-http-request-duration-distribution)
2\. HTTP Request Duration Distribution
**Metric Name**: `http_request_duration_seconds`
**Type**: Histogram
**Unit**: Seconds **Technical Description**: Measures the complete HTTP request-response cycle duration from the gateway’s perspective. This includes all processing time: authentication, middleware execution, provider communication, response processing, and network transmission back to the client. **Bucket Configuration**: `[0.1, 1, 1.5, 2, 3, 5, 7, 10, 15, 20, 30, 45, 60, 90, 120, 240, 500, 1000, 3000]`
* Optimized for typical LLM response times ranging from sub-second to several minutes
* Enables percentile analysis for SLA monitoring
**Use Cases**:
* **SLA Monitoring**: Track P95/P99 response times against service level agreements
* **Performance Regression Detection**: Identify when response times degrade
* **Endpoint Performance Analysis**: Compare performance across different API endpoints
* **Client-Side Latency Tracking**: Full end-to-end timing from client perspective
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#3-llm-provider-request-duration)
3\. LLM Provider Request Duration
**Metric Name**: `llm_request_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Measures the duration of actual requests sent to LLM providers, excluding gateway processing overhead. This metric isolates provider performance from internal gateway operations, enabling precise provider SLA monitoring and performance comparison. **Bucket Configuration**: `[0.1, 1, 2, 5, 10, 30, 50, 75, 100, 150, 200, 350, 500, 1000, 2500, 5000, 10000, 50000, 100000, 300000, 500000, 10000000]`
* High-resolution buckets for millisecond-precision analysis
* Extended range to handle very long-running requests (up to ~2.7 hours)
**Use Cases**:
* **Provider Performance Comparison**: Benchmark response times across different LLM providers
* **Model Performance Analysis**: Compare latency characteristics of different models
* **Provider SLA Monitoring**: Track provider performance against contractual agreements
* **Capacity Planning**: Understand provider response time distributions for scaling decisions
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#4-portkey-processing-time-excluding-streaming-latency)
4\. Portkey Processing Time (Excluding Streaming Latency)
**Metric Name**: `portkey_processing_time_excluding_last_byte_ms`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Measures Portkey’s internal processing time excluding the time spent waiting for the final byte of streamed responses from LLM providers. This metric isolates the gateway’s computational overhead from provider streaming characteristics, enabling precise performance optimization of internal operations. **Key Insights**:
* Excludes network latency and provider-side streaming delays
* Includes authentication, request transformation, middleware execution, and initial response processing
* Critical for identifying gateway performance bottlenecks vs. provider latency issues
**Use Cases**:
* **Gateway Performance Optimization**: Identify internal processing bottlenecks
* **Middleware Performance Analysis**: Measure impact of authentication and transformation logic
* **Scaling Decisions**: Understand processing capacity independent of provider performance
* **Performance Baseline Establishment**: Set internal SLAs for gateway operations
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#5-llm-last-byte-latency-analysis)
5\. LLM Last Byte Latency Analysis
**Metric Name**: `llm_last_byte_diff_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Captures the time difference between receiving the first response data and the final byte from LLM providers. This metric is essential for understanding streaming performance characteristics and time-to-first-token vs. total completion time patterns across different providers and models. **Streaming Analysis Value**:
* **Time-to-First-Token**: Indirectly measurable by comparing with total request duration
* **Streaming Efficiency**: Identifies providers with consistent vs. bursty streaming patterns
* **Model Behavior Analysis**: Different models exhibit different streaming characteristics
**Use Cases**:
* **User Experience Optimization**: Understand perceived responsiveness for streaming applications
* **Provider Streaming Comparison**: Compare streaming performance across providers
* **Model Selection**: Choose models based on streaming vs. batch completion preferences
* **Client Application Optimization**: Inform client-side timeout and buffering strategies
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#6-total-portkey-request-duration)
6\. Total Portkey Request Duration
**Metric Name**: `portkey_request_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Comprehensive timing metric measuring the complete duration of Portkey request processing from initial request receipt to final response transmission. This provides the most complete view of gateway performance and serves as the authoritative metric for end-to-end processing analysis. **Scope Includes**:
* Authentication and authorization processing
* Request validation and transformation
* Provider selection and routing logic
* LLM provider communication (complete)
* Response processing and transformation
* Cache operations (read/write)
* Post-processing hooks and analytics
**Use Cases**:
* **Comprehensive Performance Monitoring**: Single metric for overall gateway health
* **Capacity Planning**: Understand total processing requirements for scaling
* **Performance Baseline**: Primary metric for SLA establishment and monitoring
* **Troubleshooting**: First metric to check during performance investigations
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#7-llm-cost-accumulator)
7\. LLM Cost Accumulator
**Metric Name**: `llm_cost_sum`
**Type**: Gauge
**Unit**: Currency Units (USD) **Technical Description**: Real-time accumulator tracking the total monetary cost of LLM API usage across all providers and models. This gauge provides immediate visibility into cost burn rates and enables fine-grained cost analysis across multiple dimensions including users, applications, models, and providers. **Cost Calculation Features**:
* **Multi-Provider Support**: Normalized cost tracking across different provider pricing models
* **Token-Based Accuracy**: Precise cost calculation based on actual token consumption
* **Real-Time Updates**: Immediate cost visibility for budget monitoring
* **Dimensional Analysis**: Cost breakdown by any label dimension
**Business Value**:
* **Budget Monitoring**: Real-time tracking against spending limits
* **Cost Attribution**: Identify highest-cost users, applications, or use cases
* **ROI Analysis**: Measure cost efficiency across different models and providers
* **Chargeback/Showback**: Accurate cost allocation for internal billing
**Key Monitoring Patterns**:
Copy
Ask AI
# Cost per model over time
rate(llm_cost_sum[1h]) by (model, provider)
# Highest cost users
topk(10, sum(rate(llm_cost_sum[24h])) by (metadata_user_id))
# Cost efficiency by provider
rate(llm_cost_sum[1h]) / rate(request_count[1h]) by (provider)
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#8-authentication-performance-analysis)
8\. Authentication Performance Analysis
**Metric Name**: `authentication_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Measures the complete authentication and authorization pipeline duration, including API key validation, usage limit verification, and permission checks. This metric is critical for identifying authentication bottlenecks that could impact overall gateway performance and user experience. **Authentication Pipeline Components**:
* **API Key Validation**: Cryptographic verification and database lookups
* **Usage Limit Checks**: Real-time quota verification against configured limits
* **Permission Validation**: Role-based access control (RBAC) enforcement
* **Workspace/Organization Context**: Multi-tenant authorization processing
**Performance Optimization Value**:
* **Database Performance**: Identifies slow authentication database queries
* **Caching Effectiveness**: Measures benefit of authentication result caching
* **Security vs. Performance**: Balances security thoroughness with response time requirements
**Use Cases**:
* **Security Performance Monitoring**: Ensure security checks don’t degrade user experience
* **Database Optimization**: Identify authentication-related database performance issues
* **Caching Strategy**: Optimize authentication result caching for better performance
* **Bottleneck Identification**: Pinpoint authentication components causing delays
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#9-rate-limiting-performance-metrics)
9\. Rate Limiting Performance Metrics
**Metric Name**: `api_key_rate_limit_check_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Tracks the performance of hierarchical rate limiting checks across organization, workspace, and user levels. This metric monitors the computational overhead of the multi-level rate limiting system and identifies performance impacts of complex rate limiting policies. **Rate Limiting Hierarchy**:
* **Organization Level**: Global rate limits across entire organization
* **Workspace Level**: Team or project-specific rate limits
* **User Level**: Individual user rate limits and quotas
* **API Key Level**: Specific API key usage limits
**Technical Implementation Insights**:
* **Redis Performance**: Measures Redis-based rate limiting performance
* **Multi-Level Complexity**: Tracks overhead of hierarchical limit checking
* **Atomic Operations**: Performance of distributed rate limiting algorithms
**Use Cases**:
* **Rate Limiting Optimization**: Optimize rate limiting algorithms for better performance
* **Redis Performance Monitoring**: Track distributed rate limiting infrastructure performance
* **Policy Impact Analysis**: Measure performance impact of complex rate limiting policies
* **Scaling Capacity Planning**: Understand rate limiting overhead for capacity planning
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#10-pre-request-middleware-pipeline-performance)
10\. Pre-Request Middleware Pipeline Performance
**Metric Name**: `pre_request_processing_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Comprehensive timing of the pre-request processing pipeline that prepares requests for LLM provider execution. This metric captures the overhead of all preparatory operations required before sending requests to upstream providers. **Pipeline Components**:
* **Request Context Creation**: Building execution context and metadata
* **Prompt Template Processing**: Variable substitution and template rendering
* **Guardrails Retrieval**: Fetching and applying content safety policies
* **Provider Configuration**: Loading provider-specific settings and authentication
* **Cache Key Generation**: Computing cache identifiers for request deduplication
* **Request Transformation**: Converting requests to provider-specific formats
**Optimization Opportunities**:
* **Template Caching**: Optimize prompt template compilation and caching
* **Guardrails Performance**: Minimize overhead of content safety checks
* **Configuration Caching**: Reduce database queries for provider settings
**Use Cases**:
* **Middleware Performance Optimization**: Identify and optimize slow preprocessing steps
* **Request Preparation Monitoring**: Track overhead of request enhancement features
* **Feature Impact Analysis**: Measure performance cost of advanced features
* **Pipeline Efficiency**: Optimize the request processing pipeline for better throughput
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#11-post-request-processing-pipeline-performance)
11\. Post-Request Processing Pipeline Performance
**Metric Name**: `post_request_processing_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Measures the duration of post-request processing operations that occur after receiving responses from LLM providers but before returning results to clients. This metric captures the overhead of response enhancement, logging, analytics, and cleanup operations. **Post-Processing Components**:
* **Response Transformation**: Converting provider responses to standardized formats
* **Analytics Data Collection**: Gathering metrics and usage statistics
* **Audit Logging**: Recording detailed request/response logs for compliance
* **Cache Writing**: Storing responses for future cache hits
* **Webhook Execution**: Triggering configured post-request webhooks
* **Cost Calculation**: Computing and recording usage costs
**Business Intelligence Value**:
* **Analytics Overhead**: Measures cost of detailed usage analytics
* **Compliance Impact**: Tracks overhead of audit logging requirements
* **Cache Performance**: Monitors cache write operation performance
**Use Cases**:
* **Response Processing Optimization**: Minimize post-request processing overhead
* **Analytics Performance**: Optimize data collection and storage operations
* **Cache Write Performance**: Monitor and optimize cache storage operations
* **Compliance Monitoring**: Track overhead of regulatory compliance features
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#12-cache-system-performance-analysis)
12\. Cache System Performance Analysis
**Metric Name**: `llm_cache_processing_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Dedicated metric for measuring cache system performance including cache key computation, cache lookups, cache hits/misses, and cache storage operations. This metric is essential for optimizing cache configuration and understanding cache system impact on overall performance. **Cache Operation Types**:
* **Cache Key Generation**: Computing deterministic cache identifiers
* **Cache Lookup Operations**: Reading from distributed cache storage
* **Cache Hit Processing**: Deserializing and returning cached responses
* **Cache Miss Handling**: Managing cache misses and preparing for cache writes
* **Cache Storage Operations**: Writing new responses to cache storage
**Cache Performance Insights**:
* **Hit vs. Miss Performance**: Compare cache hit vs. miss processing times
* **Storage Backend Performance**: Monitor Redis, Memcached, or other cache backend performance
* **Serialization Overhead**: Track cost of response serialization/deserialization
* **Network Latency**: Measure distributed cache network performance
**Optimization Opportunities**:
* **Cache Strategy Tuning**: Optimize cache TTL and eviction policies
* **Serialization Optimization**: Improve response serialization performance
* **Cache Backend Scaling**: Plan cache infrastructure scaling based on performance data
* **Cache Hit Rate Optimization**: Improve cache key generation for better hit rates
**Key Monitoring Patterns**:
Copy
Ask AI
# Cache hit vs miss performance comparison
histogram_quantile(0.95, rate(llm_cache_processing_duration_milliseconds_bucket{cacheStatus="hit"}[5m])) vs
histogram_quantile(0.95, rate(llm_cache_processing_duration_milliseconds_bucket{cacheStatus="miss"}[5m]))
# Cache operation throughput
rate(llm_cache_processing_duration_milliseconds_count[5m]) by (cacheStatus)
* * *
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#13-grpc-request-conversion-performance)
13\. gRPC Request Conversion Performance
**Metric Name**: `grpc_req_conversion_duration_milliseconds`
**Type**: Histogram
**Unit**: Milliseconds **Technical Description**: Measures the performance of converting incoming gRPC requests to HTTP format before forwarding to the internal HTTP handler. This metric is critical for understanding the overhead introduced by the gRPC-to-HTTP adapter layer and identifying potential bottlenecks in the gRPC gateway implementation. **Bucket Configuration**: `[0.01, 0.1, 0.5, 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000]`
* Optimized for conversion operations typically ranging from sub-millisecond to several seconds
* Higher resolution at lower latencies to detect micro-optimizations
**Conversion Pipeline Components**:
* **gRPC Metadata Extraction**: Converting gRPC metadata to HTTP headers
* **Request Body Processing**: Transforming gRPC request body to HTTP format
* **Protocol Translation**: Converting gRPC semantics to HTTP semantics
* **Header Validation**: Filtering and validating headers for HTTP compatibility
**Performance Optimization Value**:
* **Adapter Efficiency**: Identify inefficiencies in the gRPC-to-HTTP conversion logic
* **Serialization Performance**: Monitor overhead of request format transformation
* **Memory Usage**: Track memory allocation patterns during conversion
* **Protocol Overhead**: Measure the cost of protocol translation
**Use Cases**:
* **gRPC Gateway Optimization**: Optimize the conversion layer for better throughput
* **Service Migration**: Compare gRPC vs. HTTP performance during service transitions
* **Protocol Selection**: Inform decisions about when to use gRPC vs. HTTP
* **Performance Regression Detection**: Alert on conversion performance degradation
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#configuration)
Configuration
-------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#dynamic-metadata-label-system)
Dynamic Metadata Label System
The Portkey Enterprise Gateway supports dynamic metadata labeling through request-specific metadata injection. This powerful feature enables fine-grained observability across custom dimensions specific to your organization’s structure and use cases. **Environment Variable**: `PROMETHEUS_LABELS_METADATA_ALLOWED_KEYS` **Configuration Format**: Comma-separated list of metadata keys
Copy
Ask AI
PROMETHEUS_LABELS_METADATA_ALLOWED_KEYS=user_id,organization_id,workspace_id,application_name,cost_center,environment_type
**Technical Implementation**:
* Metadata is passed via the `x-portkey-metadata` HTTP header as a JSON string
* The JSON string is parsed and keys are validated against the allowlist for security
* Invalid or missing metadata values are handled gracefully (empty object returned)
* Labels are automatically prefixed with `metadata_` to avoid naming conflicts
**Security Considerations**:
* **Cardinality Control**: Limit allowed keys to prevent metric explosion
* **Sensitive Data Protection**: Avoid including PII or sensitive information in labels
* **Performance Impact**: Each additional label increases metric storage requirements
**Example Metadata Usage**: Pass metadata via HTTP header:
Copy
Ask AI
POST /v1/chat/completions
Content-Type: application/json
x-portkey-metadata: {"user_id":"user_12345","organization_id":"org_acme","workspace_id":"workspace_engineering","application_name":"customer_support_bot","cost_center":"engineering_ops"}
{
"model": "gpt-4",
"messages": [...]
}
Resulting in Prometheus labels:
Copy
Ask AI
metadata_user_id="user_12345"
metadata_organization_id="org_acme"
metadata_workspace_id="workspace_engineering"
metadata_application_name="customer_support_bot"
metadata_cost_center="engineering_ops"
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#comprehensive-monitoring-examples)
Comprehensive Monitoring Examples
-----------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#traffic-analysis-%26-capacity-planning)
Traffic Analysis & Capacity Planning
**Request Volume Monitoring**:
Copy
Ask AI
# Requests per second by provider
rate(request_count[5m]) by (provider)
# Peak request volume (requests per hour)
increase(request_count[1h]) by (provider, model)
# Request volume growth trend (week-over-week)
(
rate(request_count[7d]) -
rate(request_count[7d] offset 7d)
) / rate(request_count[7d] offset 7d) * 100
**Traffic Distribution Analysis**:
Copy
Ask AI
# Top 10 models by request volume
topk(10, rate(request_count[1h]) by (model, provider))
# Request distribution by endpoint
(
rate(request_count[5m]) by (endpoint) /
ignoring(endpoint) group_left sum(rate(request_count[5m]))
) * 100
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#performance-monitoring-%26-sla-tracking)
Performance Monitoring & SLA Tracking
**Latency Percentile Analysis**:
Copy
Ask AI
# P50, P95, P99 response times by provider
histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m])) by (provider)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) by (provider)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) by (provider)
# LLM provider latency comparison (P95)
histogram_quantile(0.95, rate(llm_request_duration_milliseconds_bucket[5m])) by (provider, model)
**Performance Degradation Detection**:
Copy
Ask AI
# Identify performance regressions (current P95 vs. 24h ago)
(
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) -
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m] offset 24h))
) > 0.5 # Alert if P95 increased by >500ms
**Error Rate Monitoring**:
Copy
Ask AI
# Overall error rate (4xx + 5xx)
sum(rate(request_count{code=~"4..|5.."}[5m])) /
sum(rate(request_count[5m])) * 100
# Error rate by provider and model
sum(rate(request_count{code=~"4..|5.."}[5m])) by (provider, model) /
sum(rate(request_count[5m])) by (provider, model) * 100
# Critical error rate (5xx only)
sum(rate(request_count{code=~"5.."}[5m])) /
sum(rate(request_count[5m])) * 100
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#cache-performance-optimization)
Cache Performance Optimization
**Cache Effectiveness Analysis**:
Copy
Ask AI
# Cache hit rate by provider and model
sum(rate(request_count{cacheStatus="hit"}[5m])) by (provider, model) /
sum(rate(request_count{cacheStatus=~"hit|miss"}[5m])) by (provider, model) * 100
# Cache performance improvement (latency reduction)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{cacheStatus="miss"}[5m])) -
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{cacheStatus="hit"}[5m]))
# Cache system overhead
histogram_quantile(0.95, rate(llm_cache_processing_duration_milliseconds_bucket[5m])) by (cacheStatus)
**Cache Cost-Benefit Analysis**:
Copy
Ask AI
# Cost savings from cache hits
sum(rate(llm_cost_sum{cacheStatus="hit"}[1h])) * 0 # Cache hits cost $0
vs
sum(rate(llm_cost_sum{cacheStatus="miss"}[1h])) # Actual provider costs
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#cost-management-%26-business-intelligence)
Cost Management & Business Intelligence
**Real-Time Cost Monitoring**:
Copy
Ask AI
# Hourly cost burn rate
rate(llm_cost_sum[1h])
# Cost per request by model
rate(llm_cost_sum[1h]) / rate(request_count[1h]) by (model, provider)
# Top cost drivers (users/applications)
topk(10, sum(rate(llm_cost_sum[24h])) by (metadata_user_id))
topk(10, sum(rate(llm_cost_sum[24h])) by (metadata_application_name))
**Cost Trend Analysis**:
Copy
Ask AI
# Daily cost comparison (today vs. yesterday)
increase(llm_cost_sum[1d]) - increase(llm_cost_sum[1d] offset 1d)
# Monthly cost projection based on current week
increase(llm_cost_sum[7d]) * 4.33 # Approximate monthly projection
# Cost efficiency trend (cost per successful request)
rate(llm_cost_sum[1d]) / rate(request_count{code="200"}[1d])
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#security-%26-authentication-performance)
Security & Authentication Performance
**Authentication Performance Monitoring**:
Copy
Ask AI
# Authentication latency impact on total request time
histogram_quantile(0.95, rate(authentication_duration_milliseconds_bucket[5m]))
# Rate limiting overhead
histogram_quantile(0.95, rate(api_key_rate_limit_check_duration_milliseconds_bucket[5m]))
# Authentication vs. total request duration ratio
histogram_quantile(0.95, rate(authentication_duration_milliseconds_bucket[5m])) /
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) * 1000
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#advanced-performance-analysis)
Advanced Performance Analysis
**Processing Pipeline Breakdown**:
Copy
Ask AI
# Pre-processing vs. LLM vs. post-processing time distribution
avg_over_time(
histogram_quantile(0.95, rate(pre_request_processing_duration_milliseconds_bucket[5m]))[1h:]
) label_replace(..., "stage", "pre_processing", "", "")
union
avg_over_time(
histogram_quantile(0.95, rate(llm_request_duration_milliseconds_bucket[5m]))[1h:]
) label_replace(..., "stage", "llm_processing", "", "")
union
avg_over_time(
histogram_quantile(0.95, rate(post_request_processing_duration_milliseconds_bucket[5m]))[1h:]
) label_replace(..., "stage", "post_processing", "", "")
**Streaming Performance Analysis**:
Copy
Ask AI
# Time-to-first-token approximation
histogram_quantile(0.95, rate(llm_request_duration_milliseconds_bucket{stream="true"}[5m])) -
histogram_quantile(0.95, rate(llm_last_byte_diff_duration_milliseconds_bucket{stream="true"}[5m]))
# Streaming vs. non-streaming latency comparison
histogram_quantile(0.95, rate(llm_request_duration_milliseconds_bucket{stream="true"}[5m])) by (provider)
vs
histogram_quantile(0.95, rate(llm_request_duration_milliseconds_bucket{stream="false"}[5m])) by (provider)
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#grpc-gateway-performance-analysis)
gRPC Gateway Performance Analysis
**gRPC Conversion Efficiency Monitoring**:
Copy
Ask AI
# gRPC conversion latency by service
histogram_quantile(0.95, rate(grpc_req_conversion_duration_milliseconds_bucket[5m])) by (service, method)
histogram_quantile(0.95, rate(grpc_res_conversion_duration_milliseconds_bucket[5m])) by (service, method)
# Total round-trip conversion overhead
(
histogram_quantile(0.95, rate(grpc_req_conversion_duration_milliseconds_bucket[5m])) +
histogram_quantile(0.95, rate(grpc_res_conversion_duration_milliseconds_bucket[5m]))
) by (service)
# gRPC vs HTTP performance comparison
histogram_quantile(0.95, rate(grpc_req_conversion_duration_milliseconds_bucket[5m])) by (service)
vs
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{method="POST"}[5m])) by (endpoint) * 1000
**gRPC Throughput Analysis**:
Copy
Ask AI
# gRPC conversion requests per second
rate(grpc_req_conversion_duration_milliseconds_count[5m]) by (service, method)
# Conversion efficiency trend (conversions per second vs latency)
rate(grpc_req_conversion_duration_milliseconds_count[5m]) /
histogram_quantile(0.95, rate(grpc_req_conversion_duration_milliseconds_bucket[5m]))
# Service-specific conversion volume
sum(rate(grpc_req_conversion_duration_milliseconds_count[5m])) by (service)
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#alerting-rules-examples)
Alerting Rules Examples
---------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#critical-performance-alerts)
Critical Performance Alerts
Copy
Ask AI
# High error rate alert (>5% for 5 minutes)
sum(rate(request_count{code=~"5.."}[5m])) / sum(rate(request_count[5m])) > 0.05
# High latency alert (P95 > 10 seconds)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 10
# Authentication performance degradation
histogram_quantile(0.95, rate(authentication_duration_milliseconds_bucket[5m])) > 1000
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#cost-management-alerts)
Cost Management Alerts
Copy
Ask AI
# Hourly cost spike (>200% of 24h average)
rate(llm_cost_sum[1h]) > 2 * avg_over_time(rate(llm_cost_sum[1h])[24h:])
# Daily budget threshold (>80% of daily limit)
increase(llm_cost_sum[1d]) > 0.8 * $DAILY_BUDGET_LIMIT
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#security-considerations)
Security Considerations
---------------------------------------------------------------------------------------------------------------
When deploying metrics collection in production:
1. **Endpoint Security**: Consider securing the `/metrics` endpoint with authentication or network restrictions
2. **Data Sensitivity**: Avoid including sensitive information in metadata labels
3. **Cardinality Management**: Limit metadata keys to prevent metric explosion and storage issues
4. **Network Security**: Ensure secure communication between Prometheus and the gateway
5. **Access Control**: Implement appropriate access controls for monitoring dashboards
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#troubleshooting)
Troubleshooting
-----------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#common-issues)
Common Issues
**High Cardinality Metrics**:
* Symptom: Prometheus storage growth, query performance degradation
* Solution: Review `PROMETHEUS_LABELS_METADATA_ALLOWED_KEYS` configuration and limit high-cardinality labels
**Missing Metrics**:
* Check that the `/metrics` endpoint is accessible
* Verify Prometheus scraping configuration
* Review gateway logs for metric collection errors
**Performance Impact**:
* Monitor the overhead of metrics collection on gateway performance
* Consider adjusting scrape intervals for high-volume deployments
[](https://portkey.ai/docs/self-hosting/prometheus-metrics#related-documentation)
Related Documentation
-----------------------------------------------------------------------------------------------------------
* [Analytics Dashboard](https://portkey.ai/docs/product/observability/analytics)
- SaaS monitoring and analytics
* [Private Cloud Architecture](https://portkey.ai/docs/product/enterprise-offering/private-cloud-deployments/architecture)
- Deployment architecture overview
* [Observability](https://portkey.ai/docs/product/observability)
- General observability features
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/self-hosting/prometheus-metrics.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/self-hosting/prometheus-metrics)
[Azure](https://portkey.ai/docs/self-hosting/hybrid-deployments/azure)
[Upgrade to Model Catalog](https://portkey.ai/docs/support/upgrade-to-model-catalog)
⌘I
---
# December '23 Migration - Portkey Docs
[Skip to main content](https://portkey.ai/docs/support/portkeys-december-migration#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Support
December '23 Migration
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Compatibility & Deprecation List](https://portkey.ai/docs/support/portkeys-december-migration#compatibility-%26-deprecation-list)
* [Major Version Release of the SDK](https://portkey.ai/docs/support/portkeys-december-migration#major-version-release-of-the-sdk)
* [Here’s What’s New:](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new%3A)
* [Here’s What’s Changed:](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed%3A)
* [All-New APIs](https://portkey.ai/docs/support/portkeys-december-migration#all-new-apis)
* [Here’s What’s New:](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new%3A-2)
* [Here’s What’s Changed](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed)
* [Simlarly, for Prompts](https://portkey.ai/docs/support/portkeys-december-migration#simlarly%2C-for-prompts)
* [Configs 2.0](https://portkey.ai/docs/support/portkeys-december-migration#configs-2-0)
* [Here’s What’s New](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new)
* [Here’s What’s Changed](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed-2)
* [Support](https://portkey.ai/docs/support/portkeys-december-migration#support)
> **Date: 8th Dec, 2023**
This December, we’re pushing out some exciting new updates to Portkey’s **SDKs**, **APIs**, and **Configs**.
[**Portkey’s SDKs**](https://portkey.ai/docs/support/portkeys-december-migration#major-version-release-of-the-sdk)
are upped to _**major version 1.0**_ bringing parity with the new OpenAI SDK structure and adding Portkey production features to it. We are also bringing native Langchain & Llamaindex integrations inside the SDK. This is a **Breaking Change** that **Requires Migration**.
[**Portkey’s APIs**](https://portkey.ai/docs/support/portkeys-december-migration#all-new-apis)
are upgraded with _**new endpoints**_, making it simpler to do `/chat/completions` and `/completions` calls and adding Portkey’s production functionalities to them.This is a **Breaking Change** that **Requires Migration**.
[**Configs**](https://portkey.ai/docs/support/portkeys-december-migration#configs-2.0)
are upgraded to _**version**_ _**2.0**_, bringing nested gateway strategies with granular handling. For Configs saved in the Portkey dashboard, this is **NOT a Breaking Change** and we will **Auto Migrate** your old Configs. For Configs directly defined at the time of making a call, through the old SDKs or old APIS, they **will fail** on the new APIs & SDKs and **require migration**.
[](https://portkey.ai/docs/support/portkeys-december-migration#compatibility-%26-deprecation-list)
Compatibility & Deprecation List
---------------------------------------------------------------------------------------------------------------------------------------
| List | Compatibility | Deprecation Date |
| --- | --- | --- |
| **API (Old)**
`/v1/proxy`
`/v1/complete`
`/v1/chatComplete`
`/v1/embed`
`/v1/prompts/ID/generate` | SDK (Old) SDK (New) Configs (Old) Configs (New) | Q2 ‘24 |
| **API (New)**
`/v1`
`/v1/completions`
`/v1/chat/completions`
`/v1/embeddings`
`/v1/prompts/ID/completions` | SDK (Old) SDK (New) Configs (Old) Configs (New) | \- |
| **SDK Version < 1 (Old)** | API (Old) API (New) Configs (Old) Configs (new) | Q2 ‘24 |
| **SDK Version = 1 (New)** | API (Old) API (New) Configs (Old) Configs (new) | \- |
| **Configs 1.0 (Old)** | API (Old) API (New) SDK (Old) SDK (new) === Configs saved through the Portkey UI will be auto migrated. | Q2 ‘24 |
| **Configs 2.0 (New)** | API (Old) API (New) SDK (Old) SDK (new) | \- |
We recommend upgrading to these new versions promptly to take full advantage of their capabilities. While your existing code will continue to work until the deprecation date around Q2 ‘24, transitioning now ensures you stay ahead of the curve and avoid any future service interruptions. Follow along with this guide!
* * *
[](https://portkey.ai/docs/support/portkeys-december-migration#major-version-release-of-the-sdk)
Major Version Release of the SDK
-------------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new%3A)
Here’s What’s New:
1. More extensible SDK that can be used with many more LLM providers
2. Out-of-the-box support for streaming
3. Completely follows OpenAI’s SDK signature reducing your technical debt
4. Native support for Langchain & Llamaindex within the SDK (Python)
5. Support for the Portkey Feedback endpoint
6. Support for Portkey Prompt Templates
7. Older SDK versions to be deprecated soon
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed%3A)
Here’s What’s Changed:
* Portkey Python SDK
* Portkey Node SDK
**FROM**
Copy
Ask AI
import portkey
from portkey import Config, LLMOptions
portkey.config = Config(
mode="single",
llms=LLMOptions(provider="openai", api_key="OPENAI_API_KEY")
)
response = portkey.ChatCompletions.create(
model="gpt-4",
messages=[\
{"role": "user","content": "Hello World!"}\
]
)
**TO**
Copy
Ask AI
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
Authorization="OPENAI_KEY"
)
response = portkey.chat.completions.create(
messages=[{'role': 'user', 'content': 'Say this is a test'}],
model='gpt-3.5-turbo'
)
print(response)
**Installing the New SDK,**
Copy
Ask AI
pip install -U portkey-ai
[SDK\
---](https://portkey.ai/docs/api-reference/portkey-sdk-client)
[](https://portkey.ai/docs/support/portkeys-december-migration#all-new-apis)
All-New APIs
---------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new%3A-2)
Here’s What’s New:
1. Introduced 3 new routes `/chat/completions`, `/completions`, and `/embeddings`
2. Simplified the headers:
1. `x-portkey-mode` header is deprecated and replaced with `x-portkey-provider`
1. Which takes values: `openai`, `anyscale`, `cohere,` `palm`, `azure-openai`, and more.
2. New header `x-portkey-virtual-key` is introduced.
3. `/complete` and `/chatComplete` endpoints to be deprecated soon
4. Prompts endpoint `/prompts/$PROMPT_ID/generate` is upgraded to `/prompts/$PROMPT_ID/completions` and the old route will be deprecated soon
1. We now support updating the model params on-the-fly (i.e. changing temperature etc at the time of making a call)
2. Prompt response object on the `/completions` route is now fully OpenAI compliant
5. New `/gateway` endpoint that lets you make calls to third-party LLM providers easily
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed)
Here’s What’s Changed
* OpenAI Python SDK
* OpenAI Node SDK
* cURL
**FROM**
Copy
Ask AI
from openai import OpenAI
client = OpenAI(
api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
base_url="https://api.portkey.ai/v1/proxy",
default_headers= {
"x-portkey-api-key": "PORTKEY_API_KEY",
"x-portkey-mode": "proxy openai",
"Content-Type": "application/json"
}
)
chat_complete = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Say this is a test"}],
)
print(chat_complete.choices[0].message.content)
**TO**
Copy
Ask AI
# pip install -U portkey-ai
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
client = OpenAI(
api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai",
api_key="PORTKEY_API_KEY" # defaults to os.environ.get("PORTKEY_API_KEY")
)
)
chat_complete = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Say this is a test"}],
)
print(chat_complete.choices[0].message.content)
[Chat\
----](https://portkey.ai/docs/provider-endpoints/chat)
[Completions\
-----------](https://portkey.ai/docs/provider-endpoints/completions)
[Gateway for Other API Endpoints\
-------------------------------](https://portkey.ai/docs/provider-endpoints/gateway-for-other-apis)
###
[](https://portkey.ai/docs/support/portkeys-december-migration#simlarly%2C-for-prompts)
Simlarly, for Prompts
* cURL
* Changing Model Params On-the-fly (Python)
**FROM**
Copy
Ask AI
curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/generate \
-H 'x-portkey-api-key: $PORTKEY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"variables": {"variable_a": "", "variable_b": ""}}'
**TO**
Copy
Ask AI
curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/completions \
-H 'x-portkey-api-key: $PORTKEY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"variables": {"variable_a": "", "variable_b": ""}}'
[Prompts\
-------](https://portkey.ai/docs/portkey-endpoints/prompts)
[](https://portkey.ai/docs/support/portkeys-december-migration#configs-2-0)
Configs 2.0
-------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-new)
Here’s What’s New
1. New concept of `strategy` instead of standalone `mode`. You can now build bespoke gateway strategies and nest them in a single config.
2. You can also trigger a specific strategy on specific error codes.
3. New concept of `targets` that replace `options` in the previous Config
4. If you are adding `virtual_key` to the target array, you no longer need to add `provider`,Portkey will pick up the Provider directly from the Virtual Key!
5. For Azure, only now pass the `virtual_key` - it takes care of all other Azure params like Deployment name, API version etc.
The Configs UI on Portkey app will autocomplete Configs ONLY in the new format now. All your existing Configs are auto migrated.
###
[](https://portkey.ai/docs/support/portkeys-december-migration#here%E2%80%99s-what%E2%80%99s-changed-2)
Here’s What’s Changed
* Config Builder
**FROM**
Copy
Ask AI
{
"mode": "single",
"options": [\
{\
"provider": "openai",\
"virtual_key": "open-4110dd",\
}\
]
}
**TO**
Copy
Ask AI
{
"strategy": {
"mode":"single"
},
"targets": [\
{\
"virtual_key": "open-4110dd"\
}\
]
}
[Gateway Config Object\
---------------------](https://portkey.ai/docs/api-reference)
* * *
[](https://portkey.ai/docs/support/portkeys-december-migration#support)
Support
-----------------------------------------------------------------------------------
Shoot ANY questions or queries you have on the migration to the Portkey team [**on our Discord**](https://discord.gg/yn6QtVZJgV)
and we will try to get back to you ASAP.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/support/portkeys-december-migration.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/support/portkeys-december-migration)
[Common Errors & Resolutions](https://portkey.ai/docs/support/common-errors-and-resolutions)
⌘I
---
# Make Your First Request - Portkey Docs
[Skip to main content](https://portkey.ai/docs/introduction/make-your-first-request#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Introduction
Make Your First Request
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [1\. Get your Portkey API Key](https://portkey.ai/docs/introduction/make-your-first-request#1-get-your-portkey-api-key)
* [2\. Integrate Portkey](https://portkey.ai/docs/introduction/make-your-first-request#2-integrate-portkey)
* [Through the OpenAI SDK](https://portkey.ai/docs/introduction/make-your-first-request#through-the-openai-sdk)
* [Portkey SDK](https://portkey.ai/docs/introduction/make-your-first-request#portkey-sdk)
* [Other Integration Guides](https://portkey.ai/docs/introduction/make-your-first-request#other-integration-guides)
* [3\. Next Steps](https://portkey.ai/docs/introduction/make-your-first-request#3-next-steps)
[](https://portkey.ai/docs/introduction/make-your-first-request#1-get-your-portkey-api-key)
1\. Get your Portkey API Key
----------------------------------------------------------------------------------------------------------------------------
[Create](https://app.portkey.ai/signup)
or [log in](https://app.portkey.ai/login)
to your Portkey account. Grab your account’s API key from the “Settings” page.
Copy your Portkey account API key
Based on your access level, you might see the relevant permissions on the API key modal - tick the ones you’d like, name your API key, and save it.
[](https://portkey.ai/docs/introduction/make-your-first-request#2-integrate-portkey)
2\. Integrate Portkey
--------------------------------------------------------------------------------------------------------------
Portkey offers a variety of integration options, including SDKs, REST APIs, and native connections with platforms like OpenAI, Langchain, and LlamaIndex, among others.
###
[](https://portkey.ai/docs/introduction/make-your-first-request#through-the-openai-sdk)
Through the OpenAI SDK
If you’re using the **OpenAI SDK**, import the Portkey SDK and configure it within your OpenAI client object:[OpenAI\
------](https://portkey.ai/docs/integrations/llms/openai)
###
[](https://portkey.ai/docs/introduction/make-your-first-request#portkey-sdk)
Portkey SDK
You can also use the **Portkey SDK / REST APIs** directly to make the chat completion calls. This is a more versatile way to make LLM calls across any provider:[SDK\
---](https://portkey.ai/docs/api-reference/portkey-sdk-client)
Once, the integration is ready, you can view the requests reflect on your Portkey dashboard.
###
[](https://portkey.ai/docs/introduction/make-your-first-request#other-integration-guides)
Other Integration Guides
[Azure OpenAI\
------------](https://portkey.ai/docs/integrations/llms/azure-openai)
[Anthropic\
---------](https://portkey.ai/docs/integrations/llms/anthropic)
[Langchain\
---------](https://portkey.ai/docs/integrations/libraries/langchain-python)
[LlamaIndex\
----------](https://portkey.ai/docs/integrations/libraries/llama-index-python)
[Ollama\
------](https://portkey.ai/docs/integrations/llms/ollama)
[Others\
------](https://portkey.ai/docs/integrations/llms)
[](https://portkey.ai/docs/introduction/make-your-first-request#3-next-steps)
3\. Next Steps
------------------------------------------------------------------------------------------------
Now that you’re up and running with Portkey, you can dive into the various Portkey features to learn about all of the supported functionalities:
[Observability\
-------------](https://portkey.ai/docs/product/observability)
[AI Gateway\
----------](https://portkey.ai/docs/product/ai-gateway)
[Prompt Library\
--------------](https://portkey.ai/docs/product/prompt-library)
[Autonomous Fine-Tuning\
----------------------](https://portkey.ai/docs/product/autonomous-fine-tuning)
[Guardrails\
----------](https://portkey.ai/docs/product/guardrails)
[Enterprise\
----------](https://portkey.ai/docs/product/enterprise-offering)
While you’re here, why not [give us a star](https://git.new/ai-gateway-docs)
? It helps us a lot!
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/introduction/make-your-first-request.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/introduction/make-your-first-request)
[What is Portkey?](https://portkey.ai/docs/introduction/what-is-portkey)
[Portkey Features](https://portkey.ai/docs/introduction/feature-overview)
⌘I
---
# September - Portkey Docs
[Skip to main content](https://portkey.ai/docs/changelog/2025/september#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
2025
September
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Summary](https://portkey.ai/docs/changelog/2025/september#summary)
* [Platform](https://portkey.ai/docs/changelog/2025/september#platform)
* [MCP Gateway (beta)](https://portkey.ai/docs/changelog/2025/september#mcp-gateway-beta)
* [Unified token counting endpoint](https://portkey.ai/docs/changelog/2025/september#unified-token-counting-endpoint)
* [Guardrails](https://portkey.ai/docs/changelog/2025/september#guardrails)
* [Metadata-based Model Access guardrail](https://portkey.ai/docs/changelog/2025/september#metadata-based-model-access-guardrail)
* [Regex-replace guardrail](https://portkey.ai/docs/changelog/2025/september#regex-replace-guardrail)
* [Gateway and Providers](https://portkey.ai/docs/changelog/2025/september#gateway-and-providers)
* [Unified finish\_reason parameter](https://portkey.ai/docs/changelog/2025/september#unified-finish-reason-parameter)
* [Conditional router enhancement](https://portkey.ai/docs/changelog/2025/september#conditional-router-enhancement)
* [Gateway and Providers](https://portkey.ai/docs/changelog/2025/september#gateway-and-providers-2)
* [New Models](https://portkey.ai/docs/changelog/2025/september#new-models)
* [New Providers](https://portkey.ai/docs/changelog/2025/september#new-providers)
* [Improvements and Fixes](https://portkey.ai/docs/changelog/2025/september#improvements-and-fixes)
* [Community & Events](https://portkey.ai/docs/changelog/2025/september#community-%26-events)
* [Protecting your AI platform with Palo Alto Networks and Portkey](https://portkey.ai/docs/changelog/2025/september#protecting-your-ai-platform-with-palo-alto-networks-and-portkey)
* [LLMs in Prod, San Francisco](https://portkey.ai/docs/changelog/2025/september#llms-in-prod%2C-san-francisco)
* [PG&E Immersion Day](https://portkey.ai/docs/changelog/2025/september#pg%26e-immersion-day)
* [Syngenta Devcon 2025](https://portkey.ai/docs/changelog/2025/september#syngenta-devcon-2025)
* [MCP Salon](https://portkey.ai/docs/changelog/2025/september#mcp-salon)
* [Resources](https://portkey.ai/docs/changelog/2025/september#resources)
* [Community Contributors](https://portkey.ai/docs/changelog/2025/september#community-contributors)
* [Coming this month!](https://portkey.ai/docs/changelog/2025/september#coming-this-month)
* [Support](https://portkey.ai/docs/changelog/2025/september#support)
This was an eventful month at Portkey. We hosted and joined incredible events, from community gatherings like **LLMs in Prod in San Francisco**, to enterprise conversations with **Palo Alto Networks and PG&E**. A highlight was seeing **Syngenta’s teams use Portkey** in their Devcon 2025 hackathon, experimenting with new AI workflows. Each of these moments reinforced how fast enterprise AI is evolving, and how important it is to build responsibly at scale. Alongside the events, we shipped some of our most anticipated product updates yet, including the **MCP Gateway** and a series of new guardrails, routing improvements, and provider integrations, all designed to make it easier for teams to run AI in production with security, governance, and flexibility built in.
[](https://portkey.ai/docs/changelog/2025/september#summary)
Summary
------------------------------------------------------------------------
| Area | Key Highlights |
| --- | --- |
| **Platform** | • MCP Gateway beta
• Unified token counting endpoint |
| **Guardrails** | • Metadata-based model access guardrail
• Regex Replace Guardrail |
| **Gateway & Providers** | • Unified finish\_reason
• Conditional routing enhancement
• Provider updates and improvements |
| **Community & Events** | • Palo Alto Networks webinar
• LLMs in Prod, San Francisco
• PG&E Immersion Day
• Syngenta Devcon 2025
• MCP Salon |
* * *
[](https://portkey.ai/docs/changelog/2025/september#platform)
Platform
--------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/september#mcp-gateway-beta)
MCP Gateway (beta)
Enterprises can now bring MCP servers and tools into production with governance, observability, and access controls built-in. The Gateway helps teams avoid authentication sprawl, shadow tool usage, and fragmented monitoring—giving enterprises a central way to manage how agents interact with external tools.
Key benefits include:
* Centralized MCP tool access
* Unified authentication and credentials handling
* Complete visibility into agent-tool interactions
* Access controls and usage limits by team
👉 To try out the beta, [send us an email](https://portkey.ai/docs/changelog/2025/support@portkey.ai)
or [book a demo](https://portkey.sh/sept-email)
.
###
[](https://portkey.ai/docs/changelog/2025/september#unified-token-counting-endpoint)
Unified token counting endpoint
Token counting is now unified across AWS Bedrock, Vertex AI, and Anthropic. With a single endpoint, you can:
* Estimate token usage before sending a request (avoid context limit errors)
* Standardize cost/usage logic across providers
* Enforce routing and quota rules directly inside your app
[Read more here](https://portkey.ai/docs/integrations/llms/anthropic/count-tokens)
 This goes beyond dashboard reporting so developers can make smarter runtime decisions with accurate, provider-agnostic token counts.
[](https://portkey.ai/docs/changelog/2025/september#guardrails)
Guardrails
------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/september#metadata-based-model-access-guardrail)
Metadata-based Model Access guardrail
 We introduced a new guardrail that lets you restrict model access based on metadata key-value pairs at runtime. By evaluating metadata dynamically on every request, this guardrail provides granular, per-request governance—without requiring changes to your app logic.
###
[](https://portkey.ai/docs/changelog/2025/september#regex-replace-guardrail)
Regex-replace guardrail
 Our PII guardrail already covers common fields like emails, phone numbers, and credit cards. But many customers asked for a way to redact custom org-specific patterns. With the Regex Replace Guardrail, you can now:
* Define your own regex patterns
* Replace matches with a chosen string (e.g., \[masked\_user\])
* Enforce masking rules at runtime, before data reaches the model
This is particularly useful for internal IDs, employee codes, or project references that shouldn’t leave your environment. [Read more here](https://portkey.ai/docs/integrations/guardrails/regex)
.
[](https://portkey.ai/docs/changelog/2025/september#gateway-and-providers)
Gateway and Providers
----------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/september#unified-finish-reason-parameter)
Unified `finish_reason` parameter
We standardized the `finish_reason` field across all providers. By default, values are mapped to OpenAI-compatible outputs, ensuring consistent handling across multi-provider deployments. If you prefer to keep the original provider-returned value, set `x-portkey-strict-openai-compliance = false`.
###
[](https://portkey.ai/docs/changelog/2025/september#conditional-router-enhancement)
Conditional router enhancement
Conditional routing now supports **parameter-based** routing in addition to metadata. Parameter-based routing enables dynamic, per-request optimizations, giving you better performance, cost efficiency, and control over user experience. [Read more about this here](https://portkey.ai/docs/product/ai-gateway/conditional-routing#structure-of-conditions-object)
###
[](https://portkey.ai/docs/changelog/2025/september#gateway-and-providers-2)
Gateway and Providers
###
[](https://portkey.ai/docs/changelog/2025/september#new-models)
New Models
Claude Sonnet 4.5
-----------------
Anthropic’s latest model, now available via Portkey
GPT-5 Codex
-----------
OpenAI’s specialized coding model, supported with full observability
###
[](https://portkey.ai/docs/changelog/2025/september#new-providers)
New Providers
Meshy
-----
Specialized 3D generation and design workflows provider
Tripo3D
-------
Next-generation 3D modeling and visualization
Cerebras
--------
High-performance AI inference provider offering up to 70x faster speeds than GPU-based solutions
Nextbit256
----------
New provider focused on efficient inference for specialized workloads
###
[](https://portkey.ai/docs/changelog/2025/september#improvements-and-fixes)
Improvements and Fixes
* **DashScope** → Updated supported parameters
* **Vertex AI**:
* Added `timeRangeFilter` support for Google Search tool
* Added support for Mistral models
* Added support for `task_type` and `dimensions` parameters in Vertex AI batch embeddings
* Handle empty responses returned by the provider
* Support for `global` region
* **Fireworks** → Better handling of non-ASCII characters in file uploads, plus removed unnecessary response transforms for faster performance
* **OpenAI & Azure OpenAI**:
* Added new parameters for GPT-5 compatibility
* Updated the tokenizer to support streaming request token calculation for latest gpt-5 models
* **AWS Bedrock**:
* Added `video` support in chat completions
* Added support for `APAC` cross region inference profiles
* Added support for `performance_config` parameter which will be passed as-is to the provider as `performanceConfig` parameter
* Support Inference Profiles when uploading files for batches & finetuning
* KMS Support for file uploads to AWS
* **Azure Foundry and Github** → Updated the parameter mapping to support all the latest OpenAI compatible chat completions parameters
* **Adding new models to Azure OpenAI** → Now much simpler. You just need to add the target URI, and the model will be fetched and added directly to your integration.
* **Azure Foundry** → You can now enable multiple models in a single Azure Foundry integration. This simplifies management and makes it easier to experiment with different models under the same integration, without repetitive setup.
* Support custom scope for entra auth to use with deprecated azure serverless models
* Custom Header support for OTEL Export of analytics data
🚨 **Vertex AI deprecation** → Llama 3.1 + 3.2 models will be retired on Jan 15, 2026. Recommended migration: Llama 3.3 or Llama 4.
[](https://portkey.ai/docs/changelog/2025/september#community-%26-events)
Community & Events
------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/september#protecting-your-ai-platform-with-palo-alto-networks-and-portkey)
Protecting your AI platform with Palo Alto Networks and Portkey
We partnered with Palo Alto Networks for a joint webinar on securing enterprise AI platforms with guardrails + gateway. The session highlighted the most pressing AI security risks (prompt injections, data leakage, compliance gaps) and how combining a gateway with guardrails can address them without slowing down scale.
###
[](https://portkey.ai/docs/changelog/2025/september#llms-in-prod%2C-san-francisco)
LLMs in Prod, San Francisco
###
[](https://portkey.ai/docs/changelog/2025/september#pg%26e-immersion-day)
PG&E Immersion Day
We had the privilege of joining Pacfic Gas & Electricity for their internal Immersion Day. It was a hands-on session with their teams, exploring how enterprises can adopt AI responsibly and scale across critical operations.
###
[](https://portkey.ai/docs/changelog/2025/september#syngenta-devcon-2025)
Syngenta Devcon 2025
Our customer Syngenta hosted Devcon 2025, centered on AI.  [Syngenta’s](https://www.linkedin.com/posts/miragirahul_devcon2025-syngenta-hackathon-activity-7377593161826603008-X_bn)
teams ran a hackathon using Portkey + n8n, building creative workflows and experimenting with how MCP servers and digital apps can transform grower experiences. It was inspiring to see Portkey embedded directly into their innovation process, powering hands-on experimentation and ideation at scale.
###
[](https://portkey.ai/docs/changelog/2025/september#mcp-salon)
MCP Salon
We hosted some of the most illustrious MCP builders for a closed-door roundtable. The group went deep into the technical challenges of building MCP servers and clients, serving them in production, and solving real-world adoption hurdles. 👉 To stay updated on upcoming events, subscribe to our [event calendar](https://luma.com/portkey?k=c)
[](https://portkey.ai/docs/changelog/2025/september#resources)
Resources
----------------------------------------------------------------------------
* **Blog**: [MCP Message Types: Complete MCP JSON-RPC Reference Guide](https://portkey.ai/blog/mcp-message-types-complete-json-rpc-reference-guide)
* **Blog**: [Failover routing strategies for LLMs in production](https://portkey.ai/blog/failover-routing-strategies-for-llms-in-production)
* **Partnership Blog with Feedback Intelligence**: [Tracing Failures from the LLM Call to the User Experience](https://portkey.ai/blog/tracing-failures-from-the-llm-call-to-the-user-experience)
* **Blog**: [A Strategic Perspective on the MCP Registry for the Enterprise](https://portkey.ai/blog/mcp-registry)
[](https://portkey.ai/docs/changelog/2025/september#community-contributors)
Community Contributors
------------------------------------------------------------------------------------------------------
A special thanks to our contributor this month:
* [MarcNB256](https://github.com/MarcNB256)
[](https://portkey.ai/docs/changelog/2025/september#coming-this-month)
Coming this month!
---------------------------------------------------------------------------------------------
Webinar - **LibreChat in Production** [Register here →](https://luma.com/cywhfpko)
[](https://portkey.ai/docs/changelog/2025/september#support)
Support
------------------------------------------------------------------------
[Need Help?\
----------\
\
Open an issue on GitHub](https://github.com/Portkey-AI/gateway/issues)
[Join Us\
-------\
\
Get support in our Discord](https://portkey.wiki/community)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/changelog/2025/september.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/changelog/2025/september)
[August](https://portkey.ai/docs/changelog/2025/august)
⌘I
---
# Errors - Portkey Docs
[Skip to main content](https://portkey.ai/docs/api-reference/admin-api/error#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
API Reference
Errors
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Admin API Error Codes](https://portkey.ai/docs/api-reference/admin-api/error#admin-api-error-codes)
* [Error Reference](https://portkey.ai/docs/api-reference/admin-api/error#error-reference)
* [Notes on Common Errors](https://portkey.ai/docs/api-reference/admin-api/error#notes-on-common-errors)
[](https://portkey.ai/docs/api-reference/admin-api/error#admin-api-error-codes)
Admin API Error Codes
=========================================================================================================
Below is a list of error codes returned by the Portkey Admin API. These help with debugging failed requests and ensuring proper authentication, permissions, and request formatting.
[](https://portkey.ai/docs/api-reference/admin-api/error#error-reference)
Error Reference
---------------------------------------------------------------------------------------------
| Error Code | HTTP Status | Message | Type |
| --- | --- | --- | --- |
| AB01 | 400 | Request Validation Error | Client Error |
| AB02 | 404 | Request Validation Error | Client Error |
| AB03 | 403 | User not allowed to access the resource | Client Error |
| AB04 | 500 | Internal Server Error | Server Error |
| AB05 | 401 | Unauthorized access | Client Error |
| AB06 | 429 | Rate limit exceeded | Client Error |
| AB07 | 409 | Resource already exists | Client Error |
| AB08 | 404 | Resource not found | Client Error |
| AB09 | 402 | Subscription exhausted | Client Error |
* * *
[](https://portkey.ai/docs/api-reference/admin-api/error#notes-on-common-errors)
Notes on Common Errors
-----------------------------------------------------------------------------------------------------------
**AB01 – Request Validation Error (400)**This error usually happens when:
* You’re either missing required parameters
* using incorrect data types (e.g., sending a number instead of a string)
* passing values that are not within the allowed set (enum violations).
**AB05 – Unauthorized Access (401)**This indicates that the user is not authorized to access the resource, often due to incorrect API key permissions.**Common Cause**: Your API key does **not have the right permissions** for this request.**Fix**: Go to [app.portkey.ai](https://app.portkey.ai/)
→ **API Keys**, and check the permissioning for your key. Ensure it includes the required scopes for the endpoint you’re calling.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/api-reference/admin-api/error.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/api-reference/admin-api/error)
[Introduction](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Create Config](https://portkey.ai/docs/api-reference/admin-api/control-plane/configs/create-config)
⌘I
---
# August - Portkey Docs
[Skip to main content](https://portkey.ai/docs/changelog/2025/august#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
2025
August
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Summary](https://portkey.ai/docs/changelog/2025/august#summary)
* [Highlights](https://portkey.ai/docs/changelog/2025/august#highlights)
* [Making GenAI rollout easy with Ivy League institutions and Internet2](https://portkey.ai/docs/changelog/2025/august#making-genai-rollout-easy-with-ivy-league-institutions-and-internet2)
* [Protecting Your AI Platform - Palo Alto Networks x Portkey](https://portkey.ai/docs/changelog/2025/august#protecting-your-ai-platform-palo-alto-networks-x-portkey)
* [Platform](https://portkey.ai/docs/changelog/2025/august#platform)
* [List models API](https://portkey.ai/docs/changelog/2025/august#list-models-api)
* [Unified + Portkey Batching](https://portkey.ai/docs/changelog/2025/august#unified-%2B-portkey-batching)
* [OTEL-based autoinstrumentation](https://portkey.ai/docs/changelog/2025/august#otel-based-autoinstrumentation)
* [Multiple guardrails at the workspace level](https://portkey.ai/docs/changelog/2025/august#multiple-guardrails-at-the-workspace-level)
* [CRUD Guardrails API](https://portkey.ai/docs/changelog/2025/august#crud-guardrails-api)
* [New Models and Providers](https://portkey.ai/docs/changelog/2025/august#new-models-and-providers)
* [🌐 Community Highlights](https://portkey.ai/docs/changelog/2025/august#%F0%9F%8C%90-community-highlights)
* [Future of AI Platforms](https://portkey.ai/docs/changelog/2025/august#future-of-ai-platforms)
* [Enterprise Security with Falco Vanguard](https://portkey.ai/docs/changelog/2025/august#enterprise-security-with-falco-vanguard)
* [Resources](https://portkey.ai/docs/changelog/2025/august#resources)
* [Community Contributors](https://portkey.ai/docs/changelog/2025/august#community-contributors)
* [Coming this month!](https://portkey.ai/docs/changelog/2025/august#coming-this-month)
* [Support](https://portkey.ai/docs/changelog/2025/august#support)
August was all about real-world impact and community. Princeton and NYU shared how they’re deploying Portkey across their institutions. Seeing our platform in action at these universities was incredible. **Future of AI Platforms** brought together industry leaders from Doordash, Postman, Qure.ai and more for honest conversations about challenges and opportunities. Meanwhile, we shipped what you asked for: multiple workspace guardrails, OTEL instrumentation, and day-zero support for the latest models. Here’s what’s new this month:
[](https://portkey.ai/docs/changelog/2025/august#summary)
Summary
---------------------------------------------------------------------
| Area | Key Highlights |
| --- | --- |
| **Highlights** | • Making GenAI rollout easy with Princeton, NYU and Internet2
• Upcoming Palo Alto Networks webinar
• New `/models` endpoint |
| **Platform** | • Unified + Portkey batching
• OTEL-based auto-instrumentation
• Multiple guardrails at workspace level
• CRUD Guardrails API |
| **Gateway & Providers** | • Featherless.ai integration
• GPT-OSS models support
• Claude Opus 4.1 support
• Custom models and pricing |
[](https://portkey.ai/docs/changelog/2025/august#highlights)
Highlights
---------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/august#making-genai-rollout-easy-with-ivy-league-institutions-and-internet2)
Making GenAI rollout easy with Ivy League institutions and Internet2
Together with Internet2, we co-hosted a webinar where Ivy League institutions like NYU\*\* shared how universities are deploying GenAI across classrooms, research, and student services. The session highlighted how Portkey’s AI Gateway provides secure model access, guardrails, and compliance support for educational institutions. Watch the recording [here →](https://internet2.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=36f87257-f9f5-4a18-b2db-b333011b3437&start=561)
.
###
[](https://portkey.ai/docs/changelog/2025/august#protecting-your-ai-platform-palo-alto-networks-x-portkey)
Protecting Your AI Platform - Palo Alto Networks x Portkey
.png?w=2500&fit=max&auto=format&n=SnsFU5cdk_U0OW7n&q=85&s=8180a8fff4f2115b581a60efe0c252b1) Join us for an upcoming webinar in partnership with Palo Alto Networks where we’ll explore best practices for securing AI infrastructure at scale, implementing enterprise-grade guardrails, and maintaining compliance while enabling innovation. Click here to [register](https://luma.com/z84zjko5)
.
[](https://portkey.ai/docs/changelog/2025/august#platform)
Platform
-----------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/august#list-models-api)
List models API
 You can now see a list of all models available through Portkey—along with basic details for each one. This makes it easier to discover which models you can use and compare providers and options. Read more about it [here](https://portkey.ai/docs/api-reference/inference-api/models/models)
.
###
[](https://portkey.ai/docs/changelog/2025/august#unified-%2B-portkey-batching)
Unified + Portkey Batching
 Optimize throughput and reduce costs with our improved batching capabilities. We’ve implemented unified batching across all supported providers with intelligent request grouping for optimized performance. This reduces per-request overhead while providing automatic batch size optimization and configurable parameters to suit your specific needs. See how you can get started [here](http://localhost:3000/product/ai-gateway/batches)
.
###
[](https://portkey.ai/docs/changelog/2025/august#otel-based-autoinstrumentation)
**OTEL-based autoinstrumentation**
Portkey now works as an OTel endpoint. Send telemetry from any OTel-compatible source into Portkey and view it alongside LLM call logs. End-to-end observability for performance, cost, and compliance. Super powerful for agent workflows. See how you can implement this [here](https://portkey.ai/docs/product/observability/opentelemetry)
This strengthens our observability even further! 
###
[](https://portkey.ai/docs/changelog/2025/august#multiple-guardrails-at-the-workspace-level)
**Multiple guardrails at the workspace level**
 You can now define and enforce multiple guardrail policies at the workspace-level, making it easier to secure usage, filter content, and maintain compliance across your team.
###
[](https://portkey.ai/docs/changelog/2025/august#crud-guardrails-api)
CRUD Guardrails API
Managing guardrails at scale just got easier. You can now create, update, retrieve, delete, and list guardrails directly via API. This makes it simple to keep policies consistent across environments, automate changes, and integrate guardrail management into your CI/CD workflows. [Learn more](https://portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/create-guardrail)
.
[](https://portkey.ai/docs/changelog/2025/august#new-models-and-providers)
New Models and Providers
-------------------------------------------------------------------------------------------------------
* **Featherless.ai**: Access 11,900+ open-source models with unlimited tokens through a single gateway. [Learn more →](https://portkey.ai/docs/integrations/llms/featherless)
* **GPT-OSS Models**: OpenAI’s new open-weight models with strong reasoning and tool use capabilities.
* **Claude Opus 4.1**: Anthropic’s latest flagship model with 74.5% performance on SWE-bench Verified.
[](https://portkey.ai/docs/changelog/2025/august#%F0%9F%8C%90-community-highlights)
🌐 Community Highlights
---------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/changelog/2025/august#future-of-ai-platforms)
**Future of AI Platforms**
Leaders from **DoorDash, Postman, Qure.ai, and Qoala** joined us for the Future of AI Platforms. It turned into a night of sharp ideas, candid discussions, and bold visions for where AI is headed.  Where should we host it next? 👉 [Send us an email](https://portkey.ai/cdn-cgi/l/email-protection#cdbbbfb8bea5aca3a6e3bb8dbda2bfb9a6a8b4e3aca4)
if you’d be interested in attending!
###
[](https://portkey.ai/docs/changelog/2025/august#enterprise-security-with-falco-vanguard)
Enterprise Security with Falco Vanguard
 [Falco Vanguard](https://www.linkedin.com/posts/migueladelossantos_falco-cloudnativesecurity-aiengineering-activity-7358991765049122816-YHDS)
is using Portkey’s AI infrastructure to build an innovative security platform that intelligently clusters security events and prioritizes critical threats. Their offline-first approach ensures data sovereignty while their integration with Portkey provides the reliability and production readiness enterprises demand. We’re excited to see how this collaboration transforms security operations for organizations. Read more about this [here →](https://www.linkedin.com/posts/migueladelossantos_falco-cloudnativesecurity-aiengineering-activity-7358991765049122816-YHDS)
.
[](https://portkey.ai/docs/changelog/2025/august#resources)
Resources
-------------------------------------------------------------------------
* Blog: [Simplifying LLM batch reference](https://portkey.ai/blog/simplifying-llm-batch-inference)
* Blog: [OTel traces with LLM logs for end-to-end observability agent workflows](https://portkey.ai/blog/otel-with-llm-observability-for-agents)
[](https://portkey.ai/docs/changelog/2025/august#community-contributors)
Community Contributors
---------------------------------------------------------------------------------------------------
A special thanks to our contributor this month:
* [pnkvalavala](https://github.com/pnkvalavala)
* [indranil-kar-cloudesign](https://github.com/indranil-kar-cloudesign)
* [horochx](https://github.com/horochx)
[](https://portkey.ai/docs/changelog/2025/august#coming-this-month)
Coming this month!
------------------------------------------------------------------------------------------
Our MCP gateway is almost here! Reach out to us on [\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#53202623233c212713233c212738362a7d323a)
for early access!
[](https://portkey.ai/docs/changelog/2025/august#support)
Support
---------------------------------------------------------------------
[Need Help?\
----------\
\
Open an issue on GitHub](https://github.com/Portkey-AI/gateway/issues)
[Join Us\
-------\
\
Get support in our Discord](https://portkey.wiki/community)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/changelog/2025/august.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/changelog/2025/august)
[September](https://portkey.ai/docs/changelog/2025/september)
[July](https://portkey.ai/docs/changelog/2025/july)
⌘I
---
# July - Portkey Docs
[Skip to main content](https://portkey.ai/docs/changelog/2025/july#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
2025
July
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Summary](https://portkey.ai/docs/changelog/2025/july#summary)
* [Internet2 🤝 Portkey](https://portkey.ai/docs/changelog/2025/july#internet2-%F0%9F%A4%9D-portkey)
* [Platform](https://portkey.ai/docs/changelog/2025/july#platform)
* [Guardrails](https://portkey.ai/docs/changelog/2025/july#guardrails)
* [Gateway](https://portkey.ai/docs/changelog/2025/july#gateway)
* [New models and providers](https://portkey.ai/docs/changelog/2025/july#new-models-and-providers)
* [Integrations](https://portkey.ai/docs/changelog/2025/july#integrations)
* [Resources](https://portkey.ai/docs/changelog/2025/july#resources)
* [Community Contributors](https://portkey.ai/docs/changelog/2025/july#community-contributors)
* [Coming this month!](https://portkey.ai/docs/changelog/2025/july#coming-this-month)
* [Support](https://portkey.ai/docs/changelog/2025/july#support)
It’s been a landmark month. Portkey is now the **✨official AI Gateway provider for Internet2 NET+✨**, the gold standard for infrastructure across 300+ top U.S. universities including NYU, Harvard, Princeton, Cornell, and Berkeley. We’re now listed alongside AWS, Microsoft, and Google and we’re the youngest infra provider ever to make the cut! Alongside this, we shipped major updates across the stack: expanded data residency support, enhanced logging and cost tracking for Claude Code, new providers and models, and more. Let’s get into it!
[](https://portkey.ai/docs/changelog/2025/july#summary)
Summary
-------------------------------------------------------------------
| Area | Key Highlights |
| --- | --- |
| **Platform** | • Regional data residency controls
• Privacy mode
• Automatic user attribution for user keys
• Request transformations (original + final)
• Log exports via UI, API, stream, webhook |
| **Gateway & Providers** | • New models: Grok-4, Qwen3-Coder, Kimi K2
• New providers: Krutrim, Agno AI
• /v1/messages (beta): tool use, multi-turn, redacted thoughts
• Custom models & pricing |
| **Guardrails** | • Palo Alto Networks AIRS plugin for real-time security guardrails
• CRUD guardrails endpoint |
| **Integrations** | • LangSmith integration
• AWS Bedrock Knowledge Bases
• Langflow compatibility
• Enhanced Claude Code logging & cost tracking |
[](https://portkey.ai/docs/changelog/2025/july#internet2-%F0%9F%A4%9D-portkey)
Internet2 🤝 Portkey
-------------------------------------------------------------------------------------------------------
Many universities face similar challenges when adopting GenAI: fragmented provider access, budgetary uncertainty, and regulatory pressure. Portkey helps address these by sitting between institutional systems and model providers by offering a single API with granular usage controls, analytics, and guardrails. “The Internet2 NET+ Service Evaluation process for Portkey has been a demonstration of collaboration and efficiency,” said **Stratos Efstathiadis, Senior Director of Research Technology Services at NYU**. “The synergy among peer institutions ensures Portkey meets the diverse AI requirements of higher education.” **📅 Join the NET+ Portkey Launch Webinar**
NYU and Princeton leaders join us this Thursday, AUG 7 AT 1 P.M. ET to go behind the scenes of how GenAI is being deployed on campus. [Register here](https://events.internet2.edu/ereg/index.php?eventid=850237)
[](https://portkey.ai/docs/changelog/2025/july#platform)
Platform
---------------------------------------------------------------------
**Regional data residency** All your LLM data can now be hosted in any region of your choice with strict isolation, no cross-border data flow, and full compliance with residency regulations.
* Meet institutional and regional privacy mandates
* Improve latency by routing traffic through nearby zones
**Log exports** You can now export logs directly from the Portkey UI or through the API. Whether you’re debugging, auditing, or building internal dashboards, you can export and access complete logs in a structured, reliable format. **Transformed requests in logs**
Debugging just got simpler. You can now view the original request as received by Portkey, the transformed version sent to the model provider, and same for the response in logs, making it easier for debugging. **Privacy mode (logging off)** Portkey now gives org owners granular control over what gets logged, from full request/response payloads to just minimal metadata.
This can be configured org-wide or per workspace, offering flexibility for sensitive or regulated use cases.[Configure it for your org](https://portkey.ai/docs/product/administration/configuring-request-logging)
. **Automatic user attribution on API keys** Every request made using a user API key now automatically carries `_user` metadata, with an option to override.
[](https://portkey.ai/docs/changelog/2025/july#guardrails)
Guardrails
-------------------------------------------------------------------------
**Palo Alto Networks’ AIRS Plugin**
Portkey now integrates with PANW AIRS (AI Runtime Security) to enforce guardrails that block risky prompts or model responses based on real-time security analysis. [Learn more](https://portkey.ai/docs/integrations/guardrails/palo-alto-panw-prisma)
**CRUD Guardrails endpoint** We’ve added full support for managing guardrails via API — create, update, delete, and list guardrail rules and templates programmatically. [Learn more](https://portkey.ai/docs/api-reference/admin-api/control-plane/guardrails/create-guardrail)
[](https://portkey.ai/docs/changelog/2025/july#gateway)
Gateway
-------------------------------------------------------------------
**Support for messages route(beta)** In addition to `chat.completions`, Portkey now supports the `messages` route (beta) for Anthropic, AWS Bedrock, and Vertex AI. This includes tool calling, thinking, multi-turn conversations, and redacted thinking support
[](https://portkey.ai/docs/changelog/2025/july#new-models-and-providers)
New models and providers
-----------------------------------------------------------------------------------------------------
* **Krutrim**: LLM provider with regional language support
* **Grok 4**: The latest release from xAI’s Grok family, now supported on Portkey.
* **Qwen3-Coder**: Alibaba’s open-weight code model, available for direct use.
* **Kimi K2**: Moonshot’s new long-context model is now in production.
[](https://portkey.ai/docs/changelog/2025/july#integrations)
Integrations
-----------------------------------------------------------------------------
[Claude Code\
-----------\
\
Get cost tracking and usage breakdowns across teams for Claude Code generations — along with full request logs and output analysis to manage spend and performance.](https://portkey.ai/docs/integrations/libraries/claude-code)
[AWS Bedrock Knowledge Bases\
---------------------------\
\
You can now connect to Bedrock Knowledge Bases directly through Portkey, with unified auth, caching, and logging across your RAG workflows.](https://portkey.ai/docs/integrations/llms/bedrock/bedrock-knowledgebase)
[Agno AI\
-------\
\
Agno AI – Agno is a powerful framework for building autonomous agents, now supercharged with Portkey’s production-grade observability, governance, and tooling.](https://portkey.ai/docs/integrations/agents/agno-ai)
[LangSmith\
---------\
\
You can now export traces to LangSmith alongside Portkey’s built-in observability, for teams already using it for debugging or evals.](https://portkey.ai/docs/integrations/tracing-providers/langsmith)
[Langflow\
--------\
\
Portkey now works natively with Langflow. Route all your workflows through Portkey to apply guardrails, monitor requests, and manage access without leaving your visual interface.](https://portkey.ai/docs/integrations/libraries/langflow)
**Portkey + your fav AI tool = 🪄✨**
[](https://portkey.ai/docs/changelog/2025/july#resources)
Resources
-----------------------------------------------------------------------
* Cookbook: [Arize + Portkey: Multi-LLM debate with traces and evals](https://portkey.ai/docs/guides/integrations/arize-portkey)
* Blog: [How to add enterprise controls to OpenWebUI](https://portkey.ai/blog/how-to-add-enterprise-controls-to-openwebui)
* Blog: [Everything We Know About Claude Code Limits](https://portkey.ai/blog/claude-code-limits)
[](https://portkey.ai/docs/changelog/2025/july#community-contributors)
Community Contributors
-------------------------------------------------------------------------------------------------
A special thanks to our contributor this month:
* [AG2AI-Admin](https://github.com/AG2AI-Admin)
* [Mishalabdullah](https://github.com/Mishalabdullah)
[](https://portkey.ai/docs/changelog/2025/july#coming-this-month)
Coming this month!
----------------------------------------------------------------------------------------
MCP Connectors on the gateway! Reach out to us on [\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#a2d1d7d2d2cdd0d6e2d2cdd0d6c9c7db8cc3cb)
for a sneakpeek!
[](https://portkey.ai/docs/changelog/2025/july#support)
Support
-------------------------------------------------------------------
[Need Help?\
----------\
\
Open an issue on GitHub](https://github.com/Portkey-AI/gateway/issues)
[Join Us\
-------\
\
Get support in our Discord](https://portkey.wiki/community)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/changelog/2025/july.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/changelog/2025/july)
[August](https://portkey.ai/docs/changelog/2025/august)
[June](https://portkey.ai/docs/changelog/2025/june)
⌘I
---
# Supported Providers - Portkey Docs
[Skip to main content](https://portkey.ai/docs/api-reference/inference-api/supported-providers#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
API Reference
Supported Providers
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
| Provider | Chat | Vision | Tools | Portkey Prompts | Embeddings | Images | Audio | Finetuning | Batch | Files | Moderations | Assistants | Completions | Messages |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AI21 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | | |
| Anthropic | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | | ✅ |
| Anyscale | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | | |
| Azure OpenAI | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | |
| AWS Bedrock | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
| Cohere | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | |
| Deepinfra | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Fireworks AI | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | |
| Google Vertex AI | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| Google Gemini | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | |
| Groq | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | |
| Jina | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Lingyi (01.ai) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Mistral AI | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| Monster API | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Moonshot.cn | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Nomic AI | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Novita AI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | |
| Ollama | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Openrouter | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Perplexity AI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| Predibase | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Reka AI | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Segmind | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Stability AI | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Together AI | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | |
| Cloudflare Workers AI | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | |
| Zhipu AI | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| AWS Sagemaker | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Azure Foundry | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| BYOLLM | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Cerebras | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Dashscope | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| DeepBricks | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| DeepSeek | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Deepgram | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Google PaLM | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Huggingface | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Inference.net | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Lambda | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Lemon Fox | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Lepton | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Local AI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Nebius | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| NCompass | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| NScale | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Recraft AI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Replicate | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Sambanova | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| SiliconFlow | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Snowflake Cortex | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Triton | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Upstage | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| vLLM | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| Voyage AI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| xAI | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
While you’re here, why not [give us a star](https://git.new/ai-gateway-docs)
? It helps us a lot!
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/api-reference/inference-api/supported-providers.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/api-reference/inference-api/supported-providers)
[Introduction](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Authentication](https://portkey.ai/docs/api-reference/inference-api/authentication)
⌘I
---
# Prompts - Portkey Docs
[Skip to main content](https://portkey.ai/docs/guides/prompts#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
[\
\
Ultimate AI SDR\
---------------\
\
Leveraging Claude 3.7, Perplexity Sonar Pro, and o3-mini to build the ultimate AI SDR that researches the internet, writes outstanding copy, and self-evaluates its effectiveness.](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr)
[\
\
LLM as a Judge\
--------------\
\
LLM-as-a-Judge system that evaluates AI agent responses based on predefined quality standards, providing structured feedback at scale.](https://portkey.ai/docs/guides/prompts/llm-as-a-judge)
[\
\
Build a Chatbot using Prompt Templates\
--------------------------------------\
\
Leverage Portkey’s Prompt Template to build a Chatbot](https://portkey.ai/docs/guides/prompts/build-a-chatbot-using-portkeys-prompt-templates)
[\
\
LLama Prompt Ops Integration\
----------------------------\
\
This guide shows you how to combine Llama Prompt Ops with Portkey to optimize prompts for Llama models](https://portkey.ai/docs/guides/prompts/llama-prompts)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/guides/prompts.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/guides/prompts)
[Building an LLM-as-a-Judge System for AI (Customer Support) Agent](https://portkey.ai/docs/guides/prompts/llm-as-a-judge)
[Ultimate AI SDR](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr)
⌘I
---
# Authentication - Portkey Docs
[Skip to main content](https://portkey.ai/docs/api-reference/inference-api/authentication#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
API Reference
Authentication
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Obtaining Your API Key](https://portkey.ai/docs/api-reference/inference-api/authentication#obtaining-your-api-key)
* [Authentication with SDKs](https://portkey.ai/docs/api-reference/inference-api/authentication#authentication-with-sdks)
* [Portkey SDKs](https://portkey.ai/docs/api-reference/inference-api/authentication#portkey-sdks)
* [OpenAI SDK](https://portkey.ai/docs/api-reference/inference-api/authentication#openai-sdk)
* [JWT-based Authentication](https://portkey.ai/docs/api-reference/inference-api/authentication#jwt-based-authentication)
To ensure secure access to Portkey’s APIs, authentication is required for all requests. This guide provides the necessary steps to authenticate your requests using the Portkey API key, regardless of whether you are using the SDKs for Python and JavaScript, the OpenAI SDK, or making REST API calls directly.
[](https://portkey.ai/docs/api-reference/inference-api/authentication#obtaining-your-api-key)
Obtaining Your API Key
------------------------------------------------------------------------------------------------------------------------
[Create](https://app.portkey.ai/signup)
or [log in](https://app.portkey.ai/login)
to your Portkey account. Grab your account’s API key from the “Settings” page.
Based on your access level, you might see the relevant permissions on the API key modal - tick the ones you’d like, name your API key, and save it.[JWT-based Authentication\
------------------------\
\
You can also authenticate Portkey using JWT Tokens. Learn more here](https://portkey.ai/docs/api-reference/inference-api/authentication#jwt-based-authentication)
[](https://portkey.ai/docs/api-reference/inference-api/authentication#authentication-with-sdks)
Authentication with SDKs
----------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/api-reference/inference-api/authentication#portkey-sdks)
Portkey SDKs
* NodeJS SDK
* Python SDK
* cURL
Copy
Ask AI
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // Replace with your actual API key
virtualKey: "VIRTUAL_KEY" // Optional: Use for virtual key management
})
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'gpt-4o',
});
console.log(chatCompletion.choices);
###
[](https://portkey.ai/docs/api-reference/inference-api/authentication#openai-sdk)
OpenAI SDK
When integrating Portkey through the OpenAI SDK, modify the base URL and add the `x-portkey-api-key` header for authentication. Here’s an example of how to do it:
We use the `createHeaders` helper function from the Portkey SDK here to easily create Portkey headers.You can pass the raw headers (`x-portkey-api-key`, `x-portkey-provider`) directly in the `defaultHeaders` param as well.
* NodeJS
* Python
Copy
Ask AI
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'
const openai = new OpenAI({
apiKey: 'OPENAI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: "openai",
apiKey: "PORTKEY_API_KEY"
})
});
async function main() {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'gpt-4o',
});
console.log(chatCompletion.choices);
}
main();
Read more [here](https://portkey.ai/docs/integrations/llms/openai)
.
[](https://portkey.ai/docs/api-reference/inference-api/authentication#jwt-based-authentication)
JWT-based Authentication
----------------------------------------------------------------------------------------------------------------------------
Portkey supports JWT-based authentication as a secure alternative to API Key authentication. With JWT authentication, clients can authenticate API requests using a JWT token that is validated against a configured JWKS (JSON Web Key Set). This enterprise-grade authentication method is available as an add-on to any Portkey plan. JWT authentication provides enhanced security through:
* Temporary, expiring tokens
* Fine-grained permission scopes
* User identity tracking
* Centralized authentication management
[JWT Token Authentication\
------------------------\
\
Learn how to implement JWT-based authentication with Portkey](https://portkey.ai/docs/product/enterprise-offering/org-management/jwt)
**Interested in adding JWT authentication to your Portkey plan?**[Contact our sales team](https://portkey.sh/jwt)
to discuss pricing and implementation details.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/api-reference/inference-api/authentication.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/api-reference/inference-api/authentication)
[Supported Providers](https://portkey.ai/docs/api-reference/inference-api/supported-providers)
[Headers](https://portkey.ai/docs/api-reference/inference-api/headers)
⌘I
---
# Building an LLM-as-a-Judge System for AI (Customer Support) Agent - Portkey Docs
[Skip to main content](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Evals
Building an LLM-as-a-Judge System for AI (Customer Support) Agent
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Introduction](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#introduction)
* [What We’re Building](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#what-we%E2%80%99re-building)
* [System Architecture: How It Works](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#system-architecture%3A-how-it-works)
* [Working with Portkey’s Prompt Studio](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#working-with-portkey%E2%80%99s-prompt-studio)
* [Step-by-Step Guide to Building LLM-as-a-Judge](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-by-step-guide-to-building-llm-as-a-judge)
* [Step 1: Define Your Company Information in a Partial](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-1%3A-define-your-company-information-in-a-partial)
* [Step 2: Define the Evaluation Guidelines Partial](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-2%3A-define-the-evaluation-guidelines-partial)
* [Step 3: Create Golden Examples Partial](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-3%3A-create-golden-examples-partial)
* [Step 4: Create the Main Judge Prompt Template](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-4%3A-create-the-main-judge-prompt-template)
* [Step 5: Implementing the Evaluation Code with Structured Output](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-5%3A-implementing-the-evaluation-code-with-structured-output)
* [Step 6: Iterate with Domain Experts](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-6%3A-iterate-with-domain-experts)
* [Portkey Observability for Continuous Improvement](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#portkey-observability-for-continuous-improvement)
* [Visualizing Evaluation Results on the Portkey Dashboard](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#visualizing-evaluation-results-on-the-portkey-dashboard)
* [Running Evaluation on Scale](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#running-evaluation-on-scale)
* [Next Steps](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#next-steps)
* [Conclusion](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#conclusion)
> Before reading this guide: We recommend checking out [Hamel Husain’s excellent post on LLM-as-a-Judge](https://hamel.dev/blog/posts/llm-judge/)
> . This cookbook implements the principles discussed in Hamel’s post, providing a practical walkthrough of building LLM-as-a-judge evaluation .
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#introduction)
Introduction
---------------------------------------------------------------------------------------
AI-powered customer support agents are great, but how do you ensure they provide high-quality responses at scale? You need a system that can automatically evaluate customer support interactions by analyzing both the customer’s query and the AI agent’s response. This system should determine whether the response meets quality standards, provide a detailed critique explaining the reasoning behind the judgment, and scale easily to run tests on thousands of interactions. Quality assurance for customer support interactions is critical but increasingly challenging as AI Agents handle more customer conversations. Manual reviews are great but they don’t scale. The “LLM-as-a-Judge” approach offers a powerful solution to this challenge. This guide will show you how to build an automated evaluation system that scales to thousands of interactions. By the end, you’ll have a robust workflow that helps you improve AI agents responses.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#what-we%E2%80%99re-building)
What We’re Building
-------------------------------------------------------------------------------------------------------------
We’ll create an LLM as a judge workflow that evaluates customer support interactions by analyzing both the customer’s query and the AI agent’s response. For each interaction, our system will:
1. Determine whether the response meets quality standards (pass/fail)
2. Provide a detailed critique explaining the reasoning behind the judgment
3. Scale easily to run tests on thousands of interactions
**Use Case Example**: Imagine you’re building a customer support AI agent. Your challenges include:
* Ensuring consistent quality across all AI responses
* Identifying patterns of problematic responses
* Maintaining security and compliance standards
* Quickly detecting when the AI is providing incorrect information
With an LLM-as-a-Judge system, you can:
* Get specific feedback on why responses fail to meet standards
* Identify trends and systematic issues in your support system
* Provide targeted training and improvements based on detailed critiques
* Quickly validate whether changes to your AI agent have improved response quality
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#system-architecture%3A-how-it-works)
System Architecture: How It Works
-----------------------------------------------------------------------------------------------------------------------------------
**Industry Best Practices for AI Agent Evaluation** Before diving into implementation, let’s briefly look at evaluation approaches for customer support AI:
* **Human Evaluation**: The gold standard, but doesn’t scale
* **Offline Benchmarking**: Testing against curated datasets with known answers
* **Online Evaluation**: Monitoring live interactions and collecting user feedback
* **Multi-dimensional Scoring**: Evaluating across different attributes (accuracy, helpfulness, tone)
* **LLM-as-a-Judge**: Using a powerful model to simulate expert human judgment
This cookbook focuses on building a robust LLM-as-a-Judge system that balances accuracy with scalability, allowing you to evaluate thousands of customer interactions automatically.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#working-with-portkey%E2%80%99s-prompt-studio)
Working with [Portkey’s Prompt Studio](https://portkey.ai/docs/product/prompt-engineering-studio)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We will be using [Prompt Studio](https://portkey.ai/docs/product/prompt-engineering-studio)
in this cookbook. Unlike traditional approaches where prompts are written directly in code, Portkey allows you to:
* Create and manage prompts through an intuitive UI
* Version control your prompts
* Access prompts via simple API calls
* Deploy prompts to different environments
We use Mustache templating `{{variable}}` in our prompts, which allows for dynamic content insertion. This makes our prompts more flexible and reusable. **What are Prompt Partials?** Prompt partials are reusable components in Portkey that let you modularize parts of your prompts. Think of them like building blocks that can be combined to create complex prompts. In this guide, we’ll create several partials (company info, guidelines, examples) and then combine them in a main prompt. To follow this guide, you will need to create prompt partials first, then create the main template in the Portkey UI, and finally access them using the prompt\_id inside your codebase.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-by-step-guide-to-building-llm-as-a-judge)
Step-by-Step Guide to Building LLM-as-a-Judge
---------------------------------------------------------------------------------------------------------------------------------------------------------
**The Judge Prompt Structure** To build an effective LLM judge, we need to create a well-structured prompt that gives the model all the context it needs to make accurate evaluations. Our judge prompt will consist of four main components:
* Company Information - Details about your company, products, and support policies that help the judge understand the context of customer interactions
* Evaluation Guidelines - Specific criteria for what makes a good or bad response in your customer support context
* Golden Examples - Sample evaluations that demonstrate how to apply the guidelines to real iteractions
* Main Judge Template - This brings everything together and creates the Judgement System
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-1%3A-define-your-company-information-in-a-partial)
Step 1: Define Your Company Information in a Partial
First, we’ll create a partial that provides context about your company, products, and support policies. This helps the judge evaluate responses in the proper context.
Here’s an example of what your company info partial might look like:
Company Info Prompt Partial
TechConnect Electronics is a consumer electronics retailer founded in 2016 that sells smartphones, computers, audio equipment, and smart home devices through its website and 42 physical stores. The company offers a 30-day return policy on most items (15 days for opened software and select accessories), free shipping on orders over $50, and a 24/7 customer support team available via chat, email, and phone. TechConnect distinguishes itself with its “TechConnect Plus” membership program offering extended warranties and exclusive discounts, as well as its “Tech Support Plus” service providing personalized setup assistance and troubleshooting.
This partial gives the judge important context about your products, return policy, shipping policies, support channels, and special programs. Customize this to match your own company’s specifics. After creating this partial in Portkey, you’ll get a partial ID (e.g., `pl-llm-as-0badba`) that you’ll reference in your main prompt template.
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-2%3A-define-the-evaluation-guidelines-partial)
Step 2: Define the Evaluation Guidelines Partial
Next, create a partial that defines the criteria for evaluating responses. This ensures consistent quality standards.
Here’s an example of evaluation guidelines:
Judge Guidelines Partial Prompt
When evaluating AI chatbot responses, first assess whether the response directly answers the customer’s specific question with accurate and complete information. Check if the response verifies necessary customer details (like order eligibility or account status) when appropriate. Look for clarity of instructions, proactive offering of relevant next steps, and appropriate tone/empathy for the situation. Good responses address the specific query with complete information and clear next steps, while bad responses provide generic information, miss verification opportunities, lack critical details, or fail to acknowledge customer emotions. Your critique should be concise (2-4 sentences) and include a clear “good” or “bad” outcome assessment.
These guidelines define your primary evaluation criteria, secondary considerations, what constitutes good vs. bad responses, and format requirements for critiques. You can adjust these based on what matters most for your specific customer support context. After creating this partial, you’ll receive another partial ID (e.g., `pl-llm-as-1e1952`) to reference in your main template.
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-3%3A-create-golden-examples-partial)
Step 3: Create Golden Examples Partial
Now create a partial with example evaluations. These examples “teach” the LLM what good and bad responses look like in your specific context.
Here’s what your examples partial might look like:Golden Example Partial Prompt
Copy
Ask AI
Where is my order #45789?
Your order #45789 is currently in transit. According to our tracking information, it was shipped yesterday and is expected to arrive at your address by Thursday, March 27th. You can track it in real-time using the tracking number KJ853720164US on our website or mobile app.
{
"critique": "The response directly addresses the customer's question by providing the current status of the specific order number mentioned. It includes helpful details like the shipping date, expected delivery date, and tracking information the customer can use. This is a complete and informative response.",
"outcome": "good"
}
How do I return these broken headphones?
I'm sorry to hear about your broken headphones. To return them, you'll need to go to your order history, select the order containing the headphones, and click "Return Item." Follow the on-screen instructions to print a return label. Pack the headphones in their original packaging if possible, attach the label, and drop it off at any postal service location.
{
"critique": "While the response explains the general return process, it fails to verify if the customer's purchase is still within the return window or ask for the order number to confirm eligibility. Without this verification, the instructions might not apply to this specific situation, potentially causing customer frustration.",
"outcome": "bad"
}
Do you have the new iPhone 16 in stock?
Yes, we currently have the iPhone 16 in stock in all colors. The 128GB, 256GB and 512GB storage options are available for immediate shipping. However, the 1TB version is currently on backorder with an expected 2-week waiting period. Would you like me to check availability at stores near your location?
{
"critique": "The response clearly addresses the stock status of the iPhone 16 with detailed information about different storage options and their availability. The agent proactively offers to check local store inventory, which adds value beyond just answering the immediate question. The information is specific and actionable.",
"outcome": "good"
}
Can I cancel my subscription?
You can cancel your subscription at any time. Simply log into your account, go to "Subscription Settings" and click the "Cancel Subscription" button. Your benefits will continue until the end of your current billing cycle.
{
"critique": "The response provides clear instructions on how to cancel a subscription but doesn't acknowledge potential consequences like loss of data or saved preferences, nor does it offer alternatives like pausing or downgrading. It also doesn't express any interest in retaining the customer or understanding their reason for cancellation.",
"outcome": "bad"
}
When creating your examples:
* Include diverse scenarios covering different types of customer questions
* Show the reasoning process by explaining why an answer is good or bad
* Include both good and bad examples
* Match your actual use cases with examples that reflect your real customer interactions
* Be consistent with the format structure
After creating this partial, you’ll receive another partial ID (e.g., `pl-exampl-55b6e3`) to reference in your main template.
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-4%3A-create-the-main-judge-prompt-template)
Step 4: Create the Main Judge Prompt Template
Now that you have all the partials, it’s time to create the main judge prompt template that brings everything together. We will reference the partials we created earlier to provide context, guidelines, and examples to the judge using mustache variables.
Here’s what your main prompt template should look like:Main LLM as a Judge Prompt
##### System
Copy
Ask AI
You are a Customer Service query evaluator
##### User
Copy
Ask AI
You are a Customer Service query evaluator with advanced capabilities to judge if a query is good or not.
You understand the nuances of customer service including what is likely to be the most useful customer service executive.
Here is information about the customer service in X company:
{{customer_service_info}}
Here are some guidelines for evaluating queries:
{{guidelines}}
Example evaluations:
{{>pl-exampl-55b6e3}}
For the following query, first write a detailed critique explaining your reasoning,
then provide a pass/fail judgment in the same format as above.
{{user_input}}
{{generated_query}}
Some LLMs don’t support strcutured output generation. In those cases you should use LLMs function calling ability to generate JSON output.
Tool Calling Supported LLM
If your LLM does not support Structure Outputs, you will have to add this tool in your prompt template along with the prompt above.Choose `Tool Choice` = `Required` in your prompt template.
Copy
Ask AI
{
"type": "function",
"function": {
"name": "evaluation",
"parameters": {
"type": "object",
"required": [\
"critique",\
"outcome"\
],
"properties": {
"outcome": {
"enum": [\
"good",\
"bad"\
],
"type": "string",
"description": "The final judgment: 'good' or 'bad'"
},
"critique": {
"type": "string",
"description": "A detailed critique of the agent's response"
}
}
},
"description": "this fuuction evaluates the user query and agent ouptut and give a detailed repsone"
}
}
This template sets the evaluator role, inserts your company information, guidelines, and examples, and provides placeholders for the customer query and agent response. Make sure to select an appropriate model (like OpenAI o1, DeepSeek R1) when creating this template. Once you’ve created the main template, you’ll get a prompt ID that you’ll use in your code to access this prompt.
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-5%3A-implementing-the-evaluation-code-with-structured-output)
Step 5: Implementing the Evaluation Code with Structured Output
Now that you have your prompt template set up in Portkey, use this Python code to evaluate customer support interactinos. You can use either of the following ways to go ahead with this cookbook, depending upon what your LLM supports:
* Structured Output
* Tool Calling Supported LLM
Copy
Ask AI
from portkey_ai import Portkey
# Initialize Portkey client
portkey = Portkey(
api_key="YOUR_PORTKEY_API_KEY",
trace_id="customer-support-eval-run-1" # For tracing in Portkey
)
def evaluate_interaction(customer_query, agent_response):
"""
Evaluate a customer support interaction using LLM-as-a-Judge
"""
# Use response_schema to ensure structured output
response_schema = {
"name": "evaluation",
"schema": {
"type": "object",
"properties": {
"critique": {
"type": "string",
"description": "A detailed critique of the agent's response"
},
"outcome": {
"type": "string",
"enum": ["good", "bad"],
"description": "The final judgment: 'good' or 'bad'"
}
},
"required": ["critique", "outcome"]
}
}
# Call Portkey's prompt API with response_schema
response = portkey.prompts.completions.create(
prompt_id="pp-llm-judge-62a41d", # Replace with your actual prompt ID for LLM as a Judge
variables={
"user_input": customer_query,
"generated_query": agent_response,
},
response_format={"type": "json_schema", "json_schema": response_schema, }
)
# Extract structured response and log feedback to Portkey
result = response.choices[0].message.content
import json
# Parse the JSON string into a dictionary
result_dict = json.loads(result)
trace = response.get_headers()['trace-id']
print(result_dict["outcome"])
# Now you can access the dictionary with keys
portkey.feedback.create(
trace_id=trace,
value=1 if result_dict["outcome"] == "good" else 0, # Now this will work
)
return result
# Example usage
customer_query = "I've been waiting for my refund for over two weeks now. When will I receive it?"
agent_response = "Refunds typically take 7-10 business days to process. Let me check the status of your refund and get back to you."
evaluation = evaluate_interaction(customer_query, agent_response)
print(evaluation)
Example Output
Example output:
Copy
Ask AI
{
"critique": "The agent response provides general information about refund processing times (7-10 business days) but doesn't address why the customer's refund is taking over two weeks, which exceeds the standard timeline. While the agent offers to check the status, they don't acknowledge the customer's obvious frustration with the delay, missing an opportunity for empathy. There's also no clear explanation of when the agent will 'get back' with the information, leaving the timeline ambiguous.",
"outcome": "bad"
}
####
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#step-6%3A-iterate-with-domain-experts)
Step 6: Iterate with Domain Experts
The most important part of building an effective LLM-as-a-Judge is iterating on your prompt with feedback from domain experts:
1. Create a small test dataset with 20-30 representative customer support interactions
2. Have human experts evaluate these interactions using the same criteria
3. Compare the LLM judge results with human expert evaluations
4. Calculate agreement rate and identify patterns in disagreements
5. Update your prompt based on what you learn
Focus especially on adding examples that cover edge cases where the judge disagreed with experts, clarifying evaluation criteria, and adjusting the weight given to different factors based on business priorities.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#portkey-observability-for-continuous-improvement)
Portkey Observability for Continuous Improvement
---------------------------------------------------------------------------------------------------------------------------------------------------------------
One of Portkey’s key advantages is its built-in observability. Each evaluation generates detailed traces showing execution time and token usage, input and output logs for debugging, and performance metrics across evaluations. This visibility helps you identify performance bottlenecks, track costs as you scale, debug problematic evaluations, and compare different judge prompt versions.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#visualizing-evaluation-results-on-the-portkey-dashboard)
Visualizing Evaluation Results on the Portkey Dashboard
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The feedback data we collect using the `portkey.feedback.create()` method automatically appears in the Portkey dashboard, allowing you to:
1. Track evaluation outcomes over time
2. Identify specific areas where your agent consistently struggles
3. Measure improvement after making changes to your AI agent
4. Share results with stakeholders through customizable reports
The dashboard gives you a bird’s-eye view of your evaluation metrics, making it easy to spot trends and areas for improvement.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#running-evaluation-on-scale)
Running Evaluation on Scale
---------------------------------------------------------------------------------------------------------------------
Running evaluation on scale
Copy
Ask AI
import pandas as pd
from tqdm import tqdm
import time
# Load your dataset of customer interactions
df = pd.read_csv("customer_interactions.csv")
# Run evaluations on the entire dataset
results = []
for idx, row in tqdm(df.iterrows(), total=len(df)):
customer_query = row['customer_query']
agent_response = row['agent_response']
try:
# Evaluate the interaction
evaluation = evaluate_interaction(customer_query, agent_response)
# Store the result with the original data
results.append({
"customer_query": customer_query,
"agent_response": agent_response,
"critique": evaluation["critique"],
"outcome": evaluation["outcome"],
"improvement_areas": evaluation.get("improvement_areas", [])
})
# Add a small delay to avoid rate limits
time.sleep(0.5)
except Exception as e:
print(f"Error evaluating interaction {idx}: {e}")
# Save the results
results_df = pd.DataFrame(results)
results_df.to_csv("evaluation_results.csv", index=False)
# Calculate overall performance
pass_rate = (results_df['outcome'] == 'good').mean() * 100
print(f"Overall pass rate: {pass_rate:.2f}%")
This code runs your evaluator on an entire dataset, collects the results, and calculates an overall pass rate.
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#next-steps)
Next Steps
-----------------------------------------------------------------------------------
After implementing your LLM-as-a-Judge system, here are key ways to leverage it:
1. **Analyze quality trends**: Track pass rates over time to measure improvement
2. **Identify systematic issues**: Look for patterns in failing responses to address root causes
3. **Improve your support AI**: Use the detailed critiques to refine your support system
[](https://portkey.ai/docs/guides/prompts/llm-as-a-judge#conclusion)
Conclusion
-----------------------------------------------------------------------------------
An LLM-as-a-Judge system transforms how you approach customer support quality assurance. Rather than sampling a tiny fraction of interactions or relying on vague metrics, you can evaluate every interaction with consistency and depth. The detailed critiques provide actionable insights that drive continuous improvement in your customer support AI. By implementing this approach with Portkey, you create a scalable quality assurance system that grows with your support operations while maintaining the high standards your customers expect. Ready to build your own LLM-as-a-Judge system? Get started with Portkey today.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/guides/prompts/llm-as-a-judge.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/guides/prompts/llm-as-a-judge)
[How to Run Structured Output Evals at Scale](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Prompts](https://portkey.ai/docs/guides/prompts)
⌘I
---
# Headers - Portkey Docs
[Skip to main content](https://portkey.ai/docs/api-reference/inference-api/headers#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
API Reference
Headers
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey Authentication](https://portkey.ai/docs/api-reference/inference-api/headers#portkey-authentication)
* [Portkey API Key](https://portkey.ai/docs/api-reference/inference-api/headers#portkey-api-key)
* [Provider Authentication](https://portkey.ai/docs/api-reference/inference-api/headers#provider-authentication)
* [1\. Provider Slug + Auth](https://portkey.ai/docs/api-reference/inference-api/headers#1-provider-slug-%2B-auth)
* [2\. Virtual Key](https://portkey.ai/docs/api-reference/inference-api/headers#2-virtual-key)
* [3\. Config](https://portkey.ai/docs/api-reference/inference-api/headers#3-config)
* [4\. Custom Host](https://portkey.ai/docs/api-reference/inference-api/headers#4-custom-host)
* [Additional Portkey Headers](https://portkey.ai/docs/api-reference/inference-api/headers#additional-portkey-headers)
* [Trace ID](https://portkey.ai/docs/api-reference/inference-api/headers#trace-id)
* [Metadata](https://portkey.ai/docs/api-reference/inference-api/headers#metadata)
* [Cache Force Refresh](https://portkey.ai/docs/api-reference/inference-api/headers#cache-force-refresh)
* [Cache Namespace](https://portkey.ai/docs/api-reference/inference-api/headers#cache-namespace)
* [Request Timeout](https://portkey.ai/docs/api-reference/inference-api/headers#request-timeout)
* [Custom Headers](https://portkey.ai/docs/api-reference/inference-api/headers#custom-headers)
* [Forward Headers](https://portkey.ai/docs/api-reference/inference-api/headers#forward-headers)
* [Cloud-Specific Headers (Azure, Google, AWS)](https://portkey.ai/docs/api-reference/inference-api/headers#cloud-specific-headers-azure%2C-google%2C-aws)
* [Azure](https://portkey.ai/docs/api-reference/inference-api/headers#azure)
* [Google Vertex AI](https://portkey.ai/docs/api-reference/inference-api/headers#google-vertex-ai)
* [AWS Bedrock](https://portkey.ai/docs/api-reference/inference-api/headers#aws-bedrock)
* [List of All Headers](https://portkey.ai/docs/api-reference/inference-api/headers#list-of-all-headers)
* [Using Headers](https://portkey.ai/docs/api-reference/inference-api/headers#using-headers)
Portkey API accepts 4 kinds of headers for your requests:
| | | |
| --- | --- | --- |
| Portkey Authentication Header | `Required` | For Portkey auth |
| Provider Authentication Headers OR Cloud-Specific Headers | `Required` | For provider auth |
| Additional Portkey Headers | `Optional` | To pass `config`, `metadata`, `trace id`, `cache refresh` etc. |
| Custom Headers | `Optional` | To forward any other headers directly |
[](https://portkey.ai/docs/api-reference/inference-api/headers#portkey-authentication)
Portkey Authentication
-----------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#portkey-api-key)
Portkey API Key
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-api-key-api-key-api-key)
x-portkey-api-key / api\_key / apiKey
string
required
Authenticate your requests with your Portkey API key. Obtain API key from the [Portkey dashboard](https://app.portkey.ai/api-keys)
.
Environment variable: `PORTKEY_API_KEY`Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
[](https://portkey.ai/docs/api-reference/inference-api/headers#provider-authentication)
Provider Authentication
-------------------------------------------------------------------------------------------------------------------
In addition to the Portkey API key, you must provide information about the AI provider you’re using. There are **4** ways to do this:
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#1-provider-slug-%2B-auth)
1\. Provider Slug + Auth
Useful if you do not want to save your API keys to Portkey vault and make direct requests.
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-provider-provider)
x-portkey-provider / provider
string
Specifies the provider you’re using (e.g., `openai`, `anthropic`, `vertex-ai`).
List of [Portkey supported providers here](https://portkey.ai/docs/integrations/llms)
.
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-authorization)
Authorization
string
Pass the auth details for the specified provider as a `"Bearer $TOKEN"`.
If your provider expects their auth with headers such as `x-api-key` or `api-key`, you can pass the token with the `Authorization` header directly and Portkey will convert it into the provider-specific format.
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: openai" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#2-virtual-key)
2\. Virtual Key
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-virtual-key-virtual-key-virtual-key)
x-portkey-virtual-key / virtual\_key / virtualKey
string
Save your provider auth on Portkey and use a virtual key to directly make a call. [Docs](https://portkey.ai/docs/product/ai-gateway/virtual-keys)
)
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#3-config)
3\. Config
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-config-config)
x-portkey-config / config
string or JSON
Pass your Portkey config with this header. Accepts a `JSON object` or a `config ID` that can also contain gateway configuration settings, and provider details.
* Configs can be saved in the Portkey UI and referenced by their ID ([Docs](https://portkey.ai/docs/product/ai-gateway/configs)
)
* Configs also enable other optional features like Caching, Load Balancing, Fallback, Retries, and Timeouts.
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-config: openai-config" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#4-custom-host)
4\. Custom Host
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-custom-host-custom-host-custom-host)
x-portkey-custom-host / custom\_host / customHost
string
Specifies the base URL where you want to send your request
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-provider-provider-1)
x-portkey-provider / provider
string
Target provider that’s availabe on your base URL. If you are unsure of which target provider to set, you can set `openai`.
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-authorization-1)
Authorization
string
Pass the auth details for the specified provider as a `"Bearer $TOKEN"`.
If your provider expects their auth with headers such as `x-api-key` or `api-key`, you can pass the token with the `Authorization` header directly and Portkey will convert it into the provider-specific format.
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-custom-host: http://124.124.124.124/v1" \
-H "x-portkey-provider: openai" \
-H "Authorization: Bearer $TOKEN" \
* * *
[](https://portkey.ai/docs/api-reference/inference-api/headers#additional-portkey-headers)
Additional Portkey Headers
-------------------------------------------------------------------------------------------------------------------------
There are additional optional Portkey headers that enable various features and enhancements:
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#trace-id)
Trace ID
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-trace-id-trace-id-trace-id)
x-portkey-trace-id / trace\_id / traceId
string
An ID you can pass to refer to one or more requests later on. If not provided, Portkey generates a trace ID automatically for each request. ([Docs](https://portkey.ai/docs/product/observability/traces)
)
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "x-portkey-trace-id: test-request" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#metadata)
Metadata
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-metadata-metadata)
x-portkey-metadata / metadata
JSON
Allows you to attach custom metadata to your requests, which can be filtered later in the analytics and log dashboards.
You can include the special metadata type `_user` to associate requests with specific users. ([Docs](https://portkey.ai/docs/product/observability/metadata)
)
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "x-portkey-metadata: {'_user': 'user_id_123', 'foo': 'bar'}" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#cache-force-refresh)
Cache Force Refresh
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-cache-force-refresh-cache-force-refresh-cache-force-refresh)
x-portkey-cache-force-refresh / cache\_force\_refresh / cacheForceRefresh
boolean
Forces a cache refresh for your request by making a new API call and storing the updated value.
Expects `true` or `false` See the caching documentation for more information. ([Docs](https://portkey.ai/docs/product/ai-gateway/cache-simple-and-semantic)
)
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "x-portkey-cache-force-refresh: true" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#cache-namespace)
Cache Namespace
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-cache-namespace-cache-namespace-cache-namespace)
x-portkey-cache-namespace / cache\_namespace / cacheNamespace
string
Partition your cache store based on custom strings, ignoring metadata and other headers.
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "x-portkey-cache-namespace: any-string" \
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#request-timeout)
Request Timeout
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-request-timeout-request-timeout-request-timeout)
x-portkey-request-timeout / request\_timeout / requestTimeout
integer
Set timeout after which a request automatically terminates. The time is set in milliseconds.
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "x-portkey-request-timeout: 3000" \
[](https://portkey.ai/docs/api-reference/inference-api/headers#custom-headers)
Custom Headers
-------------------------------------------------------------------------------------------------
You can pass any other headers your API expects by directly forwarding them without any processing by Portkey.
This is especially useful if you want to pass send sensitive headers.
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#forward-headers)
Forward Headers
[](https://portkey.ai/docs/api-reference/inference-api/headers#param-x-portkey-forward-headers-forward-headers-forward-headers)
x-portkey-forward-headers / forward\_headers / forwardHeaders
array of strings
Pass all the headers you want to forward directly in this array. ([Docs](https://portkey.ai/docs/welcome/integration-guides/byollm#forward-sensitive-headers-securely)
)
Example
cURL
Python
JavaScript
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: openai-virtual-key" \
-H "X-Custom-Header: ...."\
-H "Another-Header: ....."\
-H "x-portkey-forward-headers: ['X-Custom-Header', 'Another-Header']" \
####
[](https://portkey.ai/docs/api-reference/inference-api/headers#python-usage)
Python Usage
With the Python SDK, you need to transform your headers to **Snake Case** and then include them while initializing the Portkey client. Example: If you have a header of the format `X-My-Custom-Header`, it should be sent as `X_My_Custom_Header` in the SDK
####
[](https://portkey.ai/docs/api-reference/inference-api/headers#javascript-usage)
JavaScript Usage
With the JS SDK, you need to transform your headers to **Camel Case** and then include them while initializing the Portkey client. Example: If you have a header of the format `X-My-Custom-Header`, it should be sent as `xMyCustomHeader` in the SDK
[](https://portkey.ai/docs/api-reference/inference-api/headers#cloud-specific-headers-azure%2C-google%2C-aws)
Cloud-Specific Headers (`Azure`, `Google`, `AWS`)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Pass more configuration headers for `Azure OpenAI`, `Google Vertex AI`, or `AWS Bedrock`
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#azure)
Azure
* `x-portkey-azure-resource-name`, `x-portkey-azure-deployment-id`, `x-portkey-azure-api-version`, `Authorization`, `x-portkey-azure-model-name`
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#google-vertex-ai)
Google Vertex AI
* `x-portkey-vertex-project-id`, `x-portkey-vertex-region`, `X-Vertex-AI-LLM-Request-Type`
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#aws-bedrock)
AWS Bedrock
* `x-portkey-aws-session-token`, `x-portkey-aws-secret-access-key`, `x-portkey-aws-region`, `x-portkey-aws-session-token`
* * *
[](https://portkey.ai/docs/api-reference/inference-api/headers#list-of-all-headers)
List of All Headers
-----------------------------------------------------------------------------------------------------------
The following is a comprehensive list of headers that can be used when initializing the Portkey client. Portkey adheres to language-specific naming conventions:
* **camelCase** for **JavaScript/Node.js** parameters
* **snake\_case** for **Python** parameters
* **hyphenated-keys** for **HTTP headers**
* NodeJS
* Python
* REST Headers
| Parameter | Type | Key |
| --- | --- | --- |
| **API Key** Your Portkey account’s API Key. | stringrequired | `apiKey` |
| **Virtual Key** The virtual key created from Portkey’s vault for a specific provider | string | `virtualKey` |
| **Config** The slug or [config object](https://portkey.ai/docs/api-reference/inference-api/config-object)
to use | stringobject | `config` |
| **Provider** The AI provider to use for your calls. ([supported providers](https://portkey.ai/docs/integrations/llms#supported-ai-providers)
). | string | `provider` |
| **Base URL** You can edit the URL of the gateway to use. Needed if you’re [self-hosting the AI gateway](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md) | string | `baseURL` |
| **Trace ID** An ID you can pass to refer to 1 or more requests later on. Generated automatically for every request, if not sent. | string | `traceID` |
| **Metadata** Any metadata to attach to the requests. These can be filtered later on in the analytics and log dashboards Can contain `_prompt`, `_user`, `_organisation`, or `_environment` that are special metadata types in Portkey. You can also send any other keys as part of this object. | object | `metadata` |
| **Cache Force Refresh** Force refresh the cache for your request by making a new call and storing that value. | boolean | `cacheForceRefresh` |
| **Cache Namespace** Partition your cache based on custom strings, ignoring metadata and other headers. | string | `cacheNamespace` |
| **Custom Host** Route to locally or privately hosted model by configuring the API URL with custom host | string | `customHost` |
| **Forward Headers** Forward sensitive headers directly to your model’s API without any processing from Portkey. | array of string | `forwardHeaders` |
| **Azure OpenAI Headers** Configuration headers for Azure OpenAI that you can send separately | string | `azureResourceName` `azureDeploymentId` `azureApiVersion` `azureModelName` |
| **Google Vertex AI Headers** Configuration headers for Vertex AI that you can send separately | string | `vertexProjectId` `vertexRegion` |
| **AWS Bedrock Headers** Configuration headers for Bedrock that you can send separately | string | `awsAccessKeyId` `awsSecretAccessKey` `awsRegion` `awsSessionToken` |
###
[](https://portkey.ai/docs/api-reference/inference-api/headers#using-headers)
Using Headers
You can send these headers in multiple ways:
REST API
Portkey SDK
Node.js SDK
OpenAI SDK
OpenAI Node.js SDK
Copy
Ask AI
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: VIRTUAL_KEY" \
-H "x-portkey-trace-id: your_trace_id" \
-H "x-portkey-metadata: {\"_user\": \"user_12345\"}" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/api-reference/inference-api/headers.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/api-reference/inference-api/headers)
[Authentication](https://portkey.ai/docs/api-reference/inference-api/authentication)
[Errors](https://portkey.ai/docs/api-reference/inference-api/error-codes)
⌘I
---
# AWS SageMaker - Portkey Docs
[Skip to main content](https://portkey.ai/docs/integrations/llms/aws-sagemaker#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
LLM Integrations
AWS SageMaker
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey SDK Integration with AWS Sagemaker](https://portkey.ai/docs/integrations/llms/aws-sagemaker#portkey-sdk-integration-with-aws-sagemaker)
* [1\. Install the Portkey SDK](https://portkey.ai/docs/integrations/llms/aws-sagemaker#1-install-the-portkey-sdk)
* [2\. Initialize Portkey with a Integration](https://portkey.ai/docs/integrations/llms/aws-sagemaker#2-initialize-portkey-with-a-integration)
* [3\. Invoke the Sagemaker model](https://portkey.ai/docs/integrations/llms/aws-sagemaker#3-invoke-the-sagemaker-model)
* [Making Requests without Virtual Keys](https://portkey.ai/docs/integrations/llms/aws-sagemaker#making-requests-without-virtual-keys)
* [Example](https://portkey.ai/docs/integrations/llms/aws-sagemaker#example)
* [Next Steps](https://portkey.ai/docs/integrations/llms/aws-sagemaker#next-steps)
Sagemaker allows users to host any ML model on their own AWS infrastructure. With portkey you can manage/restrict access, log requests, and more.
Provider Slug. `sagemaker`
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#portkey-sdk-integration-with-aws-sagemaker)
Portkey SDK Integration with AWS Sagemaker
-----------------------------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#1-install-the-portkey-sdk)
1\. Install the Portkey SDK
Add the Portkey SDK to your application to interact with Sagemaker’s API through Portkey’s gateway.
* NodeJS
* Python
Copy
Ask AI
npm install --save portkey-ai
###
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#2-initialize-portkey-with-a-integration)
2\. Initialize Portkey with a Integration
There are multiple ways to integrate Sagemaker with Portkey. You can use your AWS credentials, or use an assumed role. In this example we will create a integration and use it to interact with Sagemaker. This helps you restrict access (specific models, few endpoints, etc).
Here’s how to find your AWS credentials:
[](https://portkey.ai/docs/integrations/llms/aws-bedrock#how-to-find-your-aws-credentials)
[](https://portkey.ai/docs/integrations/llms/aws-bedrock#how-to-find-your-aws-credentials)
[AWS Access Key\
--------------](https://portkey.ai/docs/integrations/llms/aws-bedrock#how-to-find-your-aws-credentials)
[Use your `AWS Secret Access Key`, `AWS Access Key Id`, and `AWS Region` to create your Virtual key.](https://portkey.ai/docs/integrations/llms/aws-bedrock#how-to-find-your-aws-credentials)
[**Integration Guide**](https://portkey.ai/docs/integrations/llms/aws-bedrock#how-to-find-your-aws-credentials)
[](https://portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role)
[](https://portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role)
[AWS Assumed Role\
----------------](https://portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role)
[Take your `AWS Assumed Role ARN` and `AWS Region` to create the virtaul key.](https://portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role)
[**Integration Guide**](https://portkey.ai/docs/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role)
Create a new integration in the Portkey dashboard. You can select sagemaker as the provider, and fill in deployment details. Initialize the Portkey SDK with the integration. (If you are using the REST API, skip to next step)
* NodeJS SDK
* Python SDK
Copy
Ask AI
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key
provider:"@sagemaker" // Replace with your Sagemaker provider slug
})
###
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#3-invoke-the-sagemaker-model)
3\. Invoke the Sagemaker model
* Python SDK
* NodeJS SDK
* REST API
Copy
Ask AI
response = portkey.post(
url="endpoints/{endpoint_name}/invocations",
# You can pass any key value pair required by the model, apart from `url`, they are passed as kwargs to the Sagemaker endpoint
inputs="my_custom_value",
my_custom_key="my_custom_value",
)
print(response)
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#making-requests-without-virtual-keys)
Making Requests without Virtual Keys
-----------------------------------------------------------------------------------------------------------------------------------------
If you do not want to add your AWS details to Portkey vault, you can also directly pass them while instantiating the Portkey client. These are the supported headers/parameters for Sagemaker (Not required if you’re using a virtual key):
| Node SDK | Python SDK | REST Headers |
| --- | --- | --- |
| awsAccessKeyId | aws\_access\_key\_id | x-portkey-aws-access-key-id |
| awsSecretAccessKey | aws\_secret\_access\_key | x-portkey-aws-secret-access-key |
| awsRegion | aws\_region | x-portkey-aws-region |
| awsSessionToken | aws\_session\_token | x-portkey-aws-session-token |
| sagemakerCustomAttributes | sagemaker\_custom\_attributes | x-portkey-amzn-sagemaker-custom-attributes |
| sagemakerTargetModel | sagemaker\_target\_model | x-portkey-amzn-sagemaker-target-model |
| sagemakerTargetVariant | sagemaker\_target\_variant | x-portkey-amzn-sagemaker-target-variant |
| sagemakerTargetContainerHostname | sagemaker\_target\_container\_hostname | x-portkey-amzn-sagemaker-target-container-hostname |
| sagemakerInferenceId | sagemaker\_inference\_id | x-portkey-amzn-sagemaker-inference-id |
| sagemakerEnableExplanations | sagemaker\_enable\_explanations | x-portkey-amzn-sagemaker-enable-explanations |
| sagemakerInferenceComponent | sagemaker\_inference\_component | x-portkey-amzn-sagemaker-inference-component |
| sagemakerSessionId | sagemaker\_session\_id | x-portkey-amzn-sagemaker-session-id |
| sagemakerModelName | sagemaker\_model\_name | x-portkey-amzn-sagemaker-model-name |
###
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#example)
Example
* Python
* NodeJS
* cURL
Copy
Ask AI
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
provider="sagemaker",
aws_region="us-east-1", # Replace with your AWS region
aws_access_key_id="AWS_ACCESS_KEY_ID", # Replace with your AWS access key id
aws_secret_access_key="AWS_SECRET_ACCESS_KEY", # Replace with your AWS secret access key
amzn_sagemaker_inference_component="SAGEMAKER_INFERENCE_COMPONENT" # Replace with your Sagemaker inference component
)
response = portkey.post(
url="endpoints/{endpoint_name}/invocations",
# You can pass any key value pair required by the model, apart from `url`, they are passed as kwargs to the Sagemaker endpoint
inputs="my_custom_value",
my_custom_key="my_custom_value"
)
print(response)
[](https://portkey.ai/docs/integrations/llms/aws-sagemaker#next-steps)
Next Steps
-------------------------------------------------------------------------------------
The complete list of features supported in the SDK are available on the link below.[SDK\
---](https://portkey.ai/docs/api-reference/portkey-sdk-client)
You’ll find more information in the relevant sections:
1. [Add metadata to your requests](https://portkey.ai/docs/product/observability/metadata)
2. [Add gateway configs to your Sagemaker requests](https://portkey.ai/docs/product/ai-gateway/configs)
3. [Tracing Sagemaker requests](https://portkey.ai/docs/product/observability/traces)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/integrations/llms/aws-sagemaker.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/integrations/llms/aws-sagemaker)
[AWS Bedrock Knowledge Bases](https://portkey.ai/docs/integrations/llms/bedrock/bedrock-knowledgebase)
[Ollama](https://portkey.ai/docs/integrations/llms/ollama)
⌘I
---
# Google Gemini - Portkey Docs
[Skip to main content](https://portkey.ai/docs/integrations/llms/gemini#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
LLM Integrations
Google Gemini
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey SDK Integration with Google Gemini Models](https://portkey.ai/docs/integrations/llms/gemini#portkey-sdk-integration-with-google-gemini-models)
* [1\. Install the Portkey SDK](https://portkey.ai/docs/integrations/llms/gemini#1-install-the-portkey-sdk)
* [2\. Initialize Portkey with the Virtual Key](https://portkey.ai/docs/integrations/llms/gemini#2-initialize-portkey-with-the-virtual-key)
* [3\. Invoke Chat Completions with Google Gemini](https://portkey.ai/docs/integrations/llms/gemini#3-invoke-chat-completions-with-google-gemini)
* [Function Calling](https://portkey.ai/docs/integrations/llms/gemini#function-calling)
* [Advanced Multimodal Capabilities with Gemini](https://portkey.ai/docs/integrations/llms/gemini#advanced-multimodal-capabilities-with-gemini)
* [Image Processing](https://portkey.ai/docs/integrations/llms/gemini#image-processing)
* [Audio Processing](https://portkey.ai/docs/integrations/llms/gemini#audio-processing)
* [Video Processing](https://portkey.ai/docs/integrations/llms/gemini#video-processing)
* [Document Processing (PDF)](https://portkey.ai/docs/integrations/llms/gemini#document-processing-pdf)
* [Code Execution Tool](https://portkey.ai/docs/integrations/llms/gemini#code-execution-tool)
* [Grounding with Google Search](https://portkey.ai/docs/integrations/llms/gemini#grounding-with-google-search)
* [Extended Thinking (Reasoning Models) (Beta)](https://portkey.ai/docs/integrations/llms/gemini#extended-thinking-reasoning-models-beta)
* [Single turn conversation](https://portkey.ai/docs/integrations/llms/gemini#single-turn-conversation)
* [Next Steps](https://portkey.ai/docs/integrations/llms/gemini#next-steps)
Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including [Google Gemini APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini)
. With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through a [virtual key](https://portkey.ai/docs/product/ai-gateway/virtual-keys)
system.
Provider Slug. `google`
[](https://portkey.ai/docs/integrations/llms/gemini#portkey-sdk-integration-with-google-gemini-models)
Portkey SDK Integration with Google Gemini Models
------------------------------------------------------------------------------------------------------------------------------------------------------------
Portkey provides a consistent API to interact with models from various providers. To integrate Google Gemini with Portkey:
###
[](https://portkey.ai/docs/integrations/llms/gemini#1-install-the-portkey-sdk)
1\. Install the Portkey SDK
Add the Portkey SDK to your application to interact with Google Gemini’s API through Portkey’s gateway.
* NodeJS
* Python
Copy
Ask AI
npm install --save portkey-ai
###
[](https://portkey.ai/docs/integrations/llms/gemini#2-initialize-portkey-with-the-virtual-key)
2\. Initialize Portkey with the Virtual Key
To use Gemini with Portkey, [get your API key from here](https://aistudio.google.com/app/apikey)
, then add it to Portkey to create the virtual key.
* NodeJS SDK
* Python SDK
Copy
Ask AI
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
provider:"@PROVIDER" // Your Google Virtual Key
})
###
[](https://portkey.ai/docs/integrations/llms/gemini#3-invoke-chat-completions-with-google-gemini)
**3\. Invoke Chat Completions with** Google Gemini
Use the Portkey instance to send requests to Google Gemini. You can also override the virtual key directly in the API call if needed.
* NodeJS SDK
* Python SDK
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
messages: [\
{ role: 'system', content: 'You are not a helpful assistant' },\
{ role: 'user', content: 'Say this is a test' }\
],
model: 'gemini-1.5-pro',
});
console.log(chatCompletion.choices);
Portkey supports the `system_instructions` parameter for Google Gemini 1.5 - allowing you to control the behavior and output of your Gemini-powered applications with ease.Simply include your Gemini system prompt as part of the `{"role":"system"}` message within the `messages` array of your request body. Portkey Gateway will automatically transform your message to ensure seamless compatibility with the Google Gemini API.
[](https://portkey.ai/docs/integrations/llms/gemini#function-calling)
Function Calling
------------------------------------------------------------------------------------------
Portkey supports function calling mode on Google’s Gemini Models. Explore this Cookbook for a deep dive and examples: [Function Calling](https://portkey.ai/docs/guides/getting-started/function-calling)
[](https://portkey.ai/docs/integrations/llms/gemini#advanced-multimodal-capabilities-with-gemini)
Advanced Multimodal Capabilities with Gemini
--------------------------------------------------------------------------------------------------------------------------------------------------
Gemini models are inherently multimodal, capable of processing and understanding content from a wide array of file types. Portkey streamlines the integration of these powerful features by providing a unified, OpenAI-compatible API.
**The Portkey Advantage: A Unified Format for All Media**To simplify development, Portkey uses a consistent format for all multimodal requests. Whether you’re sending an image, audio, video, or document, you will use an object with `type: 'image_url'` within the user message’s `content` array.Portkey’s AI Gateway intelligently interprets your request—based on the URL or data URI you provide—and translates it into the precise format required by the Google Gemini API. This means you only need to learn one structure for all your media processing needs.
###
[](https://portkey.ai/docs/integrations/llms/gemini#image-processing)
Image Processing
Gemini can analyze images to describe their content, answer visual questions, or identify objects.[Gemini Image Understanding Docs\
-------------------------------](https://ai.google.dev/gemini-api/docs/image-understanding)
**Method 1: Sending an Image via Google Files URL** Use the Google Files API to upload your image and get a URL. This is the recommended approach for larger files or when you need persistent storage.
To upload files and get Google Files URLs, use the [Files API](https://ai.google.dev/gemini-api/docs/files)
. The URL format will be similar to: `https://generativelanguage.googleapis.com/v1beta/files/[FILE_ID]`
NodeJS
Python
cURL
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{\
type: 'image_url',\
image_url: {\
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id'\
}\
},\
{\
type: 'text',\
text: 'Describe this image in detail.'\
}\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
**Method 2: Sending a Local Image as Base64 Data** Use this method for local image files. The file is encoded into a Base64 string and sent as a data URI. This is ideal for smaller files when you don’t want to use the Files API. The data URI format is: `data:;base64,`
NodeJS
Python
cURL
Copy
Ask AI
import fs from 'fs';
const imageBytes = fs.readFileSync('local-image.png');
const base64Image = imageBytes.toString('base64');
const imageUri = `data:image/png;base64,${base64Image}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{ type: 'image_url', image_url: { url: imageUri }},\
{ type: 'text', text: 'What is in this picture?' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
Supported Image MIME types: `image/png`, `image/jpeg`, `image/webp`, `image/heic`, `image/heif`
* * *
###
[](https://portkey.ai/docs/integrations/llms/gemini#audio-processing)
Audio Processing
Gemini can transcribe speech, summarize audio content, or answer questions about sounds.[Gemini Audio Understanding Docs\
-------------------------------](https://ai.google.dev/gemini-api/docs/audio)
**Method 1: Sending Audio via Google Files URL** Upload your audio file using the Files API to get a Google Files URL.
NodeJS
Python
cURL
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{\
type: 'image_url',\
image_url: {\
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id'\
}\
},\
{ type: 'text', text: 'Please transcribe the speech in this audio.' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
**Method 2: Sending Local Audio as Base64 Data** This is the standard way to process local audio files directly through the API.
NodeJS
Python
cURL
Copy
Ask AI
import fs from 'fs';
const audioBytes = fs.readFileSync('audio-example.mp3');
const base64Audio = audioBytes.toString('base64');
const audioUri = `data:audio/mp3;base64,${base64Audio}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{ type: 'image_url', image_url: { url: audioUri }},\
{ type: 'text', text: 'Describe this audio' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
Supported Audio MIME types: `audio/wav`, `audio/mp3`, `audio/aiff`, `audio/aac`, `audio/ogg`, `audio/flac`
* * *
###
[](https://portkey.ai/docs/integrations/llms/gemini#video-processing)
Video Processing
Gemini can summarize videos, answer questions about specific events, and describe scenes.[Gemini Video Understanding Docs\
-------------------------------](https://ai.google.dev/gemini-api/docs/video-understanding)
**Method 1: Sending a Video via YouTube URL** YouTube is the only supported public URL source for videos. Simply provide the YouTube video URL.
NodeJS
Python
cURL
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{\
type: 'text',\
text: 'Describe this video in 3 sentences.'\
},\
{\
type: 'image_url',\
image_url: {\
url: 'https://www.youtube.com/watch?v=9hE5-98ZeCg'\
}\
}\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
**Method 2: Sending Local Video as Base64 Data** For smaller video files, you can encode them as base64. Note that this method has size limitations.
NodeJS
Python
cURL
Copy
Ask AI
import fs from 'fs';
const videoBytes = fs.readFileSync('video-example.mp4');
const base64Video = videoBytes.toString('base64');
const videoUri = `data:video/mp4;base64,${base64Video}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{ type: 'image_url', image_url: { url: videoUri }},\
{ type: 'text', text: 'Describe this video' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
**Method 3: Sending Video via Google Files URL** For larger video files, upload them using the Files API to get a Google Files URL.
NodeJS
Python
cURL
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{\
type: 'image_url',\
image_url: {\
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id'\
}\
},\
{ type: 'text', text: 'Please describe the main events in this video.' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
Supported Video MIME types: `video/mp4`, `video/mpeg`, `video/mov`, `video/avi`, `video/webm`, `video/wmv`
* * *
###
[](https://portkey.ai/docs/integrations/llms/gemini#document-processing-pdf)
Document Processing (PDF)
Gemini’s vision capabilities excel at understanding the content of PDF documents, including text, tables, and images.[Gemini Documents Understanding Docs\
-----------------------------------](https://ai.google.dev/gemini-api/docs/document-processing)
**Method 1: Sending a Document via Google Files URL** Upload your PDF using the Files API to get a Google Files URL.
NodeJS
Python
cURL
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{\
type: 'image_url',\
image_url: {\
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id'\
}\
},\
{ type: 'text', text: 'Summarize the key findings of this research paper.' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
**Method 2: Sending a Local Document as Base64 Data** This is suitable for smaller, local PDF files.
NodeJS
Python
cURL
Copy
Ask AI
import fs from 'fs';
const pdfBytes = fs.readFileSync('whitepaper.pdf');
const base64Pdf = pdfBytes.toString('base64');
const pdfUri = `data:application/pdf;base64,${base64Pdf}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{\
role: 'user',\
content: [\
{ type: 'image_url', image_url: { url: pdfUri }},\
{ type: 'text', text: 'What is the main conclusion of this document?' }\
]\
}],
});
console.log(chatCompletion.choices[0].message.content);
While you can send other document types like `.txt` or `.html`, they will be treated as plain text. Gemini’s native document vision capabilities are optimized for the `application/pdf` MIME type.
* * *
[](https://portkey.ai/docs/integrations/llms/gemini#code-execution-tool)
Code Execution Tool
------------------------------------------------------------------------------------------------
Gemini can use a built-in code interpreter tool to solve complex computational problems, perform calculations, and generate code. To enable this, simply include the `code_execution` tool in your request. The model will automatically decide when to invoke it.
NodeJS
Python
cURL
Copy
Ask AI
const response = await portkey.chat.completions.create({
model: "gemini-1.5-pro",
messages: [{\
"role": "user",\
"content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."\
}],
tools: [{ "type": "code_execution" }]
});
console.log(response.choices[0].message.content);
**Important:** For all file uploads (except YouTube videos), it’s recommended to use the [Google Files API](https://ai.google.dev/gemini-api/docs/files)
to upload your files first, then use the returned file URL in your requests. This approach provides better performance and reliability for larger files.
[](https://portkey.ai/docs/integrations/llms/gemini#grounding-with-google-search)
Grounding with Google Search
------------------------------------------------------------------------------------------------------------------
Vertex AI supports grounding with Google Search. This is a feature that allows you to ground your LLM responses with real-time search results. Grounding is invoked by passing the `google_search` tool (for newer models like gemini-2.0-flash-001), and `google_search_retrieval` (for older models like gemini-1.5-flash) in the `tools` array.
Copy
Ask AI
"tools": [\
{\
"type": "function",\
"function": {\
"name": "google_search" // or google_search_retrieval for older models\
}\
}]
If you mix regular tools with grounding tools, vertex might throw an error saying only one tool can be used at a time.
[](https://portkey.ai/docs/integrations/llms/gemini#extended-thinking-reasoning-models-beta)
Extended Thinking (Reasoning Models) (Beta)
--------------------------------------------------------------------------------------------------------------------------------------------
The assistants thinking response is returned in the `response_chunk.choices[0].delta.content_blocks` array, not the `response.choices[0].message.content` string.
Models like `gemini-2.5-flash-preview-04-17` `gemini-2.5-flash-preview-04-17` support [extended thinking](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude#claude-3-7-sonnet)
. This is similar to openai thinking, but you get the model’s reasoning as it processes the request as well. Note that you will have to set [`strict_open_ai_compliance=False`](https://portkey.ai/docs/product/ai-gateway/strict-open-ai-compliance)
in the headers to use this feature.
###
[](https://portkey.ai/docs/integrations/llms/gemini#single-turn-conversation)
Single turn conversation
Python
NodeJS
OpenAI NodeJS
OpenAI Python
cURL
Copy
Ask AI
from portkey_ai import Portkey
# Initialize the Portkey client
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
provider="@PROVIDER",
strict_open_ai_compliance=False
)
# Create the request
response = portkey.chat.completions.create(
model="gemini-2.5-flash-preview-04-17",
max_tokens=3000,
thinking={
"type": "enabled",
"budget_tokens": 2030
},
stream=True,
messages=[\
{\
"role": "user",\
"content": [\
{\
"type": "text",\
"text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"\
}\
]\
}\
]
)
print(response)
# in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
# response = portkey.chat.completions.create(
# ...same config as above but with stream: true
# )
# for chunk in response:
# if chunk.choices[0].delta:
# content_blocks = chunk.choices[0].delta.get("content_blocks")
# if content_blocks is not None:
# for content_block in content_blocks:
# print(content_block)
To disable thinking for gemini models like `gemini-2.5-flash-preview-04-17`, you are required to explicitly set `budget_tokens` to `0`.
Copy
Ask AI
"thinking": {
"type": "enabled",
"budget_tokens": 0
}
Gemini grounding mode may not work via Portkey SDK. Contact [\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#1a696f6a6a75686e5a6a75686e717f63347b73)
for assistance.
[](https://portkey.ai/docs/integrations/llms/gemini#next-steps)
Next Steps
------------------------------------------------------------------------------
The complete list of features supported in the SDK are available on the link below.[SDK\
---](https://portkey.ai/docs/api-reference/portkey-sdk-client)
You’ll find more information in the relevant sections:
1. [Add metadata to your requests](https://portkey.ai/docs/product/observability/metadata)
2. [Add gateway configs to your Gemini requests](https://portkey.ai/docs/product/ai-gateway/configs)
3. [Tracing Google Gemini requests](https://portkey.ai/docs/product/observability/traces)
4. [Setup a fallback from OpenAI to Gemini APIs](https://portkey.ai/docs/product/ai-gateway/fallbacks)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/integrations/llms/gemini.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/integrations/llms/gemini)
[Count Tokens](https://portkey.ai/docs/integrations/llms/anthropic/count-tokens)
[Google Vertex AI](https://portkey.ai/docs/integrations/llms/vertex-ai)
⌘I
---
# Overview - Portkey Docs
[Skip to main content](https://portkey.ai/docs/integrations/llms#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
LLM Integrations
Overview
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Supported AI Providers](https://portkey.ai/docs/integrations/llms#supported-ai-providers)
* [Endpoints Supported](https://portkey.ai/docs/integrations/llms#endpoints-supported)
* [Supported Frameworks](https://portkey.ai/docs/integrations/llms#supported-frameworks)
[](https://portkey.ai/docs/integrations/llms#supported-ai-providers)
Supported AI Providers
-----------------------------------------------------------------------------------------------
[OpenAI\
------\
\
](https://portkey.ai/docs/integrations/llms/openai)
[Anthropic\
---------\
\
](https://portkey.ai/docs/integrations/llms/anthropic)
[Azure OpenAI\
------------\
\
](https://portkey.ai/docs/integrations/llms/azure-openai)
[AWS Bedrock\
-----------\
\
](https://portkey.ai/docs/integrations/llms/aws-bedrock)
[Cerebras\
--------\
\
](https://portkey.ai/docs/integrations/llms/cerebras)
[Google Gemini\
-------------\
\
](https://portkey.ai/docs/integrations/llms/gemini)
[Vertex AI\
---------\
\
](https://portkey.ai/docs/integrations/llms/vertex-ai)
[Cohere\
------\
\
](https://portkey.ai/docs/integrations/llms/cohere)
[Perplexity AI\
-------------\
\
](https://portkey.ai/docs/integrations/llms/perplexity-ai)
[AI21\
----\
\
](https://portkey.ai/docs/integrations/llms/ai21)
[Anyscale\
--------\
\
](https://portkey.ai/docs/integrations/llms/anyscale-llama2-mistral-zephyr)
[Byollm\
------](https://portkey.ai/docs/integrations/llms/byollm)
[Dashscope\
---------\
\
](https://portkey.ai/docs/integrations/llms/dashscope)
[Deepbricks\
----------\
\
](https://portkey.ai/docs/integrations/llms/deepbricks)
[DeepInfra\
---------\
\
](https://portkey.ai/docs/integrations/llms/deepinfra)
[DeepSeek\
--------\
\
](https://portkey.ai/docs/integrations/llms/deepseek)
[Fireworks AI\
------------\
\
](https://portkey.ai/docs/integrations/llms/fireworks)
[Featherless AI\
--------------\
\
](https://portkey.ai/docs/integrations/llms/featherless)
[Github Models\
-------------\
\
](https://portkey.ai/docs/integrations/llms/github)
[Google Palm\
-----------\
\
](https://portkey.ai/docs/integrations/llms/google-palm)
[Groq\
----\
\
](https://portkey.ai/docs/integrations/llms/groq)
[Jina AI\
-------\
\
](https://portkey.ai/docs/integrations/llms/jina-ai)
[Lambda\
------\
\
](https://portkey.ai/docs/integrations/llms/lambda)
[Lemonfox AI\
-----------\
\
](https://portkey.ai/docs/integrations/llms/lemon-fox)
[Lepton AI\
---------\
\
](https://portkey.ai/docs/integrations/llms/lepton)
[Lingyi (01.ai)\
--------------\
\
](https://portkey.ai/docs/integrations/llms/lingyi-01.ai)
[LocalAI\
-------\
\
](https://portkey.ai/docs/integrations/llms/local-ai)
[Mistral AI\
----------\
\
](https://portkey.ai/docs/integrations/llms/mistral-ai)
[MonsterAPI\
----------\
\
](https://portkey.ai/docs/integrations/llms/monster-api)
[Moonshot\
--------\
\
](https://portkey.ai/docs/integrations/llms/moonshot)
[nCompass\
--------\
\
](https://portkey.ai/docs/integrations/llms/ncompass)
[Nebius\
------\
\
](https://portkey.ai/docs/integrations/llms/nebius)
[Nomic AI\
--------\
\
](https://portkey.ai/docs/integrations/llms/nomic)
[Nscale\
------\
\
](https://portkey.ai/docs/integrations/llms/nscale)
[Ollama\
------\
\
](https://portkey.ai/docs/integrations/llms/ollama)
[Openrouter\
----------\
\
](https://portkey.ai/docs/integrations/llms/openrouter)
[Predibase\
---------\
\
](https://portkey.ai/docs/integrations/llms/predibase)
[Recraft AI\
----------\
\
](https://portkey.ai/docs/integrations/llms/recraft-ai)
[Siliconflow\
-----------\
\
](https://portkey.ai/docs/integrations/llms/siliconflow)
[Snowflake Cortex\
----------------\
\
](https://portkey.ai/docs/integrations/llms/snowflake-cortex)
[Stability AI\
------------\
\
](https://portkey.ai/docs/integrations/llms/stability-ai)
[Together AI\
-----------\
\
](https://portkey.ai/docs/integrations/llms/together-ai)
[Triton\
------\
\
](https://portkey.ai/docs/integrations/llms/triton)
[Upstage\
-------\
\
](https://portkey.ai/docs/integrations/llms/upstage)
[Voyage-AI\
---------\
\
](https://portkey.ai/docs/integrations/llms/voyage-ai)
[Workers AI\
----------\
\
](https://portkey.ai/docs/integrations/llms/workers-ai)
[ZhipuAI\
-------\
\
](https://portkey.ai/docs/integrations/llms/zhipu)
[](https://portkey.ai/docs/integrations/llms#endpoints-supported)
Endpoints Supported
-----------------------------------------------------------------------------------------
For a detailed breakdown of supported endpoints and features per provider, refer to the table below:[Supported Providers\
-------------------](https://portkey.ai/docs/provider-endpoints/supported-providers)
[](https://portkey.ai/docs/integrations/llms#supported-frameworks)
Supported Frameworks
-------------------------------------------------------------------------------------------
Portkey has native integrations with the following frameworks. Click to read their getting started guides.
[Langchain\
---------\
\
](https://portkey.ai/docs/integrations/libraries/langchain-python)
[LlamaIndex\
----------\
\
](https://portkey.ai/docs/integrations/libraries/llama-index-python)
[Autogen\
-------\
\
](https://portkey.ai/docs/integrations/libraries/autogen)
[Vercel\
------\
\
](https://portkey.ai/docs/integrations/libraries/vercel)
[Instructor\
----------\
\
](https://portkey.ai/docs/integrations/libraries/instructor)
[DSPy\
----\
\
](https://portkey.ai/docs/integrations/libraries/dspy)
[Promptfoo\
---------\
\
](https://portkey.ai/docs/integrations/libraries/promptfoo)
Have a suggestion for an integration with Portkey? Tell us on [Discord](https://discord.gg/DD7vgKK299)
, or drop a message on [\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#dfacaaafafb0adab9fafb0adabb4baa6f1beb6)
. While you’re here, why not [give us a star](https://git.new/ai-gateway-docs)
? It helps us a lot!
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/integrations/llms.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/integrations/llms)
[Submit an Integration](https://portkey.ai/docs/integrations/partner)
[OpenAI](https://portkey.ai/docs/integrations/llms/openai)
⌘I
---
# June - Portkey Docs
[Skip to main content](https://portkey.ai/docs/changelog/2025/june#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
2025
June
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Summary](https://portkey.ai/docs/changelog/2025/june#summary)
* [🎉 Introducing the Model Catalog 🎉](https://portkey.ai/docs/changelog/2025/june#%F0%9F%8E%89-introducing-the-model-catalog-%F0%9F%8E%89)
* [Circuit breaker](https://portkey.ai/docs/changelog/2025/june#circuit-breaker)
* [OpenAI Agent SDK (TypeScript)](https://portkey.ai/docs/changelog/2025/june#openai-agent-sdk-typescript)
* [Strands Agents SDK](https://portkey.ai/docs/changelog/2025/june#strands-agents-sdk)
* [Gateway & Providers](https://portkey.ai/docs/changelog/2025/june#gateway-%26-providers)
* [New models and providers](https://portkey.ai/docs/changelog/2025/june#new-models-and-providers)
* [Integrations](https://portkey.ai/docs/changelog/2025/june#integrations)
* [Partnerships](https://portkey.ai/docs/changelog/2025/june#partnerships)
* [Portkey Live!](https://portkey.ai/docs/changelog/2025/june#portkey-live)
* [Resources](https://portkey.ai/docs/changelog/2025/june#resources)
* [Community Contributors](https://portkey.ai/docs/changelog/2025/june#community-contributors)
* [Coming this month!](https://portkey.ai/docs/changelog/2025/june#coming-this-month)
* [Support](https://portkey.ai/docs/changelog/2025/june#support)
June was a big one. We launched **Model Catalog**, one of our most requested features from enterprise users, and something we’ve spent the last few weeks building, testing, and refining. It’s now the central control plane for managing every model your teams use through Portkey. Along with that, we rolled out several core updates across the platform and gateway including OpenAI Agents TS SDK support, circuit breaker, global endpoints for Vertex AI, expanded Azure coverage, and more. June also saw increased GitHub momentum as we rolled out one of our largest infrastructure upgrades to date! [](https://portkey.sh/home-git)
Here’s everything that went live in June.
[](https://portkey.ai/docs/changelog/2025/june#summary)
Summary
-------------------------------------------------------------------
| Area | Key Highlights |
| --- | --- |
| **Platform** | • Model Catalog launch
• OpenAI background mode
• Circuit breaker config |
| **Gateway & Providers** | • Support for Sutra, o3-pro, Magistral, and Gemini 2.5 models
• Vertex AI global endpoints
• Anthropic Computer Use tool support
• Support for additional Azure OpenAI endpoints
• Bedrock Inference Profiles
• Prompt caching for tools (Anthropic) |
| **Integrations** | • OpenAI Agent SDK (TypeScript)
• Langroid native support
• Strands SDK & ADK integration
• Cursor integration
• Gemini CLI support |
* * *
[](https://portkey.ai/docs/changelog/2025/june#%F0%9F%8E%89-introducing-the-model-catalog-%F0%9F%8E%89)
🎉 Introducing the Model Catalog 🎉
-----------------------------------------------------------------------------------------------------------------------------------------------
Take control of which models your teams can use across every provider, from one place. With Model Catalog, you can:
* Manage access to 1,600+ models across OpenAI, Anthropic, Azure, and more
* Decide which models are available to which teams
* Set rate limits and budgets per workspace
* Automatically enable new models as they’re released
Your browser does not support the video tag.
No more scattered configuration or manual provisioning. Just clean, centralized governance for all your AI usage. To enable it in your organizattion, reach out to us at [\[email protected\]](https://portkey.ai/cdn-cgi/l/email-protection#ff8c8a8f8f908d8bbf8f908d8b949a86d19e96)
[](https://portkey.ai/docs/changelog/2025/june#circuit-breaker)
Circuit breaker
-----------------------------------------------------------------------------------
When a model starts failing or slowing down, Portkey can now temporarily route traffic to fallback targets — then automatically restore routing once things stabilize. It’s a smarter way to handle failovers without manual intervention. Available as a config option. [Learn more ->](https://portkey.ai/docs/product/ai-gateway/circuit-breaker)
[](https://portkey.ai/docs/changelog/2025/june#openai-agent-sdk-typescript)
OpenAI Agent SDK (TypeScript)
-------------------------------------------------------------------------------------------------------------
With the OpenAI Agents SDK (TS), building agents in TypeScript just got easier - no Python, no context switching, just native tooling and agent workflows. But turning those prototypes into production-ready systems reveals real gaps:
* No logging or tracing
* No retries or failover
* No cost visibility or access control
* No simple way to switch LLM providers
That’s where Portkey comes in. Route your calls through Portkey and unlock observability, guardrails, prompt versioning, and multi-provider support instantly. [Learn more ->](https://portkey.ai/docs/integrations/agents/openai-agents-ts)
[](https://portkey.ai/docs/changelog/2025/june#strands-agents-sdk)
Strands Agents SDK
-----------------------------------------------------------------------------------------
Enterprises building on Strands Agents SDK now have a cleaner path to production — thanks to Portkey as the LLM gateway. With a single abstraction layer, you can:
* Access 2,000+ LLMs with provider-agnostic logic
* Standardize tool calling across providers
* Add conditional routing for cost control
* Enable retries, rate limits, and full observability
Huge thanks to Federico Kamelhar for leading the integration effort and bringing Portkey support to Strands. If you love Strands & Portkey **[contribute to this PR](https://portkey.sh/L1tDBwa)
** and help us stabilize this integration. [](https://github.com/strands-agents/sdk-python/pull/197)
[](https://portkey.ai/docs/changelog/2025/june#gateway-%26-providers)
Gateway & Providers
---------------------------------------------------------------------------------------------
**AWS Bedrock inference profiles** You can now route Bedrock requests by inference profile, allowing access to multiple regions or compute configurations through a single virtual key. [Learn more ->](https://portkey.ai/docs/integrations/llms/bedrock#inference-profiles)
. **Vertex AI global endpoint support** Portkey helps you improve model availability and reduce 429 (rate limit) errors with support for Vertex AI’s global endpoints. You can now set `region = global` in your Vertex AI Virtual Key config to automatically access Google’s distributed infrastructure — no manual region selection needed. **OpenAI Background Mode** Reasoning models can take minutes to solve complex problems. With background mode, you can now run long-running tasks on models like `o3-pro` and `o1-pro` reliably, without worrying about timeouts or dropped connections. Portkey now supports background mode for OpenAI requests. Simply pass `background:True` as a parameter, and Portkey will handle the rest. **Anthropic’s Computer Use** Experiment confidently with Anthropic’s new Computer Use tool by adding observability, fallback logic, and cost controls from day one. [Learn more ->](https://portkey.ai/docs/integrations/libraries/anthropic-computer-use)
**Support for additional Azure OpenAI endpoints** Portkey now supports a wider range of Azure OpenAI endpoints including image generation, audio transcription, speech synthesis, file management, and batch operations. **Anthropic prompt caching (tools)** Anthropic tool-based interactions now benefit from prompt caching in Portkey, improving performance and reducing token use for repeated tool calls. **To keeping up the pace!**
[](https://portkey.ai/docs/changelog/2025/june#new-models-and-providers)
New models and providers
-----------------------------------------------------------------------------------------------------
* **Sutra**: Multilingual LLM with standout MMLU scores in Hindi & Gujarati
* **Magistral**: Mistral’s first reasoning model, built for multilingual and domain-specific logic
* **o3-pro**: OpenAI’s latest flagship model with strong reasoning and fast response times
* **Gemini 2.5 (Flash, Pro, Flash-Lite)**: Now GA and supported with full observability
* **Kluster AI**: Claude-compatible, MCP-enabled models optimized for low latency and high availability.
* **Hyperbolic AI**: OpenAI-compatible models focused on cost efficiency and speed for production-scale usage.
* **Featherless AI**: Serverless access to Hugging Face models with a lightweight setup.
* **Groq**: Support for `service_tier` flag added
[](https://portkey.ai/docs/changelog/2025/june#integrations)
Integrations
-----------------------------------------------------------------------------
[OpenAI Agent SDK (TypeScript)\
-----------------------------\
\
Portkey now works with OpenAI’s new TypeScript Agents SDK. Add retries, observability, rate limits, and multi-provider support to your agent flows, without changing core logic.](https://portkey.ai/docs/integrations/agents/openai-agents-sdk)
[Langroid\
--------\
\
Langroid now supports Portkey natively — plug it in to get multi-provider routing, observability, fallback logic, and guardrails in your agentic Python apps.](https://portkey.ai/docs/integrations/agents/langroid)
Cursor
------
Bring visibility and governance to Cursor’s coding assistant with Portkey. Choose from 1600+ models, track requests, enforce rate limits, and log usage, making it easy for enterprises to govern Cursor across the org.
Agent Development Kit (ADK)
---------------------------
Portkey now integrates with Google’s Agent Development Kit (ADK), bringing production-grade features like retries, fallback logic, and observability to ADK-based agents.
Langflow
--------
Add enterprise-grade features to your Langflow workflows with Portkey — including unified model access, full observability, usage governance, and security guardrails.
[](https://portkey.ai/docs/changelog/2025/june#partnerships)
Partnerships
-----------------------------------------------------------------------------
.png?w=2500&fit=max&auto=format&n=QFjngWBmb6CT7QXC&q=85&s=db492841fa8ec2cacdb84ecc4a512a35)
* **Prompt Security** – Secure every prompt and response in real time by embedding Prompt Security directly into Portkey’s AI Gateway. [Read more here](https://portkey.ai/blog/why-llm-security-is-non-negotiable/)
* **Lasso Security** – Combine infra-level controls and real-time behavioral monitoring to secure the entire LLM lifecycle — from access to output. [Read more here](https://portkey.ai/blog/how-to-secure-your-entire-llm-lifecycle)
* **FutureAGI** – Use Portkey as the control layer and FutureAGI as the eval layer to automate output scoring across all model traffic. [See how you can implement this](https://portkey.ai/docs/integrations/tracing-providers/future-agi)
* **Arize AI** – Connect Portkey’s routing and guardrails with Arize’s observability to monitor model drift, latency, cost, and quality in one flow. [Read more here](https://portkey.ai/docs/integrations/tracing-providers/arize)
[](https://portkey.ai/docs/changelog/2025/june#portkey-live)
Portkey Live!
------------------------------------------------------------------------------
In partnership with [**Pangea**](https://pangea.cloud/)
, we hosted a live webinar on how to build scalable, secure GenAI infrastructure. Catch the replay here!.
**Improvements**
* Renamed Model Whitelist to Allowed Models for clarity and consistency
* Improved error responses for webhook failures, making them easier to debug and handle programmatically
**Teams love Portkey!**
If you love Portkey, drop a ⭐ on [GitHub](https://portkey.sh/home-git)
[](https://portkey.ai/docs/changelog/2025/june#resources)
Resources
-----------------------------------------------------------------------
* Cookbook: [Optimizing Prompts with LLama Prompt Ops](https://portkey.ai/docs/guides/prompts/llama-prompts)
* Cookbook: [OpenAI Computer Use Tool](https://portkey.ai/docs/guides/use-cases/openai-computer-use#portkey-with-openai-computer-use)
* Blog: [Building AI agent workflows with the help of an MCP gateway](https://portkey.ai/blog/building-ai-agent-workflows-with-the-help-of-an-mcp-gateway/)
* Blog: [Balancing AI model accuracy, performance, and costs](https://portkey.ai/blog/balancing-model-accuracy-performance-and-costs-with-an-ai-gateway/)
**Docs are now open for contributions!** We’re opening up our documentation for contributions. If you’ve found parts that could be clearer, better explained, or just more complete, we’d truly appreciate your help. Every suggestion, edit, or fix helps make Portkey better for the whole community. [See how you can contribute](https://portkey.ai/docs/README)
[](https://portkey.ai/docs/changelog/2025/june#community-contributors)
Community Contributors
-------------------------------------------------------------------------------------------------
A special thanks to our community contributors this month:
* [Shubhwithai](https://github.com/Shubhwithai)
* [DarinVerheijke](https://github.com/DarinVerheijke)
* [jroberts2600](https://github.com/jroberts2600)
[](https://portkey.ai/docs/changelog/2025/june#coming-this-month)
Coming this month!
----------------------------------------------------------------------------------------
Struggling with unauthorized tool usage in MCP? Portkey is about to solve that. Stay tuned.
[](https://portkey.ai/docs/changelog/2025/june#support)
Support
-------------------------------------------------------------------
[Need Help?\
----------\
\
Open an issue on GitHub](https://github.com/Portkey-AI/gateway/issues)
[Join Us\
-------\
\
Get support in our Discord](https://portkey.wiki/community)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/changelog/2025/june.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/changelog/2025/june)
[July](https://portkey.ai/docs/changelog/2025/july)
[May](https://portkey.ai/docs/changelog/2025/may)
⌘I
---
# Universal API - Portkey Docs
[Skip to main content](https://portkey.ai/docs/product/ai-gateway/universal-api#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
AI Gateway
Universal API
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Portkey Follows OpenAI Spec](https://portkey.ai/docs/product/ai-gateway/universal-api#portkey-follows-openai-spec)
* [Switching Providers is a Breeze](https://portkey.ai/docs/product/ai-gateway/universal-api#switching-providers-is-a-breeze)
* [Integrating Local or Private Models](https://portkey.ai/docs/product/ai-gateway/universal-api#integrating-local-or-private-models)
* [Powerful Routing and Fallback Strategies](https://portkey.ai/docs/product/ai-gateway/universal-api#powerful-routing-and-fallback-strategies)
* [Using the Anthropic’s /messages Route](https://portkey.ai/docs/product/ai-gateway/universal-api#using-the-anthropic%E2%80%99s-%2Fmessages-route)
* [Multimodality](https://portkey.ai/docs/product/ai-gateway/universal-api#multimodality)
* [Supported Endpoints](https://portkey.ai/docs/product/ai-gateway/universal-api#supported-endpoints)
* [Core Endpoints](https://portkey.ai/docs/product/ai-gateway/universal-api#core-endpoints)
* [Advanced Capabilities](https://portkey.ai/docs/product/ai-gateway/universal-api#advanced-capabilities)
* [Additional Endpoints](https://portkey.ai/docs/product/ai-gateway/universal-api#additional-endpoints)
This feature is available on all Portkey plans.
So, instead of maintaining separate integrations for different multimodal LLMs, you can interact with models from OpenAI, Anthropic, Meta, Cohere, Mistral, and many more (100+ models, 15+ providers) - all using a common, unified API signature.
[](https://portkey.ai/docs/product/ai-gateway/universal-api#portkey-follows-openai-spec)
Portkey Follows OpenAI Spec
------------------------------------------------------------------------------------------------------------------------
Portkey API is powered by its [battle-tested open-source AI Gateway](https://github.com/portkey-ai/gateway)
, which converts all incoming requests to the OpenAI signature and returns OpenAI-compliant responses.
[](https://portkey.ai/docs/product/ai-gateway/universal-api#switching-providers-is-a-breeze)
Switching Providers is a Breeze
--------------------------------------------------------------------------------------------------------------------------------
* Node
* Python
Copy
Ask AI
import Portkey from 'portkey-ai';
// Calling OpenAI
const portkey = new Portkey({
provider: "openai",
Authorization: "Bearer sk-xxxxx"
})
const response = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Hello' }],
model: 'gpt-4',
});
// Swithing to Anthropic
const portkey = new Portkey({
provider: "anthropic",
Authorization: "Bearer sk-ant-xxxxx"
})
const response = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Hello' }],
model: 'claude-3-opus-20240229',
});
[](https://portkey.ai/docs/product/ai-gateway/universal-api#integrating-local-or-private-models)
Integrating Local or Private Models
----------------------------------------------------------------------------------------------------------------------------------------
Portkey can also route to and observe your locally or privately hosted LLMs, as long as the model is compliant with one of the 15+ providers supported by Portkey and the URL is exposed publicly. Simply specify the `custom_host` parameter along with the `provider` name, and Portkey will handle the communication with your local model.
* NodeJS
* Python
* cURL
Copy
Ask AI
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
provider: "mistral-ai",
customHost: "http://MODEL_URL/v1/" // Point Portkey to where the model is hosted
})
async function main(){
const response = await portkey.chat.completions.create({
messages: [{ role: 'user', content: '1729' }],
model: 'mixtral-8x22b',
});
console.log(response)
}
main()
**Note:**When using `custom_host`, include the version identifier (e.g., `/v1`) in the URL. Portkey will append the actual endpoint path (`/chat/completions`, `/completions`, or `/embeddings`) automatically. (For Ollama models, this works differently. [Check here](https://portkey.ai/docs/integrations/llms/ollama)
)
[](https://portkey.ai/docs/product/ai-gateway/universal-api#powerful-routing-and-fallback-strategies)
Powerful Routing and Fallback Strategies
--------------------------------------------------------------------------------------------------------------------------------------------------
With Portkey you can implement sophisticated routing and fallback strategies. Route requests to different providers based on various criteria, loadbalance them, set up retries or fallbacks to alternative models in case of failures or resource constraints. Here’s an example config where we set up a fallback from OpenAI to a locally hosted Llama3 on Ollama:
Copy
Ask AI
config = {
"strategy": { "mode": "loadbalance" },
"targets": [\
{\
"provider": "openai",\
"api_key": "xxx",\
"weight": 1,\
"override_params": { "model": "gpt-3.5-turbo" }\
},\
{\
"provider": "mistral-ai",\
"custom_host": "http://MODEL_URL/v1/",\
"weight": 1,\
"override_params": { "model": "mixtral-8x22b" }\
}\
]
}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
config=config
)
[](https://portkey.ai/docs/product/ai-gateway/universal-api#using-the-anthropic%E2%80%99s-%2Fmessages-route)
Using the Anthropic’s /messages Route
------------------------------------------------------------------------------------------------------------------------------------------------------
Access models on [Anthropic](https://portkey.ai/docs/integrations/llms/anthropic)
, [Bedrock](https://portkey.ai/docs/integrations/llms/bedrock)
and [Vertex AI](https://portkey.ai/docs/integrations/llms/vertex-ai)
through Anthropic’s native`/messages` endpoint using Portkey’s SDK or Anthropic’s SDK.
This route only works with Claude models on Anthropic, Bedrock and Vertex AI. For other models, use the standard OpenAI compliant endpoint.
* cURL
* Python SDK
* NodeJS SDK
* Anthropic Python SDK
* Anthropic TypeScript SDK
Copy
Ask AI
curl --location 'https://api.portkey.ai/v1/messages' \
--header 'x-portkey-provider: @your-provider-slug' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--data '{
"model": "your-model-name",
"max_tokens": 250,
"messages": [\
{\
"role": "user",\
"content": "Hello, Claude"\
}\
]
}'
[](https://portkey.ai/docs/product/ai-gateway/universal-api#multimodality)
Multimodality
--------------------------------------------------------------------------------------------
Portkey integrates with multimodal models through the same unified API and supports vision, audio, image generation, and more capabilities across providers. [Multimodal Capabilities](https://portkey.ai/docs/product/ai-gateway/multimodal-capabilities)
[](https://portkey.ai/docs/product/ai-gateway/universal-api#supported-endpoints)
Supported Endpoints
--------------------------------------------------------------------------------------------------------
Portkey’s Universal API supports a comprehensive range of endpoints across all major AI capabilities. Each endpoint follows the OpenAI specification while working seamlessly with 15+ providers.
Not all providers support every endpoint. Check our [provider compatibility matrix](https://portkey.ai/docs/api-reference/inference-api/supported-providers)
to see which endpoints are available for each provider.
###
[](https://portkey.ai/docs/product/ai-gateway/universal-api#core-endpoints)
Core Endpoints
* **[Chat Completions](https://portkey.ai/docs/api-reference/inference-api/chat)
** - An OpenAI compatible unified endpoint for generating responses with support for streaming, function calling, and multi-modal inputs across providers 50+ providers like OpenAI, Anthropic, Vertex AI, Bedrock, and more
* **[Responses](https://portkey.ai/docs/api-reference/inference-api/responses)
** - An OpenAI compatible unified endpoint for generating responses with support for OpenAI models exclusively on responses API
* **[Messages](https://portkey.ai/docs/api-reference/inference-api/messages)
** - An Anthropic-compatible unified endpoint for generating responses with support for streaming, function calling, and multi-modal inputs across providers like Anthropic, Vertex AI, Bedrock, and more.
* **[Images](https://portkey.ai/docs/api-reference/inference-api/images)
** - Generate, edit, and create variations of images using models like DALL-E, Stable Diffusion, and others
* **[Audio](https://portkey.ai/docs/api-reference/inference-api/audio)
** - Convert speech to text (transcription) and text to speech across multiple languages
###
[](https://portkey.ai/docs/product/ai-gateway/universal-api#advanced-capabilities)
Advanced Capabilities
* **[Fine-tuning](https://portkey.ai/docs/product/ai-gateway/fine-tuning)
** - Customize models on your specific datasets
* **[Batch Processing](https://portkey.ai/docs/product/ai-gateway/batches)
** - Process large volumes of requests efficiently
* **[Files](https://portkey.ai/docs/product/ai-gateway/files)
** - Upload and manage files for fine-tuning and batch operations
* **[Moderations](https://portkey.ai/docs/api-reference/inference-api/moderations)
** - Check content for safety and compliance
###
[](https://portkey.ai/docs/product/ai-gateway/universal-api#additional-endpoints)
Additional Endpoints
* **[Gateway to other APIs](https://portkey.ai/docs/api-reference/inference-api/gateway-for-other-apis)
** - Gateway to other APIs
* **[Assistants](https://portkey.ai/docs/product/ai-gateway/assistants)
** - OpenAI assistants with persistent threads and file handling
* **[Completions](https://portkey.ai/docs/api-reference/inference-api/completions)
** - Legacy text completion endpoint for backward compatibility
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/product/ai-gateway/universal-api.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/product/ai-gateway/universal-api)
[AI Gateway](https://portkey.ai/docs/product/ai-gateway)
[Configs](https://portkey.ai/docs/product/ai-gateway/configs)
⌘I
---
# Enterprise Architecture - Portkey Docs
[Skip to main content](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Hybrid Deployments
Enterprise Architecture
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Core Architecture Components](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#core-architecture-components)
* [Data Plane (Your VPC)](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-plane-your-vpc)
* [Control Plane (Portkey VPC)](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#control-plane-portkey-vpc)
* [Data Flow Between Planes](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-flow-between-planes)
* [Deployment Architecture](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#deployment-architecture)
* [Infrastructure Components](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#infrastructure-components)
* [Storage Options](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#storage-options)
* [Authentication Methods](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#authentication-methods)
* [Infrastructure Requirements](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#infrastructure-requirements)
* [Data Security & Encryption](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-security-%26-encryption)
* [Advantages of Hybrid Architecture](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#advantages-of-hybrid-architecture)
* [Technical Rationale](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#technical-rationale)
* [Sample Files](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#sample-files)
* [Resources & Next Steps](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#resources-%26-next-steps)
* [Have Questions?](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#have-questions%3F)
Portkey Enterprise offers a **secure hybrid deployment model** that balances security, flexibility, and fast deployment timelines:
* **Data Plane** runs within your VPC, keeping sensitive LLM data and AI traffic in your environment
* **Control Plane** hosted by Portkey handles administration, configs, and analytics
Portkey Enterprise Hybrid Deployment Architecture
[Schedule an Enterprise Architecture Demo\
----------------------------------------\
\
Want to learn more about our hybrid deployment model? Schedule a personalized demo with our solutions team to see how Portkey Enterprise can fit your security and compliance requirements.](https://portkey.wiki/demo-27)
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#core-architecture-components)
Core Architecture Components
--------------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-plane-your-vpc)
Data Plane (Your VPC)
The Data Plane is deployed in your cloud environment and processes all your AI traffic:
| Component | Description | Security Benefit |
| --- | --- | --- |
| **AI Gateway** | Core engine that routes traffic across LLM providers and implements metering, access control, and guardrails | All LLM requests remain in your network perimeter |
| **Cache Store** | Local cache storage for gateway consumption | Eliminates runtime dependency on Control Plane |
| **Data Store** | Storage for LLM request/response logs | Keep sensitive LLM data completely in your environment |
The AI Gateway runs as containerized workloads in your infrastructure, deployable via your preferred orchestration method (Kubernetes, ECS, etc.).
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#control-plane-portkey-vpc)
Control Plane (Portkey VPC)
The Control Plane is fully managed by Portkey and provides the administrative layer for your deployment:
* Hosts the web dashboard for managing configurations, tracking analytics, and viewing logs
* Maintains routing configs, provider integrations
* Stores non-sensitive metadata and aggregated metrics
* Automatically updates with new features and provider integrations without requiring changes to your infrastructure
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-flow-between-planes)
Data Flow Between Planes
------------------------------------------------------------------------------------------------------------------------------
AI Traffic Flow (Fully Contained in Your VPC)
All LLM traffic stays within your network boundary:
1. Your application sends requests to the AI Gateway
2. The Gateway processes the request (applying routing, caching, guardrails)
3. The Gateway forwards the request to the appropriate LLM provider
4. Responses from LLMs return through the same path
**Security Benefit**: Complete isolation of sensitive prompt data and responses
Data Sync (Gateway ↔ Control Plane)
The AI Gateway periodically synchronizes with the Control Plane:
* **Frequency**: 30-second heartbeat intervals
* **Data Retrieved**: Prompt templates, routing configs, integrations, providers, API keys
* **Process**: Data is fetched, decrypted locally, and stored in the Gateway cache
* **Resilience**: Gateway operates independently between syncs using cached configs
**Security Benefit**: Continuous operation even during Control Plane disconnection
Analytics Flow (Gateway → Control Plane)
The Gateway sends anonymized metrics to the Analytics Store:
* **Data Sent**: Non-sensitive operational metrics (model used, token counts, response times)
* **Purpose**: Powers analytics dashboards for monitoring performance and costs
* **Example**: [View sample analytics data](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#sample-files)
**Security Benefit**: Provides insights without exposing sensitive information
Log Management Options
**Option A: Logs in Your VPC** (Recommended for high-security environments)
* Logs stored in your environment’s Blob Store
* When viewing logs in Dashboard UI, Control Plane requests them from Gateway
**Option B: Logs in Portkey Cloud**
* Gateway encrypts and sends logs to Portkey Log Store
* No connections required from Portkey to your environment for viewing logs
**Security Benefit**: Flexibility to match your compliance requirements
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#deployment-architecture)
Deployment Architecture
----------------------------------------------------------------------------------------------------------------------------
Portkey AI Gateway is deployed as containerized workloads using Helm charts for Kubernetes environments, with flexible deployment options for various cloud providers.
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#infrastructure-components)
Infrastructure Components
| Component | Description | Configuration Options |
| --- | --- | --- |
| **AI Gateway** | Core container running the routing logic | Deployed as stateless containers that can scale horizontally |
| **Cache System** | Stores routing configs, integrations, providers, and more | Redis (in-cluster, AWS ElastiCache, or custom endpoint) |
| **Log Storage** | Persistence for request/response data | Multiple options (see below) |
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#storage-options)
Storage Options
* Object Storage
* Document DB
S3-compatible storage options including:
* AWS S3 (standard credentials or assumed roles)
* Google Cloud Storage (S3 compatible interoperability mode)
* Azure Blob Storage (key, managed identity, or Entra ID)
* Any S3-compatible Blob Storage
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#authentication-methods)
Authentication Methods
* Cloud Provider IAM
* Direct Authentication
* IAM roles for service accounts (IRSA) in Kubernetes
* Instance Metadata Service (IMDS) for EC2/ECS
* Managed identities in Azure environments
###
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#infrastructure-requirements)
Infrastructure Requirements
* **Kubernetes Cluster**: K8s 1.20+ with Helm 3.x
* **Outbound Network**: HTTPS access to Control Plane endpoints
* **Container Registry Access**: For pulling gateway container images
* **Recommended Resource Requirements**:
* CPU: 1-2 cores per gateway instance
* Memory: 2-4GB per gateway instance
* Storage: Dependent on logging configuration
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#data-security-%26-encryption)
Data Security & Encryption
------------------------------------------------------------------------------------------------------------------------------------
* Data Residency
* Encryption Methods
* Access Controls
**Your Sensitive Data Stays in Your VPC**
* All prompt content and LLM responses remain within your network
* Only anonymized metrics data cross network boundaries
* Log storage location is configurable based on your requirements
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#advantages-of-hybrid-architecture)
Advantages of Hybrid Architecture
------------------------------------------------------------------------------------------------------------------------------------------------
| Benefit | Technical Implementation | Business Value |
| --- | --- | --- |
| **Security & Compliance** | \- Sensitive data never leaves VPC
\- Configurable encryption methods
\- Flexible authentication options | \- Meets data residency requirements
\- Supports regulated industries
\- Simplifies security reviews |
| **Operational Efficiency** | \- No database management overhead
\- Automatic model config updates
\- Horizontally scalable architecture | \- Low operational burden
\- Always up-to-date with LLM ecosystem
\- Scales with your traffic patterns |
| **Deployment Flexibility** | \- Kubernetes-native deployment
\- Support for major cloud providers
\- Multiple storage backend options | \- Fits into existing infrastructure
\- Avoids vendor lock-in
\- Customizable to specific needs |
| **Developer Experience** | \- OpenAI-compatible API
\- Simple integration patterns
\- Comprehensive observability | \- Minimal code changes needed
\- Smooth developer onboarding
\- Full visibility into system behavior |
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#technical-rationale)
Technical Rationale
--------------------------------------------------------------------------------------------------------------------
Why Maintain Transaction DB in Control Plane?
1. **Real-time Model Updates**: LLM providers frequently change model parameters, pricing, and availability. Centralizing this data ensures all gateways operate with current information.
2. **Feature Velocity**: AI landscape evolves rapidly. Control Plane architecture allows Portkey to deliver new features multiple times per week without requiring customer-side deployments.
3. **Operational Efficiency**: Eliminates need for customers to maintain complex database infrastructure solely for non-sensitive object management.
Why Cache Objects Locally?
1. **Performance**: Eliminates network latency during LLM requests by having all routing and configs data available locally.
2. **Resilience**: Gateway continues operating even if temporarily disconnected from Control Plane.
3. **Security**: Reduces attack surface by minimizing runtime external dependencies.
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#sample-files)
Sample Files
------------------------------------------------------------------------------------------------------
These samples demonstrate the typical data patterns flowing between systems:
[Sample Log File (4KB)\
---------------------](https://github.com/Portkey-AI/docs-core/tree/main/images/enterprise/private-cloud-deployments/architecture/sample_log_file.json)
[Sample Metric File (3KB)\
------------------------](https://github.com/Portkey-AI/docs-core/tree/main/images/enterprise/private-cloud-deployments/architecture/sample_metric_file.json)
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#resources-%26-next-steps)
Resources & Next Steps
----------------------------------------------------------------------------------------------------------------------------
[Helm Repository\
---------------](https://github.com/Portkey-AI/helm)
[Enterprise Changelog\
--------------------](https://portkey.ai/docs/changelog/enterprise)
[AWS Deployment Guide\
--------------------](https://portkey.ai/docs/product/enterprise-offering/private-cloud-deployments/aws)
[Azure Deployment Guide\
----------------------](https://portkey.ai/docs/product/enterprise-offering/private-cloud-deployments/azure)
[](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture#have-questions%3F)
Have Questions?
--------------------------------------------------------------------------------------------------------------
Our solution architects are available to discuss your specific deployment requirements and security needs.[Schedule Architecture Discussion\
--------------------------------\
\
Book a personalized consultation with our enterprise team to explore how Portkey’s architecture can be tailored to your organization’s specific requirements.](https://portkey.wiki/demo-27)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/self-hosting/hybrid-deployments/architecture.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/self-hosting/hybrid-deployments/architecture)
[Feature Comparison](https://portkey.ai/docs/product/product-feature-comparison)
[EKS](https://portkey.ai/docs/self-hosting/hybrid-deployments/aws/eks)
⌘I
---
# Observability (OpenTelemetry) - Portkey Docs
[Skip to main content](https://portkey.ai/docs/product/observability#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Observability
Observability (OpenTelemetry)
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Features](https://portkey.ai/docs/product/observability#features)
If you’re working with an LLM - visibility across all your requests can be a BIG pain. How do you trace and measure cost, latency, accuracy of your requests? Portkey’s OpenTelemetry-compliant observability suite gives you complete control over all your requests. And Portkey’s analytics dashboards provide the insights you’re looking for. Fast.
[](https://portkey.ai/docs/product/observability#features)
Features
-----------------------------------------------------------------------
[Logs\
----\
\
Portkey records all your multimodal requests and responses, making it easy to view, monitor, and debug interactions.](https://portkey.ai/docs/product/observability/logs)
[Tracing\
-------\
\
Portkey supports request tracing to help you monitor your applications throughout the lifecycle of a request.](https://portkey.ai/docs/product/observability/traces)
[Analytics\
---------\
\
A comprehensive view of 21+ key metrics. Use it to analyze data, spot trends, and make informed decisions.](https://portkey.ai/docs/product/observability/analytics)
[Filters\
-------\
\
Streamline your data view with customizable filters. Zero in on data that matters most.](https://portkey.ai/docs/product/observability/filters)
[Custom Metadata\
---------------\
\
Enrich your LLM APIs with custom metadata. Assign unique tags for swift grouping and troubleshooting.](https://portkey.ai/docs/product/observability/metadata)
[Feedback\
--------\
\
Add feedback values and weights to complete the loop.](https://portkey.ai/docs/product/observability/feedback)
[Budget Limits\
-------------\
\
Set up budget limits for your provider API keys and gain confidence over your application’s costs.](https://portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/product/observability.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/product/observability)
[Portkey Features](https://portkey.ai/docs/introduction/feature-overview)
[OpenTelemetry for LLM Observability](https://portkey.ai/docs/product/observability/opentelemetry)
⌘I
---
# OpenAI - Portkey Docs
[Skip to main content](https://portkey.ai/docs/integrations/llms/openai#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
OpenAI
OpenAI
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [Using the Portkey Gateway](https://portkey.ai/docs/integrations/llms/openai#using-the-portkey-gateway)
* [Using the Responses API](https://portkey.ai/docs/integrations/llms/openai#using-the-responses-api)
* [Track End-User IDs](https://portkey.ai/docs/integrations/llms/openai#track-end-user-ids)
* [Using the Prompts API](https://portkey.ai/docs/integrations/llms/openai#using-the-prompts-api)
* [Advanced Use Cases](https://portkey.ai/docs/integrations/llms/openai#advanced-use-cases)
* [Realtime API](https://portkey.ai/docs/integrations/llms/openai#realtime-api)
* [Streaming Responses](https://portkey.ai/docs/integrations/llms/openai#streaming-responses)
* [Streaming with the Responses API](https://portkey.ai/docs/integrations/llms/openai#streaming-with-the-responses-api)
* [Using Vision Models](https://portkey.ai/docs/integrations/llms/openai#using-vision-models)
* [Vision with the Responses API](https://portkey.ai/docs/integrations/llms/openai#vision-with-the-responses-api)
* [Function Calling](https://portkey.ai/docs/integrations/llms/openai#function-calling)
* [Function Calling with the Responses API](https://portkey.ai/docs/integrations/llms/openai#function-calling-with-the-responses-api)
* [Fine-Tuning](https://portkey.ai/docs/integrations/llms/openai#fine-tuning)
* [Image Generation](https://portkey.ai/docs/integrations/llms/openai#image-generation)
* [Audio - Transcription, Translation, and Text-to-Speech](https://portkey.ai/docs/integrations/llms/openai#audio-transcription%2C-translation%2C-and-text-to-speech)
* [Intergrated Tools with Repsponses API](https://portkey.ai/docs/integrations/llms/openai#intergrated-tools-with-repsponses-api)
* [Web Search Tool](https://portkey.ai/docs/integrations/llms/openai#web-search-tool)
* [File Search Tool](https://portkey.ai/docs/integrations/llms/openai#file-search-tool)
* [Enhanced Reasoning](https://portkey.ai/docs/integrations/llms/openai#enhanced-reasoning)
* [Computer Use Assistant](https://portkey.ai/docs/integrations/llms/openai#computer-use-assistant)
* [Managing OpenAI Projects & Organizations in Portkey](https://portkey.ai/docs/integrations/llms/openai#managing-openai-projects-%26-organizations-in-portkey)
* [Using Virtual Keys](https://portkey.ai/docs/integrations/llms/openai#using-virtual-keys)
* [Using The Gateway Config](https://portkey.ai/docs/integrations/llms/openai#using-the-gateway-config)
* [While Making a Request](https://portkey.ai/docs/integrations/llms/openai#while-making-a-request)
* [Portkey Features](https://portkey.ai/docs/integrations/llms/openai#portkey-features)
Portkey has native integrations with OpenAI SDKs for Node.js, Python, and its REST APIs. For OpenAI integration using other frameworks, explore our partnerships, including [Langchain](https://portkey.ai/docs/integrations/libraries/langchain-python)
, [LlamaIndex](https://portkey.ai/docs/integrations/libraries/llama-index-python)
, among [others](https://portkey.ai/docs/integrations/llms)
.
Provider Slug. `openai`
[](https://portkey.ai/docs/integrations/llms/openai#using-the-portkey-gateway)
Using the Portkey Gateway
------------------------------------------------------------------------------------------------------------
To integrate the Portkey gateway with OpenAI,
* Set the `baseURL` to the Portkey Gateway URL
* Include Portkey-specific headers such as `provider`, `apiKey`, ‘virtualKey’ and others.
Here’s how to apply it to a **chat completion** request:
* NodeJS
* Python
* cURL
* OpenAI Python SDK
* OpenAI NodeJS SDK
* Java
Install the Portkey SDK with npm
Copy
Ask AI
npm install portkey-ai
Chat Completions
Copy
Ask AI
import Portkey from 'portkey-ai';
const client = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider:'@PROVIDER'
});
async function main() {
const response = await client.chat.completions.create({
messages: [{ role: "user", content: "Bob the builder.." }],
model: "gpt-4o",
});
console.log(response.choices[0].message.content);
}
main();
This request will be automatically logged by Portkey. You can view this in your logs dashboard. Portkey logs the tokens utilized, execution time, and cost for each request. Additionally, you can delve into the details to review the precise request and response data.
Portkey supports [OpenAI’s new “developer” role](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages)
in chat completions. With o1 models and newer, the `developer` role replaces the previous `system` role.
###
[](https://portkey.ai/docs/integrations/llms/openai#using-the-responses-api)
Using the Responses API
OpenAI has released a new Responses API that combines the best of both Chat Completions and Assistants APIs. Portkey fully supports this new API, enabling you to use it with both the Portkey SDK and OpenAI SDK.
* Python
* NodeJS
* OpenAI Python
* OpenAI NodeJS
Copy
Ask AI
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@OPENAI_PROVIDER"
)
response = portkey.responses.create(
model="gpt-4.1",
input="Tell me a three sentence bedtime story about a unicorn."
)
print(response)
The Responses API provides a more flexible foundation for building agentic applications with built-in tools that execute automatically.
[Remote MCP support on Resposes API\
----------------------------------\
\
Portkey supports Remote MCP support by OpenAI on it’s Responses API. Learn More](https://portkey.ai/docs/product/ai-gateway/remote-mcp)
[](https://portkey.ai/docs/integrations/llms/openai#track-end-user-ids)
Track End-User IDs
----------------------------------------------------------------------------------------------
Portkey allows you to track user IDs passed with the `user` parameter in OpenAI requests, enabling you to monitor user-level costs, requests, and more.
* NodeJS
* Python
Copy
Ask AI
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: "user", content: "Say this is a test" }],
model: "gpt-4o",
user: "user_12345",
});
When you include the `user` parameter in your requests, Portkey logs will display the associated user ID, as shown in the image below:
In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.[Metadata\
--------](https://portkey.ai/docs/product/observability/metadata)
* The same integration approach applies to APIs for [completions](https://platform.openai.com/docs/guides/text-generation/completions-api)
, [embeddings](https://platform.openai.com/docs/api-reference/embeddings/create)
, [vision](https://platform.openai.com/docs/guides/vision/quick-start)
, [moderation](https://platform.openai.com/docs/api-reference/moderations/create)
, [transcription](https://platform.openai.com/docs/api-reference/audio/createTranscription)
, [translation](https://platform.openai.com/docs/api-reference/audio/createTranslation)
, [speech](https://platform.openai.com/docs/api-reference/audio/createSpeech)
and [files](https://platform.openai.com/docs/api-reference/files/create)
.
* If you are looking for a way to add your **Org ID** & **Project ID** to the requests, head over to [this section](https://portkey.ai/docs/integrations/llms/openai#managing-openai-projects-and-organizations-in-portkey)
.
[](https://portkey.ai/docs/integrations/llms/openai#using-the-prompts-api)
Using the Prompts API
----------------------------------------------------------------------------------------------------
Portkey also supports creating and managing prompt templates in the [prompt library](https://portkey.ai/docs/product/prompt-library)
. This enables the collaborative development of prompts directly through the user interface.
1. Create a prompt template with variables and set the hyperparameters.
2. Use this prompt in your codebase using the Portkey SDK.
* Node
* Python
* cURL
Copy
Ask AI
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
})
// Make the prompt creation call with the variables
const promptCompletion = await portkey.prompts.completions.create({
promptID: "Your Prompt ID",
variables: {
// The variables specified in the prompt
}
})
Copy
Ask AI
// We can also override the hyperparameters
const promptCompletion = await portkey.prompts.completions.create({
promptID: "Your Prompt ID",
variables: {
// The variables specified in the prompt
},
max_tokens: 250,
presence_penalty: 0.2
})
Observe how this streamlines your code readability and simplifies prompt updates via the UI without altering the codebase.
[](https://portkey.ai/docs/integrations/llms/openai#advanced-use-cases)
Advanced Use Cases
----------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/integrations/llms/openai#realtime-api)
Realtime API
Portkey supports OpenAI’s Realtime API with a seamless integration. This allows you to use Portkey’s logging, cost tracking, and guardrail features while using the Realtime API.[Realtime API\
------------](https://portkey.ai/docs/product/ai-gateway/realtime-api)
###
[](https://portkey.ai/docs/integrations/llms/openai#streaming-responses)
Streaming Responses
Portkey supports streaming responses using Server Sent Events (SSE).
* OpenAI NodeJS
* OpenAI Python
Copy
Ask AI
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'
const openai = new OpenAI({
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: "openai",
apiKey: "PORTKEY_API_KEY" // defaults to process.env["PORTKEY_API_KEY"]
})
});
async function main() {
const stream = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Say this is a test' }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
}
main();
####
[](https://portkey.ai/docs/integrations/llms/openai#streaming-with-the-responses-api)
Streaming with the Responses API
You can also stream responses from the Responses API:
* Python
* NodeJS
* OpenAI Python
* OpenAI NodeJS
Copy
Ask AI
response = portkey.responses.create(
model="gpt-4.1",
instructions="You are a helpful assistant.",
input="Hello!",
stream=True
)
for event in response:
print(event)
###
[](https://portkey.ai/docs/integrations/llms/openai#using-vision-models)
Using Vision Models
Portkey’s multimodal Gateway fully supports OpenAI vision models as well. See this guide for more info:
[Vision](https://portkey.ai/docs/product/ai-gateway/multimodal-capabilities/vision)
####
[](https://portkey.ai/docs/integrations/llms/openai#vision-with-the-responses-api)
Vision with the Responses API
You can also use the Responses API to process images alongside text:
* Python
* NodeJS
* OpenAI Python
* OpenAI NodeJS
Copy
Ask AI
response = portkey.responses.create(
model="gpt-4.1",
input=[\
{\
"role": "user",\
"content": [\
{ "type": "input_text", "text": "What is in this image?" },\
{\
"type": "input_image",\
"image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"\
}\
]\
}\
]
)
print(response)
###
[](https://portkey.ai/docs/integrations/llms/openai#function-calling)
Function Calling
Function calls within your OpenAI or Portkey SDK operations remain standard. These logs will appear in Portkey, highlighting the utilized functions and their outputs. Additionally, you can define functions within your prompts and invoke the `portkey.prompts.completions.create` method as above.
####
[](https://portkey.ai/docs/integrations/llms/openai#function-calling-with-the-responses-api)
Function Calling with the Responses API
The Responses API also supports function calling with the same powerful capabilities:
* Python
* NodeJS
* OpenAI Python
* OpenAI NodeJS
Copy
Ask AI
tools = [\
{\
"type": "function",\
"name": "get_current_weather",\
"description": "Get the current weather in a given location",\
"parameters": {\
"type": "object",\
"properties": {\
"location": {\
"type": "string",\
"description": "The city and state, e.g. San Francisco, CA"\
},\
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}\
},\
"required": ["location", "unit"]\
}\
}\
]
response = portkey.responses.create(
model="gpt-4.1",
tools=tools,
input="What is the weather like in Boston today?",
tool_choice="auto"
)
print(response)
###
[](https://portkey.ai/docs/integrations/llms/openai#fine-tuning)
Fine-Tuning
Please refer to our fine-tuning guides to take advantage of Portkey’s advanced [continuous fine-tuning](https://portkey.ai/docs/product/autonomous-fine-tuning)
capabilities.
###
[](https://portkey.ai/docs/integrations/llms/openai#image-generation)
Image Generation
Portkey supports multiple modalities for OpenAI and you can make image generation requests through Portkey’s AI Gateway the same way as making completion calls.
* OpenAI NodeJS
* OpenAI Python
Copy
Ask AI
// Define the OpenAI client as shown above
const image = await openai.images.generate({
model:"dall-e-3",
prompt:"Lucy in the sky with diamonds",
size:"1024x1024"
})
Portkey’s fast AI gateway captures the information about the request on your Portkey Dashboard. On your logs screen, you’d be able to see this request with the request and response.
Log view for an image generation request on OpenAI
More information on image generation is available in the [API Reference](https://portkey.ai/docs/provider-endpoints/images/create-image#create-image)
.
###
[](https://portkey.ai/docs/integrations/llms/openai#audio-transcription%2C-translation%2C-and-text-to-speech)
Audio - Transcription, Translation, and Text-to-Speech
Portkey’s multimodal Gateway also supports the `audio` methods on OpenAI API. Check out the below guides for more info: Check out the below guides for more info:
[Text-to-Speech](https://portkey.ai/docs/product/ai-gateway/multimodal-capabilities/text-to-speech)
[Speech-to-Text](https://portkey.ai/docs/product/ai-gateway/multimodal-capabilities/speech-to-text)
* * *
[](https://portkey.ai/docs/integrations/llms/openai#intergrated-tools-with-repsponses-api)
Intergrated Tools with Repsponses API
------------------------------------------------------------------------------------------------------------------------------------
###
[](https://portkey.ai/docs/integrations/llms/openai#web-search-tool)
Web Search Tool
Web search delivers accurate and clearly-cited answers from the web, using the same tool as search in ChatGPT:
* Python
* NodeJS
Copy
Ask AI
response = portkey.responses.create(
model="gpt-4.1",
tools=[{\
"type": "web_search_preview",\
"search_context_size": "medium", # Options: "high", "medium" (default), or "low"\
"user_location": { # Optional - for localized results\
"type": "approximate",\
"country": "US",\
"city": "San Francisco",\
"region": "California"\
}\
}],
input="What was a positive news story from today?"
)
print(response)
**Options for `search_context_size`:**
* `high`: Most comprehensive context, higher cost, slower response
* `medium`: Balanced context, cost, and latency (default)
* `low`: Minimal context, lowest cost, fastest response
Responses include citations for URLs found in search results, with clickable references.
###
[](https://portkey.ai/docs/integrations/llms/openai#file-search-tool)
File Search Tool
File search enables quick retrieval from your knowledge base across multiple file types:
* Python
* NodeJS
Copy
Ask AI
response = portkey.responses.create(
model="gpt-4.1",
tools=[{\
"type": "file_search",\
"vector_store_ids": ["vs_1234567890"],\
"max_num_results": 20,\
"filters": { # Optional - filter by metadata\
"type": "eq",\
"key": "document_type",\
"value": "report"\
}\
}],
input="What are the attributes of an ancient brown dragon?"
)
print(response)
This tool requires you to first create a vector store and upload files to it. Supports various file formats including PDFs, DOCXs, TXT, and more. Results include file citations in the response.
###
[](https://portkey.ai/docs/integrations/llms/openai#enhanced-reasoning)
Enhanced Reasoning
Control the depth of model reasoning for more comprehensive analysis:
* Python
* NodeJS
Copy
Ask AI
response = portkey.responses.create(
model="o3-mini",
input="How much wood would a woodchuck chuck?",
reasoning={
"effort": "high" # Options: "high", "medium", or "low"
}
)
print(response)
###
[](https://portkey.ai/docs/integrations/llms/openai#computer-use-assistant)
Computer Use Assistant
Portkey also supports the Computer Use Assistant (CUA) tool, which helps agents control computers or virtual machines through screenshots and actions. This feature is available for select developers as a research preview on premium tiers.[Learn More about Computer use tool here](https://platform.openai.com/docs/guides/tools-computer-use?lang=python)
[](https://portkey.ai/docs/integrations/llms/openai#managing-openai-projects-%26-organizations-in-portkey)
Managing OpenAI Projects & Organizations in Portkey
------------------------------------------------------------------------------------------------------------------------------------------------------------------
When integrating OpenAI with Portkey, you can specify your OpenAI organization and project IDs along with your API key. This is particularly useful if you belong to multiple organizations or are accessing projects through a legacy user API key. Specifying the organization and project IDs helps you maintain better control over your access rules, usage, and costs. In Portkey, you can add your Org & Project details by,
1. Creating your Virtual Key
2. Defining a Gateway Config
3. Passing Details in a Request
Let’s explore each method in more detail.
###
[](https://portkey.ai/docs/integrations/llms/openai#using-virtual-keys)
Using Virtual Keys
When selecting OpenAI from the dropdown menu while creating a virtual key, Portkey automatically displays optional fields for the organization ID and project ID alongside the API key field. [Get your OpenAI API key from here](https://platform.openai.com/api-keys)
, then add it to Portkey to create the virtual key that can be used throughout Portkey.
[Virtual Keys](https://portkey.ai/docs/product/ai-gateway/virtual-keys)
Portkey takes budget management a step further than OpenAI. While OpenAI allows setting budget limits per project, Portkey enables you to set budget limits for each virtual key you create. For more information on budget limits, refer to this documentation:
[Budget Limits](https://portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits)
###
[](https://portkey.ai/docs/integrations/llms/openai#using-the-gateway-config)
Using The Gateway Config
You can also specify the organization and project details in the gateway config, either at the root level or within a specific target.
Copy
Ask AI
{
"provider": "openai",
"api_key": "OPENAI_API_KEY",
"openai_organization": "org-xxxxxx",
"openai_project": "proj_xxxxxxxx"
}
###
[](https://portkey.ai/docs/integrations/llms/openai#while-making-a-request)
While Making a Request
You can also pass your organization and project details directly when making a request using curl, the OpenAI SDK, or the Portkey SDK.
* OpenAI Python SDK
* OpenAI TS SDK
* cURL
* Portkey Python SDK
* Portkey Node SDK
Copy
Ask AI
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
client = OpenAI(
api_key="OPENAI_API_KEY",
organization="org-xxxxxxxxxx",
project="proj_xxxxxxxxx",
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai",
api_key="PORTKEY_API_KEY"
)
)
chat_complete = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Say this is a test"}],
)
print(chat_complete.choices[0].message.content)
* * *
###
[](https://portkey.ai/docs/integrations/llms/openai#portkey-features)
Portkey Features
Portkey supports the complete host of it’s functionality via the OpenAI SDK so you don’t need to migrate away from it. Please find more information in the relevant sections:
1. [Add metadata to your requests](https://portkey.ai/docs/product/observability/metadata)
2. [Add gateway configs to the OpenAI client or a single request](https://portkey.ai/docs/product/ai-gateway/configs)
3. [Tracing OpenAI requests](https://portkey.ai/docs/product/observability/traces)
4. [Setup a fallback to Azure OpenAI](https://portkey.ai/docs/product/ai-gateway/fallbacks)
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/integrations/llms/openai.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/integrations/llms/openai)
[Overview](https://portkey.ai/docs/integrations/llms)
[Structured Outputs](https://portkey.ai/docs/integrations/llms/openai/structured-outputs)
⌘I
---
# Ultimate AI SDR - Portkey Docs
[Skip to main content](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#content-area)
[Portkey Docs home page](https://portkey.ai/docs)
Latest
Search...
⌘K
Search...
Navigation
Prompt Engineering
Ultimate AI SDR
[Docs](https://portkey.ai/docs/introduction/what-is-portkey)
[Integrations](https://portkey.ai/docs/integrations/ecosystem)
[Gateway APIs](https://portkey.ai/docs/api-reference/inference-api/introduction)
[Admin APIs](https://portkey.ai/docs/api-reference/admin-api/introduction)
[Cookbooks](https://portkey.ai/docs/guides/use-cases/run-batch-evals)
[Changelog](https://portkey.ai/docs/changelog/2025/september)
On this page
* [The Problem: Generic Sales Outreach Doesn’t Work](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#the-problem%3A-generic-sales-outreach-doesn%E2%80%99t-work)
* [Multi-Agent Architecture](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#multi-agent-architecture)
* [Creating the Prompt Templates](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#creating-the-prompt-templates)
* [Implementing the Workflow](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#implementing-the-workflow)
* [Monitoring and Optimization](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#monitoring-and-optimization)
* [Implementation Checklist](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#implementation-checklist)
* [Troubleshooting & Best Practices](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#troubleshooting-%26-best-practices)
* [Ready to Transform Your Outreach?](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#ready-to-transform-your-outreach%3F)
###
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#the-problem%3A-generic-sales-outreach-doesn%E2%80%99t-work)
The Problem: Generic Sales Outreach Doesn’t Work
❌ Before
Dear John,
I hope this email finds you well. I wanted to reach out about our security services that might be of interest to YMU Talent Agency.
Our company provides security personnel for events. We have many satisfied customers and would like to schedule a call to discuss how we can help you.
Let me know when you’re available.
Regards,
Sales Rep
Response rate: 1.2%
✅ After
Subject: Quick security solution for YMU’s talent events
Hi John,
I noticed YMU’s been expanding its roster of A-list talent lately – congrats on that growth. Having worked event security for talent agencies before, I know how challenging it can be coordinating reliable security teams, especially on short notice.
We’ve built something I think you’ll find interesting – an on-demand security platform that’s already being used by several major talent agencies.
Best,
Ilya
Response rate: 9.5%
This cookbook shows you how to build an AI-powered system that:
* **Researches prospects in real-time** using up-to-date web data
* **Crafts personalized emails** based on prospect-specific insights
* **Self-evaluates and improves** its output before sending
* **Scales to thousands of prospects** at a fraction of the usual cost
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#multi-agent-architecture)
Multi-Agent Architecture
----------------------------------------------------------------------------------------------------------------
Our system combines three specialized AI models:
1. **Orchestrator (Claude 3.7)**: Generates research queries, drafts emails, and refines based on feedback
2. **Researcher (Perplexity)**: Gathers real-time web information about prospects and companies
3. **Evaluator (OpenAI)**: Reviews email quality, providing scores and improvement suggestions
This architecture delivers superior results because:
* Each model handles tasks it excels at
* The system includes built-in quality control
* Cost efficiency through right-sized models and targeted research
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#creating-the-prompt-templates)
Creating the Prompt Templates
--------------------------------------------------------------------------------------------------------------------------
* Orchestrator Template
* Researcher Template
* Evaluator Template
###
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#what-you%E2%80%99ll-create)
What You’ll Create
The Orchestrator template handles three different roles depending on which “mode” is activated:
1. **Research Query Generator**: Creates targeted questions for the researcher
2. **Email Drafter**: Uses research findings to write personalized outreach
3. **Email Refiner**: Incorporates evaluator feedback to improve the email
###
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#variables-you%E2%80%99ll-need)
Variables You’ll Need
| Variable | Purpose | Example |
| --- | --- | --- |
| `our_offering` | Your product/service description | ”Umbrella Corp offers ‘Uber for personal protection’…” |
| `company_name` | Prospect’s company | ”YMU Talent Agency” |
| `company_industry` | Industry sector | ”Elite Talent Management” |
| `target_person_name` | Contact name | ”John Wick” |
| `target_person_designation` | Contact’s role | ”Event Organizer” |
| `requirement_gathering_mode` | Activates research query mode | ”TRUE” or "" (empty) |
| `research_mode` | Activates email drafting mode | ”TRUE” or "" (empty) |
| `evaluator_mode` | Activates email refinement mode | ”TRUE” or "" (empty) |
| `researcher_output` | Data from the researcher | (JSON response from research) |
| `evaluator_output` | Feedback from evaluator | (JSON with score and comments) |
###
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#step-by-step-setup)
Step-by-Step Setup
1. **Create template** in [prompt.new](https://prompt.new/)
with **Claude 3.7 Sonnet**
2. **Add core partials**:
Let’s create reusable components that define our SDR’s core instructions and persona. These are added as **Prompt Partials** - reusable blocks that can be inserted in any template.
Core Agent Instructions Partial
You are the ultimate sales representative from Umbrella Corporation. Your job is to:
1. Understand the company and target person
2. Write research queries to learn more about them
3. Use research findings to write the ultimate opener email
4. Send to evaluator for improvements
5. Write final email based on feedback
SDR Persona Partial
Your name is Ilya:
* You acutely understand the exact requirements your target person and their company has
* You write short, to the point emails that feel like a friend sending a text to you
* At the same time, you understand the importance of coming across as a thorough professional
* You have yourself been on both ends - when you needed private security and when you yourself were a private security professional
We’ll insert both partials into the template’s system role like this:
3. **Add product offering**:
Next, we’ll add a section that will receive your company’s offering details from a variable:
We’ll send this variable’s content at runtime.
4. **Add Prospect Information Section**:
Now let’s add a section that will receive the prospect information variables:
We’ll send these values at runtime as well.
5. **Create Agent-Specific Sections with Conditional Logic**:
This is where the magic happens! We’ll add three “conditional sections” that only appear when a specific mode is activated:_A. Research Query Generation Mode:_ Here, we’ll explain how the research query should be generated.
At this stage, we can send a request to the researcher get the research output back._B. Email Drafting Mode (add this section next):_Once we have the research output, we can create the first email, and add the following to a new user role in the prompt template:
We’ll take this email and send it to the evaluator, which will send back a JSON with two keys: “score” and “comment”._C. Email Refinement Mode (add this final section):_With the Evaluator’s output, we’ll now create the final email.
**The Power of Conditional Variables**This approach with `{{#variable_name}}` syntax lets you use a single template for three different purposes. When you set `requirement_gathering_mode` to “TRUE”, only that section appears. When you set it to empty and instead set `research_mode` to “TRUE”, the email drafting section appears instead. This keeps your templates DRY (Don’t Repeat Yourself).
###
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#complete-template-overview)
Complete Template Overview
When finished, your template should have:
1. Core instruction and persona partials at the top
2. Company offering section
3. Prospect information section
4. Three conditional sections for different modes
This single template will now handle all three stages of the orchestrator’s job, activated by different variables in your code.
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#implementing-the-workflow)
Implementing the Workflow
------------------------------------------------------------------------------------------------------------------
1
Setup
Copy
Ask AI
from portkey_ai import Portkey
# Initialize with tracing
client = Portkey(
api_key="PORTKEY_API_KEY",
trace_id="ultimate-ai-sdr-run-1"
)
# Company offering
our_offering = """Umbrella Corp offers 'Uber for personal protection'. Using our app, you can get highly vetted,
arms-bearing ex-veterans who can accompany you to any place that's supported for any amount of hours or days..."""
# Target information
company_name = "YMU Talent Agency"
company_industry = "Elite Talent Management"
target_person_name = "John Wick"
target_person_designation = "Event Organizer"
2
Generate Research Queries
Copy
Ask AI
# Activate research query generation mode
variables = {
"our_offering": our_offering,
"company_name": company_name,
"company_industry": company_industry,
"target_person_name": target_person_name,
"target_person_designation": target_person_designation,
"requirement_gathering_mode": "TRUE" # Activate research query mode
}
gatherer = client.with_options(span_name="gatherer").prompts.completions.create(
prompt_id="your-orchestrator-template-id",
variables=variables
).choices[0].message.content
Example Output
Copy
Ask AI
Research Queries:
1. What are the typical event sizes and types that YMU Talent Agency organizes?
2. Have there been any security incidents at YMU's past events?
3. What is John Wick's specific role and experience in event organization at YMU?
4. Does YMU currently work with any security providers?
5. What are the most significant upcoming events that YMU is organizing?
...
3
Conduct Research
Copy
Ask AI
# Send queries to researcher
researcher = client.with_options(span_name="researcher").prompts.completions.create(
prompt_id="your-researcher-template-id",
variables={"requirement_gathering_output": gatherer}
).choices[0].message.content
Example Output
Copy
Ask AI
Research Summary for YMU Talent Agency:
1. Company Profile:
- YMU is one of the world's leading talent management companies
- Represents high-profile clients including Simon Cowell, Nicole Scherzinger
...
2. Security Considerations:
- Growing concern about celebrity stalking incidents in the industry
- Multiple high-profile clients have experienced security threats
...
4
Draft Initial Email
Copy
Ask AI
# Activate email drafting mode
variables = {
"our_offering": our_offering,
"company_name": company_name,
"target_person_name": target_person_name,
"researcher_output": researcher,
"requirement_gathering_mode": "", # Deactivate research mode
"research_mode": "TRUE" # Activate email drafting mode
}
email_one = client.with_options(span_name="email-one").prompts.completions.create(
prompt_id="your-orchestrator-template-id", # Same template, different mode
variables=variables
).choices[0].message.content
Example Output
Copy
Ask AI
Subject: Quick security chat - from one protection expert to another
Hi John,
As someone who's been on both sides of event security, I know the stress of finding reliable protection for high-profile talent. Especially given YMU's roster including Simon Cowell and Nicole Scherzinger.
We've built an 'Uber for security' that top agencies use - you get instant access to vetted, armed ex-veterans through an app.
...
5
Evaluate Email
Copy
Ask AI
# Send to evaluator
evaluator = client.with_options(span_name="evaluator").prompts.completions.create(
prompt_id="your-evaluator-template-id",
variables={"work_history": gatherer + " " + researcher, "email_output": email_one}
).choices[0].message.content
Example Output
Copy
Ask AI
{
"score": 7,
"comment": "The email does a good job establishing rapport by acknowledging John's background in event security and mentioning specific clients. However, it could be improved by directly addressing the specific pain point mentioned in the research - 'finding reliable security staff on short notice.'"
}
6
Refine Email
Copy
Ask AI
# Activate email refinement mode
variables = {
"our_offering": our_offering,
"company_name": company_name,
"target_person_name": target_person_name,
"researcher_output": researcher,
"evaluator_output": evaluator,
"requirement_gathering_mode": "",
"research_mode": "",
"evaluator_mode": "TRUE" # Activate refinement mode
}
email_two = client.with_options(span_name="email-two").prompts.completions.create(
prompt_id="your-orchestrator-template-id", # Same template, third mode
variables=variables
).choices[0].message.content
Final Email
Copy
Ask AI
Subject: Quick security solution for YMU's talent events
Hi John,
I noticed YMU's been expanding its roster of A-list talent lately – congrats on that growth. Having worked event security for talent agencies before, I know how challenging it can be coordinating reliable security teams, especially on short notice.
We've built something I think you'll find interesting – an on-demand security platform that's already being used by several major talent agencies. Think of it like having an elite security team in your pocket, available within hours.
...
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#monitoring-and-optimization)
Monitoring and Optimization
----------------------------------------------------------------------------------------------------------------------
Portkey’s trace view provides complete visibility to track performance, cost, latency, and opportunities for improvement.
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#implementation-checklist)
Implementation Checklist
----------------------------------------------------------------------------------------------------------------
✅ Set up Portkey account and API credentials
✅ Create prompt templates for all three agents
✅ Define your company offering and SDR persona
✅ Configure basic prospect information
✅ Implement the five-step workflow
✅ Set up tracing and monitoring
✅ Create a system for batching multiple prospects
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#troubleshooting-%26-best-practices)
Troubleshooting & Best Practices
----------------------------------------------------------------------------------------------------------------------------------
| Issue | Solution |
| --- | --- |
| Low research quality | Make research queries more specific |
| Generic emails | Ensure research findings are prominently featured |
| High token usage | Remove redundant information from prompts |
[](https://portkey.ai/docs/guides/prompts/ultimate-ai-sdr#ready-to-transform-your-outreach%3F)
Ready to Transform Your Outreach?
------------------------------------------------------------------------------------------------------------------------------------
This AI SDR system isn’t just an incremental improvement—it’s a fundamental reimagining of how sales development works. By combining specialized AI agents in an orchestrated workflow, you can achieve personalization at scale that was previously impossible. The result? More meetings, stronger relationships, and ultimately more closed deals—all while freeing your team to focus on high-value activities.
Was this page helpful?
YesNo
[Suggest edits](https://github.com/portkey-ai/docs-core/edit/main/guides/prompts/ultimate-ai-sdr.mdx)
[Raise issue](https://github.com/portkey-ai/docs-core/issues/new?title=Issue%20on%20docs&body=Path:%20/guides/prompts/ultimate-ai-sdr)
[Prompts](https://portkey.ai/docs/guides/prompts)
[Build a chatbot using Portkey's Prompt Templates](https://portkey.ai/docs/guides/prompts/build-a-chatbot-using-portkeys-prompt-templates)
⌘I
---