# Table of Contents - [Introduction](#introduction) - [JSON Mode](#json-mode) - [Quickstart](#quickstart) - [Integrations](#integrations) - [OpenAI compatibility](#openai-compatibility) - [Function calling](#function-calling) - [Serverless models](#serverless-models) - [Quickstart: Flux Tools Models](#quickstart-flux-tools-models) - [Logprobs](#logprobs) - [Llama 3.1 Native FC](#llama-3-1-native-fc) - [Chat](#chat) - [Serverless LoRA Inference](#serverless-lora-inference) - [Dedicated models](#dedicated-models) - [Images](#images) - [Code/Language](#code-language) - [Text-to-Speech](#text-to-speech) - [Quickstart: Retrieval Augmented Generation (RAG)](#quickstart-retrieval-augmented-generation-rag-) - [Quickstart: Next.js](#quickstart-next-js) - [Quickstart: Using Vercel's AI SDK with Together AI](#quickstart-using-vercel-s-ai-sdk-with-together-ai) - [Error codes](#error-codes) - [Rate limits](#rate-limits) - [Deployment Options](#deployment-options) - [Together Mixture-Of-Agents (MoA)](#together-mixture-of-agents-moa-) - [Cluster user management](#cluster-user-management) - [RAG Integrations](#rag-integrations) - [Embeddings](#embeddings) - [CodeSandbox SDK (Code execution)](#codesandbox-sdk-code-execution-) - [Slurm management system](#slurm-management-system) - [Cluster storage](#cluster-storage) - [Dedicated Endpoints Dashboard](#dedicated-endpoints-dashboard) - [Create tickets in Slack](#create-tickets-in-slack) --- # Introduction ![](https://files.readme.io/5db3cdbe7e9fa0b984a16749fc9788400049b02ba55100ee4c556f59882d8840-White_Llamas_on_Blue_Background.png) 👋 Welcome to the Together AI docs! Together AI makes it easy to run or fine-tune leading open source models with only a few lines of code. We offer a variety of generative AI services: * **Serverless models** - Use our API or [playground](https://api.together.xyz/playground) to run dozens of models with pay as you go pricing. * **Fine-Tuning** \- Fine-tune models on your own data [in 5 minutes](/docs/fine-tuning-overview) , then run the model for inference. * **Dedicated endpoints** - Run models on your own private GPUs, starting at a one month minimum commitment. [Contact us](https://www.together.ai/forms/monthly-reserved) . * **GPU Clusters** - If you're interested in private, state of the art clusters with H100 GPUs, [contact us](https://www.together.ai/forms/gpu-cluster-requests) . Quickstart [](#quickstart) ------------------------------ See our [full quickstart](/docs/quickstart) for how to get started with our API in 1 minute. PythonTypeScriptcURL `from together import Together client = Together() completion = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", messages=[{"role": "user", "content": "What are the top 3 things to do in New York?"}], ) print(completion.choices[0].message.content)` `import Together from 'together-ai'; const together = new Together(); const completion = await together.chat.completions.create({ model: 'meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo', messages: [{ role: 'user', content: 'Top 3 things to do in New York?' }], }); console.log(completion.choices[0].message.content);` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", "messages": [ {"role": "user", "content": "What are the top 3 things to do in New York?"} ] }'` Together Cookbook [](#together-cookbook) -------------------------------------------- See the **[Together Cookbook](https://github.com/togethercomputer/together-cookbook) ** – a collection of notebooks showcasing use cases of open-source models with Together AI. Examples include RAG (text + multimodal), Semantic Search, Rerankers, & Structured JSON extraction. Example apps [](#example-apps) ---------------------------------- We've built a number of full-stack open source example apps that you can reference. These are production-ready apps have over 500k users & 10k GitHub stars combined – all fully open source and built on Together AI. * [LlamaCoder](https://llamacoder.together.ai/) ([GitHub](https://github.com/Nutlope/llamacoder) ) – an OSS Claude artifacts that is able to generate full React apps from a single prompt. Built on Llama 3.1 405B powered by Together inference. * [BlinkShot](https://www.blinkshot.io/) ([GitHub](https://github.com/Nutlope/blinkshot) ) – a realtime AI image generator using Flux Schnell on Together AI. Type in a prompt and images will get generated as you type. * [TurboSeek](https://www.turboseek.io/) ([GitHub](https://github.com/Nutlope/turboseek) ) – an AI search engine inspired by Perplexity. It uses a search API (Serper) along with an LLM (Mixtral) to be able to answer any questions. * [Napkins.dev](https://www.napkins.dev/) ([GitHub](https://github.com/nutlope/napkins) ) – a wireframe to app tool. It uses Llama 3.2 vision to read in screenshots and write code for them using Llama 3.1 405B. * [PDFToChat](https://www.pdftochat.com/) ([GitHub](https://github.com/nutlope/pdftochat) ) – a site that lets you chat with your PDFs. Uses RAG with Together embeddings, inference with Llama 3, authentication with Clerk, & MongoDB/Pinecone for the vector database. * [LlamaTutor](https://llamatutor.together.ai/) ([GitHub](https://github.com/Nutlope/llamatutor) ) – a personal tutor that can explain any topic at any education level by using a search API along with Llama 3.1. * [NotesGPT](https://usenotesgpt.com/) ([GitHub](https://github.com/nutlope/notesgpt) ) – an AI note taker that converts your voice notes into organized summaries and clear action items using AI. Uses Together inference (Mixtral) with JSON mode. * [CareerExplorer](https://www.explorecareers.io/) ([GitHub](https://github.com/Nutlope/ExploreCareers) ) – a site that takes in a resume and suggests career paths based on your strengths and interests. Uses Llama 3 and demonstrates how to parse PDFs and chain multiple calls together. Which model should I use? [](#which-model-should-i-use) ----------------------------------------------------------- Together hosts many popular models via our serverless endpoints. For each of these, you'll [be charged](https://together.ai/pricing) based on the tokens you use and size of the model. Here are all the different types of models that we support: * [Chat models](/docs/chat-models) * [Vision models](/docs/vision-models) * [Image models](/docs/image-models) * [Embedding models](/docs/embedding-models) * [Rerank Models](/docs/rerank-models) * [Language and code models](/docs/language-and-code-models) Don't see a model you want to use? **[Send us a request](https://www.together.ai/forms/model-requests) ** to add or upvote the model you'd love to see us add to our serverless infrastructure. Next steps [](#next-steps) ------------------------------ * Check out [our Quickstart](/docs/quickstart) to get started with our API in 1 minute * Explore [our cookbook](https://github.com/togethercomputer/together-cookbook) for Python recipes with Together AI * Explore [our demos](https://together.ai/demos) for full-stack open source example apps. * Check out the [Together AI playground](https://api.together.xyz/playground) to try out different models. * See [our integrations](/docs/integrations) with leading LLM frameworks. Resources [](#resources) ---------------------------- * [Discord](https://discord.com/invite/9Rk6sSeWEG) * [Pricing](https://www.together.ai/pricing) * [Support](https://www.together.ai/contact) Updated 3 days ago * * * --- # JSON Mode JSON mode corrals the LLM into outputting JSON conforming to a provided schema. To activate JSON mode, provide the `response_format` parameter to the Chat Completions API with `{"type": "json_object"}`. The JSON Schema can be specified with the `schema` property of `response_format`. Supported models [](#supported-models) ========================================== * `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo` * `meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo` JSON mode example [](#json-mode-example) ============================================ > When using JSON mode, always instruct the model to produce JSON, either in a system or user message. This is very important so that it only responds in JSON, along with providing the `response_format` parameter. With JSON mode, you can specify a schema for the output of the LLM. In Python, we'll do this with Pydantic and in TypeScript, we'll do this with Zod. Here's an example of JSON mode with Python using Llama 3.1. PythonTypeScriptcurl `import json from together import Together from pydantic import BaseModel, Field import together together = Together() # Define the schema for the output class VoiceNote(BaseModel): title: str = Field(description="A title for the voice note") summary: str = Field(description="A short one sentence summary of the voice note.") actionItems: list[str] = Field( description="A list of action items from the voice note" ) def main(): transcript = ( "Good morning! It's 7:00 AM, and I'm just waking up. Today is going to be a busy day, " "so let's get started. First, I need to make a quick breakfast. I think I'll have some " "scrambled eggs and toast with a cup of coffee. While I'm cooking, I'll also check my " "emails to see if there's anything urgent." ) # Call the LLM with the JSON schema extract = together.chat.completions.create( messages=[ { "role": "system", "content": "The following is a voice message transcript. Only answer in JSON.", }, { "role": "user", "content": transcript, }, ], model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", response_format={ "type": "json_object", "schema": VoiceNote.model_json_schema(), }, ) output = json.loads(extract.choices[0].message.content) print(json.dumps(output, indent=2)) return output main()` `import Together from 'together-ai'; import { z } from 'zod'; import { zodToJsonSchema } from 'zod-to-json-schema'; const together = new Together(); // Defining the schema we want our data in const voiceNoteSchema = z.object({ title: z.string().describe('A title for the voice note'), summary: z .string() .describe('A short one sentence summary of the voice note.'), actionItems: z .array(z.string()) .describe('A list of action items from the voice note'), }); const jsonSchema = zodToJsonSchema(voiceNoteSchema, { target: 'openAi' }); async function main() { const transcript = "Good morning! It's 7:00 AM, and I'm just waking up. Today is going to be a busy day, so let's get started. First, I need to make a quick breakfast. I think I'll have some scrambled eggs and toast with a cup of coffee. While I'm cooking, I'll also check my emails to see if there's anything urgent."; const extract = await together.chat.completions.create({ messages: [ { role: 'system', content: 'The following is a voice message transcript. Only answer in JSON.', }, { role: 'user', content: transcript, }, ], model: 'meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo', // @ts-ignore response_format: { type: 'json_object', schema: jsonSchema }, }); if (extract?.choices?.[0]?.message?.content) { const output = JSON.parse(extract?.choices?.[0]?.message?.content); console.log(output); return output; } return 'No output.'; } main();` `curl -X POST https://api.together.xyz/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -d '{ "messages": [ { "role": "system", "content": "The following is a voice message transcript. Only answer in JSON." }, { "role": "user", "content": "Good morning! It'"'"'s 7:00 AM, and I'"'"'m just waking up. Today is going to be a busy day, so let'"'"'s get started. First, I need to make a quick breakfast. I think I'"'"'ll have some scrambled eggs and toast with a cup of coffee. While I'"'"'m cooking, I'"'"'ll also check my emails to see if there'"'"'s anything urgent." } ], "model": "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", "response_format": { "type": "json_object", "schema": { "properties": { "title": { "description": "A title for the voice note", "title": "Title", "type": "string" }, "summary": { "description": "A short one sentence summary of the voice note.", "title": "Summary", "type": "string" }, "actionItems": { "description": "A list of action items from the voice note", "items": { "type": "string" }, "title": "Actionitems", "type": "array" } }, "required": ["title", "summary", "actionItems"], "title": "VoiceNote", "type": "object" } } }'` `{ title: 'Morning Routine', actionItems: [ 'Make breakfast', 'Check emails' ], summary: 'The speaker is starting their day by making breakfast and checking their emails.' }` Updated 8 days ago * * * --- # Quickstart Together AI makes it easy to run leading open-source models using only a few lines of code. 1\. Register for an account [](#1-register-for-an-account) -------------------------------------------------------------- First, [register for an account](https://api.together.xyz/settings/api-keys) to get an API key. New accounts come with $1 to get started. Once you've registered, set your account's API key to an environment variable named `TOGETHER_API_KEY`: Shell `export TOGETHER_API_KEY=xxxxx` 2\. Install your preferred library [](#2-install-your-preferred-library) ---------------------------------------------------------------------------- Together provides an official library for Python and TypeScript, or you can call our HTTP API in any language you want: PythonTypeScript `pip install together` `npm install together-ai` 3\. Run your first query against a model [](#3-run-your-first-query-against-a-model) ---------------------------------------------------------------------------------------- Choose a model to query. In this example, we'll do a chat completion on Llama 3.1 8B with streaming: PythonTypeScriptcURL `from together import Together client = Together() stream = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", messages=[{"role": "user", "content": "What are the top 3 things to do in New York?"}], stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True)` `import Together from 'together-ai'; const together = new Together(); const stream = await together.chat.completions.create({ model: 'meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo', messages: [ { role: 'user', content: 'What are the top 3 things to do in New York?' }, ], stream: true, }); for await (const chunk of stream) { // use process.stdout.write instead of console.log to avoid newlines process.stdout.write(chunk.choices[0]?.delta?.content || ''); }` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", "messages": [ {"role": "user", "content": "What are the top 3 things to do in New York?"} ] }'` Congratulations – you've just made your first query to Together AI! Next steps [](#next-steps) ------------------------------ * Explore [our cookbook](https://github.com/togethercomputer/together-cookbook) for Python recipes with Together AI * Explore [our demos](https://together.ai/demos) for full-stack open source example apps. * Check out the [Together AI playground](https://api.together.xyz/playground) to try out different models. * See [our integrations](/docs/integrations) with leading LLM frameworks. Resources [](#resources) ---------------------------- * [Discord](https://discord.com/invite/9Rk6sSeWEG) * [Pricing](https://www.together.ai/pricing) * [Support](https://www.together.ai/contact) Updated about 2 months ago * * * --- # Integrations Langchain [](#langchain) ---------------------------- _LangChain is a framework for developing context-aware, reasoning applications powered by language models._ To install the LangChain x Together library, run: Shell `pip install --upgrade langchain-together` Here's sample code to get you started with Langchain + Together AI: Python `from langchain_together import ChatTogether chat = ChatTogether(model="meta-llama/Llama-3-70b-chat-hf") for m in chat.stream("Tell me fun things to do in NYC"): print(m.content, end="", flush=True)` Output: `The city that never sleeps! New York City is a hub of entertainment, culture, and adventure, offering countless fun things to do for visitors of all ages and interests. Here are some ideas to get you started: **Iconic Landmarks and Attractions:** 1. **Statue of Liberty and Ellis Island**: Take a ferry to Liberty Island to see the iconic statue up close and visit the Ellis Island Immigration Museum. 2. **Central Park**: Explore the 843-acre green oasis in the middle of Manhattan, featuring lakes, gardens, and plenty of walking paths. 3. **Empire State Building**: Enjoy panoramic views of the city from the observation deck of this iconic skyscraper. 4. **The Metropolitan Museum of Art**: One of the world's largest and most famous museums, with a collection that spans over 5,000 years of human history.` LlamaIndex [](#llamaindex) ------------------------------ _LlamaIndex is a simple, flexible data framework for connecting custom data sources to large language models (LLMs)._ Here's sample code to get you started with Llama Index + Together AI: Python `!pip install llama-index langchain from llama_index.llms import OpenAILike llm = OpenAILike( model="mistralai/Mixtral-8x7B-Instruct-v0.1", api_base="https://api.together.xyz/v1", api_key="TOGETHER_API_KEY", is_chat_model=True, is_function_calling_model=True, temperature=0.1, ) response = llm.complete("Write up to 500 words essay explaining Large Language Models") print(response)` Pinecone [](#pinecone) -------------------------- _Pinecone is a vector database that helps companies build RAG applications._ Here's some sample code to get you started with Pinecone + Together AI: Python `from pinecone import Pinecone, ServerlessSpec from together import Together pc = Pinecone( api_key="PINECONE_API_KEY", source_tag="TOGETHER_AI" ) client = Together() # Create an index in pinecone index = pc.create_index( name="serverless-index", dimension=1536, metric="cosine", spec=ServerlessSpec(cloud="aws", region="us-west-2"), ) # Create an embedding on Together AI textToEmbed = "Our solar system orbits the Milky Way galaxy at about 515,000 mph" embeddings = client.embeddings.create( model="togethercomputer/m2-bert-80M-8k-retrieval", input=textToEmbed ) # Use index.upsert() to insert embeddings and index.query() to query for similar vectors` Helicone [](#helicone) -------------------------- _Helicone is an open source LLM observability platform._ Here's some sample code to get started with using Helicone + Together AI: Python `import os from together import Together client = Together( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://together.hconeai.com/v1", supplied_headers={ "Helicone-Auth": f"Bearer {os.environ.get('HELICONE_API_KEY')}", }, ) stream = client.chat.completions.create( model="meta-llama/Llama-3-8b-chat-hf", messages=[ {"role": "user", "content": "What are some fun things to do in New York?"} ], stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True)` Updated 3 months ago * * * --- # OpenAI compatibility Together's API endpoints for chat, language and code, images, and embeddings are fully compatible with OpenAI's API. If you have an application that uses one of OpenAI's client libraries, you can easily configure it to point to Together's API servers, and start running your existing applications using our open-source models. Configuring OpenAI to use Together's API [](#configuring-openai-to-use-togethers-api) ----------------------------------------------------------------------------------------- To start using Together with OpenAI's client libraries, pass in your Together API key to the `api_key` option, and change the `base_url` to `https://api.together.xyz/v1`: PythonTypeScript `import os import openai client = openai.OpenAI( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://api.together.xyz/v1", )` `import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: "https://api.together.xyz/v1", });` You can find your API key in [your settings page](https://api.together.xyz/settings/api-keys) . If you don't have an account, you can [register for free](https://api.together.ai/) . Querying an Inference model [](#querying-an-inference-model) ---------------------------------------------------------------- Now that your OpenAI client is configured to point to Together, you can start using one of our open-source models for your inference queries. For example, you can query one of our [chat models](/docs/chat-models) , like Llama 3.1 8B: PythonTypeScript `import os import openai client = openai.OpenAI( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://api.together.xyz/v1", ) response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", messages=[ {"role": "system", "content": "You are a travel agent. Be descriptive and helpful."}, {"role": "user", "content": "Tell me the top 3 things to do in San Francisco"}, ] ) print(response.choices[0].message.content)` `import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: 'https://api.together.xyz/v1', }); const response = await client.chat.completions.create({ model: 'meta-llama/Llama-3-8b-chat-hf', messages: [ { role: 'user', content: 'What are some fun things to do in New York?' }, ], }); console.log(response.choices[0].message.content);` Or you can use a [language model](/docs/language-and-code-models) to generate a code completion: PythonTypeScript `import os import openai client = openai.OpenAI( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://api.together.xyz/v1", ) response = client.completions.create( model="meta-llama/Llama-2-70b-hf", prompt="def bubbleSort(): ", max_tokens=175 ) print(response.choices[0].text)` `import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: 'https://api.together.xyz/v1', }); const response = await client.completions.create({ model: 'codellama/CodeLlama-34b-Python-hf', prompt: 'def bubbleSort(): ', max_tokens: 175, }); console.log(response.choices[0].text);` Streaming with OpenAI [](#streaming-with-openai) ---------------------------------------------------- You can also use OpenAI's streaming capabilities to stream back your response: PythonTypeScript `import os import openai client = openai.OpenAI( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://api.together.xyz/v1", ) stream = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct-Turbo", messages=[ {"role": "system", "content": "You are a travel agent. Be descriptive and helpful."}, {"role": "user", "content": "Tell me about San Francisco"}, ], stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True)` `import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: 'https://api.together.xyz/v1', }); async function run() { const stream = await client.chat.completions.create({ model: 'mistralai/Mixtral-8x7B-Instruct-v0.1', messages: [ { role: 'system', content: 'You are an AI assistant' }, { role: 'user', content: 'Who won the world series in 2020?' }, ], stream: true, }); for await (const chunk of stream) { // use process.stdout.write instead of console.log to avoid newlines process.stdout.write(chunk.choices[0]?.delta?.content || ''); } } run();` Generating embeddings [](#generating-embeddings) ---------------------------------------------------- Use our [embedding models](/docs/embedding-models) to generate an embedding for some text input: PythonTypeScript `import os import openai client = openai.OpenAI( api_key=os.environ.get("TOGETHER_API_KEY"), base_url="https://api.together.xyz/v1", ) response = client.embeddings.create( model = "togethercomputer/m2-bert-80M-8k-retrieval", input = "Our solar system orbits the Milky Way galaxy at about 515,000 mph" ) print(response.data[0].embedding)` `import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: 'https://api.together.xyz/v1', }); const response = await client.embeddings.create({ model: 'togethercomputer/m2-bert-80M-8k-retrieval', input: 'Our solar system orbits the Milky Way galaxy at about 515,000 mph', }); console.log(response.data[0].embedding);` Community libraries [](#community-libraries) ------------------------------------------------ The Together API is also supported by most [OpenAI libraries built by the community](https://platform.openai.com/docs/libraries/community-libraries) . Feel free to [reach out to support](https://www.together.ai/contact) if you come across some unexpected behavior when using our API. Updated 3 days ago * * * --- # Function calling Introduction [](#introduction) ================================== The Chat Completions API supports function calling as introduced by OpenAI. You can describe the functions and arguments to make available to the LLM model. The model will output a JSON object containing arguments to call one or many functions. After calling any invoked functions, you can provide those results back to the model in subsequent Chat Completions API calls. tool\_choice [](#tool_choice) --------------------------------- By default, the model will automatically select which functions to call. However you can direct the model to use a function by supplying the `tool_choice` parameter. In the examples below, `tool_choice=auto` can be replaced with `tool_choice={"type": "function", "function": {"name": "get_current_weather"}}`. Supported models [](#supported-models) ========================================== * `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo` * `meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo` * `meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo` * `meta-llama/Llama-3.3-70B-Instruct-Turbo` * `mistralai/Mixtral-8x7B-Instruct-v0.1` * `mistralai/Mistral-7B-Instruct-v0.1` Simple Example [](#simple-example) ====================================== pythonTypeScript `import os import json import openai client = openai.OpenAI( base_url = "https://api.together.xyz/v1", api_key = os.environ['TOGETHER_API_KEY'], ) tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ] } } } } } ] messages = [ {"role": "system", "content": "You are a helpful assistant that can access external functions. The responses from these function calls will be appended to this dialogue. Please provide responses based on the information from these function calls."}, {"role": "user", "content": "What is the current temperature of New York, San Francisco and Chicago?"} ] response = client.chat.completions.create( model="mistralai/Mixtral-8x7B-Instruct-v0.1", messages=messages, tools=tools, tool_choice="auto", ) print(json.dumps(response.choices[0].message.model_dump()['tool_calls'], indent=2))` `import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.TOGETHER_API_KEY, baseURL: "https://api.together.xyz/v1", }); async function main() { let response = await client.chat.completions.create({ model: "mistralai/Mixtral-8x7B-Instruct-v0.1", messages: [ { role: "system", content: "You are a helpful assistant that can access external functions. The responses from these function calls will be appended to this dialogue. Please provide responses based on the information from these function calls.", }, { role: "user", content: "What is the current temperature of New York, San Francisco and Chicago?", }, ], tools: [ { type: "function", function: { name: "get_current_weather", description: "Get the current weather in a given location", parameters: { type: "object", properties: { location: { type: "string", description: "The city and state, e.g. San Francisco, CA", }, unit: { type: "string", enum: ["celsius", "fahrenheit"], }, }, }, }, }, ], tool_choice: "auto", }); console.log(response.choices[0].message); } main();` Simple Example output [](#simple-example-output) ---------------------------------------------------- JSON `[ { "id": "call_1p75qwks0etzfy1g6noxvsgs", "function": { "arguments": "{\"location\":\"New York, NY\",\"unit\":\"fahrenheit\"}", "name": "get_current_weather" }, "type": "function" }, { "id": "call_aqjfgn65d0c280fjd3pbzpc6", "function": { "arguments": "{\"location\":\"San Francisco, CA\",\"unit\":\"fahrenheit\"}", "name": "get_current_weather" }, "type": "function" }, { "id": "call_rsg8muko8hymb4brkycu3dm5", "function": { "arguments": "{\"location\":\"Chicago, IL\",\"unit\":\"fahrenheit\"}", "name": "get_current_weather" }, "type": "function" } ]` Multi-turn example [](#multi-turn-example) ============================================== Python `import os import json import openai client = openai.OpenAI( base_url = "https://api.together.xyz/v1", api_key = os.environ['TOGETHER_API_KEY'], ) # Example function to make available to model def get_current_weather(location, unit="fahrenheit"): """Get the weather for some location""" if "chicago" in location.lower(): return json.dumps({"location": "Chicago", "temperature": "13", "unit": unit}) elif "san francisco" in location.lower(): return json.dumps({"location": "San Francisco", "temperature": "55", "unit": unit}) elif "new york" in location.lower(): return json.dumps({"location": "New York", "temperature": "11", "unit": unit}) else: return json.dumps({"location": location, "temperature": "unknown"}) tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ] } } } } } ] messages = [ {"role": "system", "content": "You are a helpful assistant that can access external functions. The responses from these function calls will be appended to this dialogue. Please provide responses based on the information from these function calls."}, {"role": "user", "content": "What is the current temperature of New York, San Francisco and Chicago?"} ] response = client.chat.completions.create( model="mistralai/Mixtral-8x7B-Instruct-v0.1", messages=messages, tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls if tool_calls: for tool_call in tool_calls: function_name = tool_call.function.name function_args = json.loads(tool_call.function.arguments) if function_name == "get_current_weather": function_response = get_current_weather( location=function_args.get("location"), unit=function_args.get("unit"), ) messages.append( { "tool_call_id": tool_call.id, "role": "tool", "name": function_name, "content": function_response, } ) function_enriched_response = client.chat.completions.create( model="mistralai/Mixtral-8x7B-Instruct-v0.1", messages=messages, ) print(json.dumps(function_enriched_response.choices[0].message.model_dump(), indent=2))` Multi-turn example output [](#multi-turn-example-output) ------------------------------------------------------------ JSON `{ "content": "The current temperature in New York is 11 degrees Fahrenheit, in San Francisco it is 55 degrees Fahrenheit, and in Chicago it is 13 degrees Fahrenheit.", "role": "assistant" }` Updated about 1 month ago * * * --- # Serverless models Chat models [](#chat-models) -------------------------------- > In the table below, models marked as "Turbo" are quantized to FP8 and those marked as "Lite" are INT4. All our other models are at full precision (FP16). If you're not sure which chat model to use, we currently recommend **Llama 3.1 8B Turbo** (`meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo`) to get started. | Organization | Model Name | API Model String | Context length | Quantization | | --- | --- | --- | --- | --- | | Meta | Llama 3.3 70B Instruct Turbo | meta-llama/Llama-3.3-70B-Instruct-Turbo | 131072 | FP8 | | Meta | Llama 3.1 8B Instruct Turbo | meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo | 131072 | FP8 | | Meta | Llama 3.1 70B Instruct Turbo | meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo | 131072 | FP8 | | Meta | Llama 3.1 405B Instruct Turbo | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo | 130815 | FP8 | | Meta | Llama 3 8B Instruct Turbo | meta-llama/Meta-Llama-3-8B-Instruct-Turbo | 8192 | FP8 | | Meta | Llama 3 70B Instruct Turbo | meta-llama/Meta-Llama-3-70B-Instruct-Turbo | 8192 | FP8 | | Meta | Llama 3.2 3B Instruct Turbo | meta-llama/Llama-3.2-3B-Instruct-Turbo | 131072 | FP16 | | Meta | Llama 3 8B Instruct Lite | meta-llama/Meta-Llama-3-8B-Instruct-Lite | 8192 | INT4 | | Meta | Llama 3 70B Instruct Lite | meta-llama/Meta-Llama-3-70B-Instruct-Lite | 8192 | INT4 | | Meta | Llama 3 8B Instruct Reference | meta-llama/Llama-3-8b-chat-hf | 8192 | FP16 | | Meta | Llama 3 70B Instruct Reference | meta-llama/Llama-3-70b-chat-hf | 8192 | FP16 | | Nvidia | Llama 3.1 Nemotron 70B | nvidia/Llama-3.1-Nemotron-70B-Instruct-HF | 32768 | FP16 | | Qwen | Qwen 2.5 Coder 32B Instruct | Qwen/Qwen2.5-Coder-32B-Instruct | 32768 | FP16 | | Qwen | QwQ-32B-Preview | Qwen/QwQ-32B-Preview | 32768 | FP16 | | Microsoft | WizardLM-2 8x22B | microsoft/WizardLM-2-8x22B | 65536 | FP16 | | Google | Gemma 2 27B | google/gemma-2-27b-it | 8192 | FP16 | | Google | Gemma 2 9B | google/gemma-2-9b-it | 8192 | FP16 | | databricks | DBRX Instruct | databricks/dbrx-instruct | 32768 | FP16 | | DeepSeek | DeepSeek LLM Chat (67B) | deepseek-ai/deepseek-llm-67b-chat | 4096 | FP16 | | DeepSeek | DeepSeek-V3 | deepseek-ai/DeepSeek-V3 | 131072 | FP8 | | DeepSeek | DeepSeek-R1 | deepseek-ai/DeepSeek-R1 | 163840 | FP8 | | Google | Gemma Instruct (2B) | google/gemma-2b-it | 8192 | FP16 | | Gryphe | MythoMax-L2 (13B) | Gryphe/MythoMax-L2-13b | 4096 | FP16 | | Meta | LLaMA-2 Chat (13B) | meta-llama/Llama-2-13b-chat-hf | 4096 | FP16 | | mistralai | Mistral (7B) Instruct | mistralai/Mistral-7B-Instruct-v0.1 | 8192 | FP16 | | mistralai | Mistral (7B) Instruct v0.2 | mistralai/Mistral-7B-Instruct-v0.2 | 32768 | FP16 | | mistralai | Mistral (7B) Instruct v0.3 | mistralai/Mistral-7B-Instruct-v0.3 | 32768 | FP16 | | mistralai | Mixtral-8x7B Instruct (46.7B) | mistralai/Mixtral-8x7B-Instruct-v0.1 | 32768 | FP16 | | mistralai | Mixtral-8x22B Instruct (141B) | mistralai/Mixtral-8x22B-Instruct-v0.1 | 65536 | FP16 | | NousResearch | Nous Hermes 2 - Mixtral 8x7B-DPO (46.7B) | NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO | 32768 | FP16 | | Qwen | Qwen 2.5 7B Instruct Turbo | Qwen/Qwen2.5-7B-Instruct-Turbo | 32768 | FP8 | | Qwen | Qwen 2.5 72B Instruct Turbo | Qwen/Qwen2.5-72B-Instruct-Turbo | 32768 | FP8 | | Qwen | Qwen 2 Instruct (72B) | Qwen/Qwen2-72B-Instruct | 32768 | FP16 | | Qwen | Qwen2 VL 72B Instruct | Qwen/Qwen2-VL-72B-Instruct | 32768 | FP16 | | upstage | Upstage SOLAR Instruct v1 (11B) | upstage/SOLAR-10.7B-Instruct-v1.0 | 4096 | FP16 | Image models [](#image-models) ---------------------------------- Use our [Images](/reference/images) endpoint for Image Models. | Organization | Model Name | Model String for API | Default steps | | --- | --- | --- | --- | | Black Forest Labs | Flux.1 \[schnell\] **(free)\*** | black-forest-labs/FLUX.1-schnell-Free | N/A | | Black Forest Labs | Flux.1 \[schnell\] (Turbo) | black-forest-labs/FLUX.1-schnell | 4 | | Black Forest Labs | Flux.1 Dev | black-forest-labs/FLUX.1-dev | 28 | | Black Forest Labs | Flux.1 Canny | black-forest-labs/FLUX.1-canny | 28 | | Black Forest Labs | Flux.1 Depth | black-forest-labs/FLUX.1-depth | 28 | | Black Forest Labs | Flux.1 Redux | black-forest-labs/FLUX.1-redux | 28 | | Black Forest Labs | Flux1.1 \[pro\] | black-forest-labs/FLUX.1.1-pro | \- | | Black Forest Labs | Flux.1 \[pro\] | black-forest-labs/FLUX.1-pro | \- | | Stability AI | Stable Diffusion XL 1.0 | stabilityai/stable-diffusion-xl-base-1.0 | \- | \*Free model has reduced rate limits and performance compared to our paid Turbo endpoint for Flux Shnell named `black-forest-labs/FLUX.1-schnell` **How FLUX pricing works** For FLUX models (except for pro) pricing is based on the size of generated images (in megapixels) and the number of steps used (if the number of steps exceed the default steps). * **Default pricing:** The listed per megapixel prices are for the default number of steps. * **Using more or fewer steps:** Costs are adjusted based on the number of steps used **only if you go above the default steps**. If you use more steps, the cost increases proportionally using the formula below. If you use fewer steps, the cost _does not_ decrease and is based on the default rate. Here’s a formula to calculate cost: Cost = MP × Price per MP × (Steps ÷ Default Steps) Where: * MP = (Width × Height ÷ 1,000,000) * Price per MP = Cost for generating one megapixel at the default steps * Steps = The number of steps used for the image generation. This is only factored in if going above default steps. Vision models [](#vision-models) ------------------------------------ If you're not sure which vision model to use, we currently recommend **Llama 3.2 11B Turbo** (`meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo`) to get started. For model specific rate limits, navigate [here](/docs/rate-limits) . | Organization | Model Name | API Model String | Context length | | --- | --- | --- | --- | | Meta | **(Free)** Llama 3.2 11B Vision Instruct Turbo\* | meta-llama/Llama-Vision-Free | 131072 | | Meta | Llama 3.2 11B Vision Instruct Turbo | meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo | 131072 | | Meta | Llama 3.2 90B Vision Instruct Turbo | meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo | 131072 | \*Free model has reduced rate limits compared to paid version of Llama 3.2 Vision 11B named `Llama-3.2-11B-Vision-Instruct-Turbo` Code models [](#code-models) -------------------------------- Use our [Completions](/reference/completions) endpoint for Code Models. | Organization | Model Name | Model String for API | Context length | | --- | --- | --- | --- | | Qwen | Qwen 2.5 Coder 32B Instruct | Qwen/Qwen2.5-Coder-32B-Instruct | 32768 | Language models [](#language-models) ---------------------------------------- Use our [Completions](/reference/completions) endpoint for Language Models. | Organization | Model Name | Model String for API | Context length | | --- | --- | --- | --- | | Meta | LLaMA-2 (70B) | meta-llama/Llama-2-70b-hf | 4096 | | mistralai | Mistral (7B) | mistralai/Mistral-7B-v0.1 | 8192 | | mistralai | Mixtral-8x7B (46.7B) | mistralai/Mixtral-8x7B-v0.1 | 32768 | Moderation models [](#moderation-models) -------------------------------------------- Use our [Completions](/reference/completions) endpoint to run a moderation model as a standalone classifier, or use it alongside any of the other models above as a filter to safeguard responses from 100+ models, by specifying the parameter `"safety_model": "MODEL_API_STRING"` | Organization | Model Name | Model String for API | Context length | | --- | --- | --- | --- | | Meta | Llama Guard (7B) | Meta-Llama/Llama-Guard-7b | 4096 | Embedding models [](#embedding-models) ------------------------------------------ | Model Name | Model String for API | Model Size | Embedding Dimension | Context Window | | --- | --- | --- | --- | --- | | M2-BERT-80M-2K-Retrieval | togethercomputer/m2-bert-80M-2k-retrieval | 80M | 768 | 2048 | | M2-BERT-80M-8K-Retrieval | togethercomputer/m2-bert-80M-8k-retrieval | 80M | 768 | 8192 | | M2-BERT-80M-32K-Retrieval | togethercomputer/m2-bert-80M-32k-retrieval | 80M | 768 | 32768 | | UAE-Large-v1 | WhereIsAI/UAE-Large-V1 | 326M | 1024 | 512 | | BGE-Large-EN-v1.5 | BAAI/bge-large-en-v1.5 | 326M | 1024 | 512 | | BGE-Base-EN-v1.5 | BAAI/bge-base-en-v1.5 | 102M | 768 | 512 | | Sentence-BERT | sentence-transformers/msmarco-bert-base-dot-v5 | 110M | 768 | 512 | | BERT | bert-base-uncased | 110M | 768 | 512 | Rerank models [](#rerank-models) ------------------------------------ Our [Rerank API](/docs/rerank-overview) has built-in support for the following models, that we host via our serverless endpoints. | Organization | Model Name | Model Size | Model String for API | Max Doc Size (tokens) | Max Docs | | --- | --- | --- | --- | --- | --- | | Salesforce | LlamaRank | 8B | Salesforce/Llama-Rank-v1 | 8192 | 1024 | Updated 1 day ago * * * --- # Quickstart: Flux Tools Models Flux Tools [](#flux-tools) ============================== Black Forest labs released FLUX Tools with support on Together AI These models enable you to iterate on existing images with image → image and image + text → prompt, which unlocks lots of exciting use cases for AI developers. Three new FLUX Tools image generation models from BFL now available on Together AI: Canny for precise composition control, Depth for accurate spatial relationships, and Redux for instant image variations. Generating an image [](#generating-an-image) ================================================ To use one of these models, see the code below PythonTypeScript `from together import Together client = Together() imageCompletion = client.images.generate( model="black-forest-labs/FLUX.1-depth", width=1024, height=768, steps=28, prompt="show me this picture as a superhero", image_url="https://github.com/nutlope.png", ) print(imageCompletion.data[0].url)` `import Together from "together-ai"; const together = new Together(); async function main() { const response = await together.images.create({ model: "black-forest-labs/FLUX.1-depth", width: 1024, height: 1024, steps: 28, prompt: "give me a superhero", // @ts-ignore image_url: "https://github.com/nutlope.png", }); // @ts-ignore console.log(response.data[0].url); } main();` Check out all available Flux models here: [https://docs.together.ai/docs/serverless-models#image-models](/docs/serverless-models#image-models) Updated 2 months ago * * * --- # Logprobs Logprobs, short for log probabilities, are logarithms of probabilities that indicate the likelihood of each token occurring based on the previous tokens in the context. They allow users to gauge a model's confidence in its outputs and explore alternative responses considered by the model and are beneficial for various applications such as classification tasks, retrieval evaluations, and autocomplete suggestions. Returning logprobs [](#returning-logprobs) ============================================== To return logprobs from our API, simply add `logprobs: 1` to your API call as seen below. Shell `curl -X POST https://api.together.xyz/v1/chat/completions \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $TOGETHER_API_KEY"\ -d '{ "model": "codellama/CodeLlama-70b-Instruct-hf", "stream": false, "max_tokens": 10, "messages": [ {"role": "user", "content": "write an async function in python"} ], "logprobs": 1 }'` Response of returning logprobs [](#response-of-returning-logprobs) ---------------------------------------------------------------------- Here's the response you can expect. You'll notice both the tokens and the log probability of every token is shown. `{ "id": "85d3d3ff1a0d8c57-EWR", "object": "chat.completion", "created": 1709240335, "model": "codellama/CodeLlama-70b-Instruct-hf", "prompt": [], "choices": [ { "finish_reason": "length", "logprobs": { "tokens": [ "1", ".", " Define", " the", " function", " with", " the", " async", " keyword", "." ], "token_logprobs": [ -1.0029297, -0.07861328, -2.1367188, -0.921875, -0.6479492, -1.1318359, -0.35668945, -0.6743164, -0.023162842, -1.1484375 ] }, "index": 0, "message": { "role": "assistant", "content": "1. Define the function with the async keyword." } } ], "usage": { "prompt_tokens": 33, "completion_tokens": 10, "total_tokens": 43 } }` Returning your prompt logprobs [](#returning-your-prompt-logprobs) ====================================================================== If you want to also return your prompt along with its logprobs, simply add `echo: true` as a parameter in your API request. Shell `curl -X POST https://api.together.xyz/v1/chat/completions \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $TOGETHER_API_KEY"\ -d '{ "model": "codellama/CodeLlama-70b-Instruct-hf", "stream": false, "max_tokens": 10, "messages": [ {"role": "user", "content": "write an async function in python"} ], "logprobs": 1, "echo": true }'` Response of returning your prompt logprobs [](#response-of-returning-your-prompt-logprobs) ---------------------------------------------------------------------------------------------- You'll notice that in addition to returning the logprobs of the model output, we're also returning the logprobs of the prompt. Response `{ "id": "85d3cf2ae98c0f73-EWR", "object": "chat.completion", "created": 1709240137, "model": "codellama/CodeLlama-70b-Instruct-hf", "prompt": [ { "text": " Source: user\nDestination: assistant\n\n write an async function in python Source: assistant\nDestination: user\n\n ", "logprobs": { "tokens": [ "", "", "Source", ":", "user", "\n", "Dest", "ination", ":", "assistant", "\n", "\n", "write", "an", "async", "function", "in", "python", "<", "step", ">", "Source", ":", "assistant", "\n", "Dest", "ination", ":", "user", "\n", "\n", "" ], "token_logprobs": [ null, -3.6757812, -12.984375, -0.025985718, -14.7109375, -0.7006836, -1.5390625, -0.00027370453, -0.00031781197, -3.3457031, -0.00097322464, -0.049835205, -14, -3.6210938, -11.109375, -0.3190918, -4.3242188, -2.3574219, -15.890625, -10.1953125, -4.171875, -8.0859375, -0.0010204315, -0.000026106834, -0.0019760132, -0.0027122498, -0.00000333786, -0.0000021457672, -0.79589844, -0.0015296936, -0.0010738373, -6.2734375 ] } } ], "choices": [ { "finish_reason": "length", "logprobs": { "tokens": [ "1", ".", " Define", " the", " function", " with", " the", " async", " keyword", "." ], "token_logprobs": [ -1.0009766, -0.07861328, -2.1367188, -0.921875, -0.6479492, -1.1318359, -0.35668945, -0.6743164, -0.023162842, -1.1484375 ] }, "index": 0, "message": { "role": "assistant", "content": "1. Define the function with the async keyword." } } ], "usage": { "prompt_tokens": 33, "completion_tokens": 10, "total_tokens": 43 } }` Updated 7 months ago * * * --- # Llama 3.1 Native FC Llama 3.1 shipped natively with function calling support, but instead of specifying a `tool_choice` parameter like traditional function calling, it works with a special prompt syntax. Let's take a look at how to do function calling with Llama 3.1 models – strictly with a custom prompt! **Note: OpenAI compatible function calling with Llama 3.1 is also supported! This is usually what you want, especially when working with agent frameworks. [Learn how to do that here](/docs/function-calling) .** > According to Meta, if you want to do a full conversation with tool calling, Llama 3.1 70B and Llama 3.1 405B are the two recommended options. Llama 3.1 8B is good for zero shot tool calling, but can't hold a full conversation at the same time. Function calling w/ Llama 3.1 70B [](#function-calling-w-llama-31-70b) -------------------------------------------------------------------------- Say we have a function called `weatherTool` and want to pass it to LLama 3.1 to be able to call it when it sees fit. We'll define the function attributes, use the special prompt from [llama-agentic-system](https://github.com/meta-llama/llama-agentic-system) (from Meta) to pass the function to the model in the system prompt, and send in a prompt asking how the weather is in Tokyo. PythonTypeScript `import json from together import Together together = Together() weatherTool = { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, }, "required": ["location"], }, } toolPrompt = f""" You have access to the following functions: Use the function '{weatherTool["name"]}' to '{weatherTool["description"]}': {json.dumps(weatherTool)} If you choose to call a function ONLY reply in the following format with no prefix or suffix: {{\"example_name\": \"example_value\"}} Reminder: - Function calls MUST follow the specified format, start with - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls """ messages = [ { "role": "system", "content": toolPrompt, }, { "role": "user", "content": "What is the weather in Tokyo?", }, ] response = together.chat.completions.create( model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo", messages=messages, max_tokens=1024, temperature=0, ) messages.append(response.choices[0].message) print(response.choices[0].message.content)` ``import Together from 'together-ai'; const together = new Together(); const weatherTool = { name: 'get_current_weather', description: 'Get the current weather in a given location', parameters: { type: 'object', properties: { location: { type: 'string', description: 'The city and state, e.g. San Francisco, CA', }, }, required: ['location'], }, }; const toolPrompt = `You have access to the following functions: Use the function '${weatherTool.name}' to '${weatherTool.description}': ${JSON.stringify(weatherTool)} If you choose to call a function ONLY reply in the following format with no prefix or suffix: {"example_name": "example_value"} Reminder: - Function calls MUST follow the specified format, start with - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls `; let messages = [ { role: 'system', content: toolPrompt, }, { role: 'user', content: 'What is the weather in Casablanca?', }, ]; async function main() { const response = await together.chat.completions.create({ model: 'meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo', messages: messages, max_tokens: 1024, temperature: 0, }); if (response.choices?.[0]?.message) { messages.push({ role: response.choices[0].message.role, content: response.choices[0].message.content!, }); console.log(response.choices?.[0]?.message?.content); } } main();`` `{"location": "Tokyo, Japan"}` The output to the function above gets us the following string. We'll then need to parse this string & can call our function after. ### Parsing [](#parsing) > Note: Function calling with Llama 3.1 does not call the function for us, but it simply lets us know what function(s) to call with what parameters. To parse the function above, we can write some code to get a nice JSON object. PythonTypeScript `import re import json def parse_tool_response(response: str): function_regex = r"(.*?)" match = re.search(function_regex, response) if match: function_name, args_string = match.groups() try: args = json.loads(args_string) return { "function": function_name, "arguments": args, } except json.JSONDecodeError as error: print(f"Error parsing function arguments: {error}") return None return None parsed_response = parse_tool_response(response.choices[0].message.content) print(parse_tool_response(response.choices[0].message.content))` `function parseToolResponse(response: string) { const functionRegex = /(.*?)<\/function>/; const match = response.match(functionRegex); if (match) { const [, functionName, argsString] = match; try { return { function: functionName, arguments: JSON.parse(argsString), }; } catch (error) { console.error('Error parsing function arguments:', error); return null; } } return null; } const parsedResponse = parseToolResponse(response.choices[0].message.content); console.log(parsedResponse);` `{ "function": "get_current_weather", "arguments": { "location": "Tokyo, Japan" }, }` Now that we have the parsed function call and arguments in JSON, we can pass these into our actual weather function, pass these into the LLM, and have it respond back to the user. ### Calling the function [](#calling-the-function) Now that Llama 3.1 has told us what function(s) to call and with what parameters, we can execute this ourselves and pass it back to the LLM so it can respond to the user. PythonTypeScript `def get_current_weather(location: str) -> str: # This would be replaced by a weather API if location == "San Francisco, CA": return "62 degrees and cloudy" elif location == "Philadelphia, PA": return "83 degrees and sunny" return "Weather is unknown" if parsed_response: available_functions = {"get_current_weather": get_current_weather} function_to_call = available_functions[parsed_response["function"]] weather = function_to_call(parsed_response["arguments"]["location"]) messages.append( { "role": "tool", "content": weather, } ) print("Weather answer is: ", weather) res = together.chat.completions.create( model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo", messages=messages, max_tokens=1000, temperature=0, ) print("Answer from the LLM: ", res.choices[0].message) else: print("No function call found in the response")` ``function get_current_weather(location: string) { // This would be replaced by a weather API if (location === 'San Francisco, CA') { return '62 degrees and cloudy'; } else if (location === 'Philadelphia, PA') { return '87 degrees and sunny'; } return 'Weather is unknown'; } if (parsedResponse) { const availableFunctions = { get_current_weather: get_current_weather, }; if (parsedResponse.function in availableFunctions) { const functionToCall = availableFunctions[parsedResponse.function]; let weather = functionToCall(parsedResponse.arguments.location); messages.push({ role: 'tool', content: weather, }); console.log('Weather answer is: ', weather); let res = await together.chat.completions.create({ model: 'meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo', messages: messages as any, max_tokens: 1000, temperature: 0, }); console.log('Answer from the LLM: ', res.choices[0].message); } else { console.log(`Function ${parsedResponse.function} not found`); } } else { console.log('No function call found in the response'); }`` Text `Weather answer is: 87 degrees and sunny Answer from the LLM: { role: 'assistant', content: 'The current weather in Casablanca is 87 degrees and sunny.' }` How good is Llama 3.1 at tool calling? [](#how-good-is-llama-31-at-tool-calling) ------------------------------------------------------------------------------------ Llama 3.1 70B and Llama 3.1 405B are both excellent models for tool calling since they were trained with this functionality in mind. In fact, according to some evals, these models even [perform better than GPT-4o](https://www.braintrust.dev/docs/cookbook/recipes/LLaMa-3_1-Tools#analyzing-all-models) . Updated 5 months ago * * * --- # Chat You can use Together's APIs to send individual queries or have long-running conversations with chat models. You can also configure a system prompt to customize how a model should respond. Queries run against a model of your choice. For most use cases, we recommend using Meta Llama 3. Running a single query [](#running-a-single-query) ------------------------------------------------------ Use `chat.completions.create` to send a single query to a chat model: PythonTypeScriptHTTP `from together import Together client = Together() response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct-Turbo", messages=[{"role": "user", "content": "What are some fun things to do in New York?"}], ) print(response.choices[0].message.content)` `import Together from "together-ai"; const together = new Together(); const response = await together.chat.completions.create({ model: "meta-llama/Meta-Llama-3-8B-Instruct-Turbo", messages: [{ role: "user", content: "What are some fun things to do in New York?" }], }); console.log(response.choices[0].message.content)` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Meta-Llama-3-8B-Instruct-Turbo", "messages": [ {"role": "user", "content": "What are some fun things to do in New York?"} ] }'` The `create` method takes in a model name and a `messages` array. Each `message` is an object that has the content of the query, as well as a role for the message's author. In the example above, you can see that we're using "user" for the role. The "user" role tells the model that this message comes from the end user of our system – for example, a customer using your chatbot app. The other two roles are "assistant" and "system", which we'll talk about next. Having a long-running conversation [](#having-a-long-running-conversation) ------------------------------------------------------------------------------ Every query to a chat model is self-contained. This means that new queries won't automatically have access to any queries that may have come before them. This is exactly why the "assistant" role exists. The "assistant" role is used to provide historical context for how a model has responded to prior queries. This makes it perfect for building apps that have long-running conversations, like chatbots. To provide a chat history for a new query, pass the previous messages to the `messages` array, denoting the user-provided queries with the "user" role, and the model's responses with the "assistant" role: PythonTypeScriptHTTP `import os from together import Together client = Together() response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct-Turbo", messages=[ {"role": "user", "content": "What are some fun things to do in New York?"}, {"role": "assistant", "content": "You could go to the Empire State Building!"}, {"role": "user", "content": "That sounds fun! Where is it?"}, ], ) print(response.choices[0].message.content)` `import Together from "together-ai"; const together = new Together(); const response = await together.chat.completions.create({ model: "meta-llama/Meta-Llama-3-8B-Instruct-Turbo", messages: [ { role: "user", content: "What are some fun things to do in New York?" }, { role: "assistant", content: "You could go to the Empire State Building!"}, { role: "user", content: "That sounds fun! Where is it?" }, ], }); console.log(response.choices[0].message.content);` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Meta-Llama-3-8B-Instruct-Turbo", "messages": [ {"role": "user", "content": "What are some fun things to do in New York?"}, {"role": "assistant", "content": "You could go to the Empire State Building!"}, {"role": "user", "content": "That sounds fun! Where is it?" } ] }'` How your app stores historical messages is up to you. Customizing how the model responds [](#customizing-how-the-model-responds) ------------------------------------------------------------------------------ While you can query a model just by providing a user message, typically you'll want to give your model some context for how you'd like it to respond. For example, if you're building a chatbot to help your customers with travel plans, you might want to tell your model that it should act like a helpful travel guide. To do this, provide an initial message that uses the "system" role: PythonTypeScriptHTTP `import os from together import Together client = Together() response = client.chat.completions.create( model="meta-llama/Llama-3-8b-chat-hf", messages=[ {"role": "system", "content": "You are a helpful travel guide."}, {"role": "user", "content": "What are some fun things to do in New York?"}, ], ) print(response.choices[0].message.content)` `import Together from "together-ai"; const together = new Together(); const response = await together.chat.completions.create({ model: "meta-llama/Llama-3-8b-chat-hf", messages: [ {"role": "system", "content": "You are a helpful travel guide."}, { role: "user", content: "What are some fun things to do in New York?" }, ], }); console.log(response.choices[0].message.content);` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3-8b-chat-hf", "messages": [ {"role": "system", "content": "You are a helpful travel guide."}, {"role": "user", "content": "What are some fun things to do in New York?"} ] }'` Streaming responses [](#streaming-responses) ------------------------------------------------ Since models can take some time to respond to a query, Together's APIs support streaming back responses in chunks. This lets you display results from each chunk while the model is still running, instead of having to wait for the entire response to finish. To return a stream, set the `stream` option to true. (If using HTTP, the option name is `stream_tokens`.) PythonTypeScriptHTTP `import os from together import Together client = Together() stream = client.chat.completions.create( model="meta-llama/Llama-3-8b-chat-hf", messages=[{"role": "user", "content": "What are some fun things to do in New York?"}], stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True)` `import Together from 'together-ai'; const together = new Together(); const stream = await together.chat.completions.create({ model: 'meta-llama/Llama-3-8b-chat-hf', messages: [ { role: 'user', content: 'What are some fun things to do in New York?' }, ], stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); }` `curl -X POST "https://api.together.xyz/v1/chat/completions" \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3-8b-chat-hf", "messages": [ {"role": "user", "content": "What are some fun things to do in New York?"} ], "stream_tokens": true }' # Response will be a stream of Server-Sent Events with JSON-encoded payloads. For example: # # data: {"choices":[{"index":0,"delta":{"content":" A"}}],"id":"85ffbb8a6d2c4340-EWR","token":{"id":330,"text":" A","logprob":1,"special":false},"finish_reason":null,"generated_text":null,"stats":null,"usage":null,"created":1709700707,"object":"chat.completion.chunk"} # data: {"choices":[{"index":0,"delta":{"content":":"}}],"id":"85ffbb8a6d2c4340-EWR","token":{"id":28747,"text":":","logprob":0,"special":false},"finish_reason":null,"generated_text":null,"stats":null,"usage":null,"created":1709700707,"object":"chat.completion.chunk"} # data: {"choices":[{"index":0,"delta":{"content":" Sure"}}],"id":"85ffbb8a6d2c4340-EWR","token":{"id":12875,"text":" Sure","logprob":-0.00724411,"special":false},"finish_reason":null,"generated_text":null,"stats":null,"usage":null,"created":1709700707,"object":"chat.completion.chunk"}` A note on async support in Python [](#a-note-on-async-support-in-python) ---------------------------------------------------------------------------- Since I/O in Python is synchronous, multiple queries will execute one after another in sequence, even if they are independent. If you have multiple independent calls that you want to run in parallel, you can use our Python library's `AsyncTogether` module: Python `import os, asyncio from together import AsyncTogether async_client = AsyncTogether() messages = [ "What are the top things to do in San Francisco?", "What country is Paris in?", ] async def async_chat_completion(messages): async_client = AsyncTogether(api_key=os.environ.get("TOGETHER_API_KEY")) tasks = [ async_client.chat.completions.create( model="mistralai/Mixtral-8x7B-Instruct-v0.1", messages=[{"role": "user", "content": message}], ) for message in messages ] responses = await asyncio.gather(*tasks) for response in responses: print(response.choices[0].message.content) asyncio.run(async_chat_completion(messages))` Updated 7 days ago * * * --- # Serverless LoRA Inference LoRA (Low-Rank Adaptation of LLMs) is a popular and lightweight training technique that significantly reduces the number of trainable parameters. It works by inserting a smaller number of new weights into the model and only these are trained. During inference these updated weights are added to the frozen original model weights. This makes training with LoRA much faster, memory-efficient, and produces smaller model weights (a few hundred MBs), which are easier to store and share. Running LoRA Inference on Together [](#running-lora-inference-on-together) ------------------------------------------------------------------------------ The Together API now supports LoRA inference on select base models, allowing you to either: 1. Do LoRA fine-tuning on the many available models through Together AI, then run inference right away 2. Bring Your Own Adapters: If you have custom LoRA adapters, that you've trained or obtained from HuggingFace, you can upload them and run inference You can follow the instructions provided in the [Fine-Tuning Overview](/docs/fine-tuning-overview) to get started with LoRA Fine-tuning. Otherwise, follow the instructions below. Adapters trained previous to 12/17 will not be available for LoRA serverless at the moment. We will be migrating your previous adapters to work with LoRA Serverless. A workaround is to download the adapter and re-upload it using Option 2 below. Supported Base Models [](#supported-base-models) ---------------------------------------------------- Currently, LoRA inference is supported for adapters based on the following base models in Together API. Whether using pre-fine-tuned models or bringing your own adapters, these are the only compatible models: | Organization | Base Model Name | Base Model String | Quantization | | --- | --- | --- | --- | | Meta | Llama 3.2 1B Instruct | meta-llama/Llama-3.2-1B-Instruct | FP8 | | Meta | Llama 3.2 3B Instruct | meta-llama/Llama-3.2-3B-Instruct | FP8 | | Meta | Llama 3.1 8B Instruct | meta-llama/Meta-Llama-3.1-8B-Instruct | FP8 | | Meta | Llama 3.1 70B Instruct | meta-llama/Meta-Llama-3.1-70B-Instruct | FP8 | | Alibaba | Qwen2.5 14B Instruct | Qwen/Qwen2.5-14B-Instruct | FP8 | | Alibaba | Qwen2.5 72B Instruct | Qwen/Qwen2.5-72B-Instruct | FP8 | Option 1: Fine-tune your LoRA model and run inference on it on Together [](#option-1-fine-tune-your-lora-model-and-run-inference-on-it-on-together) ------------------------------------------------------------------------------------------------------------------------------------------------------- The Together API supports both LoRA and full fine-tuning. For serverless LoRA inference, follow these steps: **Step 1: Fine-Tune with LoRA on Together API:** To start a Fine-tuning job with LoRA, follow the detailed instructions in the [Fine-Tuning Overview,](/docs/fine-tuning-overview) or follow the below snippets as a quick start: CLIPython `together files upload "your-datafile.jsonl"` `import os from together import Together client = Together(api_key=os.environ.get("TOGETHER_API_KEY")) resp = client.files.upload(file="your-datafile.jsonl") print(resp.model_dump())` CLIPython `together fine-tuning create \ --training-file "file-629e58b4-ff73-438c-b2cc-f69542b27980" \ --model "meta-llama/Meta-Llama-3.1-8B-Instruct-Reference" \ --lora` `import os from together import Together client = Together(api_key=os.environ.get("TOGETHER_API_KEY")) response = client.fine_tuning.create( training_file = file_resp.id, model = 'meta-llama/Meta-Llama-3.1-8B-Instruct-Reference', lora = True, ) print(response)` **Step 2: Run LoRA Inference**: Once you submit the fine-tuning job you should be able to see the model name in the response: JSON `{ "id": "ft-44129430-ac08-4136-9774-aed81e0164a4", "training_file": "file-629e58b4-ff73-438c-b2cc-f69542b27980", "validation_file": "", "model": "meta-llama/Meta-Llama-3.1-8B-Instruct-Reference", "output_name": "zainhas/Meta-Llama-3.1-8B-Instruct-Reference-my-demo-finetune-4224205a", ... }` You can also see the status of the job and get the model name if you navigate to your fine-tuned model in the 'Model' or 'Jobs' tab in the Together dashboard. You'll see a model string – use it through the Together API. ![](https://files.readme.io/a9c3fb15e77e1df27b72195dc80dea4d0748fcf5f958a15d5884bdb982d3d9b9-image.png) cURLPythonTypeScript `MODEL_NAME_FOR_INFERENCE="zainhas/Meta-Llama-3.1-8B-Instruct-Reference-my-demo-finetune-4224205a" curl -X POST https://api.together.xyz/v1/completions \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "'$MODEL_NAME_FOR_INFERENCE'", "messages": [ { "role": "user", "content": "debate the pros and cons of AI", }, ], "max_tokens": 128 }'` `import os from together import Together client = Together(api_key = TOGETHERAI_API_KEY) user_prompt = "debate the pros and cons of AI" response = client.chat.completions.create( model="zainhas/Meta-Llama-3.1-8B-Instruct-Reference-my-demo-finetune-4224205a", messages=[ { "role": "user", "content": user_prompt, } ], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content)` `import Together from 'together-ai'; const together = new Together(); const stream = await together.chat.completions.create({ model: 'zainhas/Meta-Llama-3.1-8B-Instruct-Reference-my-demo-finetune-4224205a', messages: [ { role: 'user', content: '"ebate the pros and cons of AI' }, ], stream: true, }); for await (const chunk of stream) { // use process.stdout.write instead of console.log to avoid newlines process.stdout.write(chunk.choices[0]?.delta?.content || ''); }` Expected Response: JSON `{ "id": "8f2cb236c80ea20e-YYZ", "object": "text.completion", "created": 1734331375, "model": "zainhas/Meta-Llama-3.1-8B-Instruct-Reference-my-demo-finetune-4224205a", "prompt": [], "choices": [ { "text": "Here's a debate on the pros and cons of AI:\n\n**Moderator:** Welcome to today's debate on the pros and cons of AI. We have two debaters, Alex and Ben, who will present their arguments on the topic. Alex will argue in favor of AI, while Ben will argue against it. Let's begin with opening statements.\n\n**Alex (In Favor of AI):** Thank you, Moderator. AI has revolutionized the way we live and work. It has improved efficiency, productivity, and accuracy in various industries, such as healthcare, finance, and transportation. AI-powered systems can analyze vast amounts of data, identify", "finish_reason": "length", "seed": 5626645655383684000, "logprobs": null, "index": 0 } ], "usage": { "prompt_tokens": 18, "completion_tokens": 128, "total_tokens": 146, "cache_hit_rate": 0 } }` Your first couple queries may have slow TTFT (up to 10 seconds) but subsequent queries should be fast! Option 2: Upload a Custom Adapter & run inference on it on Together [](#option-2-upload-a-custom-adapter--run-inference-on-it-on-together) ---------------------------------------------------------------------------------------------------------------------------------------------- The Together API also allows you to upload your own private LoRA adapter files for inference. To upload a custom adapter: ### **Step 1: Prepare Adapter File:** [](#step-1-prepare-adapter-file) Ensure your adapter file is compatible with the above supported base models. If you are getting the adapter from HuggingFace you can find information about the base model there as well. You need to make sure that the adapter you are trying to upload has an `adapter_config.json` and `adapter_model.safetensors` files. ### **Step 2: Upload Adapter Using Together API:** [](#step-2-upload-adapter-using-together-api) **Source 1: Source the adapter from an AWS s3 bucket:** cURL `#!/bin/bash # uploadadapter.sh # Generate presigned adapter url ADAPTER_URL="s3://test-s3-presigned-adapter/my-70B-lora-1.zip" PRESIGNED_ADAPTER_URL=$(aws s3 presign ${ADAPTER_URL}) # Specify additional params MODEL_TYPE="adapter" ADAPTER_MODEL_NAME="test-lora-model-70B-1" BASE_MODEL="meta-llama/Meta-Llama-3.1-70B-Instruct" DESCRIPTION="test_70b_lora_description" # Lazy curl replace below, don't put spaces here. # Upload curl -v https://api.together.xyz/v0/models \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -d '{ "model_name": "'${ADAPTER_MODEL_NAME}'", "model_source": "'${PRESIGNED_ADAPTER_URL}'", "model_type": "'${MODEL_TYPE}'", "base_model": "'${BASE_MODEL}'", "description": "'${DESCRIPTION}'" }'` **Source 2: Source the adapter from HuggingFace:** Make sure that the adapter contains `adapter_config.json` and `adapter_model.safetensors` files in Files and versions tab on HuggingFace. cURL `# From HuggingFace PRESIGNED_ADAPTER_URL="https://huggingface.co/RayBernard/llama3.2-3B-ft-reasoning" MODEL_TYPE="adapter" BASE_MODEL="meta-llama/Llama-3.2-3B-Instruct" DESCRIPTION="test_lora_3B" ADAPTER_MODEL_NAME=test-lora-model-creation-3b # Uplaod curl -v https://api.together.xyz/v0/models \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -d '{ "model_name": "'${ADAPTER_MODEL_NAME}'", "model_source": "'${PRESIGNED_ADAPTER_URL}'", "model_type": "'${MODEL_TYPE}'", "description": "'${DESCRIPTION}'" }'` For both Option 1 and 2 the output contains the "job\_id" and "model\_name". JSON `{ "data": { "job_id": "job-b641db51-38e8-40f2-90a0-5353aeda6f21", <------- Job ID "model_name": "devuser/test-lora-model-creation-3b", "model_source": "remote_archive" }, "message": "job created" }` You can poll our API using the "job\_id" until the adapter has finished uploading. cURL `curl https://api.together.xyz/v1/jobs/job-b641db51-38e8-40f2-90a0-5353aeda6f21 \ -H "Authorization: Bearer $TOGETHER_API_KEY" | jq .` The output contains a "status" field. When the "status" is "Complete", your adapter is ready! JSON `{ "type": "adapter_upload", "job_id": "job-b641db51-38e8-40f2-90a0-5353aeda6f21", "status": "Complete", "status_updates": [] }` ### **Step 3: Run LoRA Inference**: [](#step-3-run-lora-inference) Take the model\_name string you get from the adapter upload output below, then use it through the Together API. JSON `{ "data": { "job_id": "job-b641db51-38e8-40f2-90a0-5353aeda6f21", "model_name": "devuser/test-lora-model-creation-3b", <------ Model Name "model_source": "remote_archive" }, "message": "job created" }` Make Together API call to the model: cURL `MODEL_NAME_FOR_INFERENCE="devuser/test-lora-model-creation-3b" curl -X POST https://api.together.xyz/v1/completions \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "'$MODEL_NAME_FOR_INFERENCE'", "prompt": "Q: The capital of France is?\nA:", "temperature": 0.8, "max_tokens": 128 }'` Expected Response: JSON `{ "id": "8f3317dd3c3a39ef-YYZ", "object": "text.completion", "created": 1734398453, "model": "devuser/test-lora-model-creation-3b", "prompt": [], "choices": [ { "text": " Paris\nB: Berlin\nC: Warsaw\nD: London\nAnswer: A", "finish_reason": "eos", "seed": 13424880326038300000, "logprobs": null, "index": 0 } ], "usage": { "prompt_tokens": 10, "completion_tokens": 18, "total_tokens": 28, "cache_hit_rate": 0 } }` LoRA Adapter Limits [](#lora-adapter-limits) ------------------------------------------------ You are limited to the following number of LoRA adapters hosted based on build tier: ![](https://files.readme.io/bc98169540fb29568558f6882c1a1ac8d2fb988269c49742b26a91afa1241094-image.png) Updated 10 days ago * * * --- # Dedicated models Chat models [](#chat-models) -------------------------------- | Organization | Model Name | API Model String | Context length | Quantization | | --- | --- | --- | --- | --- | | 01.AI | 01-ai Yi Chat (34B) | zero-one-ai/Yi-34B-Chat | 4096 | FP16 | | AllenAI | OLMo Instruct (7B) | allenai/OLMo-7B-Instruct | 2048 | FP16 | | Austism | Chronos Hermes (13B) | Austism/chronos-hermes-13b | 2048 | FP16 | | carson | carson ml318br | carson/ml318br | 8192 | FP16 | | cognitivecomputations | Dolphin 2.5 Mixtral 8x7b | cognitivecomputations/dolphin-2.5-mixtral-8x7b | 32768 | FP16 | | Databricks | DBRX Instruct | databricks/dbrx-instruct | 32768 | FP16 | | DeepSeek | DeepSeek LLM Chat (67B) | deepseek-ai/deepseek-llm-67b-chat | 4096 | FP16 | | DeepSeek | Deepseek Coder Instruct (33B) | deepseek-ai/deepseek-coder-33b-instruct | 16384 | FP16 | | garage-bAInd | Platypus2 Instruct (70B) | garage-bAInd/Platypus2-70B-instruct | 4096 | FP16 | | google | Gemma-2 Instruct (9B) | google/gemma-2-9b-it | 8192 | FP16 | | Google | Gemma Instruct (2B) | google/gemma-2b-it | 8192 | FP16 | | Google | Gemma-2 Instruct (27B) | google/gemma-2-27b-it | 8192 | FP16 | | Google | Gemma Instruct (7B) | google/gemma-7b-it | 8192 | FP16 | | gradientai | Llama-3 70B Instruct Gradient 1048K | gradientai/Llama-3-70B-Instruct-Gradient-1048k | 1048576 | FP16 | | Gryphe | MythoMax-L2 (13B) | Gryphe/MythoMax-L2-13b | 4096 | FP16 | | Gryphe | Gryphe MythoMax L2 Lite (13B) | Gryphe/MythoMax-L2-13b-Lite | 4096 | FP16 | | Haotian Liu | LLaVa-Next (Mistral-7B) | llava-hf/llava-v1.6-mistral-7b-hf | 4096 | FP16 | | HuggingFace | Zephyr-7B-ß | HuggingFaceH4/zephyr-7b-beta | 32768 | FP16 | | LM Sys | Koala (7B) | togethercomputer/Koala-7B | 2048 | FP16 | | LM Sys | Vicuna v1.3 (7B) | lmsys/vicuna-7b-v1.3 | 2048 | FP16 | | LM Sys | Vicuna v1.5 16K (13B) | lmsys/vicuna-13b-v1.5-16k | 16384 | FP16 | | LM Sys | Vicuna v1.5 (13B) | lmsys/vicuna-13b-v1.5 | 4096 | FP16 | | LM Sys | Vicuna v1.3 (13B) | lmsys/vicuna-13b-v1.3 | 2048 | FP16 | | LM Sys | Koala (13B) | togethercomputer/Koala-13B | 2048 | FP16 | | LM Sys | Vicuna v1.5 (7B) | lmsys/vicuna-7b-v1.5 | 4096 | FP16 | | Meta | Code Llama Instruct (34B) | codellama/CodeLlama-34b-Instruct-hf | 16384 | FP16 | | Meta | Llama3 8B Chat HF INT4 | togethercomputer/Llama-3-8b-chat-hf-int4 | 8192 | FP16 | | Meta | Meta Llama 3.2 90B Vision Instruct Turbo | meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo | 131072 | FP16 | | Meta | Meta Llama 3.2 11B Vision Instruct Turbo | meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo | 131072 | FP16 | | Meta | Meta Llama 3.2 3B Instruct Turbo | meta-llama/Llama-3.2-3B-Instruct-Turbo | 131072 | FP16 | | Meta | Togethercomputer Llama3 8B Instruct Int8 | togethercomputer/Llama-3-8b-chat-hf-int8 | 8192 | FP16 | | Meta | Meta Llama 3.1 70B Instruct Turbo | meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo | 32768 | FP8 | | Meta | LLaMA-2 Chat (13B) | meta-llama/Llama-2-13b-chat-hf | 4096 | FP16 | | Meta | Meta Llama 3 70B Instruct Lite | meta-llama/Meta-Llama-3-70B-Instruct-Lite | 8192 | INT4 | | Meta | Meta Llama 3 8B Instruct Reference | meta-llama/Llama-3-8b-chat-hf | 8192 | FP16 | | Meta | Meta Llama 3 70B Instruct Reference | meta-llama/Llama-3-70b-chat-hf | 8192 | FP16 | | Meta | Meta Llama 3 8B Instruct Turbo | meta-llama/Meta-Llama-3-8B-Instruct-Turbo | 8192 | FP8 | | Meta | Meta Llama 3 8B Instruct Lite | meta-llama/Meta-Llama-3-8B-Instruct-Lite | 8192 | INT4 | | Meta | Meta Llama 3.1 405B Instruct Turbo | meta-llama/Meta-Llama-3.1-405B-Instruct-Lite-Pro | 4096 | FP16 | | Meta | LLaMA-2 Chat (7B) | meta-llama/Llama-2-7b-chat-hf | 4096 | FP16 | | Meta | Meta Llama 3.1 405B Instruct Turbo | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo | 130815 | FP8 | | Meta | Meta Llama Vision Free | meta-llama/Llama-Vision-Free | 131072 | FP16 | | Meta | Meta Llama 3 70B Instruct Turbo | meta-llama/Meta-Llama-3-70B-Instruct-Turbo | 8192 | FP8 | | Meta | Meta Llama 3.1 8B Instruct Turbo | meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo | 32768 | FP8 | | Meta | Code Llama Instruct (7B) | togethercomputer/CodeLlama-7b-Instruct | 16384 | FP16 | | Meta | Code Llama Instruct (34B) | togethercomputer/CodeLlama-34b-Instruct | 16384 | FP16 | | Meta | Code Llama Instruct (13B) | codellama/CodeLlama-13b-Instruct-hf | 16384 | FP16 | | Meta | Code Llama Instruct (13B) | togethercomputer/CodeLlama-13b-Instruct | 16384 | FP16 | | Meta | LLaMA-2 Chat (13B) | togethercomputer/llama-2-13b-chat | 4096 | FP16 | | Meta | LLaMA-2 Chat (7B) | togethercomputer/llama-2-7b-chat | 4096 | FP16 | | Meta | Meta Llama 3 8B Instruct | meta-llama/Meta-Llama-3-8B-Instruct | 8192 | FP16 | | Meta | Meta Llama 3 70B Instruct | meta-llama/Meta-Llama-3-70B-Instruct | 8192 | FP16 | | Meta | Code Llama Instruct (70B) | codellama/CodeLlama-70b-Instruct-hf | 4096 | FP16 | | Meta | LLaMA-2 Chat (70B) | togethercomputer/llama-2-70b-chat | 4096 | FP16 | | Meta | Code Llama Instruct (7B) | codellama/CodeLlama-7b-Instruct-hf | 16384 | FP16 | | Meta | LLaMA-2 Chat (70B) | meta-llama/Llama-2-70b-chat-hf | 4096 | FP16 | | Meta | Meta Llama 3.1 8B Instruct | meta-llama/Meta-Llama-3.1-8B-Instruct-Reference | 16384 | FP16 | | Meta | Meta Llama 3.1 70B Instruct | meta-llama/Meta-Llama-3.1-70B-Instruct-Reference | 8192 | FP16 | | microsoft | WizardLM-2 (8x22B) | microsoft/WizardLM-2-8x22B | 65536 | FP16 | | mistralai | Mistral (7B) Instruct | mistralai/Mistral-7B-Instruct-v0.1 | 4096 | FP16 | | mistralai | Mistral (7B) Instruct v0.2 | mistralai/Mistral-7B-Instruct-v0.2 | 32768 | FP16 | | mistralai | Mistral (7B) Instruct v0.3 | mistralai/Mistral-7B-Instruct-v0.3 | 32768 | FP16 | | mistralai | Mixtral-8x7B Instruct v0.1 | mistralai/Mixtral-8x7B-Instruct-v0.1 | 32768 | FP16 | | mistralai | Mixtral-8x22B Instruct v0.1 | mistralai/Mixtral-8x22B-Instruct-v0.1 | 65536 | FP16 | | NousResearch | Nous Hermes 2 - Mixtral 8x7B-DPO | NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO | 32768 | FP16 | | NousResearch | Nous Hermes LLaMA-2 (70B) | NousResearch/Nous-Hermes-Llama2-70b | 4096 | FP16 | | NousResearch | Nous Hermes 2 - Mixtral 8x7B-SFT | NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT | 32768 | FP16 | | NousResearch | Nous Hermes Llama-2 (13B) | NousResearch/Nous-Hermes-Llama2-13b | 4096 | FP16 | | NousResearch | Nous Hermes 2 - Mistral DPO (7B) | NousResearch/Nous-Hermes-2-Mistral-7B-DPO | 32768 | FP16 | | NousResearch | Nous Hermes LLaMA-2 (7B) | NousResearch/Nous-Hermes-llama-2-7b | 4096 | FP16 | | NousResearch | Nous Capybara v1.9 (7B) | NousResearch/Nous-Capybara-7B-V1p9 | 8192 | FP16 | | NousResearch | Hermes 2 Theta Llama-3 70B | NousResearch/Hermes-2-Theta-Llama-3-70B | 8192 | FP16 | | OpenChat | OpenChat 3.5 | openchat/openchat-3.5-1210 | 8192 | FP16 | | OpenOrca | OpenOrca Mistral (7B) 8K | Open-Orca/Mistral-7B-OpenOrca | 8192 | FP16 | | Qwen | Qwen 2 Instruct (72B) | Qwen/Qwen2-72B-Instruct | 32768 | FP16 | | Qwen | Qwen2.5 72B Instruct Turbo | Qwen/Qwen2.5-72B-Instruct-Turbo | 32768 | FP8 | | Qwen | Qwen2.5 7B Instruct Turbo | Qwen/Qwen2.5-7B-Instruct-Turbo | 32768 | FP8 | | Qwen | Qwen 1.5 Chat (110B) | Qwen/Qwen1.5-110B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (72B) | Qwen/Qwen1.5-72B-Chat | 32768 | FP16 | | Qwen | Qwen 2 Instruct (1.5B) | Qwen/Qwen2-1.5B-Instruct | 32768 | FP16 | | Qwen | Qwen 2 Instruct (7B) | Qwen/Qwen2-7B-Instruct | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (14B) | Qwen/Qwen1.5-14B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (1.8B) | Qwen/Qwen1.5-1.8B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (32B) | Qwen/Qwen1.5-32B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (7B) | Qwen/Qwen1.5-7B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (0.5B) | Qwen/Qwen1.5-0.5B-Chat | 32768 | FP16 | | Qwen | Qwen 1.5 Chat (4B) | Qwen/Qwen1.5-4B-Chat | 32768 | FP16 | | Snorkel AI | Snorkel Mistral PairRM DPO (7B) | snorkelai/Snorkel-Mistral-PairRM-DPO | 32768 | FP16 | | Snowflake | Snowflake Arctic Instruct | Snowflake/snowflake-arctic-instruct | 4096 | FP16 | | Stanford | Alpaca (7B) | togethercomputer/alpaca-7b | 2048 | FP16 | | teknium | OpenHermes-2-Mistral (7B) | teknium/OpenHermes-2-Mistral-7B | 8192 | FP16 | | teknium | OpenHermes-2.5-Mistral (7B) | teknium/OpenHermes-2p5-Mistral-7B | 8192 | FP16 | | test | Test 11 | test/test11 | 4096 | FP16 | | Tim Dettmers | Guanaco (65B) | togethercomputer/guanaco-65b | 2048 | FP16 | | Tim Dettmers | Guanaco (13B) | togethercomputer/guanaco-13b | 2048 | FP16 | | Tim Dettmers | Guanaco (33B) | togethercomputer/guanaco-33b | 2048 | FP16 | | Tim Dettmers | Guanaco (7B) | togethercomputer/guanaco-7b | 2048 | FP16 | | Undi95 | ReMM SLERP L2 (13B) | Undi95/ReMM-SLERP-L2-13B | 4096 | FP16 | | Undi95 | Toppy M (7B) | Undi95/Toppy-M-7B | 4096 | FP16 | | upstage | Upstage SOLAR Instruct v1 (11B) | upstage/SOLAR-10.7B-Instruct-v1.0 | 4096 | FP16 | | upstage | Upstage SOLAR Instruct v1 (11B)-Int4 | togethercomputer/SOLAR-10.7B-Instruct-v1.0-int4 | 4096 | FP16 | | WizardLM | WizardLM v1.2 (13B) | WizardLM/WizardLM-13B-V1.2 | 4096 | FP16 | Image models [](#image-models) ---------------------------------- | Organization | Model Name | API Model String | | --- | --- | --- | | Black Forest Labs | FLUX.1 \[pro\] | black-forest-labs/FLUX.1-pro | | Black Forest Labs | FLUX.1 \[schnell\] | black-forest-labs/FLUX.1-schnell | | Black Forest Labs | FLUX1.1 \[pro\] | black-forest-labs/FLUX.1.1-pro | | Black Forest Labs | FLUX.1 \[schnell\] Free | black-forest-labs/FLUX.1-schnell-Free | | Prompt Hero | Openjourney v4 | prompthero/openjourney | | Runway ML | Stable Diffusion 1.5 | runwayml/stable-diffusion-v1-5 | | SG161222 | Realistic Vision 3.0 | SG161222/Realistic\_Vision\_V3.0\_VAE | | Stability AI | Stable Diffusion XL 1.0 | stabilityai/stable-diffusion-xl-base-1.0 | | Stability AI | Stable Diffusion 2.1 | stabilityai/stable-diffusion-2-1 | | Wavymulder | Analog Diffusion | wavymulder/Analog-Diffusion | Language models [](#language-models) ---------------------------------------- | Organization | Model Name | API Model String | Context length | | --- | --- | --- | --- | | 01.AI | 01-ai Yi Base (34B) | zero-one-ai/Yi-34B | 4096 | | 01.AI | 01-ai Yi Base (6B) | zero-one-ai/Yi-6B | 4096 | | AllenAI | OLMo (7B) | allenai/OLMo-7B | 2048 | | EleutherAI | Llemma (7B) | EleutherAI/llemma\_7b | 4096 | | google | Gemma 2 (9B) | google/gemma-2-9b | 8192 | | Google | Gemma (7B) | google/gemma-7b | 8192 | | Google | Gemma (2B) | google/gemma-2b | 8192 | | Meta | Meta Llama 3 8B | meta-llama/Meta-Llama-3-8B | 8192 | | Meta | LLaMA-2 (70B) | meta-llama/Llama-2-70b-hf | 4096 | | Meta | LLaMA-2 (7B) | togethercomputer/llama-2-7b | 4096 | | Meta | LLaMA (7B) | huggyllama/llama-7b | 2048 | | Meta | LLaMA (65B) | huggyllama/llama-65b | 2048 | | Meta | LLaMA-2 (13B) | togethercomputer/llama-2-13b | 4096 | | Meta | LLaMA-2 (70B) | togethercomputer/llama-2-70b | 4096 | | Meta | LLaMA-2 (13B) | meta-llama/Llama-2-13b-hf | 4096 | | Meta | LLaMA (13B) | huggyllama/llama-13b | 2048 | | Meta | LLaMA (30B) | huggyllama/llama-30b | 2048 | | Meta | Meta Llama 3 70B | meta-llama/Meta-Llama-3-70B | 8192 | | Meta | Meta Llama 3 8B | meta-llama/Llama-3-8b-hf | 8192 | | Meta | LLaMA-2 (7B) | meta-llama/Llama-2-7b-hf | 4096 | | Meta | Meta Llama 3 70B HF | meta-llama/Llama-3-70b-hf | 8192 | | Meta | Meta Llama 3.1 8B | meta-llama/Meta-Llama-3.1-8B-Reference | 8192 | | Meta | Meta Llama 3.1 70B | meta-llama/Meta-Llama-3.1-70B-Reference | 8192 | | Microsoft | Microsoft Phi-2 | microsoft/phi-2 | 2048 | | mistralai | Mixtral-8x7B v0.1 | mistralai/Mixtral-8x7B-v0.1 | 32768 | | mistralai | Mistral (7B) | mistralai/Mistral-7B-v0.1 | 4096 | | mistralai | Mixtral-8x22B | mistralai/Mixtral-8x22B | 65536 | | Nexusflow | NexusRaven (13B) | Nexusflow/NexusRaven-V2-13B | 16384 | | Nous Research | Nous Hermes (13B) | NousResearch/Nous-Hermes-13b | 2048 | | Qwen | Qwen 2 (72B) | Qwen/Qwen2-72B | 32768 | | Qwen | Qwen 1.5 (0.5B) | Qwen/Qwen1.5-0.5B | 32768 | | Qwen | Qwen 1.5 (1.8B) | Qwen/Qwen1.5-1.8B | 32768 | | Qwen | Qwen 1.5 (4B) | Qwen/Qwen1.5-4B | 32768 | | Qwen | Qwen 1.5 (7B) | Qwen/Qwen1.5-7B | 32768 | | Qwen | Qwen 1.5 (72B) | Qwen/Qwen1.5-72B | 4096 | | Qwen | Qwen 2 (7B) | Qwen/Qwen2-7B | 32768 | | Qwen | Qwen 2 (1.5B) | Qwen/Qwen2-1.5B | 32768 | | Qwen | Qwen 1.5 (32B) | Qwen/Qwen1.5-32B | 32768 | | Qwen | Qwen 1.5 (14B) | Qwen/Qwen1.5-14B | 32768 | | Together | StripedHyena Hessian (7B) | togethercomputer/StripedHyena-Hessian-7B | 32768 | | Together | LLaMA-2-32K (7B) | togethercomputer/LLaMA-2-7B-32K | 32768 | | Together | Evo-1 Base (131K) | togethercomputer/evo-1-131k-base | 131073 | | Together | Evo-1 Base (8K) | togethercomputer/evo-1-8k-base | 8192 | | WizardLM | WizardLM v1.0 (70B) | WizardLM/WizardLM-70B-V1.0 | 4096 | Code models [](#code-models) -------------------------------- | Organization | Model Name | API Model String | Context length | | --- | --- | --- | --- | | Meta | Code Llama Python (34B) | codellama/CodeLlama-34b-Python-hf | 16384 | | Meta | Code Llama Python (70B) | codellama/CodeLlama-70b-Python-hf | 4096 | | Meta | Code Llama Python (34B) | togethercomputer/CodeLlama-34b-Python | 16384 | | Meta | Code Llama (34B) | togethercomputer/CodeLlama-34b | 16384 | | Meta | Code Llama (13B) | codellama/CodeLlama-13b-hf | 16384 | | Meta | Code Llama (34B) | codellama/CodeLlama-34b-hf | 16384 | | Meta | Code Llama Python (7B) | togethercomputer/CodeLlama-7b-Python | 16384 | | Meta | Code Llama (70B) | codellama/CodeLlama-70b-hf | 16384 | | Meta | Code Llama Python (13B) | togethercomputer/CodeLlama-13b-Python | 16384 | | Meta | Code Llama (7B) | codellama/CodeLlama-7b-hf | 16384 | | Meta | Code Llama Python (13B) | codellama/CodeLlama-13b-Python-hf | 16384 | | Meta | Code Llama Python (7B) | codellama/CodeLlama-7b-Python-hf | 16384 | | Numbers Station | NSQL LLaMA-2 (7B) | NumbersStation/nsql-llama-2-7B | 4096 | | Phind | Phind Code LLaMA v2 (34B) | Phind/Phind-CodeLlama-34B-v2 | 16384 | | Phind | Phind Code LLaMA Python v1 (34B) | Phind/Phind-CodeLlama-34B-Python-v1 | 16384 | | WizardLM | WizardCoder Python v1.0 (34B) | WizardLM/WizardCoder-Python-34B-V1.0 | 8192 | Moderation models [](#moderation-models) -------------------------------------------- | Organization | Model Name | API Model String | Context length | | --- | --- | --- | --- | | Meta | Meta Llama Guard 3 8B | meta-llama/Meta-Llama-Guard-3-8B | 8192 | | Meta | Meta Llama Guard 2 8B | meta-llama/LlamaGuard-2-8b | 8192 | | Meta | Meta Llama Guard 3 11B Vision Turbo | meta-llama/Llama-Guard-3-11B-Vision-Turbo | 131072 | | Meta | Llama Guard (7B) | Meta-Llama/Llama-Guard-7b | 4096 | Embedding models [](#embedding-models) ------------------------------------------ | Organization | Model Name | API Model String | Context length | | --- | --- | --- | --- | | BAAI | BAAI-Bge-Base-1p5 | BAAI/bge-base-en-v1.5 | undefined | | BAAI | BAAI-Bge-Large-1p5 | BAAI/bge-large-en-v1.5 | undefined | | Google | Bert Base Uncased | bert-base-uncased | undefined | | HazyResearch | M2-BERT 2K Retrieval Encoder V1 | hazyresearch/M2-BERT-2k-Retrieval-Encoder-V1 | 2048 | | Together | M2-BERT-Retrieval-32k | togethercomputer/m2-bert-80M-32k-retrieval | 32768 | | Together | M2-BERT-Retrieval-2K | togethercomputer/m2-bert-80M-2k-retrieval | undefined | | Together | M2-BERT-Retrieval-8k | togethercomputer/m2-bert-80M-8k-retrieval | 8192 | | Together | Sentence-BERT | sentence-transformers/msmarco-bert-base-dot-v5 | 512 | | WhereIsAI | UAE-Large-V1 | WhereIsAI/UAE-Large-V1 | undefined | Rerank models [](#rerank-models) ------------------------------------ | Organization | Model Name | API Model String | Max Doc Size (tokens) | Max Docs | | --- | --- | --- | --- | --- | | salesforce | Salesforce Llama Rank V1 (8B) | Salesforce/Llama-Rank-V1 | 8192 | 1024 | Updated about 2 months ago * * * --- # Images Generating an image [](#generating-an-image) ------------------------------------------------ To query an image model, use the `.images` method and specify the image model you want to use. PythonTypeScript `from together import Together client = Together() response = client.images.generate( prompt="a flying cat", model="black-forest-labs/FLUX.1-schnell", steps=4 ) print(response.data[0].url)` `import Together from "together-ai"; const together = new Together(); async function main() { const response = await together.images.create({ prompt: "a flying cat", model: "black-forest-labs/FLUX.1-schnell", steps: 4, }); // @ts-ignore console.log(response.data[0].url); } main();` Supported image models [](#supported-image-models) ------------------------------------------------------ See **[our models page](/docs/serverless-models#image-models) ** for supported image models. Safety Checker [](#safety-checker) -------------------------------------- We have a built in safety checker that detects NSFW words but you can disable it by passing in `"disable_safety_checker": "true"`. This works for every model except Flux Schnell Free and Flux Pro. PythonTypeScript `from together import Together client = Together() response = client.images.generate( prompt="a flying cat", model="black-forest-labs/FLUX.1-schnell", steps=4, disable_safety_checker=True, ) print(response.data[0].url)` `import Together from 'together-ai'; const together = new Together(); async function main() { const response = await together.images.create({ prompt: 'a flying cat', model: 'black-forest-labs/FLUX.1-schnell', steps: 4, // @ts-ignore disable_safety_checker: true, }); console.log(response.data[0].url); } main();` Updated 17 days ago * * * --- # Code/Language Creating a completion [](#creating-a-completion) ---------------------------------------------------- Use `client.completions.create` to create a completion for a code or language models: PythonTypeScript `import os from together import Together client = Together() response = client.completions.create( model="codellama/CodeLlama-34b-Instruct-hf", prompt="def fibonacci(n): ", stream=True, ) for chunk in response: print(chunk.choices[0].text or "", end="", flush=True)` `import Together from 'together-ai'; const together = new Together(); const response = await together.completions.create({ model: "codellama/CodeLlama-13b-Python-hf", prompt: 'def bubbleSort(): ', stream: true }); for chunk in response: console.log(chunk.choices[0].text)` Updated 3 months ago * * * --- # Text-to-Speech Quickstart [](#quickstart) ============================== 1\. Register for an account [](#1-register-for-an-account) -------------------------------------------------------------- First, [register for an account](https://api.together.xyz/settings/api-keys) to get an API key. New accounts come with $5 to get started. Once you've registered, set your account's API key to an environment variable named `TOGETHER_API_KEY`: Shell `export TOGETHER_API_KEY=xxxxx` 2\. Query the models via our API [](#2-query-the-models-via-our-api) ------------------------------------------------------------------------ In this example, we're giving it a string and asking the model to return a `.wav` back to us. PythonTypeScriptcURL `import requests url = "https://api.together.ai/v1/audio/generations" headers = {"Authorization": f"Bearer {TOGETHER_API_KEY}"} data = { "input": text, "voice": voice, "response_format": "raw", "sample_rate": 44100, "stream": False, "model": "cartesia/sonic", } response = requests.post(url, headers=headers, json=data) response.content` `import Together from 'together-ai'; const together = new Together(); async function generateAudio() { const res = await together.audio.create({ input: 'Hello, how are you today?', voice: 'laidback woman', response_format: 'wav', sample_rate: 44100, stream: false, model: 'cartesia/sonic', }); if (res.body) { console.log(res.body); const nodeStream = Readable.from(res.body as ReadableStream); const fileStream = createWriteStream('./text2.wav'); nodeStream.pipe(fileStream); } } generateAudio();` `curl --location 'https://api.together.ai/v1/audio/generations' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer $TOGETHER_API_KEY' \ --output test2.wav \ --data '{ "input": "hello, how are you?", "voice": "laidback woman", "response_format": "wav", "sample_rate": 44100, "stream": false, "model": "cartesia/sonic" }'` This will output a `test2.wav`file that can be played. For a complete example using this API to generate podcasts refer to our [notebook](https://github.com/togethercomputer/together-cookbook/blob/main/Agents/Serial_Chain_Agent_Workflow.ipynb) ### Output Raw Bytes [](#output-raw-bytes) If you want to extract out raw audio bytes use the settings below: PythonTypeScriptcURL `import requests url = "https://api.together.ai/v1/audio/generations" headers = {"Authorization": f"Bearer {TOGETHER_API_KEY}"} data = { "input": text, "voice": voice, "response_format": "raw", "response_encoding": "pcm_f32le", "sample_rate": 44100, "stream": False, "model": "cartesia/sonic", } response = requests.post(url, headers=headers, json=data) with open("text2.pcm", "wb") as f: f.write(response.content)` `import Together from 'together-ai'; const together = new Together(); async function generateRawBytes() { const res = await together.audio.create({ input: 'Hello, how are you today?', voice: 'laidback woman', response_format: 'raw', response_encoding: 'pcm_f32le', sample_rate: 44100, stream: false, model: 'cartesia/sonic', }); console.log(res.body); } generateRawBytes();` `curl --location 'https://api.together.ai/v1/audio/generations' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer $TOGETHER_API_KEY' \ --output test2.pcm \ --data '{ "input": text, "voice": voice, "response_format": "raw", "response_encoding": "pcm_f32le", "sample_rate": 44100, "stream": false, "model": "cartesia/sonic" }'` This will output a raw bytes `test2.pcm`file. ### Voices Available [](#voices-available) All valid voice model strings: `'german conversational woman', 'nonfiction man', 'friendly sidekick', 'french conversational lady', 'french narrator lady', 'german reporter woman', 'indian lady', 'british reading lady', 'british narration lady', 'japanese children book', 'japanese woman conversational', 'japanese male conversational', 'reading lady', 'newsman', 'child', 'meditation lady', 'maria', "1920's radioman", 'newslady', 'calm lady', 'helpful woman', 'mexican woman', 'korean narrator woman', 'russian calm lady', 'russian narrator man 1', 'russian narrator man 2', 'russian narrator woman', 'hinglish speaking lady', 'italian narrator woman', 'polish narrator woman', 'chinese female conversational', 'pilot over intercom', 'chinese commercial man', 'french narrator man', 'spanish narrator man', 'reading man', 'new york man', 'friendly french man', 'barbershop man', 'indian man', 'australian customer support man', 'friendly australian man', 'wise man', 'friendly reading man', 'customer support man', 'dutch confident man', 'dutch man', 'hindi reporter man', 'italian calm man', 'italian narrator man', 'swedish narrator man', 'polish confident man', 'spanish-speaking storyteller man', 'kentucky woman', 'chinese commercial woman', 'middle eastern woman', 'hindi narrator woman', 'sarah', 'sarah curious', 'laidback woman', 'reflective woman', 'helpful french lady', 'pleasant brazilian lady', 'customer support lady', 'british lady', 'wise lady', 'australian narrator lady', 'indian customer support lady', 'swedish calm lady', 'spanish narrator lady', 'salesman', 'yogaman', 'movieman', 'wizardman', 'australian woman', 'korean calm woman', 'friendly german man', 'announcer man', 'wise guide man', 'midwestern man', 'kentucky man', 'brazilian young man', 'chinese call center man', 'german reporter man', 'confident british man', 'southern man', 'classy british man', 'polite man', 'mexican man', 'korean narrator man', 'turkish narrator man', 'turkish calm man', 'hindi calm man', 'hindi narrator man', 'polish narrator man', 'polish young man', 'alabama male', 'australian male', 'anime girl', 'japanese man book', 'sweet lady', 'commercial lady', 'teacher lady', 'princess', 'commercial man', 'asmr lady', 'professional woman', 'tutorial man', 'calm french woman', 'new york woman', 'spanish-speaking lady', 'midwestern woman', 'sportsman', 'storyteller lady', 'spanish-speaking man', 'doctor mischief', 'spanish-speaking reporter man', 'young spanish-speaking woman', 'the merchant', 'stern french man', 'madame mischief', 'german storyteller man', 'female nurse', 'german conversation man', 'friendly brazilian man', 'german woman', 'southern woman', 'british customer support lady', 'chinese woman narrator', 'pleasant man', 'california girl', 'john', 'anna'` Pricing [](#pricing) ------------------------ | Model | Price | | --- | --- | | Cartesia Sonic | $65 per 1 Million characters | Updated about 8 hours ago * * * --- # Quickstart: Retrieval Augmented Generation (RAG) In this Quickstart you'll learn how to build a RAG workflow using Together AI in 6 quick steps that can be ran in under 5 minutes! We will leverage the embedding, reranking and inference endpoints. 1\. Register for an account [](#1-register-for-an-account) -------------------------------------------------------------- First, [register for an account](https://api.together.xyz/settings/api-keys) to get an API key. New accounts come with $1 to get started. Once you've registered, set your account's API key to an environment variable named `TOGETHER_API_KEY`: Shell `export TOGETHER_API_KEY=xxxxx` 2\. Install your preferred library [](#2-install-your-preferred-library) ---------------------------------------------------------------------------- Together provides an official library for Python: Python `pip install together --upgrade` Python `from together import Together client = Together(api_key = TOGETHER_API_KEY)` 3\. Data Processing and Chunking [](#3-data-processing-and-chunking) ------------------------------------------------------------------------ We will RAG over Paul Grahams latest essay titled [Founder Mode](https://paulgraham.com/foundermode.html) . The code below will scrape and load the essay into memory. Python `import requests from bs4 import BeautifulSoup def scrape_pg_essay(): url = 'https://paulgraham.com/foundermode.html' try: # Send GET request to the URL response = requests.get(url) response.raise_for_status() # Raise an error for bad status codes # Parse the HTML content soup = BeautifulSoup(response.text, 'html.parser') # Paul Graham's essays typically have the main content in a font tag # You might need to adjust this selector based on the actual HTML structure content = soup.find('font') if content: # Extract and clean the text text = content.get_text() # Remove extra whitespace and normalize line breaks text = ' '.join(text.split()) return text else: return "Could not find the main content of the essay." except requests.RequestException as e: return f"Error fetching the webpage: {e}" # Scrape the essay pg_essay = scrape_pg_essay()` Chunk the essay: Python `# Naive fixed sized chunking with overlaps def create_chunks(document, chunk_size=300, overlap=50): return [document[i : i + chunk_size] for i in range(0, len(document), chunk_size - overlap)] chunks = create_chunks(pg_essay, chunk_size=250, overlap=30)` 4\. Generate Vector Index and Perform Retrieval [](#4-generate-vector-index-and-perform-retrieval) ------------------------------------------------------------------------------------------------------ We will now use `bge-large-en-v1.5` to embed the augmented chunks above into a vector index. Python `from typing import List import numpy as np def generate_embeddings(input_texts: List[str], model_api_string: str) -> List[List[float]]: """Generate embeddings from Together python library. Args: input_texts: a list of string input texts. model_api_string: str. An API string for a specific embedding model of your choice. Returns: embeddings_list: a list of embeddings. Each element corresponds to the each input text. """ outputs = client.embeddings.create( input=input_texts, model=model_api_string, ) return [x.embedding for x in outputs.data] embeddings = generate_embeddings(chunks, "BAAI/bge-large-en-v1.5")` The function below will help us perform vector search: Python `def vector_retreival(query: str, top_k: int = 5, vector_index: np.ndarray = None) -> List[int]: """ Retrieve the top-k most similar items from an index based on a query. Args: query (str): The query string to search for. top_k (int, optional): The number of top similar items to retrieve. Defaults to 5. index (np.ndarray, optional): The index array containing embeddings to search against. Defaults to None. Returns: List[int]: A list of indices corresponding to the top-k most similar items in the index. """ query_embedding = np.array(generate_embeddings([query], 'BAAI/bge-large-en-v1.5')[0]) similarity_scores = np.dot(query_embedding, vector_index.T) return list(np.argsort(-similarity_scores)[:top_k]) top_k_indices = vector_retreival(query = "What are 'skip-level' meetings?", top_k = 5, vector_index = embeddings) top_k_chunks = [chunks[i] for i in top_k_indices]` We now have a way to retrieve from the vector index given a query. 5\. Rerank To Improve Quality [](#5-rerank-to-improve-quality) ------------------------------------------------------------------ We will use a reranker model to improve retrieved chunk relevance quality: Python `def rerank(query: str, chunks: List[str], top_k = 3) -> List[int]: response = client.rerank.create( model = "Salesforce/Llama-Rank-V1", query = query, documents = chunks, top_n = top_k ) return [result.index for result in response.results] rerank_indices = rerank("What are 'skip-level' meetings?", chunks = top_k_chunks, top_k=3) reranked_chunks = '' for index in rerank_indices: reranked_chunks += top_k_chunks[index] + '\n\n' print(reranked_chunks)` 6\. Call Generative Model - Llama 405b [](#6-call-generative-model---llama-405b) ------------------------------------------------------------------------------------ We will pass the final 3 concatenated chunks into an LLM to get our final answer. Python `query = "What are 'skip-level' meetings?" response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo", messages=[ {"role": "system", "content": "You are a helpful chatbot."}, {"role": "user", "content": f"Answer the question: {query}. Use only information provided here: {reranked_chunks}"}, ], ) response.choices[0].message.content` If you want to learn more about how to best use open models refer to our [docs](/docs) here! Updated about 2 months ago * * * --- # Quickstart: Next.js In this guide you'll learn how to use Together AI and Next.js to build two common AI features: * Ask a question and getting a response * Have a long-running chat with a bot [Here's the live demo](https://together-nextjs-chat.vercel.app/) , and [here's the source on GitHub](https://github.com/samselikoff/together-nextjs-chat) . Let's get started! Installation [](#installation) ================================== After [creating a new Next.js app](https://nextjs.org/docs/app/getting-started/installation) , install the [Together AI TypeScript SDK](https://www.npmjs.com/package/together-ai) : `npm i together-ai` Ask a single question [](#ask-a-single-question) ==================================================== To ask a question with Together AI, we'll need an API route, and a page with a form that lets the user submit their question. **1\. Create the API route** Make a new POST route that takes in a `question` and returns a chat completion as a stream: TypeScript `// app/api/answer/route.ts import Together from "together-ai"; const together = new Together(); export async function POST(request: Request) { const { question } = await request.json(); const res = await together.chat.completions.create({ model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", messages: [{ role: "user", content: question }], stream: true, }); return new Response(res.toReadableStream()); }` **2\. Create the page** Add a form that sends a POST request to your new API route, and use the `ChatCompletionStream` helper to read the stream and update some React state to display the answer: TypeScript `// app/page.tsx "use client"; import { FormEvent, useState } from "react"; import { ChatCompletionStream } from "together-ai/lib/ChatCompletionStream"; export default function Chat() { const [question, setQuestion] = useState(""); const [answer, setAnswer] = useState(""); const [isLoading, setIsLoading] = useState(false); async function handleSubmit(e: FormEvent) { e.preventDefault(); setIsLoading(true); setAnswer(""); const res = await fetch("/api/answer", { method: "POST", body: JSON.stringify({ question }), }); if (!res.body) return; ChatCompletionStream.fromReadableStream(res.body) .on("content", (delta) => setAnswer((text) => text + delta)) .on("end", () => setIsLoading(false)); } return (
setQuestion(e.target.value)} placeholder="Ask me a question" required />

{answer}

); }` That's it! Submitting the form will update the page with the LLM's response. You can now use the `isLoading` state to add additional styling, or a Reset button if you want to reset the page. Have a long-running chat [](#have-a-long-running-chat) ========================================================== To build a chatbot with Together AI, we'll need an API route that accepts an array of messages, and a page with a form that lets the user submit new messages. The page will also need to store the entire history of messages between the user and the AI assistant. **1\. Create an API route** Make a new POST route that takes in a `messages` array and returns a chat completion as a stream: TypeScript `// app/api/chat/route.ts import Together from "together-ai"; const together = new Together(); export async function POST(request: Request) { const { messages } = await request.json(); const res = await together.chat.completions.create({ model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", messages, stream: true, }); return new Response(res.toReadableStream()); }` **2\. Create a page** Create a form to submit a new message, and some React state to stores the `messages` for the session. In the form's submit handler, send over the new array of messages, and use the `ChatCompletionStream` helper to read the stream and update the last message with the LLM's response. TypeScript `// app/page.tsx "use client"; import { FormEvent, useState } from "react"; import Together from "together-ai"; import { ChatCompletionStream } from "together-ai/lib/ChatCompletionStream"; export default function Chat() { const [prompt, setPrompt] = useState(""); const [messages, setMessages] = useState< Together.Chat.Completions.CompletionCreateParams.Message[] >([]); const [isPending, setIsPending] = useState(false); async function handleSubmit(e: FormEvent) { e.preventDefault(); setPrompt(""); setIsPending(true); setMessages((messages) => [...messages, { role: "user", content: prompt }]); const res = await fetch("/api/chat", { method: "POST", body: JSON.stringify({ messages: [...messages, { role: "user", content: prompt }], }), }); if (!res.body) return; ChatCompletionStream.fromReadableStream(res.body) .on("content", (delta, content) => { setMessages((messages) => { const lastMessage = messages.at(-1); if (lastMessage?.role !== "assistant") { return [...messages, { role: "assistant", content }]; } else { return [...messages.slice(0, -1), { ...lastMessage, content }]; } }); }) .on("end", () => { setIsPending(false); }); } return (
setPrompt(e.target.value)} />
{messages.map((message, i) => (

{message.role}: {message.content}

))}
); }` You've just built a simple chatbot with Together AI! Updated 1 day ago * * * --- # Quickstart: Using Vercel's AI SDK with Together AI The Vercel AI SDK is a powerful Typescript library designed to help developers build AI-powered applications. Using Together AI and the Vercel AI SDK, you can easily integrate AI into your TypeScript, React, or Next.js project. In this tutorial, we'll look into how easy it is to use Together AI's models and the Vercel AI SDK. QuickStart: 15 lines of code [](#quickstart-15-lines-of-code) ================================================================= 1. Install both the Vercel AI SDK and OpenAI's Vercel package. npmyarn `npm i ai @ai-sdk/openai` `yarn add ai @ai-sdk/openai` 2. Instantiate the Together client and call the `generateText` function with Llama 3.1 8B to generate some text. TypeScript `import { createOpenAI } from "@ai-sdk/openai"; import { generateText } from "ai"; const together = createOpenAI({ apiKey: process.env.TOGETHER_API_KEY ?? "", baseURL: "https://api.together.xyz/v1", }); async function main() { const { text } = await generateText({ model: together("meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"), prompt: "Write a vegetarian lasagna recipe for 4 people.", }); console.log(text); } main();` ### Output [](#output) `Here's a delicious vegetarian lasagna recipe for 4 people: **Ingredients:** - 8-10 lasagna noodles - 2 cups marinara sauce (homemade or store-bought) - 1 cup ricotta cheese - 1 cup shredded mozzarella cheese - 1 cup grated Parmesan cheese - 1 cup frozen spinach, thawed and drained - 1 cup sliced mushrooms - 1 cup sliced bell peppers - 1 cup sliced zucchini - 1 small onion, chopped - 2 cloves garlic, minced - 1 cup chopped fresh basil - Salt and pepper to taste - Olive oil for greasing the baking dish **Instructions:** 1. **Preheat the oven:** Preheat the oven to 375°F (190°C). 2. **Prepare the vegetables:** Sauté the mushrooms, bell peppers, zucchini, and onion in a little olive oil until they're tender. Add the garlic and cook for another minute. 3. **Prepare the spinach:** Squeeze out as much water as possible from the thawed spinach. Mix it with the ricotta cheese and a pinch of salt and pepper. 4. **Assemble the lasagna:** Grease a 9x13-inch baking dish with olive oil. Spread a layer of marinara sauce on the bottom. Arrange 4 lasagna noodles on top. 5. **Layer 1:** Spread half of the spinach-ricotta mixture on top of the noodles. Add half of the sautéed vegetables and half of the shredded mozzarella cheese. 6. **Layer 2:** Repeat the layers: marinara sauce, noodles, spinach-ricotta mixture, sautéed vegetables, and mozzarella cheese. 7. **Top layer:** Spread the remaining marinara sauce on top of the noodles. Sprinkle with Parmesan cheese and a pinch of salt and pepper. 8. **Bake the lasagna:** Cover the baking dish with aluminum foil and bake for 30 minutes. Remove the foil and bake for another 10-15 minutes, or until the cheese is melted and bubbly. 9. **Let it rest:** Remove the lasagna from the oven and let it rest for 10-15 minutes before slicing and serving. **Tips and Variations:** - Use a variety of vegetables to suit your taste and dietary preferences. - Add some chopped olives or artichoke hearts for extra flavor. - Use a mixture of mozzarella and Parmesan cheese for a richer flavor. - Serve with a side salad or garlic bread for a complete meal. **Nutrition Information (approximate):** Per serving (serves 4): - Calories: 450 - Protein: 25g - Fat: 20g - Saturated fat: 8g - Cholesterol: 30mg - Carbohydrates: 40g - Fiber: 5g - Sugar: 10g - Sodium: 400mg Enjoy your delicious vegetarian lasagna!` Streaming with the Vercel AI SDK [](#streaming-with-the-vercel-ai-sdk) ========================================================================== To stream from Together AI models using the Vercel AI SDK, simply use `streamText` as seen below. TypeScript `import { createOpenAI } from "@ai-sdk/openai"; import { streamText } from "ai"; const together = createOpenAI({ apiKey: process.env.TOGETHER_API_KEY ?? "", baseURL: "https://api.together.xyz/v1", }); async function main() { const result = await streamText({ model: together("meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"), prompt: "Invent a new holiday and describe its traditions.", }); for await (const textPart of result.textStream) { process.stdout.write(textPart); } } main();` ### Output [](#output-1) `Introducing "Luminaria Day" - a joyous holiday celebrated on the spring equinox, marking the return of warmth and light to the world. This festive occasion is a time for family, friends, and community to come together, share stories, and bask in the radiance of the season. **Date:** Luminaria Day is observed on the spring equinox, typically around March 20th or 21st. **Traditions:** 1. **The Lighting of the Lanterns:** As the sun rises on Luminaria Day, people gather in their neighborhoods, parks, and public spaces to light lanterns made of paper, wood, or other sustainable materials. These lanterns are adorned with intricate designs, symbols, and messages of hope and renewal. 2. **The Storytelling Circle:** Families and friends gather around a central fire or candlelight to share stories of resilience, courage, and triumph. These tales are passed down through generations, serving as a reminder of the power of human connection and the importance of learning from the past. 3. **The Luminaria Procession:** As the sun sets, communities come together for a vibrant procession, carrying their lanterns and sharing music, dance, and laughter. The procession winds its way through the streets, symbolizing the return of light and life to the world. 4. **The Feast of Renewal:** After the procession, people gather for a festive meal, featuring dishes made with seasonal ingredients and traditional recipes. The feast is a time for gratitude, reflection, and celebration of the cycle of life. 5. **The Gift of Kindness:** On Luminaria Day, people are encouraged to perform acts of kindness and generosity for others. This can take the form of volunteering, donating to charity, or simply offering a helping hand to a neighbor in need. **Symbolism:** * The lanterns represent the light of hope and guidance, illuminating the path forward. * The storytelling circle symbolizes the power of shared experiences and the importance of learning from one another. * The procession represents the return of life and energy to the world, as the seasons shift from winter to spring. * The feast of renewal celebrates the cycle of life, death, and rebirth. * The gift of kindness embodies the spirit of generosity and compassion that defines Luminaria Day. **Activities:** * Create your own lanterns using recycled materials and decorate them with symbols, messages, or stories. * Share your own stories of resilience and triumph with family and friends. * Participate in the Luminaria Procession and enjoy the music, dance, and laughter. * Prepare traditional dishes for the Feast of Renewal and share them with loved ones. * Perform acts of kindness and generosity for others, spreading joy and positivity throughout your community. Luminaria Day is a time to come together, celebrate the return of light and life, and honor the power of human connection.` Updated 1 day ago * * * --- # Error codes | Code | Cause | Solution | | --- | --- | --- | | 400 - Invalid Request | Misconfigured request | Ensure your request is a [Valid JSON](/docs/inference-rest#create-your-json-formatted-object)
and your [API Key](https://api.together.xyz/settings/api-keys)
is correct. Also ensure you're using the right prompt format - which is different for Mistral and LLaMA models. | | 401 - Authentication Error | Missing or Invalid API Key | Ensure you are using the correct [API Key](https://api.together.xyz/settings/api-keys)
and [supplying it correctly](/reference/inference) | | 402 - Payment Required | The account associated with the API key has reached its maximum allowed monthly spending limit. | Adjust your [billing settings](https://api.together.xyz/settings/billing)
or make a payment to resume service. | | 403 - Bad Request | Input token count + `max_tokens` parameter must be less than the [context](/docs/inference-models)
length of the model being queried. | Set `max_tokens` to a lower number. If querying a chat model, you may set `max_tokens` to `null` and let the model decide when to stop generation. | | 404 - Not Found | Invalid Endpoint URL or model name | Check your request is being made to the correct endpoint (see the [API reference](/reference/inference)
page for details) and that the [model being queried is available](/docs/inference-models) | | 429 - Rate limit | Too many requests sent in a short period of time | Throttle the rate at which requests are sent to our servers (see our [rate limits](/docs/rate-limits)
) | | 500 - Server Error | Unknown server error | This error is caused by an issue on our servers. Please try again after a brief wait. If the issue persists, please [contact support](https://www.together.ai/contact) | | 503 - Engine Overloaded | Our servers are seeing high amounts of traffic | Please try again after a brief wait. If the the issue persists, please [contact support](https://www.together.ai/contact) | If you are seeing other error codes or the solutions do not work, please [contact support](https://www.together.ai/contact) for help. Updated 4 months ago * * * --- # Rate limits Rate limiting refers to the constraints our API enforces on how frequently a user or client can access our services within a given timeframe. Rate limits are denoted as HTTP status code 429s. Read more about our rate limit tiers below, and find out how you can increase them here: * If you have a high volume of steady traffic and good payment history for this traffic, you can request a higher limit [here](https://together.ai/forms/scale-ent) . * If you are interested in our Scale or Enterprise packages, with custom requests per minute (RPM) and unlimited tokens per minute (TPM), please reach out to sales [here](https://together.ai/forms/scale-ent) . ### What is the purpose of rate limits? [](#what-is-the-purpose-of-rate-limits) Rate limits in APIs are a standard approach, and they serve to safeguard against abuse or misuse of the API, helping to ensure equitable access to the API with consistent performance. ### How are our rate limits implemented? [](#how-are-our-rate-limits-implemented) Our rate limits are currently measured in requests per second (RPS) and tokens per second (TPS) for each model type. If you exceed any of the rate limits you will get a 429 error. We show you the values per minute below, as its the industry standard. ### Rate limit tiers [](#rate-limit-tiers) You can view your rate limit by navigating to Settings > Billing. As your usage of the Together API and your spend on our API increases, we will automatically increase your rate limits. **Chat, language & code models** | Tier | Qualification criteria | RPM | TPM | | --- | --- | --- | --- | | Free | User must be in an allowed geography | 60 | 60,000 | | Tier 1 | Credit card added | 600 | 180,000 | | Tier 2 | $50 paid | 1,800 | 250,000 | | Tier 3 | $100 paid | 3,000 | 500,000 | | Tier 4 | $250 paid | 4,500 | 1,000,000 | | Tier 5 | $1,000 paid | 6,000 | 2,000,000 | **Embedding models** | Tier | Qualification criteria | RPM | TPM | | --- | --- | --- | --- | | Free | User must be in an allowed geography | 3,000 | 1,000,000 | | Tier 1 | Credit card added | 3,000 | 2,000,000 | | Tier 2 | $50 paid | 5,000 | 2,000,000 | | Tier 3 | $100 paid | 5,000 | 10,000,000 | | Tier 4 | $250 paid | 10,000 | 10,000,000 | | Tier 5 | $1,000 paid | 10,000 | 20,000,000 | **Re-rank models** | Tier | Qualification criteria | RPM | TPM | | --- | --- | --- | --- | | Free | User must be in an allowed geography | 1,000 | 150,000 | | Tier 1 | Credit card added | 2,500 | 500,000 | | Tier 2 | $50 paid | 3,500 | 1,500,000 | | Tier 3 | $100 paid | 4,000 | 2,000,000 | | Tier 4 | $250 paid | 7,500 | 3,000,000 | | Tier 5 | $1,000 paid | 9,000 | 5,000,000 | **Image models** | Tier | Qualification criteria | Img/min | | --- | --- | --- | | Free | User must be in an allowed geography | 60 | | Tier 1 | Credit card added | 240 | | Tier 2 | $50 paid | 480 | | Tier 3 | $100 paid | 600 | | Tier 4 | $250 paid | 960 | | Tier 5 | $1,000 paid | 1200 | Note: FLUX.1 \[schnell\] Free has a model specific rate limit of 10 img/min. You may experience congestion based on traffic from other users, and may be throttled to a lower level because of that. If you want committed capacity, [contact](https://together.ai/forms/scale-ent) our sales team to inquire about our Scale and Enterprise plans, which include custom RPM and unlimited TPM. **Rate limits in headers** The API response includes headers that display the rate limit enforcement, current usage, and when the limit will reset. We enforce limits per second and minute for token usage and per second for request rates, but the headers display per second limits only. | Field | Description | | --- | --- | | x-ratelimit-limit | The maximum number of requests per sec that are permitted before exhausting the rate limit. | | x-ratelimit-remaining | The remaining number of requests per sec that are permitted before exhausting the rate limit. | | x-ratelimit-reset | The time until the rate limit (based on requests per sec) resets to its initial state. | | x-tokenlimit-limit | The maximum number of tokens per sec that are permitted before exhausting the rate limit. | | x-tokenlimit-remaining | The remaining number of tokens per sec that are permitted before exhausting the rate limit. | Updated 4 months ago * * * --- # Deployment Options Together Enterprise Platform Deployment Options [](#together-enterprise-platform-deployment-options) ======================================================================================================== Together AI offers a flexible and powerful platform that enables organizations to deploy in a way that best suits their needs. Whether you're looking for a fully-managed cloud solution, or secure VPC deployment on any cloud, Together AI provides robust tools, superior performance, and comprehensive support. Deployment Options Overview [](#deployment-options-overview) ---------------------------------------------------------------- Together AI provides two key deployment options: * **Together AI Cloud**: A fully-managed, inference platform that is fast, scalable, and cost-efficient. * **VPC Deployment**: Deploy Together AI's Enterprise Platform within your own Virtual Private Cloud (VPC) on any cloud platform for enhanced security and control. The following sections provide an overview of each deployment type, along with a detailed responsibility matrix comparing the features and benefits of each option. * * * Together AI Cloud [](#together-ai-cloud) -------------------------------------------- Together AI Cloud is a fully-managed service that runs in Together AI's cloud infrastructure. With seamless access to Together's products, this option is ideal for companies that want to get started quickly without the overhead of managing their own infrastructure. ### Key Features [](#key-features) * **Fully Managed**: Together AI handles infrastructure, scaling, and orchestration. * **Fast and Scalable**: Both Dedicated and Serverless API endpoints ensure optimal performance and scalability. * **Cost-Effective**: Pay-as-you-go pricing with the option for reserved endpoints at a discount. * **Privacy & Security**: Full control over your data; Together AI ensures SOC 2 and HIPAA compliance. * **Ideal Use Case**: Best suited for AI-native startups and companies that need fast, easy deployment without infrastructure management. For more information on Together AI Cloud, [contact our team](#) . * * * Together AI VPC Deployment [](#together-ai-vpc-deployment) -------------------------------------------------------------- Together AI VPC Deployment allows you to deploy the platform in your own Virtual Private Cloud (VPC) on any cloud provider (such as Google Cloud, Azure, AWS, or others). This option is ideal for enterprises that need enhanced security, control, and compliance while benefiting from Together AI's powerful AI stack. ### Key Features [](#key-features-1) * **Cloud-Agnostic**: Deploy within your VPC on any cloud platform of your choice (e.g., AWS, Azure, Google Cloud). * **Full Control**: Complete administrative access, enabling you to manage and control ingress and egress traffic within your VPC. * **High Performance**: Achieve up to 2x faster performance on your existing infrastructure, optimized for your environment. * **Data Sovereignty**: Data never leaves your controlled environment, ensuring complete security and compliance. * **Customization**: Tailor scaling, performance, and resource allocation to fit your infrastructure’s specific needs. * **Ideal Use Case**: Perfect for enterprises with strict security, privacy, and compliance requirements who want to retain full control over their cloud infrastructure. ### Example: VPC Deployment in AWS [](#example-vpc-deployment-in-aws) Below is an example of how Together AI VPC Deployment works in an AWS environment. This system diagram illustrates the architecture and flow: ![VPC Deployment in AWS](https://files.readme.io/4d91f4692c4c1c0daa087650e7af7b59d55f2d4fb5e87daecbfcccb90a3a3382-Screenshot_2024-09-18_at_3.17.04_PM.png) 1. **Secure VPC Peering**: Together AI connects to your AWS environment via secure VPC peering, ensuring data remains entirely within your AWS account. 2. **Private Subnets**: All data processing and model inference happens within private subnets, isolating resources from the internet. 3. **Control of Ingress/Egress Traffic**: You have full control over all traffic entering and leaving your VPC, including restrictions on external network access. 4. **Data Sovereignty**: Since all computations are performed within your VPC, data never leaves your controlled environment. 5. **Custom Scaling**: Leverage AWS autoscaling groups to ensure that your AI workloads scale seamlessly with demand, while maintaining complete control over resources. Although this example uses AWS, the architecture can be adapted to other cloud providers such as Azure or Google Cloud with similar capabilities. For more information on VPC deployment, [get in touch with us](#) . * * * Comparison of Deployment Options [](#comparison-of-deployment-options) -------------------------------------------------------------------------- | Feature | Together AI Cloud | Together AI VPC Deployment | | --- | --- | --- | | **How It Works** | Fully-managed, serverless API endpoints.

On-demand and reserved dedicated endpoints for production workloads - with consistent performance and no rate limits. | Deploy Together's Platform and inference stack in your VPC on any cloud platform. | | **Performance** | Optimal performance with Together inference stack and Together Turbo Endpoints. | Better performance on your infrastructure: Up to 2x better speed on existing infrastructure | | **Cost** | Pay-as-you-go, or discounts for reserved endpoints. | Lower TCO through faster performance and optimized GPU usage. | | **Management** | Fully-managed service, no infrastructure to manage. | You manage your VPC, with Together AI’s support. Managed service offering also available. | | **Scaling** | Automatic scaling to meet demand. | Intelligent scaling based on your infrastructure. Fully customizable. | | **Data Privacy & Security** | Data ownership with SOC 2 and HIPAA compliance. | Data never leaves your environment. | | **Compliance** | SOC 2 and HIPAA compliant. | Implement security and compliance controls to match internal standards. | | **Support** | 24/7 support with guaranteed SLAs. | Dedicated support with engineers on call. | | **Ideal For** | Startups and companies that want quick, easy access to AI infrastructure without managing it. | Enterprises with stringent security and privacy needs, or those leveraging existing cloud infrastructure. | * * * Next Steps [](#next-steps) ------------------------------ To get started with Together AI’s platform, **we recommend [trying the Together AI Cloud](https://api.together.ai/signin) ** for quick deployment and experimentation. If your organization has specific security, infrastructure, or compliance needs, consider Together AI VPC. For more information, or to find the best deployment option for your business, [contact our team](https://www.together.ai/forms/contact-sales) . Updated about 1 month ago * * * --- # Together Mixture-Of-Agents (MoA) ![](https://files.readme.io/8c88157-image.png) What is Together MoA? [](#what-is-together-moa) --------------------------------------------------- Mixture of Agents (MoA) is a novel approach that leverages the collective strengths of multiple LLMs to enhance performance, achieving state-of-the-art results. By employing a layered architecture where each layer comprises several LLM agents, **MoA significantly outperforms** GPT-4 Omni’s 57.5% on AlpacaEval 2.0 with a score of 65.1%, using only open-source models! The way Together MoA works is that given a prompt, like `tell me the best things to do in SF`, it sends it to 4 different OSS LLMs. It then combines results from all 4, sends it to a final LLM, and asks it to combine all 4 responses into an ideal response. That’s it! It’s just the idea of combining the results of 4 different LLMs to produce a better final output. It’s obviously slower than using a single LLM but it can be great for use cases where latency doesn't matter as much like synthetic data generation. For a quick summary and 3-minute demo on how to implement MoA with code, watch the video below: Together MoA in 50 lines of code [](#together-moa-in-50-lines-of-code) -------------------------------------------------------------------------- To get to get started with using MoA in your own apps, you'll need to install the Together python library, get your Together API key, and run the code below which uses our chat completions API to interact with OSS models. 1. Install the Together Python library Shell `pip install together` 2. Get your [Together API key](https://api.together.xyz/settings/api-keys) & export it Shell `export TOGETHER_API_KEY='xxxx'` 3. Run the code below, which interacts with our chat completions API. This implementation of MoA uses 2 layers and 4 LLMs. We’ll define our 4 initial LLMs and our aggregator LLM, along with our prompt. We’ll also add in a prompt to send to the aggregator to combine responses effectively. Now that we have this, we’ll simply send the prompt to the 4 LLMs and compute all results simultaneously. Finally, we'll send the results from the four LLMs to our final LLM, along with a system prompt instructing it to combine them into a final answer, and we’ll stream results back. Python `# Mixture-of-Agents in 50 lines of code import asyncio import os from together import AsyncTogether, Together client = Together() async_client = AsyncTogether() user_prompt = "What are some fun things to do in SF?" reference_models = [ "Qwen/Qwen2-72B-Instruct", "meta-llama/Llama-3.3-70B-Instruct-Turbo", "mistralai/Mixtral-8x22B-Instruct-v0.1", "databricks/dbrx-instruct", ] aggregator_model = "mistralai/Mixtral-8x22B-Instruct-v0.1" aggreagator_system_prompt = """You have been provided with a set of responses from various open-source models to the latest user query. Your task is to synthesize these responses into a single, high-quality response. It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability. Responses from models:""" async def run_llm(model): """Run a single LLM call with a reference model.""" response = await async_client.chat.completions.create( model=model, messages=[{"role": "user", "content": user_prompt}], temperature=0.7, max_tokens=512, ) print(model) return response.choices[0].message.content async def main(): results = await asyncio.gather(*[run_llm(model) for model in reference_models]) finalStream = client.chat.completions.create( model=aggregator_model, messages=[ {"role": "system", "content": aggreagator_system_prompt}, {"role": "user", "content": ",".join(str(element) for element in results)}, ], stream=True, ) for chunk in finalStream: print(chunk.choices[0].delta.content or "", end="", flush=True) asyncio.run(main())` Advanced MoA example [](#advanced-moa-example) -------------------------------------------------- In the previous example, we went over how to implement MoA with 2 layers (4 LLMs answering and one LLM aggregating). However, one strength of MoA is being able to go through several layers to get an even better response. In this example, we'll go through how to run MoA with 3+ layers. ![](https://files.readme.io/ddb138e-moa-3layer.png) Python `# Advanced Mixture-of-Agents example – 3 layers import asyncio import os import together from together import AsyncTogether, Together client = Together() async_client = AsyncTogether() user_prompt = "What are 3 fun things to do in SF?" reference_models = [ "Qwen/Qwen2-72B-Instruct", "meta-llama/Llama-3.3-70B-Instruct-Turbo", "mistralai/Mixtral-8x22B-Instruct-v0.1", "databricks/dbrx-instruct", ] aggregator_model = "mistralai/Mixtral-8x22B-Instruct-v0.1" aggreagator_system_prompt = """You have been provided with a set of responses from various open-source models to the latest user query. Your task is to synthesize these responses into a single, high-quality response. It is crucial to critically evaluate the information provided in these responses, recognizing that some of it may be biased or incorrect. Your response should not simply replicate the given answers but should offer a refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards of accuracy and reliability. Responses from models:""" layers = 3 def getFinalSystemPrompt(system_prompt, results): """Construct a system prompt for layers 2+ that includes the previous responses to synthesize.""" return ( system_prompt + "\n" + "\n".join([f"{i+1}. {str(element)}" for i, element in enumerate(results)]) ) async def run_llm(model, prev_response=None): """Run a single LLM call with a model while accounting for previous responses + rate limits.""" for sleep_time in [1, 2, 4]: try: messages = ( [ { "role": "system", "content": getFinalSystemPrompt( aggreagator_system_prompt, prev_response ), }, {"role": "user", "content": user_prompt}, ] if prev_response else [{"role": "user", "content": user_prompt}] ) response = await async_client.chat.completions.create( model=model, messages=messages, temperature=0.7, max_tokens=512, ) print("Model: ", model) break except together.error.RateLimitError as e: print(e) await asyncio.sleep(sleep_time) return response.choices[0].message.content async def main(): """Run the main loop of the MOA process.""" results = await asyncio.gather(*[run_llm(model) for model in reference_models]) for _ in range(1, layers - 1): results = await asyncio.gather( *[run_llm(model, prev_response=results) for model in reference_models] ) finalStream = client.chat.completions.create( model=aggregator_model, messages=[ { "role": "system", "content": getFinalSystemPrompt(aggreagator_system_prompt, results), }, {"role": "user", "content": user_prompt}, ], stream=True, ) for chunk in finalStream: print(chunk.choices[0].delta.content or "", end="", flush=True) asyncio.run(main())` Resources [](#resources) ---------------------------- * [Together MoA GitHub Repo](https://github.com/togethercomputer/MoA) (includes an interactive demo) * [Together MoA blog post](https://www.together.ai/blog/together-moa) * [MoA Technical Paper](https://arxiv.org/abs/2406.04692) Updated 17 days ago * * * --- # Cluster user management Prior to adding any user to your cluster, please make sure the user has created an account and added an SSH key in the [Together Playground](https://api.together.xyz/playground/) . Users can add an SSH key [here](https://api.together.xyz/settings/ssh-key) . For more information, please see [Quickstart](/docs/quickstart) . To add users to your cluster, please follow these steps: Log in to your Together AI account. In the circle in the right hand corner, click into your avatar and select “Settings” from the drop down menu. ![](https://files.readme.io/7039280-Screenshot_2024-06-17_at_3.50.42_PM.png) On the left hand side, select Members. ![](https://files.readme.io/df1e4dc-Screenshot_2024-06-17_at_3.52.04_PM.png) At the top of Members, select “Add User”. ![](https://files.readme.io/c632c65-Screenshot_2024-06-17_at_3.54.41_PM.png) A popup will appear. In this popup, please enter the email of the user. ![](https://files.readme.io/b141846-Screenshot_2024-06-17_at_3.55.26_PM.png) If the user does not have an Playground account or SSH key, you will see an error indicating that the user cannot be added. ![](https://files.readme.io/2545b20-Screenshot_2024-06-17_at_3.56.14_PM.png) Once you click add user, the user will appear in the grid. ![](https://files.readme.io/c17a715-Screenshot_2024-06-17_at_3.56.48_PM.png) To remove this user, press the 3 dots on the right side and select “Remove user”. ![](https://files.readme.io/6edcd0e-Screenshot_2024-06-17_at_3.57.34_PM.png) Updated 7 months ago * * * --- # RAG Integrations Using MongoDB [](#using-mongodb) ------------------------------------ See [this tutorial blog](https://www.together.ai/blog/rag-tutorial-mongodb) for the RAG implementation details using Together and MongoDB. Using LangChain [](#using-langchain) ---------------------------------------- See [this tutorial blog](https://www.together.ai/blog/rag-tutorial-langchain) for the RAG implementation details using Together and LangChain. * [LangChain TogetherEmbeddings](https://python.langchain.com/docs/integrations/text_embedding/together) * [LangChain Together](https://python.langchain.com/docs/integrations/llms/together) Using LlamaIndex [](#using-llamaindex) ------------------------------------------ See [this tutorial blog](https://www.together.ai/blog/rag-tutorial-llamaindex) for the RAG implementation details using Together and LlamaIndex. * [LlamaIndex TogetherEmbeddings](https://docs.llamaindex.ai/en/stable/examples/embeddings/together.html) * [LlamaIndex TogetherLLM](https://docs.llamaindex.ai/en/stable/examples/llm/together.html) Using Pixeltable [](#using-pixeltable) ------------------------------------------ See [this tutorial blog](https://pixeltable.readme.io/docs/together-ai) for the RAG implementation details using Together and Pixeltable. Updated 5 months ago * * * --- # Embeddings Together's Embeddings API lets you turn some input text (the _input_) into an array of numbers (the _embedding_). The resulting embedding can be compared against other embeddings to determine how closely related the two input strings are. Embeddings from large datasets can be stored in vector databases for later retrieval or comparison. Common use cases for embeddings are search, classification, and recommendations. They're also used for building Retrieval Augmented Generation (RAG) applications. Generating a single embedding [](#generating-a-single-embedding) -------------------------------------------------------------------- Use `client.embeddings.create` to generate an embedding for some input text, passing in a model name and input string: PythonTypeScriptHTTP `import os from together import Together client = Together() response = client.embeddings.create( model = "togethercomputer/m2-bert-80M-8k-retrieval", input = "Our solar system orbits the Milky Way galaxy at about 515,000 mph" )` `import Together from "together-ai"; const together = new Together(); const response = await client.embeddings.create({ model: 'togethercomputer/m2-bert-80M-8k-retrieval', input: 'Our solar system orbits the Milky Way galaxy at about 515,000 mph', });` `curl -X POST https://api.together.xyz/v1/embeddings \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input": "Our solar system orbits the Milky Way galaxy at about 515,000 mph.", "model": "togethercomputer/m2-bert-80M-8k-retrieval" }'` The response will be an object that contains the embedding under the `data` key, as well as some metadata: JSON `{ model: 'togethercomputer/m2-bert-80M-8k-retrieval', object: 'list', data: [ { index: 0, object: 'embedding', embedding: [0.2633975, 0.13856208, ..., 0.04331574], }, ], };` Generating multiple embeddings [](#generating-multiple-embeddings) ---------------------------------------------------------------------- You can also pass an array of input strings to the `input` option: PythonHTTP `import os from together import Together client = Together() response = client.embeddings.create( model = "togethercomputer/m2-bert-80M-8k-retrieval", input = [ "Our solar system orbits the Milky Way galaxy at about 515,000 mph", "Jupiter's Great Red Spot is a storm that has been raging for at least 350 years." ] )` `curl -X POST https://api.together.xyz/v1/embeddings \ -H "Authorization: Bearer $TOGETHER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "togethercomputer/m2-bert-80M-8k-retrieval", "input": [ "Our solar system orbits the Milky Way galaxy at about 515,000 mph", "Jupiter'\''s Great Red Spot is a storm that has been raging for at least 350 years." ] }'` The `response.data` key will contain an array of objects for each input string you provide: JSON `{ model: 'togethercomputer/m2-bert-80M-8k-retrieval', object: 'list', data: [ { index: 0, object: 'embedding', embedding: [0.2633975, 0.13856208, ..., 0.04331574], }, { index: 1, object: 'embedding', embedding: [-0.14496337, 0.21044481, ..., -0.16187587] }, ], };` Updated 3 months ago * * * --- # CodeSandbox SDK (Code execution) The CodeSandbox SDK enables you to programmatically spin up development environments and run untrusted code. It provides a programmatic API to create and run sandboxes quickly and securely. Use Cases [](#use-cases) ---------------------------- The main use cases for the SDK are: * **Agentic workflows**: Build coding agents that can run code to make decisions by calling APIs, processing data & performing calculations * **Data analysis & viz**: Analyze datasets to provide on-the-fly analysis and visualizations like charts and graphs * **Cloud code environments**: Spin up personalized VM sandboxes that can render a code editor in the browser for each user * **Dynamic file processing**: Automate the handling and processing of user-uploaded files, such as converting formats or extracting data Getting Started [](#getting-started) ======================================== > Note: The CodeSandbox SDK is only available on TypeScript for now, Python support is coming! To get started, install the SDK: `npm install @codesandbox/sdk` Then, create an API token by going to [https://codesandbox.io/t/api](https://codesandbox.io/t/api) , and clicking on the "Create API Token" button. You can then use this token to authenticate with the SDK: TypeScript ``import { CodeSandbox } from "@codesandbox/sdk"; // Create the client with your token const sdk = new CodeSandbox(token); // This creates a new sandbox by forking our default template sandbox. // You can also pass in other template ids, or create your own template to fork from. const sandbox = await sdk.sandbox.create(); // You can run JS code directly await sandbox.shells.js.run("console.log(1+1)"); // Or Python code (if it's installed in the template) await sandbox.shells.python.run("print(1+1)"); // Or anything else await sandbox.shells.run("echo 'Hello, world!'"); // We have a FS API to interact with the filesystem await sandbox.fs.writeTextFile("./hello.txt", "world"); // And you can clone sandboxes! This does not only clone the filesystem, processes that are running in the original sandbox will also be cloned! const sandbox2 = await sandbox.fork(); // Check that the file is still there await sandbox2.fs.readTextFile("./hello.txt"); // You can also get the opened ports, with the URL to access those console.log(sandbox2.ports.getOpenedPorts()); // Getting metrics... const metrics = await sandbox2.getMetrics(); console.log( `Memory: ${metrics.memory.usedKiB} KiB / ${metrics.memory.totalKiB} KiB` ); console.log(`CPU: ${(metrics.cpu.used / metrics.cpu.cores) * 100}%`); // Finally, you can hibernate a sandbox. This will snapshot the sandbox and stop it. Next time you start the sandbox, it will continue where it left off, as we created a memory snapshot. await sandbox.hibernate(); await sandbox2.hibernate(); // Open the sandbox again const resumedSandbox = await sdk.sandbox.open(sandbox.id);`` CodeSandbox Integration [](#codesandbox-integration) -------------------------------------------------------- This SDK uses the API token from your workspace in CodeSandbox to authenticate and create sandboxes. Because of this, the sandboxes will be created inside your workspace, and the resources will be billed to your workspace. You could, for example, create a private template in your workspace that has all the dependencies you need (even running servers), and then use that template to fork sandboxes from. This way, you can control the environment that the sandboxes run in. Features [](#features) -------------------------- Under the hood, the SDK uses the microVM infrastructure of CodeSandbox to spin up sandboxes. It supports: * Starting fresh VMs within 4 seconds * Snapshotting/restoring VMs (checkpointing) at any point in time * With snapshot restore times of less than 2 seconds * Cloning VMs within 3 seconds * Source control (git, GitHub, CodeSandbox SCM) * Running any Dockerfile Updated about 1 month ago * * * --- # Slurm management system Slurm [](#slurm) ==================== Slurm is a cluster management system that allows users to manage and schedule jobs on a cluster of computers. A Together GPU Cluster provides Slurm configured out-of-the-box for distributed training and the option to use your own scheduler.  Users can submit computing jobs to the Slurm head node where the scheduler will assign the tasks to available GPU nodes based on resource availability. For more information on Slurm, see the [Slurm Quick Start User Guide](https://slurm.schedmd.com/quickstart.html) . **Slurm Basic Concepts** [](#slurm-basic-concepts) ------------------------------------------------------ 1. **Jobs**: A job is a unit of work that is submitted to the cluster. Jobs can be scripts, programs, or other types of tasks. 2. **Nodes**: A node is a computer in the cluster that can run jobs. Nodes can be physical machines or virtual machines. 3. **Head Node**: Each Together GPU Cluster cluster is configured with head node. A user will login to the head node to write jobs, submit jobs to the GPU cluster, and retrieve the results. 4. **Partitions**: A partition is a group of nodes that can be used to run jobs. Partitions can be configured to have different properties, such as the number of nodes and the amount of memory available. 5. **Priorities**: Priorities are used to determine which jobs should be run first. Jobs with higher priorities are given preference over jobs with lower priorities. **Using Slurm** [](#using-slurm) ------------------------------------ 1. **Job Submission**: Jobs can be submitted to the cluster using the **`sbatch`** command. Jobs can be submitted in batch mode or interactively using the **`srun`** command. 2. **Job Monitoring**: Jobs can be monitored using the **`squeue`** command, which displays information about the jobs that are currently running or waiting to run. 3. **Job Control**: Jobs can be controlled using the **`scancel`** command, which allows users to cancel or interrupt jobs that are running. Slurm Job Arrays [](#slurm-job-arrays) ------------------------------------------ You can use Slurm job arrays to partition input files into k chunks and distribute the chunks across the nodes. See this example on processing RPv1 which will need to be adapted to your processing: [arxiv-clean-slurm.batch](https://github.com/togethercomputer/RedPajama-Data/blob/rp_v1/data_prep/arxiv/scripts/arxiv-clean-slurm.sbatch) **Troubleshooting Slurm** [](#troubleshooting-slurm) -------------------------------------------------------- 1. **Error Messages**: Slurm provides error messages that can help users diagnose and troubleshoot problems. 2. **Log Files**: Slurm provides log files that can be used to monitor the status of the cluster and diagnose problems. Updated 8 months ago * * * --- # Cluster storage A Together GPU Cluster has 3 types of storage: ### 1\. Local disks [](#1-local-disks) Each server has NVME drives which can be used for high speed local read/writes. ### 2\. Shared `/home` folder [](#2-shared-home-folder) The `/home` folder is shared across all nodes, mounted as an NFS volume from the head node. This should be used for code, configs, logs, etc. It should not be used for training data or checkpointing, as it is slower. We recommend logging into the Slurm head node first to properly set up your user folder with the right permissions. ### 3\. Shared remote attached storage [](#3-shared-remote-attached-storage) The GPU nodes all have a mounted volume from a high-speed storage cluster, which is useful for reading training data and writing checkpoints to/from a central location. Updated 8 months ago * * * --- # Dedicated Endpoints Dashboard With Together AI, you can create on-demand dedicated endpoints with the following advantages: * Consistent, predictable performance, unaffected by other users' load in our serverless environment * No rate limits, with a high maximum load capacity * More cost-effective under high utilization * Access to a broader selection of models Creating an on demand dedicated endpoint [](#creating-an-on-demand-dedicated-endpoint) ------------------------------------------------------------------------------------------ Navigate to the [Models page](https://api.together.xyz/models) in our playground. Under "All models" click "Dedicated." Search across 179 available models. ![](https://files.readme.io/ce8b4c9-image.png) Select your hardware. We have multiple hardware options available, all with varying prices (e.g. RTX-6000, L40, A100 SXM, A100 PCIe, and H100). ![](https://files.readme.io/7c19b1a-image.png) Click the Play button, and wait up to 10 minutes for the endpoint to be deployed. ![](https://files.readme.io/4f32241-image.png) We will provide you the string you can use to call the model, as well as additional information about your deployment. ![](https://files.readme.io/a39b5e8-image.png) You can navigate away while your model is being deployed. Click open when it's ready: ![](https://files.readme.io/77384f6-image.png) Start using your endpoint! ![](https://files.readme.io/f49c65e-image.png) You can now find your endpoint in the My Models Page, and upon clicking the Model, under "Endpoints" ![](https://files.readme.io/f39da4f-image.png) ![](https://files.readme.io/4a27bcd-image.png) **Looking for custom configurations?** [Contact us.](https://www.together.ai/forms/monthly-reserved) Updated about 2 months ago * * * --- # Create tickets in Slack **Emoji Ticketing** 1. Send a message in the Together shared channel 2. Add the 🎫 (ticket) emoji reaction to convert the thread into a ticket 3. A message will pop-up in the channel. Click on the `File ticket` button to proceed 4. In the form modal, fill out the required information and click `File ticket` to submit 5. Check the thread for ticket details **Note:** The best practice is to use Slack threads by adding replies to the original post. ![](https://files.readme.io/5f8217647987df303502fc5b355197432458cac2c983c601c66ae17dc620b197-Creating_Intercom_tickets_with_Forms.gif) Updated 5 months ago * * * ---