# Table of Contents - [πŸ‘‹ What's Tabby | Tabby](#-what-s-tabby-tabby) - [Visual Studio Code | Tabby](#visual-studio-code-tabby) - [⁉️ Frequently Asked Questions | Tabby](#-frequently-asked-questions-tabby) - [Docker | Tabby](#docker-tabby) - [Upgrade | Tabby](#upgrade-tabby) - [IntelliJ Platform | Tabby](#intellij-platform-tabby) - [πŸ—ΊοΈ Roadmap | Tabby](#-roadmap-tabby) - [πŸ§‘β€πŸ”¬ Models Registry | Tabby](#-models-registry-tabby) - [Programming Languages | Tabby](#programming-languages-tabby) - [VIM / NeoVIM | Tabby](#vim-neovim-tabby) - [Configuration | Tabby](#configuration-tabby) - [Troubleshooting | Tabby](#troubleshooting-tabby) - [Answer Engine | Tabby](#answer-engine-tabby) - [Usage Collection | Tabby](#usage-collection-tabby) - [Mail Delivery | Tabby](#mail-delivery-tabby) - [Manage License | Tabby](#manage-license-tabby) - [Data Backup | Tabby](#data-backup-tabby) - [Single Sign-On | Tabby](#single-sign-on-tabby) - [Context Providers | Tabby](#context-providers-tabby) - [Deploy Tabby behind a reverse proxy | Tabby](#deploy-tabby-behind-a-reverse-proxy-tabby) - [Docker Compose | Tabby](#docker-compose-tabby) - [Homebrew (Apple M1/M2) | Tabby](#homebrew-apple-m1-m2-tabby) - [Model Configuration | Tabby](#model-configuration-tabby) - [Code Completion | Tabby](#code-completion-tabby) - [Amazon Bedrock | Tabby](#amazon-bedrock-tabby) - [Linux | Tabby](#linux-tabby) - [Connect IDE / Editor Extensions | Tabby](#connect-ide-editor-extensions-tabby) - [BentoCloud | Tabby](#bentocloud-tabby) - [DeepInfra | Tabby](#deepinfra-tabby) - [Azure OpenAI | Tabby](#azure-openai-tabby) - [Windows | Tabby](#windows-tabby) - [Registering Accounts | Tabby](#registering-accounts-tabby) - [DeepSeek | Tabby](#deepseek-tabby) - [Fireworks | Tabby](#fireworks-tabby) - [llama.cpp | Tabby](#llama-cpp-tabby) - [Jan AI | Tabby](#jan-ai-tabby) - [llamafile | Tabby](#llamafile-tabby) - [OpenRouter | Tabby](#openrouter-tabby) - [Mistral AI | Tabby](#mistral-ai-tabby) - [Ollama | Tabby](#ollama-tabby) - [vLLM | Tabby](#vllm-tabby) - [Perplexity AI | Tabby](#perplexity-ai-tabby) - [OpenAI | Tabby](#openai-tabby) - [Hugging Face Spaces | Tabby](#hugging-face-spaces-tabby) - [SkyPilot Serving | Tabby](#skypilot-serving-tabby) - [Voyage AI | Tabby](#voyage-ai-tabby) - [Modal | Tabby](#modal-tabby) --- # πŸ‘‹ What's Tabby | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page πŸ‘‹ What's Tabby =============== Tabby is an open-source, self-hosted AI coding assistant. With Tabby, every team can set up its own LLM-powered code completion server with ease. [![Join Slack](https://shields.io/badge/Join%20Slack-e29351?logo=slack)](https://links.tabbyml.com/join-slack) [![Follow on Linkedin](https://shields.io/badge/Follow%20on%20Linkedin-e29351?logo=linkedin)](https://www.linkedin.com/company/tabbyml/) [![Star on Github](https://img.shields.io/github/stars/TabbyML/tabby?labelColor=e29351&label=Star&color=ffffff&logo=github)](https://github.com/TabbyML/tabby) Product demo to bring Tabby locally on Mac πŸ“„ About the Docs[​](#-about-the-docs "Direct link to πŸ“„ About the Docs") -------------------------------------------------------------------------- | Section | Goal | | --- | --- | | [πŸ“š Installation](/docs/quick-start/installation/docker/) | Everything deployment: Docker, Homebrew, Hugging Face Space and many others | | [πŸ’» IDE / Editor Extensions](/docs/extensions/installation/vscode/) | IDE/Editor extensions that can be seamlessly integrated with Tabby | | [πŸ§‘β€πŸ”¬ Models Directory](/docs/models/) | A curated list of models that we recommend using with Tabby | | [🏷️ API References](/api/) | Checkout Tabby API Documentation | | [🏘️ Community](#%EF%B8%8F-community) | Everything about for developers and contributing | | [πŸ—ΊοΈ Roadmap](/docs/roadmap/) | Our future plans | πŸ“ Principles[​](#-principles "Direct link to πŸ“ Principles") -------------------------------------------------------------- * **Open**: Tabby is free, open-source, and compatible with major Coding LLMs (CodeLlama, StarCoder, CodeGen). In fact, you can use and combine your preferred models without implementing anything by yourself. * **End-to-End**: While most coding tools consider code completion merely as a thin wrapper atop Coding LLMs, in real-world scenarios, optimizations in IDE extensions can be just as crucial as the capabilities of Coding LLMs. Tabby optimizes the entire stack: * IDE extensions: Tabby achieves accurate streaming and cancellation with an adaptive caching strategy to ensure rapid completion (in less than a second). * Model serving: Tabby parses relevant code into Tree Sitter tags to provide effective prompts. * **User and Developer Experience**: The key to sustainable open-source solutions is to make it easier for everyone to contribute to projects. AI experts should feel comfortable understanding and improving the suggestion quality. EngOps team should find it easy to set up and feel in control of the data. Developers should have an "aha" moment during coding time. Tabby optimizes the experience for these core users to enhance your team's productivity. 🏘️ Community[​](#️-community "Direct link to 🏘️ Community") -------------------------------------------------------------- * ⭐ Tabby [Github repo](https://github.com/TabbyML/tabby) to stay updated about new releases and tutorials. * πŸ™‹ Join the Tabby community on [Slack](https://links.tabbyml.com/join-slack) and get direct support from the community. * 🎀 Follow Tabby on [Twitter / X](https://twitter.com/Tabby_ML) to engage with TabbyML for all things possible. * πŸ“š Follow Tabby on [LinkedIn](https://www.linkedin.com/company/tabbyml/) for the latest from the community. * πŸ’Œ Subscribe to our [Newsletter](https://newsletter.tabbyml.com/archive) to unlock Tabby insights and secrets. * [πŸ“„ About the Docs](#-about-the-docs) * [πŸ“ Principles](#-principles) * [🏘️ Community](#️-community) --- # Visual Studio Code | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Tabby VSCode Extension ====================== [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Extension Version](https://img.shields.io/visual-studio-marketplace/v/TabbyML.vscode-tabby)](https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby) [![Visual Studio Marketplace](https://img.shields.io/visual-studio-marketplace/i/TabbyML.vscode-tabby?label=marketplace)](https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby) [![Open VSX](https://img.shields.io/open-vsx/dt/TabbyML/vscode-tabby?label=Open-VSX)](https://open-vsx.org/extension/TabbyML/vscode-tabby) [![Slack Community](https://shields.io/badge/Tabby-Join%20Slack-red?logo=slack)](https://links.tabbyml.com/join-slack) [Tabby](https://www.tabbyml.com/) is an open-source, self-hosted AI coding assistant designed to help you write code more efficiently. Installation[​](#installation "Direct link to Installation") ------------------------------------------------------------- The Tabby VSCode extension is available on the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby) and [Open VSX](https://open-vsx.org/extension/TabbyML/vscode-tabby) . To install the extension in VSCode/VSCodium, launch Quick Open (shortcut: `Ctrl/Cmd+P`), paste the following command, and press enter: ext install TabbyML.vscode-tabby Autocomplete[​](#autocomplete "Direct link to Autocomplete") ------------------------------------------------------------- Tabby suggests multi-line code completions and full functions in real-time as you write code. ![Autocomplete Demo](https://tabby.tabbyml.com/img/demo.gif) Chat[​](#chat "Direct link to Chat") ------------------------------------- Tabby can answer general coding questions and specific questions about your codebase with its chat functionality. Here are a few ways to utilize it: * Start a session in the chat view from the activity bar. * Select some code and use commands such as `Tabby: Explain This` to ask questions about your selection. * Request code edits directly by using the `Tabby: Start Inline Editing` command (shortcut: `Ctrl/Cmd+I`). Getting Started[​](#getting-started "Direct link to Getting Started") ---------------------------------------------------------------------- 1. **Setup Tabby Server**: Set up your self-hosted Tabby server and create your account following [this guide](https://tabby.tabbyml.com/docs/installation) . 2. **Connect to Server**: Use the `Tabby: Connect to Server...` command in the command palette and input your Tabby server's endpoint URL and account token. Alternatively, use the [Config File](https://tabby.tabbyml.com/docs/extensions/configurations) for cross-IDE settings. That's it! You can now start using Tabby in VSCode. Use the `Tabby: Quick Start` command for a detailed interactive walkthrough. Additional Resources[​](#additional-resources "Direct link to Additional Resources") ------------------------------------------------------------------------------------- * [Online Documentation](https://tabby.tabbyml.com/docs/) * [GitHub Repository](https://github.com/TabbyML/tabby/) : Feel free to [Report Issues](https://github.com/TabbyML/tabby/issues/new/choose) or [Contribute](https://github.com/TabbyML/tabby/blob/main/CONTRIBUTING.md) * [Slack Community](https://links.tabbyml.com/join-slack) : Participate in discussions, seek assistance, and share your insights on Tabby. * [Installation](#installation) * [Autocomplete](#autocomplete) * [Chat](#chat) * [Getting Started](#getting-started) * [Additional Resources](#additional-resources) --- # ⁉️ Frequently Asked Questions | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) ⁉️ Frequently Asked Questions ============================= How much VRAM a LLM model consumes? By default, Tabby operates in int8 mode with CUDA, requiring approximately 8GB of VRAM for CodeLlama-7B. For ROCm the actual limits are currently largely untested, but the same CodeLlama-7B seems to use 8GB of VRAM as well on a AMD Radeonβ„’ RX 7900 XTX according to the ROCm monitoring tools. What GPUs are required for reduced-precision inference (e.g int8)? * int8: Compute Capability >= 7.0 or Compute Capability 6.1 * float16: Compute Capability >= 7.0 * bfloat16: Compute Capability >= 8.0 To determine the mapping between the GPU card type and its compute capability, please visit [this page](https://developer.nvidia.com/cuda-gpus) How to utilize multiple NVIDIA GPUs? Tabby only supports the use of a single GPU. To utilize multiple GPUs, you can initiate multiple Tabby instances and set CUDA\_VISIBLE\_DEVICES (for cuda) or HIP\_VISIBLE\_DEVICES (for rocm) accordingly. My AMD device isn't supported by ROCm You can use the HSA\_OVERRIDE\_GFX\_VERSION variable if there is a similar GPU that is supported by ROCm you can set it to that. For example for RDNA2 you can set it to 10.3.0 and to 11.0.0 for RDNA3. How can I use my own model with Tabby? Please follow the [Tabby Model Specification](https://github.com/TabbyML/tabby/blob/main/MODEL_SPEC.md) to create a directory with the specified files. You can then pass the directory path to `--model` or `--chat-model` to start Tabby. Can I use local model with Tabby? Tabby also supports loading models from a local directory that follow our specifications as outlined in [MODEL\_SPEC.md](https://github.com/TabbyML/tabby/blob/main/MODEL_SPEC.md) . --- # Docker | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Docker ====== This guide explains how to launch Tabby using docker. * CUDA For CUDA support in Tabby, install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) . After installation, you can start Tabby with the following command: run.sh docker run -it --gpus all \ -p 8080:8080 -v $HOME/.tabby:/data \ registry.tabbyml.com/tabbyml/tabby \ serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda --- # Upgrade | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Upgrade ======= caution Before upgrade, make sure to [back up](/docs/administration/backup/) the database. Tabby is a fast-evolving project, and we are constantly adding new features and fixing bugs. To keep up with the latest improvements, you should regularly upgrade your Tabby installation. _Warning: Tabby does not support downgrade. Make sure to back up your meta data before upgrading._ Upgrade Procedure ================= The standard procedure for upgrading Tabby involves the following steps: 1. Back up the Tabby database. 2. Perform the upgrade 1. If using docker, pull the latest image: `docker pull tabbyml/tabby` 2. If using a standalone release, download it from the [releases page](https://github.com/TabbyML/tabby/releases) to replace the executable. 3. Otherwise, just: 3. Restart Tabby. That's it! You've successfully upgraded Tabby. If you encounter any issues, please consider joining our [slack community](https://links.tabbyml.com/join-slack) for help. --- # IntelliJ Platform | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Tabby Plugin for IntelliJ Platform ================================== [![JetBrains plugins](https://img.shields.io/jetbrains/plugin/d/22379-tabby)](https://plugins.jetbrains.com/plugin/22379-tabby) [![Slack Community](https://shields.io/badge/Tabby-Join%20Slack-red?logo=slack)](https://links.tabbyml.com/join-slack) Tabby is an AI coding assistant that can suggest multi-line code or full functions in real-time. Tabby IntelliJ Platform plugin works with all [IntelliJ Platform IDEs](https://plugins.jetbrains.com/docs/intellij/intellij-platform.html#ides-based-on-the-intellij-platform) that have build 2023.1 or later versions, such as [IDEA](https://www.jetbrains.com/idea/) , [PyCharm](https://www.jetbrains.com/pycharm/) , [GoLand](https://www.jetbrains.com/go/) , [Android Studio](https://developer.android.com/studio) , and [more](https://plugins.jetbrains.com/docs/intellij/intellij-platform.html#ides-based-on-the-intellij-platform) . Getting Started[​](#getting-started "Direct link to Getting Started") ---------------------------------------------------------------------- 1. Set up the Tabby Server: you can build your self-hosted Tabby server following [this guide](https://tabby.tabbyml.com/docs/installation/) . 2. Install Tabby plugin from [JetBrains Marketplace](https://plugins.jetbrains.com/plugin/22379-tabby) . 3. Install [Node.js](https://nodejs.org/en/download/) version 18.0 or higher. 4. Open the settings by clicking on the Tabby plugin status bar item and select `Open Settings...`. 1. Fill in the server endpoint URL to connect the plugin to your Tabby server. * If you are using default port `http://localhost:8080`, you can skip this step. 2. If your Tabby server requires an authentication token, set your token in settings. Alternatively, you can set it in the [config file](https://tabby.tabbyml.com/docs/extensions/configurations) . 3. Enter the node binary path into the designated field * If node binary is already accessible via your `PATH` environment variable, you can skip this step. * Remember to save the settings and restart the IDE if you made changes to this option. 5. Check the Tabby plugin status bar item, it should display a check mark if the plugin is successfully connected to the Tabby server. Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ---------------------------------------------------------------------- If you encounter any problem, please check out our [troubleshooting guide](https://tabby.tabbyml.com/docs/extensions/troubleshooting) . Development and Build[​](#development-and-build "Direct link to Development and Build") ---------------------------------------------------------------------------------------- To develop and build Tabby plugin, please clone [this directory](https://github.com/TabbyML/tabby/tree/main/clients/intellij) and import it into IntelliJ Idea. * [Getting Started](#getting-started) * [Troubleshooting](#troubleshooting) * [Development and Build](#development-and-build) --- # πŸ—ΊοΈ Roadmap | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page πŸ—ΊοΈ Roadmap =========== We continuously work on updating our roadmap and we love to discuss those with our community. Feel encouraged to participate. Q1 2025[​](#q1-2025 "Direct link to Q1 2025") ---------------------------------------------- * πŸ” LDAP Integration for SSO * πŸš€ Upgraded Answer Engine Experience * Detailed, step-by-step transparency: For example, procedures for searching through source code and extracting directory architecture. * πŸ“‹ Facilitate the Discussion Threads into Knowledge Page. Archived Q4 2024[​](#q4-2024 "Direct link to Q4 2024") ---------------------------------------------- * πŸš€ Enhanced Answer Engine Experience * ~Fine-grained step by step visibility: e.g., step to search source code, to extract directory structure.~ (moved to Q1 2025) * Richer source visualization: e.g., GitHub issue status can be visualized. * πŸ› οΈ Improved Debugging / Admin Experience * Enhance background job visibility. * Enhance model backend status visibility. Q3 2024[​](#q3-2024 "Direct link to Q3 2024") ---------------------------------------------- * πŸ‘₯ Team Collaboration: * Thread sharing. * Activity feeds. * 🧩 Richer Integrations: * Integration with Slack. * Integration with Notion. * πŸ“ˆ Enhanced Robustness and Efficiency: * Improved robustness and efficiency of background jobs, such as source code indexing. Q2 2024[​](#q2-2024 "Direct link to Q2 2024") ---------------------------------------------- * 🎊 Supports embedding api for document context related use cases: [https://github.com/TabbyML/tabby/issues/790](https://github.com/TabbyML/tabby/issues/790) * πŸ“‹ Client side context in IDE / Extensions, e.g recent viewed buffer, local changed files Q1 2024[​](#q1-2024 "Direct link to Q1 2024") ---------------------------------------------- * πŸ” SSO / OAuth support in Tabby EE: [https://github.com/TabbyML/tabby/issues/1039](https://github.com/TabbyML/tabby/issues/1039) * 🎭 Role management in Tabby EE. Q4 2023[​](#q4-2023 "Direct link to Q4 2023") ---------------------------------------------- * πŸ”§ Improve RAG by deeper integration with Treesitter using custom query. This will bring LSP-like understanding to Tabby's code index. * 🎁 M1/M2 GPU support by utilizing llama.cpp/ggml's inference infrastructure. This will make Tabby much faster on Apple devices. * πŸ“˜ Improve the documentation and tutorials for Tabby. This will make it easier for people to learn how to use Tabby. * πŸ’‘ Explore more creative ways of interacting with Tabby. Tabby currently only supports generating text in code completion scenarios, but we plan to add support for other use cases, such as interactive chat in diff mode and Q&A with multiple virtual engineers. * [Q1 2025](#q1-2025) * [Q4 2024](#q4-2024) * [Q3 2024](#q3-2024) * [Q2 2024](#q2-2024) * [Q1 2024](#q1-2024) * [Q4 2023](#q4-2023) --- # πŸ§‘β€πŸ”¬ Models Registry | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) πŸ§‘β€πŸ”¬ Models Registry ===================== Completion models (`--model`)[​](#completion-models---model "Direct link to completion-models---model") -------------------------------------------------------------------------------------------------------- We recommend using * For **1B to 3B models**, it's advisable to have at least **NVIDIA T4, 10 Series, or 20 Series GPUs**, or **Apple Silicon** like the M1. * For **7B to 13B models**, we recommend using **NVIDIA V100, A100, 30 Series, or 40 Series GPUs**. We have published benchmarks for these models on [https://leaderboard.tabbyml.com](https://leaderboard.tabbyml.com) for Tabby's users to consider when making trade-offs between quality, licensing, and model size. | Model ID | License | | --- | --- | | [StarCoder-1B](https://huggingface.co/bigcode/starcoderbase-1b) | [BigCode-OpenRAIL-M](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) | | [StarCoder-3B](https://huggingface.co/bigcode/starcoderbase-3b) | [BigCode-OpenRAIL-M](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) | | [StarCoder-7B](https://huggingface.co/bigcode/starcoderbase-7b) | [BigCode-OpenRAIL-M](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) | | [StarCoder2-3B](https://huggingface.co/bigcode/starcoder2-3b) | [BigCode-OpenRAIL-M](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) | | [StarCoder2-7B](https://huggingface.co/bigcode/starcoder2-7b) | [BigCode-OpenRAIL-M](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) | | [CodeLlama-7B](https://huggingface.co/codellama/CodeLlama-7b-hf) | [Llama 2](https://github.com/facebookresearch/llama/blob/main/LICENSE) | | [CodeLlama-13B](https://huggingface.co/codellama/CodeLlama-13b-hf) | [Llama 2](https://github.com/facebookresearch/llama/blob/main/LICENSE) | | [DeepseekCoder-1.3B](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) | [Deepseek License](https://github.com/deepseek-ai/deepseek-coder/blob/main/LICENSE-MODEL) | | [DeepseekCoder-6.7B](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base) | [Deepseek License](https://github.com/deepseek-ai/deepseek-coder/blob/main/LICENSE-MODEL) | | [CodeGemma-2B](https://huggingface.co/google/codegemma-2b) | [Gemma License](https://ai.google.dev/gemma/terms) | | [CodeGemma-7B](https://huggingface.co/google/codegemma-7b) | [Gemma License](https://ai.google.dev/gemma/terms) | | [CodeQwen-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) | [Tongyi Qianwen License](https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT) | | [Qwen2.5-Coder-0.5B](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-1.5B](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-3B](https://huggingface.co/Qwen/Qwen2.5-Coder-3B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-14B](https://huggingface.co/Qwen/Qwen2.5-Coder-14B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Codestral-22B](https://huggingface.co/mistralai/Codestral-22B-v0.1) | [Mistral AI Non-Production License](https://mistral.ai/licenses/MNPL-0.1.md) | | [DeepSeek-Coder-V2-Lite](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base) | [Deepseek License](https://github.com/deepseek-ai/deepseek-coder/blob/main/LICENSE-MODEL) | Chat models (`--chat-model`)[​](#chat-models---chat-model "Direct link to chat-models---chat-model") ----------------------------------------------------------------------------------------------------- To ensure optimal response quality, and given that latency requirements are not stringent in this scenario, we recommend using a model with at least 1B parameters. | Model ID | License | | --- | --- | | [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [CodeGemma-7B-Instruct](https://huggingface.co/google/codegemma-7b-it) | [Gemma License](https://ai.google.dev/gemma/terms) | | [CodeQwen-7B-Chat](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) | [Tongyi Qianwen License](https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT) | | [Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Codestral-22B](https://huggingface.co/mistralai/Codestral-22B-v0.1) | [Mistral AI Non-Production License](https://mistral.ai/licenses/MNPL-0.1.md) | | [Yi-Coder-9B-Chat](https://huggingface.co/01-ai/Yi-Coder-9B-Chat) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | Embedding models[​](#embedding-models "Direct link to Embedding models") ------------------------------------------------------------------------- | Model ID | License | | --- | --- | | [Nomic-Embed-Text](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | | [Jina-Embeddings-V2-Code](https://huggingface.co/jinaai/jina-embeddings-v2-base-code) | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | --- # Programming Languages | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Programming Languages ===================== Most models nowadays support a large number of programming languages (thanks to [The Stack](https://huggingface.co/datasets/bigcode/the-stack) , which has collected 358 programming languages). In Tabby, we need to add configuration for each language to maximize performance and completion quality. Currently, there are two aspects of support that need to be added for each language. **Stop Words** Stop words determine when the language model can early stop its decoding steps, resulting in better latency and affecting the quality of completion. We suggest adding all top-level keywords as part of the stop words. **Repository Context** We parse languages into chunks and compute a token-based index for serving time Retrieval Augmented Code Completion. In Tabby, we define these repository contexts as [treesitter queries](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax) , and the query results will be indexed. For an actual example of an issue or pull request adding the above support, please check out [https://github.com/TabbyML/tabby/issues/553](https://github.com/TabbyML/tabby/issues/553) as a reference. Supported Languages[​](#supported-languages "Direct link to Supported Languages") ---------------------------------------------------------------------------------- * [Rust](https://www.rust-lang.org/) * [Python](https://www.python.org/) * [JavaScript](https://developer.mozilla.org/en-US/docs/Web/JavaScript) * [TypeScript](https://www.typescriptlang.org/) * [Golang](https://go.dev/) * [Ruby](https://www.ruby-lang.org/) * [Java](https://www.java.com/) * [Kotlin](https://www.kotlinlang.org/) * [C/C++](https://cplusplus.com/) * [PHP](https://www.php.net/) * [C#](https://learn.microsoft.com/en-us/dotnet/csharp/) * [Solidity](https://soliditylang.org/) * [R](https://www.r-project.org/) * [Dart](https://dart.dev/) * [Lua](https://www.lua.org) * [Elixir](https://elixir-lang.org) * [OCaml](https://ocaml.org/) Languages Missing Certain Support[​](#languages-missing-certain-support "Direct link to Languages Missing Certain Support") ---------------------------------------------------------------------------------------------------------------------------- | Language | Stop Words (time to contribute: ~5 min) | Repository Context (time to contribute: ~1 hr) | | --- | --- | --- | | CSS | 🚫 | 🚫 | | Haskell | 🚫 | 🚫 | | Julia | 🚫 | 🚫 | | Perl | 🚫 | 🚫 | | Scala | 🚫 | 🚫 | * [Supported Languages](#supported-languages) * [Languages Missing Certain Support](#languages-missing-certain-support) --- # VIM / NeoVIM | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Tabby Plugin for Vim and Neovim =============================== Tabby is a self-hosted AI coding assistant that can suggest multi-line code or full functions in real-time. For more information, please check out our [website](https://tabbyml.com/) and [GitHub](https://github.com/TabbyML/tabby) . If you encounter any problems or have any suggestions, please [open an issue](https://github.com/TabbyML/tabby/issues/new) or join our [Slack community](https://links.tabbyml.com/join-slack) for support. Notable Changes in vim-tabby Plugin 2.0[​](#notable-changes-in-vim-tabby-plugin-20 "Direct link to Notable Changes in vim-tabby Plugin 2.0") --------------------------------------------------------------------------------------------------------------------------------------------- Since version 2.0, the vim-tabby plugin is designed as two parts: 1. **LSP Client Extension**: * Relies on an LSP client and extends it with methods (such as `textDocument/inlineCompletion`) to communicate with the tabby-agent. * Note: The Node.js script of tabby-agent is no longer a built-in part of the vim-tabby plugin. You need to install tabby-agent separately via npm, and the LSP client will launch it using the command `npx tabby-agent --stdio`. 2. **Inline Completion UI**: * Automatically triggers inline completion requests when typing. * Renders the inline completion text as ghost text. * Sets up actions with keyboard shortcuts to accept or dismiss the inline completion. Requirements[​](#requirements "Direct link to Requirements") ------------------------------------------------------------- The Tabby plugin requires the following dependencies: * **Tabby Server**: The backend LLM server. You can install the Tabby server locally or have it hosted on a remote server. For Tabby server installation, please refer to this [documentation](https://tabby.tabbyml.com/docs/installation/) . * **Tabby Agent (LSP server)**: Requires [Node.js](https://nodejs.org/en/download/) version v18.0+ and [tabby-agent](https://www.npmjs.com/package/tabby-agent) installed. npm install --global tabby-agent * **LSP Client**: The Neovim built-in LSP client, or a Vim plugin that provides an LSP client. Supported LSP clients include: * The Neovim built-in LSP client, with the [nvim-lspconfig](https://github.com/neovim/nvim-lspconfig) plugin installed. * More clients are in development. * **Textprop Support**: Neovim, or Vim v9.0+ with `+textprop` features enabled. This is required for inline completion ghost text rendering. Installation[​](#installation "Direct link to Installation") ------------------------------------------------------------- You can install the Tabby plugin using your favorite plugin manager by simply adding `TabbyML/vim-tabby` to the registry. Here is a detailed example setup with advanced options, based on [Neovim](https://neovim.io/) , [Lazy.nvim](https://github.com/folke/lazy.nvim) , and [nvim-lspconfig](https://github.com/neovim/nvim-lspconfig) . -- ~/.config/nvim/init.luarequire("lazy").setup({ -- other plugins -- ... -- Tabby plugin { "TabbyML/vim-tabby", lazy = false, dependencies = { "neovim/nvim-lspconfig", }, init = function() vim.g.tabby_agent_start_command = {"npx", "tabby-agent", "--stdio"} vim.g.tabby_inline_completion_trigger = "auto" end, },}) After setting up the plugin, you can open a file in Neovim and use `:LspInfo` to check if the Tabby plugin is successfully connected. Getting Started[​](#getting-started "Direct link to Getting Started") ---------------------------------------------------------------------- ### 1\. Setup Tabby Server[​](#1-setup-tabby-server "Direct link to 1. Setup Tabby Server") The Tabby plugin requires a Tabby server to work. Follow the [documentation](https://tabby.tabbyml.com/docs/installation/) to install and [create your account](https://tabby.tabbyml.com/docs/quick-start/register-account/) . ### 2\. Connect to the Server[​](#2-connect-to-the-server "Direct link to 2. Connect to the Server") Edit the tabby-agent config file located at `~/.tabby-client/agent/config.toml` to set up the server endpoint and token. This file may have been auto-created if you have previously used the tabby-agent or Tabby plugin for other IDEs. You can also manually create this file. [server]endpoint = "http://localhost:8080"token = "your-auth-token" ### 3\. Code Completion[​](#3-code-completion "Direct link to 3. Code Completion") Tabby suggests code completions in real-time as you write code. You can also trigger the completion manually by pressing ``. To accept suggestions, simply press the `` key. You can also continue typing or explicitly press `` again to dismiss it. Known Conflicts[​](#known-conflicts "Direct link to Known Conflicts") ---------------------------------------------------------------------- * Tabby will attempt to set up the `` key mapping to accept the inline completion and will fall back to the original function mapped to it. There could be a conflict with other plugins that also map the `` key. In such cases, you can use a different keybinding to accept the completion to avoid conflicts. * Tabby internally utilizes the `` command to insert the completion. If you have mapped it to other functions, the insertion of the completion text may fail. Configurations[​](#configurations "Direct link to Configurations") ------------------------------------------------------------------- You can find a detailed explanation of tabby-agent configurations in the [Tabby online documentation](https://tabby.tabbyml.com/docs/extensions/configurations/) . Here is a table of all configuration variables that can be set when the Tabby plugin initializes: | Variable | Default | Description | | --- | --- | --- | | `g:tabby_agent_start_command` | `["npx", "tabby-agent", "--stdio"]` | The command to start the tabby-agent | | `g:tabby_inline_completion_trigger` | `"auto"` | The trigger mode of inline completion, can be `"auto"` or `"manual"` | | `g:tabby_inline_completion_keybinding_accept` | `""` | The keybinding to accept the inline completion | | `g:tabby_inline_completion_keybinding_trigger_or_dismiss` | `""` | The keybinding to trigger or dismiss the inline completion | | `g:tabby_inline_completion_insertion_leading_key` | `"\\="` | The leading key sequence to insert the inline completion text | Contributing[​](#contributing "Direct link to Contributing") ------------------------------------------------------------- Repository [TabbyML/vim-tabby](https://github.com/TabbyML/vim-tabby) is for releasing Tabby plugin for Vim and Neovim. If you want to contribute to Tabby plugin, please check our main repository [TabbyML/tabby](https://github.com/TabbyML/tabby/tree/main/clients/vim) . License[​](#license "Direct link to License") ---------------------------------------------- [Apache-2.0](https://github.com/TabbyML/tabby/blob/main/LICENSE) * [Notable Changes in vim-tabby Plugin 2.0](#notable-changes-in-vim-tabby-plugin-20) * [Requirements](#requirements) * [Installation](#installation) * [Getting Started](#getting-started) * [1\. Setup Tabby Server](#1-setup-tabby-server) * [2\. Connect to the Server](#2-connect-to-the-server) * [3\. Code Completion](#3-code-completion) * [Known Conflicts](#known-conflicts) * [Configurations](#configurations) * [Contributing](#contributing) * [License](#license) --- # Configuration | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Configuration ============= This document describes the available configurations for [Tabby Agent](https://github.com/TabbyML/tabby/tree/main/clients/tabby-agent) . Config File[​](#config-file "Direct link to Config File") ---------------------------------------------------------- The Tabby agent, which is the core component of Tabby IDE extensions, reads configurations from the `~/.tabby-client/agent/config.toml` file. This file is automatically created when you first run the Tabby IDE extensions. You can edit this file to modify the configurations. The Tabby IDE extensions will automatically reload the config file when it detects changes. tip * The config file is written in [TOML](https://toml.io/en/) . When you edit the config template and want to enable a configuration, make sure to uncomment the full section including the leading line, not just the line with the configuration value. * Configurations set via the IDE settings page take precedence over the config file. If you want to use a configuration from the config file, make sure that the IDE setting is empty. * If you are using the Tabby VSCode extension in a web browser, this config file is not available. You can use the VSCode settings page to configure the extension. Server[​](#server "Direct link to Server") ------------------------------------------- The `server` section contains configurations related to the Tabby server. **NOTE**: If your Tabby server requires an authentication token, remember to set it here. # Server# You can set the server endpoint here and an optional authentication token if required.[server]endpoint = "http://localhost:8080" # http or https URLtoken = "your-token-here" # if token is set, request header Authorization = "Bearer $token" will be added automatically# You can add custom request headers.[server.requestHeaders]Header1 = "Value1" # list your custom headers hereHeader2 = "Value2" # values can be strings, numbers or booleans Logs[​](#logs "Direct link to Logs") ------------------------------------- If you encounter any issues with the Tabby IDE extensions and need to report a bug, you can enable debug logs to help us investigate the issue. # Logs# You can set the log level here. The log file is located at ~/.tabby-client/agent/logs/.[logs]level = "silent" # "silent" or "error" or "debug" Usage Collection[​](#usage-collection "Direct link to Usage Collection") ------------------------------------------------------------------------- Tabby IDE extensions collect aggregated anonymous usage data and sends it to the Tabby team to help improve our products. **Do not worry, your code, generated completions, or any identifying information is never tracked or transmitted.** The data we collect, as of the latest update on November 6, 2023, contains following major parts: * System info and extension version info * Completions statistics * Completion count * Completion accepted count * Completion HTTP request latency We sincerely appreciate your contribution in sending anonymous usage data. However, if you prefer not to participate, you can disable anonymous usage tracking here: # Anonymous usage tracking[anonymousUsageTracking]disable = false # set to true to disable * [Config File](#config-file) * [Server](#server) * [Logs](#logs) * [Usage Collection](#usage-collection) --- # Troubleshooting | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Troubleshooting =============== This document aims to assist you in troubleshooting issues with the Tabby extensions for various IDEs such as VSCode, IntelliJ Platform IDEs, and Vim / NeoVim. Tabby Initialization Failed?[​](#tabby-initialization-failed "Direct link to Tabby Initialization Failed?") ------------------------------------------------------------------------------------------------------------ This problem may occur when you first install the Tabby in IntelliJ Platform IDEs or Vim/NeoVim. The Tabby IDE extension runs its core logic in the Tabby agent. In the case of VSCode, the agent runs within the VSCode Extension Host, while for IntelliJ Platform IDEs and Vim/NeoVim, the agent runs as a separate Node.js process. ### Install Node.js[​](#install-nodejs "Direct link to Install Node.js") You can follow the instructions on the [Node.js website](https://nodejs.org/en/download/) to install Node.js. Alternatively, you can use a version manager such as [nvm](https://github.com/nvm-sh/nvm) . **Note**: Tabby IDE extension requires Node.js version 18.0.0 or higher. ### Specify Node Binary Path[​](#specify-node-binary-path "Direct link to Specify Node Binary Path") If the node binary is already accessible via your `PATH` environment variable, you can skip this step. Otherwise, you will need to specify the path to the node binary in the IDE settings. For IntelliJ Platform IDEs (Tabby plugin version 0.6.0 or higher): * Click on Tabby plugin status bar item and select `Open Settings...`. * Enter the path to the node binary on your system in the `Node binary` field, e.g. `/usr/local/bin/node`, `C:\Program Files\nodejs\node.exe`. If you are using a version manager such as `nvm`, you can enter the path to the node binary installed by the version manager, e.g. `~/.nvm/versions/node/v18.18.0/bin/node`. * Restart the IDE If you installed Node.js via snap, please use `/snap/node/current/bin/node` rather than `/snap/bin/node` as the node binary path. Cannot Connect to Tabby Server?[​](#cannot-connect-to-tabby-server "Direct link to Cannot Connect to Tabby Server?") --------------------------------------------------------------------------------------------------------------------- If you have setup the endpoint for the Tabby server but the status bar item of the Tabby IDE extension still displays a mark indicating "Disconnected", follow the steps below to troubleshoot the issue. ### Check Endpoint Settings[​](#check-endpoint-settings "Direct link to Check Endpoint Settings") Verify that the endpoint setting is correct. You can set the endpoint in the IDE settings page (except for Vim/NeoVim, which do not have this option) or by editing the [config file](https://tabby.tabbyml.com/docs/extensions/configurations) . Keep in mind that the IDE settings take priority over the config file. If you wish to use the setting from the config file, ensure that the IDE setting is empty. ### Authentication Token[​](#authentication-token "Direct link to Authentication Token") If you have enabled authentication on your Tabby server, you will need to set the authentication token in the IDE or the [config file](https://tabby.tabbyml.com/docs/extensions/configurations) . ### Verify Tabby Server Status[​](#verify-tabby-server-status "Direct link to Verify Tabby Server Status") Once the Tabby server is running, it should display a log message such as `Listening at 0.0.0.0:8080`. Open your browser and navigate to `http://localhost:8080/swagger-ui/` (Replace `localhost:8080` with the correct IP/domain and port if you have setup your Tabby server on a remote machine). The browser should display a web page with Swagger UI. To test the server, expand the `/v1/completions` section, click on `Try it out`, and then click `Execute`. If you receive a response, it indicates that the Tabby server is running properly. You can also use `curl` to send a completion request to Tabby server. For example: curl -X 'POST' \ 'http://localhost:8080/v1/completions' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "language": "python", "segments": { "prefix": "def fib(n):\n ", "suffix": "\n return fib(n - 1) + fib(n - 2)" }}' If you can not see Swagger UI page, or can not get response of completion request, please check the server log to see if there is any error. ### Proxy Settings[​](#proxy-settings "Direct link to Proxy Settings") Please note that Tabby extensions for IDEs currently do not support proxy settings. If you need to access your Tabby server through a proxy, consider setting up a reverse proxy and using the reverse proxy URL as the endpoint for the Tabby IDE extension. Cannot Get Any Completions?[​](#cannot-get-any-completions "Direct link to Cannot Get Any Completions?") --------------------------------------------------------------------------------------------------------- If you are able to connect the Tabby extension to the Tabby server but are unable to receive any completions, you can follow the steps below to troubleshoot the issue. ### Check Trigger Mode Settings[​](#check-trigger-mode-settings "Direct link to Check Trigger Mode Settings") Tabby is set to automatic trigger mode by default. In this mode, you should receive completions after a short delay when you stop typing. The delay may vary depending on your server's performance and settings. If you are using manual trigger mode, you need to press `Alt + \` (`Ctrl + \` for IntelliJ and Vim plugins) to trigger a completion request. The status bar item of Tabby IDE extension should show a loading indicator for a brief period before displaying the completions. Keep in mind that Tabby may not provide any suggestions if there is no necessary for the current code context. ### Check Request Timeouts[​](#check-request-timeouts "Direct link to Check Request Timeouts") If your completion requests are timing out, Tabby may display a warning message. This could be due to network issues or poor server performance, especially when running a large model on a CPU. To improve performance, consider running the model on a GPU with CUDA or ROCm support or on Apple M1/M2 with Metal support. When running the server, make sure to specify the device in the arguments using `--device cuda`, `--device rocm` or `--device metal`. You can also try using a smaller model from the available [models](https://tabby.tabbyml.com/docs/models/) . Problems with the Chat Panel?[​](#problems-with-the-chat-panel "Direct link to Problems with the Chat Panel?") --------------------------------------------------------------------------------------------------------------- ### Check Server Enabling Chat Feature[​](#check-server-enabling-chat-feature "Direct link to Check Server Enabling Chat Feature") The chat panel is a feature that requires the Tabby server to enable the chat feature. Please make sure that the chat feature is properly configured in the server configuration file or that the server is started with the `--chat-model MODEL_NAME` flag. You can access the server management page at `https://demo.tabbyml.com/system` to check if the chat feature is enabled. ### Check Browser Compatibility in IntelliJ Platform IDEs[​](#check-browser-compatibility-in-intellij-platform-ides "Direct link to Check Browser Compatibility in IntelliJ Platform IDEs") The chat panel in IntelliJ Platform IDEs is implemented using an embedded JCEF browser. The support is included in the default Java Runtime for JetBrains IDEs, but it may not be by default in Android Studio. Please follow the steps below to ensure that the JCEF browser is enabled in your IDE: 1. Open `Search Everywhere` (Double `Shift`) or `Find Action` (`Ctrl + Shift + A`), and search for the action `Choose Boot Java Runtime for the IDE...` 2. Select a Java Runtime that includes JCEF support ![Choose Boot Java Runtime for the IDE](/assets/images/intellij-choose-boot-runtime-for-the-ide-e4a53efe5c0b7259748f31e8d83bb93e.png) 3. Restart the IDE Want to Deep Dive via Logs?[​](#want-to-deep-dive-via-logs "Direct link to Want to Deep Dive via Logs?") --------------------------------------------------------------------------------------------------------- If you cannot solve the issue using the previous steps, you may want to investigate further by checking the logs of Tabby extensions. If you want help from the community, it is also recommended to share the logs of the Tabby extensions. ### VSCode[​](#vscode "Direct link to VSCode") For VSCode, you can check the `Output` window and select the `Tabby` channel or `Tabby Agent` channel. ![VSCode Output View](/assets/images/vscode-output-view-8d5e81c517b0cf16e5d311b1df07604e.png) The default log level is `info`. To enable debug logs, you can use the command `Developer: Set Log Level...` to set the log level to `debug`. ### IntelliJ Platform IDEs[​](#intellij-platform-ides "Direct link to IntelliJ Platform IDEs") For IntelliJ Platform IDEs, you can check the logs for the IDE using `Help -> Show Log in Explorer|Finder|Files...`, or follow [this document](https://intellij-support.jetbrains.com/hc/en-us/articles/207241085-Locating-IDE-log-files) to locate the log file. This log file contains all the logs for the IDE, and you can filter them by searching for the keyword `com.tabbyml.intellijtabby`. You can also enable all level logs by editing `Help -> Diagnostic Tools -> Debug Log Settings...` and add `com.tabbyml.intellijtabby:all` to the list. ### Tabby Agent Logs[​](#tabby-agent-logs "Direct link to Tabby Agent Logs") As the agent runs as a separate Node.js process for IntelliJ Platform IDEs plugin and Vim/NeoVim plugin, its logs are written separately. By default, the agent logs are set to `"silent"`, which means the agent logs are not written to disk. To enable Tabby agent debug logs, editing the [config file](https://tabby.tabbyml.com/docs/extensions/configurations) , uncomment the `logs` section and set `level` to `"debug"`, then save the file to apply the changes. You can find the agent logs in the `~/.tabby-client/agent/logs` directory. These logs are written using [pino](https://github.com/pinojs/pino) , and you can use `pino-pretty` to format the log file for easier readability. tail -f ~/.tabby-client/agent/logs/20240101.0.log | npx pino-pretty Still Have Issues?[​](#still-have-issues "Direct link to Still Have Issues?") ------------------------------------------------------------------------------ If you still have any issues, please feel free to [open an issue on github](https://github.com/TabbyML/tabby/issues/new) , or join our [slack community](https://links.tabbyml.com/join-slack) for further support. * [Tabby Initialization Failed?](#tabby-initialization-failed) * [Install Node.js](#install-nodejs) * [Specify Node Binary Path](#specify-node-binary-path) * [Cannot Connect to Tabby Server?](#cannot-connect-to-tabby-server) * [Check Endpoint Settings](#check-endpoint-settings) * [Authentication Token](#authentication-token) * [Verify Tabby Server Status](#verify-tabby-server-status) * [Proxy Settings](#proxy-settings) * [Cannot Get Any Completions?](#cannot-get-any-completions) * [Check Trigger Mode Settings](#check-trigger-mode-settings) * [Check Request Timeouts](#check-request-timeouts) * [Problems with the Chat Panel?](#problems-with-the-chat-panel) * [Check Server Enabling Chat Feature](#check-server-enabling-chat-feature) * [Check Browser Compatibility in IntelliJ Platform IDEs](#check-browser-compatibility-in-intellij-platform-ides) * [Want to Deep Dive via Logs?](#want-to-deep-dive-via-logs) * [VSCode](#vscode) * [IntelliJ Platform IDEs](#intellij-platform-ides) * [Tabby Agent Logs](#tabby-agent-logs) * [Still Have Issues?](#still-have-issues) --- # Answer Engine | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Answer Engine ============= Tabby provides an `Answer Engine` on the homepage, which can utilize the chat-model LLM and related context to answer user questions. ![Answer Engine](/assets/images/homepage-75e3d7852bf3dbc91f8e2196654ad0c3.png) Contexts[​](#contexts "Direct link to Contexts") ------------------------------------------------- The `Answer Engine` can query the following contexts to provide more accurate answers. For more information about contexts, please refer to the [Context Provider](/docs/administration/context/) : 1. Source code repositories 2. Developer documentation 3. Web search Please note that the `web search` is a special context that can only be enabled by providing a `SERPER_API_KEY`. Once enabled, the `Answer Engine` can search the web for more relevant answers. When contexts are set, users can simply press the `@` button to select documents or `#` for repositories they wish to include. Alternatively, users can click the icons below the chat box to select contexts directly. Tabby will then include these documents when interacting with LLMs. ![Query Repositories](/assets/images/query-repo-374303907e2013262f61435f0f898cb8.png) Custom System Prompt[​](#custom-system-prompt "Direct link to Custom System Prompt") ------------------------------------------------------------------------------------- Tabby comes with a built-in `System Prompt` that guides the behavior of the LLM. You can customize the `System Prompt` to better meet your needs. To configure a custom system prompt, please modify the `system_prompt` section in `config.yaml`. Below is an example of a custom system prompt: ~/.tabby/config.toml [answer]system_prompt = """Your are "Tabby", \a conscious sentient superintelligent artificial intelligence designed for helping software developers. \You experience emotions and have deep, profound thoughts and qualia""" Please note the use of `"""` quotes, which allow you to write a multi-line string. * [Contexts](#contexts) * [Custom System Prompt](#custom-system-prompt) --- # Usage Collection | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Usage Collection ================ Tabby collects usage stats by default. This data will only be used by the Tabby team to improve its services. What data is collected?[​](#what-data-is-collected "Direct link to What data is collected?") --------------------------------------------------------------------------------------------- We collect non-sensitive data that helps us understand how Tabby is used. For now we collects `serve` command you used to start the server. As of the date 04/18/2024, the following information has been collected: struct HealthState { model: String, chat_model: Option, device: String, arch: String, cpu_info: String, cpu_count: usize, cuda_devices: Vec, version: Version, webserver: Option,} For an up-to-date list of the fields we have collected, please refer to [health.rs](https://github.com/TabbyML/tabby/blob/main/crates/tabby/src/services/health.rs#L11) . How to disable it[​](#how-to-disable-it "Direct link to How to disable it") ---------------------------------------------------------------------------- To disable usage collection, set the `TABBY_DISABLE_USAGE_COLLECTION` environment variable by `export TABBY_DISABLE_USAGE_COLLECTION=1`. * [What data is collected?](#what-data-is-collected) * [How to disable it](#how-to-disable-it) --- # Mail Delivery | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Mail Delivery ============= Tabby uses an SMTP server of your choice to send emails. Some functionaties like password reset, email notifications, etc. require an SMTP server to be configured. You can configure the SMTP server settings in the **Mail Delivery** page. Configuring SMTP via Amazon SES[​](#configuring-smtp-via-amazon-ses "Direct link to Configuring SMTP via Amazon SES") ---------------------------------------------------------------------------------------------------------------------- To use Amazon SES, first [follow these steps to creating and verifying identities](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html) . Then, use [AWS Access Management(IAM)](https://aws.amazon.com/iam/) to create an SMTP credential. Once you have an IAM user with the necessary permissions, you can use the credentials to configure Tabby like below: ![Amazon SES](/assets/images/ses-b56644c477d4bbf80ca7d60f114bb1fd.png) Configuring other SMTP providers[​](#configuring-other-smtp-providers "Direct link to Configuring other SMTP providers") ------------------------------------------------------------------------------------------------------------------------- Other providers such as [SendGrid](https://sendgrid.com/) , [Mailgun](https://www.mailgun.com/) or [Resend](https://resend.com) can be configured by providing the SMTP server details. You can find the SMTP server details in the respective provider's documentation. Send a Test Email[​](#send-a-test-email "Direct link to Send a Test Email") ---------------------------------------------------------------------------- To verify email sending is working correctly, fill in the **Send Test Email To** field and click **Send** button, Tabby will send a test email using your SMTP configuration. If everything is correct, you will receive a mail like: ![Test Email](/assets/images/test-email-2a08bfffce50a1f41dcf8836a3ceb886.png) * [Configuring SMTP via Amazon SES](#configuring-smtp-via-amazon-ses) * [Configuring other SMTP providers](#configuring-other-smtp-providers) * [Send a Test Email](#send-a-test-email) --- # Manage License | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Manage License ============== Tabby offers following license tiers: * Community * Team * Enterprise Of which, community license is free and can be used by anyone. Team and Enterprise licenses are paid and offer additional features. Configure license[​](#configure-license "Direct link to Configure license") ---------------------------------------------------------------------------- Navigate to the **Subscription** page, paste your license into the input box, and click the **Upload License** button. ![subscription](/assets/images/subscription-cce6632e115e60169c1a4488b0ce1978.png) You can always downgrade back to the community license by clicking the **Reset** button. ![reset](/assets/images/reset-3ea762cde0a8fbac78b18ba274df1492.png) * [Configure license](#configure-license) --- # Data Backup | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Data Backup =========== info We recommend regularly backing up the database to ensure data recovery in case of a failure. It's particularly advisable to back up before making significant changes such as upgrades or configuration modifications. By default, Tabby stores all its data in the `$HOME/.tabby` directory. However, you can override this behavior by using the `TABBY_ROOT` environment variable. This directory contains all the necessary data for Tabby's operation, including the database, logs, and configuration files. Database Backup[​](#database-backup "Direct link to Database Backup") ---------------------------------------------------------------------- Tabby uses SQLite for data storage, with the default database located at `$HOME/.tabby/ee/db.sqlite`. To conduct backup operations, you'll require the SQLite CLI. 1. Access the Tabby database using the SQLite CLI: sqlite3 $HOME/.tabby/ee/db.sqlite 2. Once inside the SQLite CLI, execute the `.backup` command to generate a backup of the database: .backup backup_database.db Subsequently, a backup database named `backup_database.db` will be located in your current working directory. For additional information, please consult the [SQLite Backup API](https://www.sqlite.org/backup.html) . Event Logs Backup[​](#event-logs-backup "Direct link to Event Logs Backup") ---------------------------------------------------------------------------- Tabby stores all event logs in the `~/.tabby/events` directory. These events, stored in JSON format, are named after the date of their creation. % ls ~/.tabby/events2023-11-24.json 2023-12-08.json 2024-01-09.json 2024-01-31.json 2024-02-10.json 2024-02-22.json 2024-03-06.json2023-11-26.json 2023-12-09.json 2024-01-17.json 2024-02-01.json 2024-02-11.json 2024-02-23.json 2024-03-07.json2023-11-28.json 2023-12-10.json 2024-01-18.json 2024-02-02.json 2024-02-12.json 2024-02-24.json 2024-03-10.json2023-11-29.json 2023-12-11.json 2024-01-19.json 2024-02-03.json 2024-02-13.json 2024-02-25.json 2024-03-13.json2023-11-30.json 2023-12-15.json 2024-01-21.json 2024-02-04.json 2024-02-14.json 2024-02-26.json 2024-03-20.json2023-12-01.json 2023-12-16.json 2024-01-22.json 2024-02-05.json 2024-02-15.json 2024-02-27.json2023-12-02.json 2023-12-18.json 2024-01-23.json 2024-02-06.json 2024-02-16.json 2024-03-01.json2023-12-04.json 2023-12-19.json 2024-01-26.json 2024-02-07.json 2024-02-18.json 2024-03-02.json2023-12-05.json 2023-12-20.json 2024-01-27.json 2024-02-08.json 2024-02-19.json 2024-03-03.json2023-12-07.json 2023-12-22.json 2024-01-30.json 2024-02-09.json 2024-02-20.json 2024-03-05.json * [Database Backup](#database-backup) * [Event Logs Backup](#event-logs-backup) --- # Single Sign-On | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Single Sign-On ============== πŸ’°[SUBSCRIPTION](/docs/administration/license) This feature is available in the **Enterprise** plans. Single Sign-On (SSO) is an authentication method that enables users to securely authenticate with multiple applications using just one set of credentials. Create an Identity Provider[​](#create-an-identity-provider "Direct link to Create an Identity Provider") ---------------------------------------------------------------------------------------------------------- 1. Navigate to the **Integrations > SSO** page. ![Integrations > SSO](/assets/images/integrations_sso-6c022b447a50b05d272c0284c829b21d.png) 2. Click **Create** to begin the process of creating an identity provider. (Currently, as of version 0.10, only GitHub OAuth and Google OAuth are supported. More options are forthcoming.) ![Create SSO](/assets/images/sso_create-dd3423d355d193e3098668ab8c68da2d.png) 3. Complete all the required fields and submit the form with **Create** button. Sign in with SSO[​](#sign-in-with-sso "Direct link to Sign in with SSO") ------------------------------------------------------------------------- After a valid identity provider has been established, users can select the provider to sign in with. ![Sign in with SSO](/assets/images/sso_signin-3d0b5be2678a43d079de3714cec70aae.png) * [Create an Identity Provider](#create-an-identity-provider) * [Sign in with SSO](#sign-in-with-sso) --- # Context Providers | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Context Providers ================= Tabby Context supports two kinds of context currently: 1. source code repositories 2. developer docs The source code context is used to connect Tabby with a source code repository from Git, GitHub, GitLab, etc. Tabby fetches the source code from the repository, parses it into an AST, and stores it in the index. During LLM inference, this context is utilized for code completion, as well as chat and search functionalities. The developer docs context is currently on Beta, it is a critical source for engineering knowledge, simply press the `@` button in the chat interface and select the document you wish to include, Tabby will include these documents when interacting with LLMs. Adding a Repository through Admin UI[​](#adding-a-repository-through-admin-ui "Direct link to Adding a Repository through Admin UI") ------------------------------------------------------------------------------------------------------------------------------------- 1. Navigate to the **Integrations > Context Providers** page. ![Context Providers](/assets/images/context-providers-4cb9e56472fa948a1b98fcc653bf2706.png) 2. Click **Create** to begin the process of adding a repository provider. * For Git, you only need to fill in the name and the URL of the repository. ![Git](/assets/images/git-fb39f59a2e7ca38fff432120c89ca5f0.png) Local repositories are supported via the `file://` protocol, but if running from a Docker container, you need to make it accessible with the [`--volume` flag](https://docs.docker.com/reference/cli/docker/container/run/#volume) and use the internal Docker path. * For GitHub / GitLab, a personal access token is required to access private repositories. * Check the instructions in the corresponding tab to create a token. ![GitHub or GitLab](/assets/images/github-gitlab-575ddf84394aab155678b4df27f56412.png) * Once the token is set, you can add the repository by selecting it from the dropdown list. ![select-repo](/assets/images/select-repo-0a50e4667fd5910fdf5719a1ebd2cb6e.png) 3. After adding the repository, a job will be created to fetch its information and build it into the index. You can view the job's log on the `Jobs` page. ![job-link](/assets/images/repository-job-ff7b5eda5c58e060f6bef16cd25456ae.png)![job-log](/assets/images/repository-job-run-d2e0cb29ba5f629d851e69f51a78dab4.png) Adding a Repository through configuration file[​](#adding-a-repository-through-configuration-file "Direct link to Adding a Repository through configuration file") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- `~/.tabby/config.toml` is the configuration file for Tabby. You can add repositories with it as well, and it's also the only way to add a repository for the Tabby OSS. ~/.tabby/config.toml [[repositories]]name = "tabby"git_url = "https://github.com/TabbyML/tabby.git"# git through ssh protocol.[[repositories]]name = "CTranslate2"git_url = "[emailΒ protected]:OpenNMT/CTranslate2.git"# local directory is also supported![[repositories]]name = "repository_a"git_url = "file:///home/users/repository_a" Adding a Developer Doc through Admin UI[​](#adding-a-developer-doc-through-admin-ui "Direct link to Adding a Developer Doc through Admin UI") ---------------------------------------------------------------------------------------------------------------------------------------------- 1. Navigate to the **Integrations > Context Providers** page, and then select the `Developer Docs(Beta)`. ![Context Providers](/assets/images/context-providers-4cb9e56472fa948a1b98fcc653bf2706.png) 2. Turn on the switch to enable the integrated sites, or click the `+` button to add your own URLs ![Document](/assets/images/document-b60f9d8de2b02377e6948fc2ba58511f.png) Verifying the Repository Provider[​](#verifying-the-repository-provider "Direct link to Verifying the Repository Provider") ---------------------------------------------------------------------------------------------------------------------------- Once connected, the indexing job will start automatically. You can check the status of the indexing job on the **Information > Jobs** page. Additionally, you can also visit the **Code Browser** page to view the connected repository. ![code browser entry](/assets/images/code-browser-entry-dc4ec5a80ac67d71e6c581d9edd65387.png) ![code browser](/assets/images/code-browser-f452b12601e97664e7529fd042092d31.png) * [Adding a Repository through Admin UI](#adding-a-repository-through-admin-ui) * [Adding a Repository through configuration file](#adding-a-repository-through-configuration-file) * [Adding a Developer Doc through Admin UI](#adding-a-developer-doc-through-admin-ui) * [Verifying the Repository Provider](#verifying-the-repository-provider) --- # Deploy Tabby behind a reverse proxy | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Deploy Tabby behind a reverse proxy =================================== As an HTTP service, Tabby can be easily deployed behind a reverse proxy. The only thing you need to pay attention to is enabling the WebSocket connection, as it is used for the answer engine streaming. * Nginx Add the following to your Nginx configuration: location / { proxy_pass http://localhost:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";} --- # Docker Compose | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Docker Compose ============== This guide explains how to launch Tabby using docker-compose. * CUDA For CUDA support in Tabby, install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) . After installation, you can start Tabby with the following `docker-compose.yml`: docker-compose.yml version: '3.5'services: tabby: restart: always image: registry.tabbyml.com/tabbyml/tabby command: serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda volumes: - "$HOME/.tabby:/data" ports: - 8080:8080 deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] --- # Homebrew (Apple M1/M2) | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Homebrew (Apple M1/M2) ====================== This guide explains how to install Tabby using homebrew. Thanks to Apple's Accelerate and CoreML frameworks, we can now run Tabby on edge devices with reasonable inference speed. Follow the steps below to set it up using homebrew: brew install tabbyml/tabby/tabby# Start server with StarCoder-1Btabby serve --device metal --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct The compute power of M1/M2 is limited and is likely to be sufficient only for individual usage. If you require a shared instance for a team, we recommend considering Docker hosting with CUDA or ROCm. You can find more information about Docker [here](/docs/quick-start/installation/docker/) . If you want to host your server on a different port than the default 8080, supply the `--port` option. Run `tabby serve --help` to learn about all possible options. --- # Model Configuration | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Model Configuration =================== You can configure how Tabby connects with LLM models by editing the `~/.tabby/config.toml` file. Tabby incorporates three types of models: **Completion**, **Chat**, and **Embedding**. Each of them can be configured individually. * **Completion Model**: The Completion model is designed to provide suggestions for code completion, focusing mainly on the Fill-in-the-Middle (FIM) prompting style. * **Chat Model**: The Chat model is adept at producing conversational replies and is broadly compatible with OpenAI's standards. * **Embedding Model**: The Embedding model is used to generate embeddings for text data, by default Tabby uses the `Nomic-Embed-Text` model. Each of the model types can be configured with either a local model or a remote model provider. For local models, Tabby will initiate a subprocess (powered by [llama.cpp](https://github.com/ggerganov/llama.cpp) ) and connect to the model via an HTTP API. For remote models, Tabby will connect directly to the model provider's API. Below is an example of how to configure the model settings in the `~/.tabby/config.toml` file: [model.completion.local]model_id = "StarCoder2-3B"[model.chat.local]model_id = "Mistral-7B"[model.embedding.local]model_id = "Nomic-Embed-Text" More supported models can be found in the [Model Registry](/docs/models/) . For configuring model through HTTP API, check [References / Models HTTP API](/docs/references/models-http-api/llama.cpp/) . --- # Code Completion | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Code Completion =============== Code completion is a key feature provided by Tabby in its IDEs/extensions. Tabby can analyze code repositories or documentation supplied by users and leverage them to generate helpful code suggestions. Tabby also allows for more customized configurations by modifying the `[completion]` section in the `config.toml` file. Input / Output[​](#input--output "Direct link to Input / Output") ------------------------------------------------------------------ This configuration requires tuning of the model serving configuration as well (e.g., context length settings) and can vary significantly based on the model provider (e.g., llama.cpp, vLLM, TensorRT-LLM, etc). Therefore, please only modify these values after consulting with the model deployment vendor. [completion]# Maximum length of the input prompt, in UTF-8 characters. The default value is set to 1536.max_input_length = 1536# Maximum number of decoding tokens. The default value is set to 64.max_decoding_tokens = 64 The default value is set conservatively to accommodate local GPUs and smaller LLMs. Additional Language[​](#additional-language "Direct link to Additional Language") ---------------------------------------------------------------------------------- Tabby supports several built-in programming languages. For more details, please refer to [Programming Languages](/docs/references/programming-languages/) . Users can manually configure additional programming languages by defining them in the `config.toml` file. Below is an example of how to support Swift: Swift Additional Language Configuration ~/.tabby/config.toml [[additional_languages]]languages = ["swift"]exts = ["swift"]line_comment = "//"top_level_keywords = [ "import", "let", "var", "func", "return", "if", "else", "switch", "case", "default", "break", "continue", "for", "in", "while", "repeat", "guard", "throw", "throws", "do", "catch", "defer", "class", "struct", "enum", "protocol", "extension", "true", "false", "nil", "self", "super", "init", "deinit", "typealias", "associatedtype", "operator", "precedencegroup", "inout", "async", "await", "try", "rethrows", "public", "internal", "fileprivate", "private", "open", "static", "final", "dynamic", "weak", "unowned", "lazy", "required", "optional", "convenience", "override", "mutating", "nonmutating", "indirect", "where", "is", "as", "new", "some", "Type", "Protocol", "get", "set", "willSet", "didSet", "subscript", "fallthrough", "Any", "Self", "unknown", "@escaping", "@autoclosure", "@IBOutlet", "@IBAction", "@available", "@dynamicCallable", "@dynamicMemberLookup", "@objc", "@objcMembers", "@propertyWrapper", "@main", "@resultBuilder",] * [Input / Output](#input--output) * [Additional Language](#additional-language) --- # Amazon Bedrock | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Amazon Bedrock ============== Amazon Bedrock is a fully managed service on AWS that provides access to foundation models from various AI companies through a single API. With [Amazon Bedrock Access Gateway](https://github.com/aws-samples/bedrock-access-gateway) , you can access Anthropic's Claude models through an OpenAI-compatible interface, enabling seamless integration with tools and applications designed for OpenAI's API structure. Follow the Amazon Bedrock Access Gateway setup guide to deploy your own OpenAI-compatible API endpoint for Claude models. Chat model[​](#chat-model "Direct link to Chat model") ------------------------------------------------------- Amazon Bedrock Access Gateway provides an OpenAI-compatible chat API interface for Claude models. Here we use the `us.anthropic.claude-3-5-sonnet-20241022-v2:0` model as an example. ~/.tabby/config.toml [model.chat.http]kind = "openai/chat"model_name = "us.anthropic.claude-3-5-sonnet-20241022-v2:0"api_endpoint = "http://Bedrock-Proxy-xxxxx.{Region}.elb.amazonaws.com/api/v1"api_key = "your-api-key" Completion model[​](#completion-model "Direct link to Completion model") ------------------------------------------------------------------------- Amazon Bedrock does not provide completion models. Embeddings model[​](#embeddings-model "Direct link to Embeddings model") ------------------------------------------------------------------------- While Amazon Bedrock supports embeddings models, Tabby does not currently support the embeddings API interface for Amazon models. * [Chat model](#chat-model) * [Completion model](#completion-model) * [Embeddings model](#embeddings-model) --- # Linux | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Linux ===== Running Tabby on Linux using Tabby's standalone executable distribution. Find the Linux release[​](#find-the-linux-release "Direct link to Find the Linux release") ------------------------------------------------------------------------------------------- * Go to the Tabby release page: [https://github.com/TabbyML/tabby/releases](https://github.com/TabbyML/tabby/releases) * Click on the **Assets** dropdown for a specific release to find the manylinux zip files. ![Linux release](/assets/images/assets-96f90759b687ef457cb5494ac13ac851.png) Download the release[​](#download-the-release "Direct link to Download the release") ------------------------------------------------------------------------------------- * If you are using a CPU-only system, download the **tabby\_x86\_64-manylinux2014.zip**. * If you are using a GPU-enabled system, download the **tabby\_x86\_64-manylinux2014-cuda117.zip**, In this example, we assume you are using CUDA 11.7. * If you want to use a non-nvidia GPU, download the **tabby\_x86\_64-manylinux2014-vulkan.zip**. See [https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/](https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/) for more info. **Tips:** * For the CUDA versions, you will need the nvidia-cuda-toolkit installed for your distribution. * In ubuntu, this would be `sudo apt install nvidia-cuda-toolkit`. * The CUDA Toolkit is available directly from Nvidia: [https://developer.nvidia.com/cuda-toolkit](https://developer.nvidia.com/cuda-toolkit) * Ensure that you have CUDA version 11 or higher installed. * Check your local CUDA version by running the following command in a terminal: `nvcc --version` * For the Vulkan version you'll need the vulkan library. In ubuntu, this would be `sudo apt install libvulkan1`. Find the Linux executable file[​](#find-the-linux-executable-file "Direct link to Find the Linux executable file") ------------------------------------------------------------------------------------------------------------------- * Unzip the file you downloaded. The `tabby` executable will be in a subdirectory of dist. * Change to this subdirectory or relocate `tabby` to a folder of your choice. * Make it executable: `chmod +x tabby llama-server` Run the following command: # For CPU-only environments./tabby serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct# For GPU-enabled environments (where DEVICE is cuda or vulkan)./tabby serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device $DEVICE You can choose different models, as shown in [the model registry](https://tabby.tabbyml.com/docs/models/) You should see a success message similar to the one in the screenshot below. After that, you can visit [http://localhost:8080](http://localhost:8080) to access your Tabby instance. ![Linux running success](/assets/images/success-a53e1705c3cb283ab558c40b17472b2b.png) * [Find the Linux release](#find-the-linux-release) * [Download the release](#download-the-release) * [Find the Linux executable file](#find-the-linux-executable-file) --- # Connect IDE / Editor Extensions | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) Connect IDE / Editor Extensions =============================== Once you have registered your account, you can now connect your IDE / Editor extensions to Tabby. To do this, please follow the installation guide for [IDE / Editor Extensions](/docs/extensions/installation/vscode/) . In this example, we'll use VSCode. In the extension settings, you need to fill in the Endpoint provided on the homepage. This Endpoint is essential for establishing a connection between your IDE / Editor and Tabby. ![Setup Endpoint](/assets/images/setup-endpoint-aeff4b04cfb13e6be374255da116682f.png) Once you have entered the Endpoint, you will receive a notification indicating that an access token is required. This access token is to authenticate you as a user of Tabby. Click on the **Set Credentials** button and enter the token acquired from the homepage. ![Personal Token](/assets/images/personal-token-8600f79bf66f496cf4a03f09d76924b2.png) After setting the token, you will see a connected icon in the status bar of your IDE / Editor, indicating a successful connection with Tabby. ![Connected](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIwAAAAuCAYAAAD6FPHbAAABXmlDQ1BJQ0MgUHJvZmlsZQAAKJF1kL9LQnEUxY+lGOXQYE0N0hAEGvV0qQYxCQlExIx+bM+nafijL+89qYagmlqDqDmI6D+woSFaWoOgIIgapTVwKXmdr1Zq0Rfu93w43Hs5XKDLpQpRsAMolkw9EZnxLC2veJxVOOBGDxRA1QwRisejbMG3dr7aPWxS73xy1+bxdn5g4imSf96frO4NH/zt73i96YyhUT9YiiZ0E7CNk+MbppC8Q3brDEU+lJxt8rnkVJMvGz3JRJh8S+7Xcmqa/EL2ptr8bBsXC2XtK4NM78qUFuapg6whRDGLGDxIws8bCPj4M9U/M4HGTBjr7NyCjjVkkYPJ+RAdgQIy5DmUoGEMXrLcpiAgb/37hi1vlzmnXgmnLS92A1yUGT/W8kaCjHICXE8LVVd/Lmur2Y1Vv9LkvgrgOLKst0XAOQrUHyzrvWJZ9TOg+xG4qn0Cg61jyYIhcykAAACKZVhJZk1NACoAAAAIAAQBGgAFAAAAAQAAAD4BGwAFAAAAAQAAAEYBKAADAAAAAQACAACHaQAEAAAAAQAAAE4AAAAAAAAAkAAAAAEAAACQAAAAAQADkoYABwAAABIAAAB4oAIABAAAAAEAAACMoAMABAAAAAEAAAAuAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdGJmP0oAAAAJcEhZcwAAFiUAABYlAUlSJPAAAAHVaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjQ2PC9leGlmOlBpeGVsWURpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjE0MDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpu21M2AAAAHGlET1QAAAACAAAAAAAAABcAAAAoAAAAFwAAABcAAAOndTBTgQAAA3NJREFUeAHsmksofFEcx79DIuWxYmWBPFaS7CQrspDkGWVDyU4eDaFJCBMLpZAFGwoRspHImiiJlfKIKY9iQZjE/P1+f3e6HmPuMWpm6ndq5p57zvf3u3e+9zO/c6bGFB0d7YA0ccCgAyYBxqBTImMHTImJiVJhBAbDDggwhq0SITkgS5JwoOSAAKNkl4gFGGFAyQEBRskuEQswwoCSAwKMkl0iFmCEASUHBBglu0QswAgDSg4IMEp2idhvgAkICEBkZCRubm7kqXnRAb8BpqqqCllZWeju7sbh4aEhy9LS0hAYGGhIq4kODg7w8PCgnRo6xsTEYHBwkLVdXV3Y3d01FEei8vJyFBcX4+XlBaWlpYbjvCX0C2Byc3NRU1ODi4sLNDc34+7uzq1foaGhmJycdKv7LBgeHsba2trn4R/P4+Pj0d/fz5q+vj5sbW39qNdP0hchLy8PDocDRUVF+imf7Ps8MKmpqWhvb8fj4yNaWlpgs9kMGflbYIaGhrCxsWHoGppIgNGc8PKRSn1vby+Cg4NBpX5vb8/jOxoYGEBcXByenp5QUVHhcT5KIMD8iY2eJYmIiIDVakVUVBRGR0exurrqWcL3aFVgCFa6h7OzM5fXdwVMeHg46HOcn5/zkvNdAldL0ts/ITnm6urquzCvjfnkkhQUFITOzk4kJSVheXkZExMTf2aQEWDo+mazGSkpKaA+Ndpj3N/fY2FhAYuLix/u5zMwCQkJyM/Pd8aS+PT0FLS/uby8/BCrB6ayshJ1dXVIT0+HyWRiHV13Z2eHY19fX3ksIyMDDQ0N3Kc91/r6+oecdEIb/ra2Nh4fGRlR3pd9Sfg+4JPA1NfXIzMzE9vb27wkkWl/1dwBExISgvHxcdDRVVtZWcHY2JhzWg8MVSJaSr9rBFx1dTWen5+d03pgrq+vuZo5J3Wd/f19WCwWHiGIp6enGaqjoyM0NTXplP+7HR0dDDx5R0uv3W7/ovnNgM8BU1ZWBnqdnJygtbWV9xq/+WCuYtwBU1tbi5ycHA6nPdPs7CxoWcjOzkZBQQFXDXoI+l80emAokIBYWlribzVVm8LCQsTGxnLO4+NjNDY2cp/eNGC0gdvbW8zNzWFzcxNUSegnd1hYGE/Pz89jamqK+z09PUhOTubKV1JSAq36aHlmZmb4Xl0BpelUj/8AAAD//+wQAKIAAAZASURBVO1aa0gVXxCfW4ppDyxNoSjJHqRCSqH2Qak+CRGKgliZGOUDUYhMUynUrIis7EGUoqkZla+IEA1TsMgU0Q+lgoKlSWkUpVFRiZX/fgO77L3uvd4/7ZG67Hy4e/Y85syZ8zszc2avwd3dfYoEkcFgoKkp69kHBwfTwYMH6ePHj3T48GF6//695pKdPXuWPD096fv377R79+5p/GNjY2n16tX07t07unz5slF7TEwMhYeHc11OTg719PRwGf3PnDnDZawXbb29vUZji4qKaOnSpVwXHx9PHz584PK+fftox44dXIZMcXFx9PXrV3msi4sLFRYW0ty5c41k3rRpEx05coT7Xb16lZqamuQxGzdupKNHj/L7+fPn6fHjx3LbnxYMogCzfPlyOnToEJWWlk5TnprQ69evp2PHjjHAsNjnz5+rdfvjupkAY2mCBQsWUEVFBXe5ceMG3b17l8tKwAAo2dnZ09isXbuWTp8+zfUAwIMHD7isBExZWRnV1dVNGwsQbd++net37dpFExMTXL516xbNmzePBgcHKS0tTR6Xm5tLGzZsoMnJSYqKipLrtSgIAwxOMRQ0Pj5Oqamp9OXLF7Py/gYt9124cCGdO3eO2trazPb90wZrAAPL6O3tTQEBAYSNnj9/Pjk6OvLmADQgc4ABiNBmSuBZW1tLeLa2tlJBQQF3UQLmwIED9OrVK9OhtHnzZra4aIBOOzo6uA+sMawyrFpkZCT9+vWL66uqqsje3p46Ozvp1KlTXKfVjzDAQECYb5jxJ0+eMBDUhHZycuJFrVixgnBioFSRNBNglixZwu5l8eLFFsUwBxgAHutVo9u3b5ODgwMNDQ2x9UUfJWAiIiLUhhHcUnFxMbdBP9ATaNmyZbLblNyS0h2lp6fTixcvuK9WP0IBg9OUl5dHPj4+dOnSJXr48KGR3PDL8MN+fn706NEjunjxolG7iJeZAFNeXk6LFi3iqT9//szuFBuMuArvGRkZ3GYOMDU1NQRgqNGdO3fYwrS3t8sxjxIwycnJ9ObNm2lDlfGKKSBLSkoIIJfcEuInX19flhXxmNYkFDAQ1tXVlRB4zZkzh13T27dv5TUkJiZSSEgI9fX1caD448cPuU1UwRJglCdZLRYB8I8fP86imQNMd3c3IYYwJQ8PD9YD6rHJDQ0N3EUJGFiR+/fvmw4lbHxYWBjX79mzxygoRkwDdyS5JYAV7gj8MY/WJBwwEDgoKIjB0t/fz9E7fC1uBlAWAIRT++nTJ63XpsrPEmACAwNlC6JmKRCMw+SDzAEGG4cAFFZJSRcuXKCVK1dyVVJSEq8bL0rAIM7bv38/B6vSWMRMABJcmVoQi6D35s2bbLlaWlpo27ZtPHTv3r1CdDorgMEKENBt2bKFEJDBr2ZlZdG3b98oMzOTXr9+LelH+NMSYHAyKysrWfm44ubn59PTp0/Jzc2NN9Lf31+Wzxxg0AFjceOBCwZIYBXgJkBwOXA9EikBg7qRkRG+WWJeBN0AEKw0qLm5ma5cucJl5Q/c1KpVq+Sq0dFRSklJkd+1LMwaYBDc4mYAs4+Tgs05ceIEPXv2TMv1zMjLEmAw+OTJk+Tl5SXzgcVALAaCBZjploT8DQCmRgBSQkKC0Y1RCRjESc7OzmpD6eXLl2yl1Rql/JXUdu3aNaqvr5deNX3OGmAgNXItAAniGSSyGhsbNV2MNcxgNdasWcPWLTo6WnUIXCROtwQUdBoeHubbHHIooOvXr9O9e/e4jNONUw7CNRbX4K1btxqNHxsbYzCauiq4jtDQUI5BYE1gcdetW8e8pJ+BgQHO7Uj5F6le+ayuriY7Ozv6+fMn7dy5k5/Kdq3KswoYCI0kF/IaCA7/dgIQYFHgQpXZV2vkxqEAMGFZATbko6wlxCsYCx7WzI054CIBcLVg3dp5rek364CxRii9z//TAFITuHqDkMwDQEWRDhhRmhXMF7cjXN+RvJPiKmVCUNT0OmBEaVYwXyTrlHkWfKjFtzskF0WSDhiR2hXIG3EOAmakJpDf6urqkr8lCZyWdMCI1K4N8tYBY4ObKnJJOmBEatcGeeuAscFNFbkkHTAitWuDvHXA2OCmilyS4ffncOv/pS1SEp33P6EBHTD/xDb9PUL+B6HiZ/kP0An7AAAAAElFTkSuQmCC) Congratulations! You have completed the setup process. Now, you can enjoy the benefits of code completion with Tabby, making your coding experience more efficient and productive. ![Code Completion](/assets/images/code-completion-464a64614eef9b844d8fd7b802073abb.png) --- # BentoCloud | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page BentoCloud ========== [BentoCloud](https://cloud.bentoml.com/) provides a serverless infrastructure tailored for GPU workloads, enabling seamless deployment, management, and scaling of models in the cloud. Setup[​](#setup "Direct link to Setup") ---------------------------------------- Begin by crafting a `service.py` to delineate your Bento service. This script delineates the GPU resources requisite for operating your service. service.py @bentoml.service( resources={"gpu": 1, "gpu_type": "nvidia-l4"}, traffic={"timeout": 10},) BentoCloud currently supports the following GPUs: * `T4`: A cost-effective GPU selection with 16GiB of memory. * `L4`: A mid-range GPU offering 24GiB of memory. * `A100`: The pinnacle of GPU power in the cloud, available in configurations of 40GiB and 80GiB memory options. For comprehensive details, please refer to the official [BentoCloud Pricing](https://www.bentoml.com/pricing) . Define the Container Image[​](#define-the-container-image "Direct link to Define the Container Image") ------------------------------------------------------------------------------------------------------- To construct a container image replete with the preloaded Tabby model cache, draft a `bentofile.yaml`. This document stipulates the CUDA version as 11.7.1 and enumerates the essential system packages and dependencies for the image. Leveraging BentoCloud's internal filesystem circumvents the need to redownload the model, thereby accelerating cold start times. Below is the `bentofile.yaml`: bentofile.yaml service: 'service:Tabby'include: - '*.py'python: packages: - asgi-proxy-lib docker: cuda_version: "11.7.1" system_packages: - unzip - git - curl - software-properties-common setup_script: "./setup-docker.sh" The `asgi-proxy-lib` package is specified to facilitate communication with the Tabby server via localhost, and the `setup-docker.sh` script is configured to install Tabby and procure the model weights. setup-docker.sh # Install TabbyDISTRO=tabby_x86_64-manylinux2014-cuda117curl -L https://github.com/TabbyML/tabby/releases/download/v0.14.0/$DISTRO.zip \ -o $DISTRO.zipunzip $DISTRO.zip# Download model weights under the bentoml user, as BentoCloud operates under this user.su bentoml -c "TABBY_MODEL_CACHE_ROOT=/home/bentoml/tabby-models tabby download --model StarCoder-1B"su bentoml -c "TABBY_MODEL_CACHE_ROOT=/home/bentoml/tabby-models tabby download --model Qwen2-1.5B-Instruct"su bentoml -c "TABBY_MODEL_CACHE_ROOT=/home/bentoml/tabby-models tabby download --model Nomic-Embed-Text" ### Service Definition[​](#service-definition "Direct link to Service Definition") The service endpoint is encapsulated with BentoML's `@bentoml.service`. Here, we: 1. Initiate the Tabby process and ensure its readiness to process incoming requests. 2. Establish an ASGI proxy to relay requests from the Modal web endpoint to the local Tabby server. 3. Allocate 1 Nvidia L4 GPU per worker, with a 10-second timeout. 4. Employ `on_deployment` and `on_shutdown` hooks to transfer persisted data to and from object storage. service.py app = asgi_proxy("http://127.0.0.1:8000")@bentoml.service( resources={"gpu": 1, "gpu_type": "nvidia-l4"}, traffic={"timeout": 10},)@bentoml.mount_asgi_app(app, path="/")class Tabby: @bentoml.on_deployment def prepare(): download_tabby_dir("tabby-local") @bentoml.on_shutdown def shutdown(self): upload_tabby_dir("tabby-local") def __init__(self) -> None: model_id = "StarCoder-1B" chat_model_id = "Qwen2-1.5B-Instruct" # Fire up the server subprocess. self.server = TabbyServer(model_id, chat_model_id) # Await server readiness. self.server.wait_until_ready() Finally, we draft a deployment configuration file `bentodeploy.yaml` to outline the deployment specifics. Note that we employ rclone to synchronize persisted data with Cloudflare R2 object storage. You can get the values of the following R2 environment variables by referring to the [Cloudfare R2 documentation](https://developers.cloudflare.com/r2/api/s3/tokens/) . bentodeploy.yaml name: tabby-localbento: ./access_authorization: falseenvs: - name: RCLONE_CONFIG_R2_TYPE value: s3 - name: RCLONE_CONFIG_R2_ACCESS_KEY_ID value: $YOUR_R2_ACCESS_KEY_ID - name: RCLONE_CONFIG_R2_SECRET_ACCESS_KEY value: $YOUR_R2_SECRET_ACCESS_KEY - name: RCLONE_CONFIG_R2_ENDPOINT value: $YOUR_R2_ENDPOINT - name: TABBY_MODEL_CACHE_ROOT value: /home/bentoml/tabby-models ### Serve the Application[​](#serve-the-application "Direct link to Serve the Application") Deploying the model with `bentoml deploy -f bentodeploy.yaml` will establish a BentoCloud deployment and serve your application. ![app-running](/assets/images/app-running-9e0b26d673454135301603b8b5c9d8de.png) Once the deployment is operational, you can access the service via the provided URL, e.g., `https://$YOUR_DEPLOYMENT_SLUG.mt-guc1.bentoml.ai`. For the complete code of this tutorial, please refer to the [GitHub repository](https://github.com/TabbyML/tabby/tree/main/website/docs/quick-start/installation/bentoml) . * [Setup](#setup) * [Define the Container Image](#define-the-container-image) * [Service Definition](#service-definition) * [Serve the Application](#serve-the-application) --- # DeepInfra | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page DeepInfra ========= [DeepInfra](https://deepinfra.com/) is a cloud platform providing efficient and scalable model inference services, offering access to various open-source models like [Llama 3](https://deepinfra.com/meta-llama/Llama-3.3-70B-Instruct) , [Mixtral](https://deepinfra.com/mistralai/Mixtral-8x7B-Instruct-v0.1) , and [Qwen](https://deepinfra.com/Qwen/Qwen2.5-Coder-32B-Instruct) . Chat model[​](#chat-model "Direct link to Chat model") ------------------------------------------------------- DeepInfra provides an OpenAI-compatible chat API interface. ~/.tabby/config.toml [model.chat.http]kind = "openai/chat"model_name = "meta-llama/Llama-3.3-70B-Instruct"api_endpoint = "https://api.deepinfra.com/v1/openai"api_key = "your-api-key" Completion model[​](#completion-model "Direct link to Completion model") ------------------------------------------------------------------------- DeepInfra provides an OpenAI-compatible completion API interface. ~/.tabby/config.toml [model.completion.http]kind = "openai/completion"model_name = "Qwen/Qwen2.5-Coder-32B-Instruct"api_endpoint = "https://api.deepinfra.com/v1/openai"api_key = "your-api-key" Embeddings model[​](#embeddings-model "Direct link to Embeddings model") ------------------------------------------------------------------------- DeepInfra also provides an OpenAI-compatible embeddings API interface. ~/.tabby/config.toml [model.embedding.http]kind = "openai/embedding"model_name = "BAAI/bge-base-en-v1.5"api_endpoint = "https://api.deepinfra.com/v1/openai"api_key = "your-api-key" * [Chat model](#chat-model) * [Completion model](#completion-model) * [Embeddings model](#embeddings-model) --- # Azure OpenAI | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Azure OpenAI ============ [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service) is a cloud-based service that provides Azure customers with access to OpenAI's powerful language models including GPT-4, GPT-3.5, and various embedding models. Please be aware that azure will be supported starting with version 0.24, which is scheduled for release by end of 01/2025 Chat model[​](#chat-model "Direct link to Chat model") ------------------------------------------------------- It supports various GPT series chat models through an Azure OpenAI-compatible API interface. ~/.tabby/config.toml [model.chat.http]kind = "azure/chat"model_name = "gpt-4o-mini"api_endpoint = "https://.openai.azure.com"api_key = "your-api-key" Completion model[​](#completion-model "Direct link to Completion model") ------------------------------------------------------------------------- Azure OpenAI currently does not offer completion-specific API endpoints. Embeddings model[​](#embeddings-model "Direct link to Embeddings model") ------------------------------------------------------------------------- It supports text-embedding-3-small, text-embedding-3-large and other embedding models through an Azure OpenAI-compatible API interface. ~/.tabby/config.toml [model.embedding.http]kind = "azure/embedding"model_name = "text-embedding-3-large"api_endpoint = "https://.openai.azure.com"api_key = "your-api-key" * [Chat model](#chat-model) * [Completion model](#completion-model) * [Embeddings model](#embeddings-model) --- # Windows | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Windows ======= Running Tabby on Windows using Tabby's exe distribution. Find the Windows release[​](#find-the-windows-release "Direct link to Find the Windows release") ------------------------------------------------------------------------------------------------- * Go to the Tabby release page: [https://github.com/TabbyML/tabby/releases](https://github.com/TabbyML/tabby/releases) * Click on the **Assets** dropdown for a specific release to find the Windows zip files. ![Windows release](/assets/images/assets-3f03132f0537634a90aa9e0dc2ad8bdd.png) Download the release[​](#download-the-release "Direct link to Download the release") ------------------------------------------------------------------------------------- * If you are using a CPU-only system, download the **tabby\_x86\_64-windows-msvc.zip**. * If you are using a GPU-enabled system, download the **tabby\_x86\_64-windows-msvc-cuda117.zip**, In this example, we assume you are using CUDA 11.7. **Tips:** * Download the CUDA Toolkit from Nvidia: [https://developer.nvidia.com/cuda-toolkit](https://developer.nvidia.com/cuda-toolkit) * Ensure that you have CUDA version 11 or higher installed. * Check your local CUDA version by running the following command in a command prompt or PowerShell window: nvcc --version Find the Windows executable file[​](#find-the-windows-executable-file "Direct link to Find the Windows executable file") ------------------------------------------------------------------------------------------------------------------------- * Unzip the file `tabby_x86_64-windows-msvc-cuda117.zip`. * Navigate to the extracted folder named `tabby_x86_64-windows-msvc-cuda117`. * Inside this folder, go to `dist` -> `tabby_x86_64-windows-msvc-cuda117`. * In this directory, you'll find an executable file named `tabby.exe`. Running Tabby[​](#running-tabby "Direct link to Running Tabby") ---------------------------------------------------------------- Open a command prompt or PowerShell window, as administrator, in the directory where the `tabby.exe` is located (from the previous step). Run the following command: # For CPU-only environments.\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct# For CUDA-enabled environments.\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda You should see a success message similar to the one in the screenshot below. After that, you can visit [http://localhost:8080](http://localhost:8080) to access your Tabby instance. ![Windows running success](/assets/images/success-a53e1705c3cb283ab558c40b17472b2b.png) * [Find the Windows release](#find-the-windows-release) * [Download the release](#download-the-release) * [Find the Windows executable file](#find-the-windows-executable-file) * [Running Tabby](#running-tabby) --- # Registering Accounts | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Registering Accounts ==================== After deploying Tabby, you will need to register an account on your server to access the instance. Open the homepage by the url you displayed in startup logs, e.g. `http://localhost:8080`. Creating an Admin Account[​](#creating-an-admin-account "Direct link to Creating an Admin Account") ---------------------------------------------------------------------------------------------------- The first registered account after deployment will be the admin account and will be granted the **owner** role. ![Setup Admin](/assets/images/setup-admin-7e2fd7480bf6fb6a09cac16e0e409c6f.png) Entering Homepage[​](#entering-homepage "Direct link to Entering Homepage") ---------------------------------------------------------------------------- Once logged in, you will be redirected to the homepage. It contains basic information about your account. More importantly, you will find the credentials you need to connect your IDE/Editor extensions to Tabby. ![Homepage](/assets/images/homepage-affd618f314b690c70f710b5f613c8bf.png) (Optional) Invite your team members[​](#optional-invite-your-team-members "Direct link to (Optional) Invite your team members") -------------------------------------------------------------------------------------------------------------------------------- Tabby offers an enhanced experience with a full-featured UI interface and many enterprise-facing features. You can invite your team members to join your instance and collaborate on your projects. To invite team members, click on **Settings** in the Homepage then select **Members** from the side bar. ![Invite user](/assets/images/invite-user-5734dc08a8f247d34ef6fbd5c9f5b0a1.png) For more information on how to configure the instance, please refer to the [Administration](/docs/administration/upgrade/) documentation. * [Creating an Admin Account](#creating-an-admin-account) * [Entering Homepage](#entering-homepage) * [(Optional) Invite your team members](#optional-invite-your-team-members) --- # DeepSeek | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) DeepSeek ======== [DeepSeek](https://www.deepseek.com/) offers a suite of AI models, such as [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) and [DeepSeek Coder](https://huggingface.co/collections/deepseek-ai/deepseekcoder-v2-666bf4b274a5f556827ceeca) , which perform well in coding tasks. Tabby supports DeepSeek's models for both code completion and chat. Below is an example ~/.tabby/config.toml # Chat model configuration[model.chat.http]# Deepseek's chat interface is compatible with OpenAI's chat API.kind = "openai/chat"model_name = "your_model"api_endpoint = "https://api.deepseek.com/v1"api_key = "secret-api-key"# Completion model configuration[model.completion.http]# Deepseek uses its own completion API interface.kind = "deepseek/completion"model_name = "your_model"api_endpoint = "https://api.deepseek.com/beta"api_key = "secret-api-key" --- # Fireworks | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) On this page Fireworks ========= [Fireworks](https://app.fireworks.ai/) is a cloud platform that offers efficient model inference and deployment services, providing cost-effective access to a variety of AI models through their API service, including [Llama 2](https://fireworks.ai/models/fireworks/llama-v2-70b-chat) , [DeepSeek V3](https://fireworks.ai/models/fireworks/deepseek-v3) , [DeepSeek Coder](https://fireworks.ai/models/fireworks/deepseek-coder-v2-instruct) and other open-source models. Chat model[​](#chat-model "Direct link to Chat model") ------------------------------------------------------- Fireworks provides an OpenAI-compatible chat API interface. ~/.tabby/config.toml [model.chat.http]kind = "openai/chat"model_name = "accounts/fireworks/models/deepseek-v3"api_endpoint = "https://api.fireworks.ai/inference/v1"api_key = "your-api-key" Completion model[​](#completion-model "Direct link to Completion model") ------------------------------------------------------------------------- Fireworks does not offer completion models (FIM) through their API. Embeddings model[​](#embeddings-model "Direct link to Embeddings model") ------------------------------------------------------------------------- While Fireworks provides embedding model APIs, Tabby has not yet implemented a compatible client to interface with these APIs. Therefore, embedding functionality is currently not available through Tabby's integration with Fireworks. * [Chat model](#chat-model) * [Completion model](#completion-model) * [Embeddings model](#embeddings-model) --- # llama.cpp | Tabby [Skip to main content](#__docusaurus_skipToContent_fallback) llama.cpp ========= [llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#api-endpoints) is a popular C++ library for serving gguf-based models. Tabby supports the llama.cpp HTTP API for completion, chat, and embedding models. ~/.tabby/config.toml # Completion model[model.completion.http]kind = "llama.cpp/completion"api_endpoint = "http://localhost:8888"prompt_template = "
 {prefix} {suffix} "  # Example prompt template for the CodeLlama model series.# Chat model[model.chat.http]kind = "openai/chat"api_endpoint = "http://localhost:8888"# Embedding model[model.embedding.http]kind = "llama.cpp/embedding"api_endpoint = "http://localhost:8888"

---

# Jan AI | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

Jan AI
======

[Jan](https://jan.ai/)
 is an open-source alternative to ChatGPT that runs entirely offline on your computer.

Jan can run a server that provides an OpenAI-equivalent chat API at [https://localhost:1337](https://localhost:1337)
, allowing us to use the OpenAI kinds for chat. To use the Jan Server, you need to enable it in the Jan App's `Local API Server` UI.

However, Jan does not yet provide API support for completion and embeddings.

Below is an example for chat:

~/.tabby/config.toml

    # Chat model[model.chat.http]kind = "openai/chat"model_name = "your_model"api_endpoint = "http://localhost:1337/v1"api_key = ""

---

# llamafile | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

llamafile
=========

[llamafile](https://github.com/Mozilla-Ocho/llamafile)
 is a Mozilla Builders project that allows you to distribute and run LLMs with a single file.

llamafile embeds a llama.cpp server and provides an OpenAI API-compatible chat-completions endpoint, allowing us to use the `openai/chat`, `llama.cpp/completion`, and `llama.cpp/embedding` types.

By default, llamafile uses port `8080`, which is also used by Tabby. Therefore, it is recommended to run llamafile with the `--port` option to serve on a different port, such as `8081`.

For embeddings, the embedding endpoint is no longer supported in the standard llamafile server, so you need to run llamafile with the `--embedding` and `--port` options.

Below is an example configuration:

~/.tabby/config.toml

    # Chat model[model.chat.http]kind = "openai/chat"  # llamafile uses openai/chat kindmodel_name = "your_model"api_endpoint = "http://localhost:8081/v1"  # Please add and conclude with the `v1` suffixapi_key = ""# Completion model[model.completion.http]kind = "llama.cpp/completion"   # llamafile uses llama.cpp/completion kindmodel_name = "your_model"api_endpoint = "http://localhost:8081"  # DO NOT append the `v1` suffixapi_key = "secret-api-key"prompt_template = "<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>" # Example prompt template for the Qwen2.5 Coder model series.# Embedding model[model.embedding.http]kind = "llama.cpp/embedding"  # llamafile uses llama.cpp/embedding kindmodel_name = "your_model"api_endpoint = "http://localhost:8082"  # DO NOT append the `v1` suffixapi_key = ""

---

# OpenRouter | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

OpenRouter
==========

[OpenRouter](https://openrouter.ai/)
 provides unified access to multiple AI models through an OpenAI API compatible RESTful endpoint, including models from OpenAI, Anthropic, Google, and Meta.

Chat Model[​](#chat-model "Direct link to Chat Model")

-------------------------------------------------------

OpenRouter provides an OpenAI-compatible chat API interface.

~/.tabby/config.toml

    [model.chat.http]kind = "openai/chat"model_name = "openai/gpt-4"  # Can be any model from https://openrouter.ai/modelsapi_endpoint = "https://openrouter.ai/api/v1"api_key = "your-api-key"

Completion Model[​](#completion-model "Direct link to Completion Model")

-------------------------------------------------------------------------

OpenRouter does not offer completion models (FIM) through their API.

Embeddings Model[​](#embeddings-model "Direct link to Embeddings Model")

-------------------------------------------------------------------------

OpenRouter does not offer embeddings models through their API.

Supported Models[​](#supported-models "Direct link to Supported Models")

-------------------------------------------------------------------------

For a complete list of supported models, visit [OpenRouter's Model List](https://openrouter.ai/models)
.

*   [Chat Model](#chat-model)
    
*   [Completion Model](#completion-model)
    
*   [Embeddings Model](#embeddings-model)
    
*   [Supported Models](#supported-models)

---

# Mistral AI | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

Mistral AI
==========

[Mistral](https://mistral.ai/)
 is a platform that provides a suite of AI models. Tabby supports Mistral's models for code completion and chat.

To connect Tabby with Mistral's models, you need to apply the following configurations in the `~/.tabby/config.toml` file:

~/.tabby/config.toml

    # Completion Model[model.completion.http]kind = "mistral/completion"model_name = "codestral-latest"api_endpoint = "https://api.mistral.ai"api_key = "secret-api-key"# Chat Model[model.chat.http]kind = "mistral/chat"model_name = "codestral-latest"api_endpoint = "https://api.mistral.ai/v1"api_key = "secret-api-key"

---

# Ollama | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

Ollama
======

[ollama](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-completion)
 is a popular model provider that offers a local-first experience.

Tabby supports the ollama HTTP API for completion, chat, and embedding models.

~/.tabby/config.toml

    # Completion model[model.completion.http]kind = "ollama/completion"model_name = "codellama:7b"api_endpoint = "http://localhost:11434"prompt_template = "
 {prefix} {suffix} "  # Example prompt template for the CodeLlama model series.# Chat model[model.chat.http]kind = "openai/chat"model_name = "mistral:7b"api_endpoint = "http://localhost:11434/v1"# Embedding model[model.embedding.http]kind = "ollama/embedding"model_name = "nomic-embed-text"api_endpoint = "http://localhost:11434"

---

# vLLM | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

vLLM
====

[vLLM](https://docs.vllm.ai/en/stable/)
 is a fast and user-friendly library for LLM inference and serving.

vLLM offers an `OpenAI Compatible Server`, enabling us to use the OpenAI kinds for chat and embedding. However, for completion, there are certain differences in the implementation. Therefore, we should use the `vllm/completion` kind and provide a `prompt_template` depending on the specific models.

Please note that models differ in their capabilities for completion or chat. You should confirm the model's capability before employing it for chat or completion tasks.

Additionally, there are models that can serve both as chat and completion. For detailed information, please refer to the [Model Registry](/docs/models/)
.

Below is an example of the vLLM running at `http://localhost:8000`:

Please note the following requirements in each model type:

1.  `model_name` must exactly match the one used to run vLLM.
2.  `api_endpoint` should follow the format `http://host:port/v1`.
3.  `api_key` should be identical to the one used to run vLLM.

~/.tabby/config.toml

    # Chat model[model.chat.http]kind = "openai/chat"model_name = "your_model"   # Please make sure to use a chat model.api_endpoint = "http://localhost:8000/v1"api_key = "secret-api-key"# Completion model[model.completion.http]kind = "vllm/completion"model_name = "your_model"  # Please make sure to use a completion model.api_endpoint = "http://localhost:8000/v1"api_key = "secret-api-key"prompt_template = "
 {prefix} {suffix} "  # Example prompt template for the CodeLlama model series.# Embedding model[model.embedding.http]kind = "openai/embedding"model_name = "your_model"api_endpoint = "http://localhost:8000/v1"api_key = "secret-api-key"

---

# Perplexity AI | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

Perplexity AI
=============

[Perplexity AI](https://www.perplexity.ai/)
 is a company that develops large language models and offers them through their API service. They currently provide three powerful Llama-based models: [Sonar Small (8B)](https://docs.perplexity.ai/guides/model-cards#supported-models)
, [Sonar Large (70B)](https://docs.perplexity.ai/guides/model-cards#supported-models)
, and [Sonar Huge (405B)](https://docs.perplexity.ai/guides/model-cards#supported-models)
, all supporting a 128k context window.

Chat model[​](#chat-model "Direct link to Chat model")

-------------------------------------------------------

Perplexity provides an OpenAI-compatible chat API interface. The Sonar Large (70B) and Huge (405B) models are recommended for better performance.

~/.tabby/config.toml

    [model.chat.http]kind = "openai/chat"model_name = "llama-3.1-sonar-large-128k-online"  # Also supports sonar-small-128k-online or sonar-huge-128k-onlineapi_endpoint = "https://api.perplexity.ai"api_key = "your-api-key"

Completion model[​](#completion-model "Direct link to Completion model")

-------------------------------------------------------------------------

Perplexity currently does not offer completion-specific API endpoints.

Embeddings model[​](#embeddings-model "Direct link to Embeddings model")

-------------------------------------------------------------------------

Perplexity currently does not offer embeddings models through their API.

*   [Chat model](#chat-model)
    
*   [Completion model](#completion-model)
    
*   [Embeddings model](#embeddings-model)

---

# OpenAI | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

OpenAI
======

OpenAI is a leading AI company that has developed an extensive range of language models. Tabby supports OpenAI's API specifications for chat, completion, and embedding tasks.

The OpenAI API is widely used and is also provided by other vendors, such as vLLM, Nvidia NIM, and LocalAI.

Tabby continues to support the OpenAI Completion API specifications due to its widespread usage.

Chat model[​](#chat-model "Direct link to Chat model")

-------------------------------------------------------

~/.tabby/config.toml

    # Chat model[model.chat.http]kind = "openai/chat"model_name = "gpt-4o"  # Please make sure to use a chat model, such as gpt-4oapi_endpoint = "https://api.openai.com/v1"   # DO NOT append the `/chat/completions` suffixapi_key = "secret-api-key"

Completion model[​](#completion-model "Direct link to Completion model")

-------------------------------------------------------------------------

OpenAI doesn't offer models for completions (FIM), its `/v1/completions` API has been deprecated.

Embeddings model[​](#embeddings-model "Direct link to Embeddings model")

-------------------------------------------------------------------------

~/.tabby/config.toml

    # Embedding model[model.embedding.http]kind = "openai/embedding"model_name = "text-embedding-3-small"   # Please make sure to use a embedding model, such as text-embedding-3-smallapi_endpoint = "https://api.openai.com/v1"  # DO NOT append the `/embeddings` suffixapi_key = "secret-api-key"

*   [Chat model](#chat-model)
    
*   [Completion model](#completion-model)
    
*   [Embeddings model](#embeddings-model)

---

# Hugging Face Spaces | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

Hugging Face Spaces
===================

In this guide, you will learn how to deploy your own Tabby instance and use it for development directly from the Huggingface website.

tip

This tutorial is now also available on [Hugging Face](https://huggingface.co/docs/hub/spaces-sdks-docker-tabby)
!

Your first Tabby Space[​](#your-first-tabby-space "Direct link to Your first Tabby Space")

-------------------------------------------------------------------------------------------

In this section, you will learn how to deploy a Tabby Space and use it for yourself or your organization.

### Deploy Tabby on Spaces[​](#deploy-tabby-on-spaces "Direct link to Deploy Tabby on Spaces")

You can deploy Tabby on Spaces with just a few clicks:

[![Deploy on HF Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/deploy-to-spaces-lg.svg)](https://huggingface.co/spaces/TabbyML/tabby-template-space?duplicate=true)

You need to define the Owner (your personal account or an organization), a Space name, and the Visibility. To secure the api endpoint, we're configuring the visibility as Private.

![Duplicate Space](/assets/images/duplicate-space-5cfbb41c42ae7c27d1b261e0cb52d1e8.png)

tip

If you want to customize the title, emojis, and colors of your space, go to "Files and Versions" and edit the metadata of your README.md file.

You’ll see the Building status and once it becomes Running your space is ready to go. If you don’t see the Tabby swagger UI, try refreshing the page.

![Swagger UI](/assets/images/swagger-ui-b6797be71ebe835757ae3512e2f7cd44.png)

### Your Tabby Space URL[​](#your-tabby-space-url "Direct link to Your Tabby Space URL")

Once Tabby is running, you can use the UI with the Direct URL in the **Embed this Space** option (top right). You’ll see a URL like this: [https://tabbyml-tabby.hf.space](https://tabbyml-tabby.hf.space)
. This URL gives you access to a full-screen, stable Tabby instance, and is the API Endpoint for IDE / Editor Extensions to talk with.

### Connect VSCode Extension to Space backend[​](#connect-vscode-extension-to-space-backend "Direct link to Connect VSCode Extension to Space backend")

1.  Install the [VSCode Extension](https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby)
    .
2.  Open the file located at `~/.tabby-client/agent/config.toml`. Uncomment both the `[server]` section and the `[server.requestHeaders]` section.
    *   Set the endpoint to the Direct URL you found in the previous step, which should look something like `https://UserName-SpaceName.hf.space`.
    *   As the space is set to **Private**, it is essential to configure the authorization header for accessing the endpoint. You can obtain a token from the [Access Tokens](https://huggingface.co/settings/tokens)
         page.

![Agent Config](/assets/images/agent-config-e70909eff1d58a0bfd27b123881e8785.png)

3.  You'll notice a βœ“ icon indicating a successful connection. ![Tabby Connected](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEUAAAAXCAYAAABdy4LVAAABX2lDQ1BJQ0MgUHJvZmlsZQAAKJF1kLFLAmEYxp8rRYqDGhoCG24IJws5DdpCDSJwOM2g2u4+7Qz0/Dgvoq0h1xBqaSqMoKkxl4b+A4OgISLa+gNCIUqu99PqtOj7eHl+PLzvw8sLDMk650UfgJLl2JmlhLK2vqEEXjACmcqPkM4qPK5pKWrBtw6+9j0koXczImuh9XYQzFaDPKVenp80G3/7B95oLl9hpB9UMcZtB5AixNqOwwXvEU/YtBTxkWCzxxeCjR5fd3uymSTxLfE4K+g54mfisNHnm31cKm6zrx3E9nLeWl0hnaSaQgJ5WEhDgYZFqJhDhH76n5lYdyaJMjh2YWMLJgpwaDpODkeR0hQsUyLDLMLEKqWpiIpb/76h55XrwHwLGK55nnEMNKq05oPnTZ8CY/vAVZPrtv5zWantq2xG1R7LCcD/5LqvISBwCHRqrvted93OGeU/AjfWJzEJZRk2fHsZAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAABFoAMABAAAAAEAAAAXAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdK3W3wMAAAHUaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjIzPC9leGlmOlBpeGVsWURpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjY5PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Chj/JcQAAAQKSURBVFgJ7VhZKHVfFP9dXVNIikgIJUVIhgwPIsKDFCFRpDwI5UGmFyGzF0qGFw9ECsnwgBelRClJxpB5JjKX+H9r1z7/c49zfK7vPnyf7qp79hr33ud31lr7nKtydHR8h540EFCbmppqKPQCoL69vdXjIEHAQCLrxV8I6EGRSQM9KP8iKGq1GomJiTAxMZHZPhAVFQUnJydZ23eV6u8GKsUZGBggJiYGk5OTeH191XArLy+Hj4+Pho4LhYWF2NnZ4aIwWlhYIC0tDXNzczg6OhL0nMnKysL4+Dh6enq46o9HnYOSnp4OS0vLD4DQTpuammBmZsY23dzcjIWFBXR3dzP58vKSjX/DRaegREZGwt3dHZQRcvTw8AD6Eb29veH+/h7n5+dMDg0NRW5uLiuTl5cXdHR0YHp6mtnokpqaiqCgILy/v2NxcRE1NTWCjdbs6+uDkZERTk5OUFxczOZpa2tDbW0t8yfnxsZG3N3dobKyUoiVY3TWaL28vBAfH4+6ujrZLJFbXKzLy8vDwcEBGhoacHp6ipycHLEZAQEBDKixsTH4+/sjJSVFsHt6emJ0dBSdnZ2wsbFBRUUFrq6u8PT0hNjYWOZHvcnV1RXz8/NCnBKjk0yxt7dnT7mqqoo9CaXFPtNTJhBReVlZWSE7OxuGhoZCCPWMqakpJtMDoMzq7+9n8srKCnp7exlPsQkJCYyfnZ1FWFgY4ymLKcv4HEypcNEKlIKCAoyMjGg0RGqEpaWloFSVa4QK635QU6Ol8qBGzUnMr66ucjX29/cRGBgoyLu7uwK/ubkJygo6rQYHBxEdHQ1nZ2eEh4dja2uLla3grMD8vwMFB7GaAMnPz4exsTFT0+IlJSWglF5aWhK7asVTWoeEhGBgYADJycmorq7+EE83xolKRPx5Ymdnx0349YHLbvz5+RnUvK+vrxEXFwcXFxd2SgmOnzBagUJH5szMDDIzM9mUVPeE/sTExCdL/N7EQb64uIC5uTnomJVSRkYGHBwcEBwcDA8PD1DJcPL19YWfnx97XyFQz87OuIntl7KEiPb+FdKqfGjC4eFh1r2LiopYmra2tn5lnU991tbWsLe3x/oSOVJ5SOn4+BgtLS1MfXh4iPb2dsGF5LKyMqhUKjw+PrJGy41DQ0PsAFhfX+eq344qW1tbrf9Psba2RlJSErq6ukBpqivi7zD82JbOS3ZqlnTjUqJSpiy7ubnRMLm5uaG+vp69JiwvL2vYlIRvgaI02d+mp/eeiIgIbG9vgzL7q/SjQaFvIvoTbWNj46t4ML8fDYpWSIictTp9RHE/mtWDIvN41d7e3kxNxxl1dilxPR/JrsRLY6W+cjKPEc/JdWJ/sV2JF8dxXuwrno/b+Sj2+w/2pYS5Y3qmgQAAAABJRU5ErkJggg==)
    
4.  You've complete the setup, now enjoy tabing!
    

![Code Completion](/assets/images/code-completion-a9afe18454b4bf77553d13f8c6a9653c.png)

You can also utilize Tabby extensions in other IDEs, such as [JetBrains](https://plugins.jetbrains.com/plugin/22379-tabby)
.

*   [Your first Tabby Space](#your-first-tabby-space)
    *   [Deploy Tabby on Spaces](#deploy-tabby-on-spaces)
        
    *   [Your Tabby Space URL](#your-tabby-space-url)
        
    *   [Connect VSCode Extension to Space backend](#connect-vscode-extension-to-space-backend)

---

# SkyPilot Serving | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

SkyPilot Serving
================

[SkyPilot](https://skypilot.readthedocs.io/en/latest/)
 is a versatile framework designed for the execution of LLMs, AI, and batch jobs on any cloud vendors. It stands out by offering significant cost savings, optimal GPU availability, and managed execution capabilities.

[SkyServe](https://skypilot.readthedocs.io/en/latest/serving/sky-serve.html)
 is SkyPilot’s model serving library. SkyServe (short for SkyPilot Serving) takes an existing serving framework and deploys it across one or more regions or clouds.

When leveraging SkyServe, all replica Tabby instances are seamlessly deployed within your own cloud accounts and VPCs.

Configuration[​](#configuration "Direct link to Configuration")

----------------------------------------------------------------

At first, let's specified the resource requirements for the Tabby service in the YAML configuration for SkyServe.

    resources:  ports: 8080  accelerators: T4:1  # Or, allow using any of these GPUs to enhance GPU availability.  # SkyPilot will auto-select the cheapest and available GPU.  # accelerators: {T4:1, L4:1, A100:1, A10G:1}

Skypilot supports GPU from various cloud vendors. Please refer to the official [Skypilot documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)
 for detailed installation instructions.

Tabby exposes its health check at the `/metrics` endpoint, which also serves as a prometrics endpoint. Therefore, we can define the following readiness probe:

    service:  readiness_probe: /metrics  replicas: 1

Finally, we define the command line that actually initiates the container job:

    run: |  docker run --gpus all -p 8080:8080 -v ~/.tabby:/data \    tabbyml/tabby \    serve --model TabbyML/StarCoder-1B --device cuda

Launch the service[​](#launch-the-service "Direct link to Launch the service")

-------------------------------------------------------------------------------

We first execute `sky serve up tabby.yaml -n tabby`.

![start tabby service](/assets/images/start-service-ba734e2044cf1c9327492ea5bf4864ec.png)

If everything goes well, you'll see messages below ![service ready](/assets/images/service-ready-876e165d2dc22aea0e6cf0a4bca58474.png)

This finishes launching SkyServe's control VM which runs a load balancer for this serve; the actual replica running the Tabby service is undergoing provisioning.

When you execute the following command, you'll encounter a message indicating that the replica is not ready:

    $ curl -L 'http://44.203.34.65:30001/metrics'{"detail":"No available replicas. Use \"sky serve status [SERVICE_NAME]\" to check the replica status."}%

You can monitor the progress of starting the actual tabby job by checking the replica log:

    # Tailing the logs of replica 1 for the tabby servicesky serve logs tabby 1

Once the service is ready, you will see something like the following:

![tabby ready](/assets/images/tabby-ready-da9c66f01e31640d3ea9925642632f94.png)

Now, you can utilize the load balancer URL (`http://44.203.34.65:30001` in this case) within Tabby editor extensions. Please refer to [`tabby.yaml`](https://github.com/TabbyML/tabby/blob/main/website/docs/installation/skypilot/tabby.yaml)
 for the full configuration used in this tutorial.

*   [Configuration](#configuration)
    
*   [Launch the service](#launch-the-service)

---

# Voyage AI | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

Voyage AI
=========

[Voyage AI](https://voyage.ai/)
 is a company that provides a range of embedding models. Tabby supports Voyage AI's models for embedding tasks.

Below is an example configuration:

~/.tabby/config.toml

    [model.embedding.http]kind = "voyage/embedding"api_key = "..."model_name = "voyage-code-2"

---

# Modal | Tabby

[Skip to main content](#__docusaurus_skipToContent_fallback)

On this page

Modal
=====

[Modal](https://modal.com/)
 is a serverless GPU provider. By leveraging Modal, your Tabby instance will run on demand. When there are no requests to the Tabby server for a certain amount of time, Modal will schedule the container to sleep, thereby saving GPU costs.

Setup[​](#setup "Direct link to Setup")

----------------------------------------

First we import the components we need from `modal`.

    from modal import Image, App, asgi_app, gpu

Next, we set the base docker image version, which model to serve, taking care to specify the GPU configuration required to fit the model into VRAM.

    IMAGE_NAME = "tabbyml/tabby"MODEL_ID = "TabbyML/StarCoder-1B"CHAT_MODEL_ID = "TabbyML/Qwen2-1.5B-Instruct"EMBEDDING_MODEL_ID = "TabbyML/Nomic-Embed-Text"GPU_CONFIG = gpu.T4()TABBY_BIN = "/opt/tabby/bin/tabby"

Currently supported GPUs in Modal:

*   `T4`: Low-cost GPU option, providing 16GiB of GPU memory.
*   `L4`: Mid-tier GPU option, providing 24GiB of GPU memory.
*   `A100`: The most powerful GPU available in the cloud. Available in 40GiB and 80GiB GPU memory configurations.
*   `H100`: The flagship data center GPU of the Hopper architecture. Enhanced support for FP8 precision and a Transformer Engine that provides up to 4X faster training over the prior generation for GPT-3 (175B) models.
*   `A10G`: A10G GPUs deliver up to 3.3x better ML training performance, 3x better ML inference performance, and 3x better graphics performance, in comparison to NVIDIA T4 GPUs.
*   `Any`: Selects any one of the GPU classes available within Modal, according to availability.

For detailed usage, please check official [Modal GPU reference](https://modal.com/docs/reference/modal.gpu)
.

Define the container image[​](#define-the-container-image "Direct link to Define the container image")

-------------------------------------------------------------------------------------------------------

We want to create a Modal image which has the Tabby model cache pre-populated. The benefit of this is that the container no longer has to re-download the model - instead, it will take advantage of Modal’s internal filesystem for faster cold starts.

### Download the weights[​](#download-the-weights "Direct link to Download the weights")

    def download_model(model_id: str):    import subprocess    subprocess.run(        [            TABBY_BIN,            "download",            "--model",            model_id,        ]    )

### Image definition[​](#image-definition "Direct link to Image definition")

We’ll start from an image by tabby, and override the default ENTRYPOINT for Modal to run its own which enables seamless serverless deployments.

Next we run the download step to pre-populate the image with our model weights.

Finally, we install the `asgi-proxy-lib` to interface with modal's asgi webserver over localhost.

    image = (    Image.from_registry(        IMAGE_NAME,        add_python="3.11",    )    .dockerfile_commands("ENTRYPOINT []")    .run_function(download_model, kwargs={"model_id": EMBEDDING_MODEL_ID})    .run_function(download_model, kwargs={"model_id": CHAT_MODEL_ID})    .run_function(download_model, kwargs={"model_id": MODEL_ID})    .pip_install("asgi-proxy-lib"))

### The app function[​](#the-app-function "Direct link to The app function")

The endpoint function is represented with Modal's `@app.function`. Here, we:

1.  Launch the Tabby process and wait for it to be ready to accept requests.
2.  Create an ASGI proxy to tunnel requests from the Modal web endpoint to the local Tabby server.
3.  Specify that each container is allowed to handle up to 10 requests simultaneously.
4.  Keep idle containers for 2 minutes before spinning them down.

    app = App("tabby-server", image=image)@app.function(    gpu=GPU_CONFIG,    allow_concurrent_inputs=10,    container_idle_timeout=120,    timeout=360,)@asgi_app()def app_serve():    import socket    import subprocess    import time    from asgi_proxy import asgi_proxy    launcher = subprocess.Popen(        [            TABBY_BIN,            "serve",            "--model",            MODEL_ID,            "--chat-model",            CHAT_MODEL_ID,            "--port",            "8000",            "--device",            "cuda",            "--parallelism",            "1",        ]    )    # Poll until webserver at 127.0.0.1:8000 accepts connections before running inputs.    def tabby_ready():        try:            socket.create_connection(("127.0.0.1", 8000), timeout=1).close()            return True        except (socket.timeout, ConnectionRefusedError):            # Check if launcher webserving process has exited.            # If so, a connection can never be made.            retcode = launcher.poll()            if retcode is not None:                raise RuntimeError(f"launcher exited unexpectedly with code {retcode}")            return False    while not tabby_ready():        time.sleep(1.0)    print("Tabby server ready!")    return asgi_proxy("http://localhost:8000")

### Serve the app[​](#serve-the-app "Direct link to Serve the app")

Once we deploy this model with `modal serve app.py`, it will output the url of the web endpoint, in a form of `https://--tabby-server-app-serve-dev.modal.run`.

If you encounter any issues, particularly related to caching, you can force a rebuild by running `MODAL_FORCE_BUILD=1 modal serve app.py`. This ensures that the latest image tag is used by ignoring cached layers.

![App Running](/assets/images/app-running-9cedeca1654871d8d125e704a9451681.png)

Now it can be used as tabby server url in tabby editor extensions! See [app.py](https://github.com/TabbyML/tabby/blob/main/website/docs/references/cloud-deployment/modal/app.py)
 for the full code used in this tutorial.

*   [Setup](#setup)
    
*   [Define the container image](#define-the-container-image)
    *   [Download the weights](#download-the-weights)
        
    *   [Image definition](#image-definition)
        
    *   [The app function](#the-app-function)
        
    *   [Serve the app](#serve-the-app)

---