# Table of Contents - [Unsloth Docs | Unsloth Documentation](#unsloth-docs-unsloth-documentation) - [初学者微调 | Unsloth Documentation](#-unsloth-documentation) - [Unsloth 文档 | Unsloth Documentation](#unsloth-unsloth-documentation) - [FAQ + 微调适合我吗？ | Unsloth Documentation](#faq-unsloth-documentation) - [Unsloth 需求 | Unsloth Documentation](#unsloth-unsloth-documentation) - [在 MacOS 上安装 Unsloth | Unsloth Documentation](#-macos-unsloth-unsloth-documentation) - [Unsloth 安装 | Unsloth Documentation](#unsloth-unsloth-documentation) - [通过 pip 和 uv 安装 Unsloth | Unsloth Documentation](#-pip-uv-unsloth-unsloth-documentation) - [更新 Unsloth | Unsloth Documentation](#-unsloth-unsloth-documentation) - [使用 Unsloth 在 AMD GPU 上微调 LLM 指南 | Unsloth Documentation](#-unsloth-amd-gpu-llm-unsloth-documentation) - [通过 Docker 安装 Unsloth | Unsloth Documentation](#-docker-unsloth-unsloth-documentation) - [我应该用什么模型来微调？ | Unsloth Documentation](#-unsloth-documentation) - [如何使用 Unsloth 在 Windows 上微调 LLM（分步指南） | Unsloth Documentation](#-unsloth-windows-llm-unsloth-documentation) - [Unsloth 笔记本 | Unsloth Documentation](#unsloth-unsloth-documentation) - [RL 奖励作弊 | Unsloth Documentation](#rl-unsloth-documentation) - [GSPO 强化学习 | Unsloth Documentation](#gspo-unsloth-documentation) - [RL 中的 FP16 与 BF16 | Unsloth Documentation](#rl-fp16-bf16-unsloth-documentation) - [使用 Unsloth Studio 导出模型 | Unsloth Documentation](#-unsloth-studio-unsloth-documentation) - [使用 Unsloth 在 Intel GPU 上微调 LLM | Unsloth Documentation](#-unsloth-intel-gpu-llm-unsloth-documentation) - [视觉强化学习（VLM RL） | Unsloth Documentation](#-vlm-rl-unsloth-documentation) - [如何使用 Unsloth Studio 运行模型 | Unsloth Documentation](#-unsloth-studio-unsloth-documentation) - [偏好优化训练 - DPO、ORPO 和 KTO | Unsloth Documentation](#-dpo-orpo-kto-unsloth-documentation) - [具有 7 倍更长上下文的强化学习 GRPO | Unsloth Documentation](#-7-grpo-unsloth-documentation) - [使用 Blackwell、RTX 50 系列和 Unsloth 微调 LLM | Unsloth Documentation](#-blackwell-rtx-50-unsloth-llm-unsloth-documentation) - [使用 NVIDIA DGX Spark 和 Unsloth 微调 LLM | Unsloth Documentation](#-nvidia-dgx-spark-unsloth-llm-unsloth-documentation) - [Blackwell、RTX 50 シリーズと Unsloth を使った LLM のファインチューニング | Unsloth Documentation](#blackwell-rtx-50-unsloth-llm-unsloth-documentation) - [Conda でインストール | Unsloth Documentation](#conda-unsloth-documentation) - [Unsloth と Colab GPU を使って VS Code で LLM をファインチューニングする方法 | Unsloth Documentation](#unsloth-colab-gpu-vs-code-llm-unsloth-documentation) - [Unsloth 推論 | Unsloth Documentation](#unsloth-unsloth-documentation) - [Google Colab | Unsloth Documentation](#google-colab-unsloth-documentation) - [使用 Hugging Face Jobs 部署 LLM | Unsloth Documentation](#-hugging-face-jobs-llm-unsloth-documentation) - [Text-to-Speech (TTS) Fine-tuning Guide | Unsloth Documentation](#text-to-speech-tts-fine-tuning-guide-unsloth-documentation) - [Fine-tuning Embedding Models with Unsloth Guide | Unsloth Documentation](#fine-tuning-embedding-models-with-unsloth-guide-unsloth-documentation) - [Vision Fine-tuning | Unsloth Documentation](#vision-fine-tuning-unsloth-documentation) - [Fine-tuning for Beginners | Unsloth Documentation](#fine-tuning-for-beginners-unsloth-documentation) - [Unsloth Notebooks | Unsloth Documentation](#unsloth-notebooks-unsloth-documentation) - [Get started with Unsloth Studio | Unsloth Documentation](#get-started-with-unsloth-studio-unsloth-documentation) - [Unsloth Updates | Unsloth Documentation](#unsloth-updates-unsloth-documentation) - [Fine-tuning LLMs Guide | Unsloth Documentation](#fine-tuning-llms-guide-unsloth-documentation) - [Reinforcement Learning (RL) Guide | Unsloth Documentation](#reinforcement-learning-rl-guide-unsloth-documentation) - [Unsloth Installation | Unsloth Documentation](#unsloth-installation-unsloth-documentation) - [Unsloth Environment Flags | Unsloth Documentation](#unsloth-environment-flags-unsloth-documentation) - [Continued Pretraining | Unsloth Documentation](#continued-pretraining-unsloth-documentation) - [Multi-GPU Fine-tuning with Unsloth | Unsloth Documentation](#multi-gpu-fine-tuning-with-unsloth-unsloth-documentation) - [Finetuning from Last Checkpoint | Unsloth Documentation](#finetuning-from-last-checkpoint-unsloth-documentation) - [Unsloth Benchmarks | Unsloth Documentation](#unsloth-benchmarks-unsloth-documentation) - [Quantization-Aware Training (QAT) | Unsloth Documentation](#quantization-aware-training-qat-unsloth-documentation) - [Fine-tuning LLMs with NVIDIA DGX Spark and Unsloth | Unsloth Documentation](#fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth-unsloth-documentation) - [Fine-Tuning LLMs on NVIDIA DGX Station with Unsloth | Unsloth Documentation](#fine-tuning-llms-on-nvidia-dgx-station-with-unsloth-unsloth-documentation) - [Multi-GPU Fine-tuning with Distributed Data Parallel (DDP) | Unsloth Documentation](#multi-gpu-fine-tuning-with-distributed-data-parallel-ddp-unsloth-documentation) - [Fine-tuning LLMs with Blackwell, RTX 50 series & Unsloth | Unsloth Documentation](#fine-tuning-llms-with-blackwell-rtx-50-series-unsloth-unsloth-documentation) - [Unsloth Model Catalog | Unsloth Documentation](#unsloth-model-catalog-unsloth-documentation) - [How to Fine-tune LLMs with Unsloth & Docker | Unsloth Documentation](#how-to-fine-tune-llms-with-unsloth-docker-unsloth-documentation) - [Chat Templates | Unsloth Documentation](#chat-templates-unsloth-documentation) - [Export models with Unsloth Studio | Unsloth Documentation](#export-models-with-unsloth-studio-unsloth-documentation) - [How to Run Local LLMs with OpenAI Codex | Unsloth Documentation](#how-to-run-local-llms-with-openai-codex-unsloth-documentation) - [500K Context Length Fine-tuning | Unsloth Documentation](#500k-context-length-fine-tuning-unsloth-documentation) - [MiniMax-M2.7 - How to Run Locally | Unsloth Documentation](#minimax-m2-7-how-to-run-locally-unsloth-documentation) --- # Unsloth Docs | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth lets you run and train AI models on your own local hardware. Our docs will guide you through running & training your own model locally. [Get started](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners) [Our GitHub](https://github.com/unslothai/unsloth) [](https://unsloth.ai/docs/models/gemma-4) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkEjWOJqBWCtIN9Cg6CdI%252FGemma%25204%2520landscape.png%3Falt%3Dmedia%26token%3D57d3f596-dae8-4eab-80e6-0847794ffc8d&width=752&dpr=3&quality=100&sign=47cf35e4&sv=2) #### [hashtag](https://unsloth.ai/docs#google-gemma-4) Google Gemma 4 Run and train Google's new Gemma 4 models! [](https://unsloth.ai/docs/new/studio) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FstfdTMsoBMmsbQsgQ1Ma%252Flandscape%2520clip%2520gemma.gif%3Falt%3Dmedia%26token%3Deec5f2f7-b97a-4c1c-ad01-5a041c3e4013&width=752&dpr=3&quality=100&sign=e4b21b2d&sv=2) #### [hashtag](https://unsloth.ai/docs#introducing-unsloth-studio) **Introducing Unsloth Studio** A new open, no-code web UI to train and run LLMs. [](https://unsloth.ai/docs/models/qwen3.5) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fvw6yRxJDCeBl1CIsQkki%252Fqwen35.png%3Falt%3Dmedia%26token%3D28fe0357-351a-49e1-a176-bb21ecc8542a&width=490&dpr=3&quality=100&sign=70c1769a&sv=2) **Qwen3.5** New Qwen3.5 Small & Medium LLMs are here! [](https://unsloth.ai/docs/models/glm-5.1) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FK69rPUGatLzCBK9uaqxU%252Fglm51%2520logo.png%3Falt%3Dmedia%26token%3D934ef701-0233-47fd-ad49-6c1a5959b684&width=490&dpr=3&quality=100&sign=3113b30a&sv=2) **GLM-5.1** Run the new SOTA open model locally. [](https://unsloth.ai/docs/models/minimax-m27) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FoWNqL9cQDvNtxEcwGfLY%252Fm27.png%3Falt%3Dmedia%26token%3Daed5db72-4961-4089-b93a-01d9c4d19f0c&width=490&dpr=3&quality=100&sign=69cfbd7a&sv=2) **MiniMax-M2.7** Run the new 230B sized model. [](https://unsloth.ai/docs/models/nemotron-3) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FllPS7l6rpEr68mytlxXU%252Fnemotron%25203%2520logo.png%3Falt%3Dmedia%26token%3D7bd05673-6b97-41c2-b657-530b7e6e4e3c&width=490&dpr=3&quality=100&sign=a25c871f&sv=2) **NVIDIA Nemotron 3** Run the new 4B and 120B models by NVIDIA. [](https://unsloth.ai/docs/basics/faster-moe) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh9BrTJR8CZghHOe1Yrgj%252Ffaster%2520moe%25201920.png%3Falt%3Dmedia%26token%3D404e70ea-6aa1-4af0-a01c-7490d8147c4e&width=490&dpr=3&quality=100&sign=5d2e21b8&sv=2) **Faster MoE is here!** Train MoE LLMs 12x faster with less VRAM. [](https://unsloth.ai/docs/basics/claude-code) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FM3el6W6XCMc0iBEgdeov%252Fclaude%2520code%2520codex.png%3Falt%3Dmedia%26token%3De45dbc05-9af6-40f7-bcf8-59b79ac44909&width=490&dpr=3&quality=100&sign=a52cee87&sv=2) **Claude Code & Codex** Learn to run local LLMs via Claude & OpenAI. [🧬Fine-tuning Guidechevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) [📒Unsloth Notebookschevron-right](https://unsloth.ai/docs/get-started/unsloth-notebooks) [🔮All Our Modelschevron-right](https://unsloth.ai/docs/get-started/unsloth-model-catalog) [🚀Complete LLM Directorychevron-right](https://unsloth.ai/docs/models/tutorials) ### [hashtag](https://unsloth.ai/docs#why-unsloth) 🦥 Why Unsloth? * We directly collab with teams behind [gpt-ossarrow-up-right](https://docs.unsloth.ai/new/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss) , [Qwen3arrow-up-right](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/) , [Llama 4arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/12889) , [Mistral](https://unsloth.ai/docs/models/tutorials/devstral-how-to-run-and-fine-tune) , [Gemma 1-3arrow-up-right](https://news.ycombinator.com/item?id=39671146) and [Phi-4arrow-up-right](https://unsloth.ai/blog/phi4) , where we’ve **fixed critical bugs** that greatly improved model accuracy. Andrej Karpathy for example has [praised our workarrow-up-right](https://x.com/karpathy/status/1765473722985771335) . * Unsloth streamlines local training, inference, data, and deployment * Unsloth supports inference and training for 500+ models: [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) , [TTS](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [embedding](https://unsloth.ai/docs/basics/embedding-finetuning) , [RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) ### [hashtag](https://unsloth.ai/docs#features) ⭐ Features Unsloth lets you run and train models for text, [audioarrow-up-right](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [embeddingarrow-up-right](https://unsloth.ai/docs/new/embedding-finetuning) , [visionarrow-up-right](https://unsloth.ai/docs/basics/vision-fine-tuning) and more. Unsloth provides many key features for both inference and training: #### [hashtag](https://unsloth.ai/docs#inference) Inference * Search + download + run any model like GGUFs, LoRA adapters, safetensors. * [Self-healing tool calling](https://unsloth.ai/docs/new/studio/chat#auto-healing-tool-calling) / web search and call OpenAI-compatible APIs. * [Auto inference parameter](https://unsloth.ai/docs/new/studio/chat#auto-parameter-tuning) tuning and edit chat templates. * [Export or save](https://unsloth.ai/docs/new/studio/export) your model to GGUF, 16-bit safetensor etc. * [Compare outputs](https://unsloth.ai/docs/new/studio/chat#model-arena) with two different model side by side. #### [hashtag](https://unsloth.ai/docs#training) Training * Train and [RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) 500+ models ~2x faster with ~70% less VRAM (no accuracy loss) * Supports full fine-tuning, pre-training, 4-bit, 16-bit and FP8 training. * [Auto-create datasets](https://unsloth.ai/docs/new/studio/data-recipe) from PDF, CSV, DOCX files. Edit data in a visual node workflow. * Observability: Monitor training live, track loss, GPU usage, customize graphs * Most efficient [**reinforcement learning**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) library, using 80% less VRAM for GRPO, [FP8](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) etc. * [Multi-GPU](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) works but a much better version is coming! ### [hashtag](https://unsloth.ai/docs#quickstart) Quickstart Unsloth supports MacOS, Linux, [Windows](https://unsloth.ai/docs/get-started/install/windows-installation) , [NVIDIA](https://unsloth.ai/docs/get-started/install/pip-install) , Intel and CPU setups. See: [Unsloth Requirements](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) . Use the same commands to update: #### [hashtag](https://unsloth.ai/docs#macos-linux-wsl) **MacOS, Linux, WSL:** #### [hashtag](https://unsloth.ai/docs#windows-powershell) **Windows PowerShell:** #### [hashtag](https://unsloth.ai/docs#docker) Docker Use our official **Docker image**: [`unsloth/unsloth`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) which currently works for Windows, WSL and Linux. MacOS support coming soon. #### [hashtag](https://unsloth.ai/docs#launch-unsloth) Launch Unsloth #### [hashtag](https://unsloth.ai/docs#new-models) New Models [](https://unsloth.ai/docs/models/kimi-k2.5) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FgcSsB0cPhjj8inDt1bqf%252Fkimi%2520k25%2520logo.png%3Falt%3Dmedia%26token%3D19aec00a-7e0f-4980-b2b7-98b65a23123e&width=490&dpr=3&quality=100&sign=b4c082eb&sv=2) **Kimi K2.5** [](https://unsloth.ai/docs/models/tutorials/minimax-m25) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F0yrdjCKbV8qnqyTrQ1pZ%252Fminimax2.5%2520logo.png%3Falt%3Dmedia%26token%3D183839fe-6750-4c95-b058-c991ec8a5dec&width=490&dpr=3&quality=100&sign=3d5749a0&sv=2) **MiniMax-M2.5** [](https://unsloth.ai/docs/models/glm-4.7-flash) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6PQZ23CoUdZs1EZCjtYn%252Fglm4.7flash.png%3Falt%3Dmedia%26token%3Dd3dc776e-ef3e-4eb3-ad4e-bf45e7b5745a&width=490&dpr=3&quality=100&sign=c13d9e53&sv=2) **GLM-4.7-Flash** ### [hashtag](https://unsloth.ai/docs#what-is-fine-tuning-and-rl-why) What is Fine-tuning and RL? Why? [**Fine-tuning** an LLM](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) customizes its behavior, enhances domain knowledge, and optimizes performance for specific tasks. By fine-tuning a pre-trained model (e.g. Llama-3.1-8B) on a dataset, you can: * **Update Knowledge**: Introduce new domain-specific information. * **Customize Behavior**: Adjust the model’s tone, personality, or response style. * **Optimize for Tasks**: Improve accuracy and relevance for specific use cases. [**Reinforcement Learning (RL)**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) is where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**. * **Action:** What the model generates (e.g. a sentence). * **Reward:** A signal indicating how good or bad the model's action was (e.g. did the response follow instructions? was it helpful?). * **Environment:** The scenario or task the model is working on (e.g. answering a user’s question). **Example fine-tuning or RL use-cases**: * Enables LLMs to predict if a headline impacts a company positively or negatively. * Can use historical customer interactions for more accurate and custom responses. * Fine-tune LLM on legal texts for contract analysis, case law research, and compliance. You can think of a fine-tuned model as a specialized agent designed to do specific tasks more effectively and efficiently. **Fine-tuning can replicate all of RAG's capabilities**, but not vice versa. [🤔FAQ + Is Fine-tuning Right For Me?chevron-right](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) [🖥️Inference & Deploymentchevron-right](https://unsloth.ai/docs/basics/inference-and-deployment) [💡Reinforcement Learning Guidechevron-right](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) [🦥Dynamic 2.0 GGUFschevron-right](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-134302f2507d4313b9575917c9a43b0a0028856c%252Flarge%2520sloth%2520wave.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=4e1b22de&sv=2) [NextBeginner? Start here!chevron-right](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners) Last updated 1 day ago Was this helpful? * [🦥 Why Unsloth?](https://unsloth.ai/docs#why-unsloth) * [⭐ Features](https://unsloth.ai/docs#features) * [Quickstart](https://unsloth.ai/docs#quickstart) * [What is Fine-tuning and RL? Why?](https://unsloth.ai/docs#what-is-fine-tuning-and-rl-why) Was this helpful? sun-brightdesktopmoon Copy curl -fsSL https://unsloth.ai/install.sh | sh Copy irm https://unsloth.ai/install.ps1 | iex Copy unsloth studio -H 0.0.0.0 -p 8888 sun-brightdesktopmoon --- # 初学者微调 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 如果你是初学者，这里可能是你在第一次微调之前会问的第一个问题。你也可以通过加入我们的社区随时询问我们的成员， [Reddit 页面arrow-up-right](https://www.reddit.com/r/unsloth/) . [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 🧬[Fine-tuning Guide](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 逐步了解如何进行微调！ 学习训练的核心基础知识。 [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use) ❓[What Model Should I Use?](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use) 是使用指令型模型还是基础模型？ 我的数据集应该有多大？ [](https://unsloth.ai/docs/zh/mo-xing/tutorials) 🚀[Complete LLM Directory](https://unsloth.ai/docs/zh/mo-xing/tutorials) 如何运行并微调 DeepSeek？ 运行 Gemma 3 时我应该设置哪些参数？ [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) 🤔[FAQ + 微调适合我吗？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) 微调能为我做些什么？ 检索增强生成（RAG）与微调？ [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install) 📥[Installation](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install) 如何在本地安装 Unsloth？ 如何更新 Unsloth？ 📈[数据集指南](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/datasets-guide) 如何构建/准备我的数据集？ 我如何收集数据？ [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) 🛠️[Unsloth 需求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) Unsloth 在我的 GPU 上能运行吗？ 我需要多少显存（VRAM）？ [](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment) 🖥️[推理与部署](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment) 如何将我的模型保存到本地？ 如何通过 Ollama 或 vLLM 运行我的模型？ 🧠[Hyperparameters Guide](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/lora-hyperparameters-guide) 当我更改参数时会发生什么？ 我应该更改哪些参数？ ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-559e7890f607e34fd6004517296e65e942c93b41%252FLarge%2520sloth%2520Question%2520mark.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=cd5e8928&sv=2) [上一页Homepagechevron-left](https://unsloth.ai/docs/zh) [下一页Unsloth 需求chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) 最后更新于 3个月前 这有帮助吗？ 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth 文档 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 让你可以在自己的本地硬件上运行和训练 AI 模型。 我们的文档将指导你在本地运行并训练自己的模型。 [开始使用](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners) [我们的 GitHub](https://github.com/unslothai/unsloth) [](https://unsloth.ai/docs/zh/mo-xing/gemma-4) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkEjWOJqBWCtIN9Cg6CdI%252FGemma%25204%2520landscape.png%3Falt%3Dmedia%26token%3D57d3f596-dae8-4eab-80e6-0847794ffc8d&width=752&dpr=3&quality=100&sign=72ae85de&sv=2) #### [hashtag](https://unsloth.ai/docs/zh#google-gemma-4) Google Gemma 4 运行并训练 Google 的全新 Gemma 4 模型！ [](https://unsloth.ai/docs/zh/xin/studio) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FstfdTMsoBMmsbQsgQ1Ma%252Flandscape%2520clip%2520gemma.gif%3Falt%3Dmedia%26token%3Deec5f2f7-b97a-4c1c-ad01-5a041c3e4013&width=752&dpr=3&quality=100&sign=fd891523&sv=2) #### [hashtag](https://unsloth.ai/docs/zh#jie-shao-unsloth-studio) **介绍 Unsloth Studio** 一个全新的、无需代码的网页 UI，用于训练和运行 LLM。 [](https://unsloth.ai/docs/zh/mo-xing/qwen3.5) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fvw6yRxJDCeBl1CIsQkki%252Fqwen35.png%3Falt%3Dmedia%26token%3D28fe0357-351a-49e1-a176-bb21ecc8542a&width=490&dpr=3&quality=100&sign=9099ca1c&sv=2) **Qwen3.5** 全新的 Qwen3.5 小型和中型 LLM 来了！ [](https://unsloth.ai/docs/zh/mo-xing/glm-5.1) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FK69rPUGatLzCBK9uaqxU%252Fglm51%2520logo.png%3Falt%3Dmedia%26token%3D934ef701-0233-47fd-ad49-6c1a5959b684&width=490&dpr=3&quality=100&sign=a131dc3c&sv=2) **GLM-5.1** 在本地运行全新的 SOTA 开源模型。 [](https://unsloth.ai/docs/zh/mo-xing/minimax-m27) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FoWNqL9cQDvNtxEcwGfLY%252Fm27.png%3Falt%3Dmedia%26token%3Daed5db72-4961-4089-b93a-01d9c4d19f0c&width=490&dpr=3&quality=100&sign=7ac329f0&sv=2) **MiniMax-M2.7** 运行全新的 230B 规模模型。 [](https://unsloth.ai/docs/zh/mo-xing/nemotron-3) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FllPS7l6rpEr68mytlxXU%252Fnemotron%25203%2520logo.png%3Falt%3Dmedia%26token%3D7bd05673-6b97-41c2-b657-530b7e6e4e3c&width=490&dpr=3&quality=100&sign=535b6a61&sv=2) **NVIDIA Nemotron 3** 运行 NVIDIA 全新的 4B 和 120B 模型。 [](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/faster-moe) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh9BrTJR8CZghHOe1Yrgj%252Ffaster%2520moe%25201920.png%3Falt%3Dmedia%26token%3D404e70ea-6aa1-4af0-a01c-7490d8147c4e&width=490&dpr=3&quality=100&sign=ffaf7096&sv=2) **更快的 MoE 来了！** 以更少的 VRAM 将 MoE LLM 的训练速度提升 12 倍。 [](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/claude-code) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FM3el6W6XCMc0iBEgdeov%252Fclaude%2520code%2520codex.png%3Falt%3Dmedia%26token%3De45dbc05-9af6-40f7-bcf8-59b79ac44909&width=490&dpr=3&quality=100&sign=7a11e23d&sv=2) **Claude Code 与 Codex** 学习通过 Claude 和 OpenAI 运行本地 LLM。 [🧬Fine-tuning Guidechevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) [📒Unsloth 笔记本chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) [🔮All Our Modelschevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-model-catalog) [🚀Complete LLM Directorychevron-right](https://unsloth.ai/docs/zh/mo-xing/tutorials) ### [hashtag](https://unsloth.ai/docs/zh#wei-shen-me-xuan-ze-unsloth) 🦥 为什么选择 Unsloth？ * 我们直接与以下团队合作 [gpt-ossarrow-up-right](https://docs.unsloth.ai/new/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss) , [Qwen3arrow-up-right](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/) , [Llama 4arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/12889) , [Mistral](https://unsloth.ai/docs/zh/mo-xing/tutorials/devstral-how-to-run-and-fine-tune) , [Gemma 1-3arrow-up-right](https://news.ycombinator.com/item?id=39671146) 和 [Phi-4arrow-up-right](https://unsloth.ai/blog/phi4) ，我们已经 **修复了关键 bug** ，这些修复极大提升了模型准确率。例如 Andrej Karpathy 就 [称赞过我们的工作arrow-up-right](https://x.com/karpathy/status/1765473722985771335) . * Unsloth 简化了本地训练、推理、数据处理和部署 * Unsloth 支持 500+ 个模型的推理和训练： [视觉](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/vision-fine-tuning) , [TTS](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/text-to-speech-tts-fine-tuning) , [嵌入](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/embedding-finetuning) , [RL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) ### [hashtag](https://unsloth.ai/docs/zh#gong-neng) ⭐ 功能 Unsloth 让你可以运行和训练文本模型， [音频arrow-up-right](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [嵌入arrow-up-right](https://unsloth.ai/docs/new/embedding-finetuning) , [视觉arrow-up-right](https://unsloth.ai/docs/basics/vision-fine-tuning) 以及更多类型的模型。Unsloth 为推理和训练提供了许多关键功能： #### [hashtag](https://unsloth.ai/docs/zh#tui-li) 推理 * 搜索 + 下载 + 运行任何模型，例如 GGUF、LoRA 适配器、safetensors。 * [自我修复式工具调用](https://unsloth.ai/docs/zh/xin/studio/chat#auto-healing-tool-calling) / 网页搜索并调用兼容 OpenAI 的 API。 * [自动推理参数](https://unsloth.ai/docs/zh/xin/studio/chat#auto-parameter-tuning) 调优并编辑聊天模板。 * [导出或保存](https://unsloth.ai/docs/zh/xin/studio/export) 你的模型为 GGUF、16 位 safetensor 等格式。 * [对比输出](https://unsloth.ai/docs/zh/xin/studio/chat#model-arena) 并排比较两个不同模型的输出。 #### [hashtag](https://unsloth.ai/docs/zh#xun-lian) 训练 * 训练并 [RL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 让 500+ 个模型的速度提升约 2 倍，VRAM 占用减少约 70%（且不损失准确率） * 支持全参数微调、预训练、4 位、16 位和 FP8 训练。 * [自动创建数据集](https://unsloth.ai/docs/zh/xin/studio/data-recipe) 从 PDF、CSV、DOCX 文件生成。在可视化节点工作流中编辑数据。 * 可观测性：实时监控训练，跟踪损失、GPU 使用情况，自定义图表 * 最高效的 [**强化学习**](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 库，GRPO 的 VRAM 占用减少 80%， [FP8](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/fp8-reinforcement-learning) 等。 * [多 GPU](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/multi-gpu-training-with-unsloth) 已可使用，但更好的版本即将到来！ ### [hashtag](https://unsloth.ai/docs/zh#kuai-su-kai-shi) 快速开始 Unsloth 支持 MacOS、Linux、 [Windows](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation) , [NVIDIA](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install) 、Intel 和 CPU 配置。请查看： [Unsloth 需求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) 。使用相同命令进行更新： #### [hashtag](https://unsloth.ai/docs/zh#macos-linux-wsl) **MacOS、Linux、WSL：** #### [hashtag](https://unsloth.ai/docs/zh#windows-powershell) **Windows PowerShell：** #### [hashtag](https://unsloth.ai/docs/zh#docker) Docker 使用我们的官方 **Docker 镜像**: [`unsloth/unsloth`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) ，目前可用于 Windows、WSL 和 Linux。MacOS 支持即将推出。 #### [hashtag](https://unsloth.ai/docs/zh#qi-dong-unsloth) 启动 Unsloth #### [hashtag](https://unsloth.ai/docs/zh#xin-mo-xing) 新模型 [](https://unsloth.ai/docs/zh/mo-xing/kimi-k2.5) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FgcSsB0cPhjj8inDt1bqf%252Fkimi%2520k25%2520logo.png%3Falt%3Dmedia%26token%3D19aec00a-7e0f-4980-b2b7-98b65a23123e&width=490&dpr=3&quality=100&sign=76c90735&sv=2) **Kimi K2.5** [](https://unsloth.ai/docs/zh/mo-xing/tutorials/minimax-m25) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F0yrdjCKbV8qnqyTrQ1pZ%252Fminimax2.5%2520logo.png%3Falt%3Dmedia%26token%3D183839fe-6750-4c95-b058-c991ec8a5dec&width=490&dpr=3&quality=100&sign=2ed083da&sv=2) **MiniMax-M2.5** [](https://unsloth.ai/docs/zh/mo-xing/glm-4.7-flash) ![Cover](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6PQZ23CoUdZs1EZCjtYn%252Fglm4.7flash.png%3Falt%3Dmedia%26token%3Dd3dc776e-ef3e-4eb3-ad4e-bf45e7b5745a&width=490&dpr=3&quality=100&sign=11fce541&sv=2) **GLM-4.7-Flash** ### [hashtag](https://unsloth.ai/docs/zh#shen-me-shi-wei-tiao-he-rl-wei-shen-me-yao-yong) 什么是微调和 RL？为什么要用？ [**微调** 一个 LLM](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 可以定制其行为、增强领域知识，并针对特定任务优化性能。通过在数据集上微调一个预训练模型（例如 Llama-3.1-8B），你可以： * **更新知识**：引入新的领域特定信息。 * **定制行为**：调整模型的语气、个性或回复风格。 * **针对任务优化**：提升特定用例的准确性和相关性。 [**强化学习（RL）**](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 是指一个“智能体”通过与环境交互并接收 **反馈** ，以 **奖励** 或 **惩罚**. * **的形式** 动作： * **奖励：** 表示模型动作好坏的信号（例如：回复是否遵循指令？是否有帮助？）。 * **环境：** 模型正在处理的场景或任务（例如：回答用户的问题）。 **微调或 RL 的示例用例**: * 使 LLM 能够预测标题是否会对公司产生正面或负面影响。 * 可以利用历史客户互动来生成更准确、更个性化的回复。 * 在法律文本上微调 LLM，用于合同分析、案例法研究和合规。 你可以把微调后的模型看作一个专门设计的智能体，用于更有效、更高效地完成特定任务。 **微调可以复制 RAG 的所有能力**，但反过来则不行。 [🤔FAQ + 微调适合我吗？chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) [🖥️推理与部署chevron-right](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment) [💡Reinforcement Learning Guidechevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) [🦥Dynamic 2.0 GGUFschevron-right](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/unsloth-dynamic-2.0-ggufs) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-134302f2507d4313b9575917c9a43b0a0028856c%252Flarge%2520sloth%2520wave.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c5a08d8&sv=2) [下一页Beginner? Start here!chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners) 最后更新于 1天前 这有帮助吗？ * [🦥 为什么选择 Unsloth？](https://unsloth.ai/docs/zh#wei-shen-me-xuan-ze-unsloth) * [⭐ 功能](https://unsloth.ai/docs/zh#gong-neng) * [快速开始](https://unsloth.ai/docs/zh#kuai-su-kai-shi) * [什么是微调和 RL？为什么要用？](https://unsloth.ai/docs/zh#shen-me-shi-wei-tiao-he-rl-wei-shen-me-yao-yong) 这有帮助吗？ sun-brightdesktopmoon 复制 curl -fsSL https://unsloth.ai/install.sh | sh 复制 irm https://unsloth.ai/install.ps1 | iex 复制 unsloth studio -H 0.0.0.0 -p 8888 sun-brightdesktopmoon --- # FAQ + 微调适合我吗？ | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#li-jie-wei-tiao) 理解微调 -------------------------------------------------------------------------------------------------------------------------------------------- 微调大型语言模型（LLM）可以定制其行为、加深其领域专业知识，并针对特定任务优化其性能。通过使用专门数据对预训练模型（例如： _Llama-3.1-8B_）进行精炼，您可以： * **更新知识** – 引入基础模型最初未包含的新领域特定信息。 * **定制行为** – 调整模型的语气、个性或响应风格以符合特定需求或品牌语言。 * **为任务优化** – 提高在您的用例所需的特定任务或查询上的准确性和相关性。 把微调想象成将一个通用模型打造为专门的专家。有人争论是否应使用检索增强生成（RAG）替代微调，但微调可以以 RAG 无法实现的方式将知识和行为直接内嵌到模型中。实际上，结合这两种方法通常能获得最佳效果——带来更高的准确性、更好的可用性和更少的幻觉。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-de-shi-ji-ying-yong) 微调的实际应用 微调可以应用于各种领域和需求。下面是一些它能产生明显差异的实际示例： * **金融情感分析** – 训练 LLM 判断新闻标题对某公司的影响是正面还是负面，并将其理解调整为金融语境。 * **客户支持聊天机器人** – 在过去的客户互动数据上进行微调，以便以公司的风格和术语提供更准确、更个性化的回复。 * **法律文档辅助** – 在法律文本（合同、判例法、法规）上进行微调，用于合同分析、判例研究或合规支持等任务，确保模型使用精确的法律用语。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-de-hao-chu) 微调的好处 ------------------------------------------------------------------------------------------------------------------------------------------------- 微调提供了若干显著优势，这是基础模型或纯检索系统无法完全提供的： #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-vs.-rag-you-shen-me-bu-tong) 微调 vs. RAG：有什么不同？ 微调几乎可以做 RAG 能做的大多数事情——但反过来则不然。在训练过程中，微调会将外部知识直接嵌入模型中。这使得模型能够处理小众查询、总结文档并维持上下文，而无需依赖外部检索系统。这并不意味着 RAG 没有优势——RAG 在访问来自外部数据库的最新信息方面表现出色。事实上，通过微调也可以获取新鲜数据，但为了高效起见，最好将 RAG 与微调结合使用。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#te-ding-ren-wu-de-jing-tong) 特定任务的精通 微调将领域知识深度整合到模型中。这使得模型在处理结构化、重复或微妙的查询时非常有效，而这些场景通常是仅有 RAG 的系统难以胜任的。换句话说，经过微调的模型会成为其训练任务或内容方面的专家。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#du-li-yu-jian-suo) 独立于检索 微调后的模型在推理时不依赖外部数据源。即使连接的检索系统失败或不完整，它仍然可靠，因为所有所需信息已包含在模型自身参数中。这种自给自足意味着生产环境中的故障点更少。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#geng-kuai-de-xiang-ying) 更快的响应 微调模型在生成时无需调用外部知识库。跳过检索步骤意味着它们可以更快地产生答案。这种速度使微调模型非常适合对时间敏感的应用场景。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#ding-zhi-xing-wei-yu-yu-qi) 定制行为与语气 微调允许对模型的沟通方式进行精确控制。这能确保模型的回应与品牌语调一致、符合监管要求或匹配特定的语气偏好。您将得到一个不仅知道 _说_ 什么 _如何_ 以期望的风格表达它。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#ke-kao-de-xing-neng) 可靠的性能 即使在同时使用微调与 RAG 的混合设置中，微调模型也提供了可靠的后备。如果检索组件未能找到正确的信息或返回不准确的数据，模型内置的知识仍然可以生成有用的答案。这为系统保证了更一致和更稳健的性能。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#chang-jian-wu-jie) 常见误解 ---------------------------------------------------------------------------------------------------------------------------------------------- 尽管微调有很多优点，但仍存在一些流传的误解。下面来澄清两个关于微调的常见误区： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-hui-wei-mo-xing-tian-jia-xin-zhi-shi-ma) 微调会为模型添加新知识吗？ **会——绝对会。** 一个常见误解认为微调不会引入新知识，但实际上它会。如果您的微调数据集包含新的领域特定信息，模型将在训练期间学习并将其纳入回答。实际上，微调 _可以并且确实会_ 从头教授模型新的事实和模式。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#rag-zong-shi-bi-wei-tiao-geng-hao-ma) RAG 总是比微调更好吗？ **不一定。** 许多人认为 RAG 始终会优于微调模型，但当微调做得恰当时情况并非如此。事实上，良好微调的模型在专门任务上往往能匹敌甚至超越基于 RAG 的系统。所谓“RAG 总是更好”的说法通常源于微调配置不当的尝试——例如，使用不正确的 [LoRA 参数](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/lora-hyperparameters-guide) 或训练不足。 Unsloth 会处理这些复杂性，自动为您选择最佳参数配置。您只需准备高质量的数据集，就能获得性能发挥到极致的微调模型。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-ang-gui-ma) 微调昂贵吗？ **一点也不！** 虽然完全微调或预训练可能代价高昂，但这些通常不是必须的（尤其不需要预训练）。在大多数情况下，LoRA 或 QLoRA 微调即可以极低成本完成。事实上，借助 Unsloth 提供的 [免费笔记本arrow-up-right](https://docs.unsloth.ai/get-started/unsloth-notebooks) （用于 Colab 或 Kaggle），您可以零费用进行微调。更好的是，您甚至可以在本地设备上进行微调。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#chang-jian-wen-ti) 常见问题： ----------------------------------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-shen-me-yao-jie-he-rag-yu-wei-tiao) 为什么要结合 RAG 与微调 与其在 RAG 与微调之间二选一，不如考虑同时使用 **两者** 以获得最佳效果。将检索系统与微调模型结合可以发挥各自的优势。原因如下： * **特定任务的专业性** – 微调擅长专门任务或特定格式（使模型在某一领域成为专家），而 RAG 则使模型能够获取最新的外部知识。 * **更好的适应性** – 即便检索组件失败或返回不完整信息，微调模型仍能提供有用的回答。同时，RAG 确保系统保持最新，而无需您为每一条新数据重新训练模型。 * **效率** – 微调在模型内部提供了坚实的基础知识库，而 RAG 处理动态或快速变化的细节，从而无需从头进行耗时的再训练。此种平衡带来高效的工作流程并降低整体计算成本。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#lora-vs.-qlora-gai-xuan-na-ge) LoRA vs. QLoRA：该选哪个？ 在实施微调时，有两种流行技术可以显著降低计算和内存需求： **LoRA** 和 **QLoRA**。以下是两者的简要比较： * **LoRA（低秩适配）** – 仅微调一小组额外的“适配器”权重矩阵（以 16 位精度），同时保持原始模型的大部分参数不变。这显著减少了训练期间需要更新的参数数量。 * **QLoRA（量化 LoRA）** – 将 LoRA 与模型权重的 4 位量化相结合，使在极少硬件资源上对非常大型模型进行高效微调成为可能。通过在可行位置使用 4 位精度，它大幅降低了内存使用和计算开销。 我们建议从 **QLoRA**开始，因为它是最有效且最易接近的方法之一。多亏了 Unsloth 的 [动态 4 位arrow-up-right](https://unsloth.ai/blog/dynamic-4bit) 量化，与标准 16 位 LoRA 微调相比，现在精度损失已经可以忽略不计。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#shi-yan-wei-guan-jian) 实验为关键 没有单一的“最佳”微调方法——只有针对不同场景的最佳实践。重要的是对不同方法和配置进行试验，以找到最适合您数据集和用例的方案。一个很好的起点是 **QLoRA（4 位）**，它提供了一种非常具有成本效益且资源友好的微调方式，无需大量计算资源。 [🧠Hyperparameters Guidechevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/lora-hyperparameters-guide) [上一页Unsloth 需求chevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) [下一页Unsloth 笔记本chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) 最后更新于 3个月前 这有帮助吗？ * [理解微调](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#li-jie-wei-tiao) * [微调的实际应用](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-de-shi-ji-ying-yong) * [微调的好处](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-de-hao-chu) * [常见误解](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#chang-jian-wu-jie) * [微调会为模型添加新知识吗？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-hui-wei-mo-xing-tian-jia-xin-zhi-shi-ma) * [RAG 总是比微调更好吗？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#rag-zong-shi-bi-wei-tiao-geng-hao-ma) * [微调昂贵吗？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-tiao-ang-gui-ma) * [常见问题：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#chang-jian-wen-ti) * [为什么要结合 RAG 与微调](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#wei-shen-me-yao-jie-he-rag-yu-wei-tiao) * [LoRA vs. QLoRA：该选哪个？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#lora-vs.-qlora-gai-xuan-na-ge) * [实验为关键](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#shi-yan-wei-guan-jian) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth 需求 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 可以通过两种方式使用：通过 [Unsloth Studio](https://unsloth.ai/docs/zh/xin/studio/install) ，Web UI，或通过 [Unsloth Core](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-core-requirements) ，原始的基于代码版本。每种都有不同的要求。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-studio-yao-qiu) **Unsloth Studio 要求** ------------------------------------------------------------------------------------------------------------------------------------------------------- * **Mac：** 类似 CPU - [聊天](https://unsloth.ai/docs/zh/xin/studio/chat#using-unsloth-studio-chat) + [数据食谱](https://unsloth.ai/docs/zh/xin/studio/data-recipe) 目前可用。 **MLX** 训练很快就会推出。 * **CPU：Unsloth 在没有 GPU 的情况下仍可运行**，用于聊天 + 数据食谱。 * **训练：** 可在以下设备上运行 **NVIDIA**：RTX 30、40、50、Blackwell、DGX Spark/Station 等 + **Intel** GPU * **即将推出：** 支持 **Apple MLX** 和 **AMD**. ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#windows) windows Window**s** Unsloth Studio 可直接在 Windows 上运行，无需 WSL。要训练模型，请确保你的系统满足以下要求： **要求** * Windows 10 或 Windows 11（64 位） * 已安装驱动程序的 NVIDIA GPU * **App Installer** （包括 `winget`): [这里arrow-up-right](https://learn.microsoft.com/en-us/windows/msix/app-installer/install-update-app-installer) * **Git**: `winget install --id Git.Git -e --source winget` * **Python**：版本 3.11 到 3.14 之前（不含 3.14） * 在 Python 环境中工作，例如 **uv**, **venv**，或 **conda/mamba** ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#macos) apple MacOS Unsloth Studio 可在 Mac 设备上用于 [聊天](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#run-models-locally) 用于 GGUF 模型和 [数据食谱](https://unsloth.ai/docs/zh/xin/studio/data-recipe) ([导出](https://unsloth.ai/docs/zh/xin/studio/export) 即将推出）。 **MLX 训练即将推出！** * macOS 12 Monterey 或更新版本（Intel 或 Apple Silicon） * 安装 Homebrew： [这里arrow-up-right](https://brew.sh/) * Git： `brew install git` * cmake： `brew install cmake` * openssl： `brew install openssl` * Python：版本 3.11 到 3.14 之前（不含 3.14） * 在 Python 环境中工作，例如 **uv**, **venv**，或 **conda/mamba** ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#linux-he-wsl) linux Linux 和 WSL * Ubuntu 20.04+ 或类似发行版（64 位） * 已安装驱动程序的 NVIDIA GPU * CUDA 工具包（推荐 12.4+，blackwell 需 12.8+） * Git： `sudo apt install git` * Python：版本 3.11 到 3.14 之前（不含 3.14） * 在 Python 环境中工作，例如 **uv**, **venv**，或 **conda/mamba** ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#jin-cpu) microchip 仅 CPU Unsloth Studio 支持 CPU 设备用于 [聊天](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#run-models-locally) 用于 GGUF 模型和 [数据食谱](https://unsloth.ai/docs/zh/xin/studio/data-recipe) ([导出](https://unsloth.ai/docs/zh/xin/studio/export) 即将推出） * 与上面提到的 Linux 要求相同（不包括 NVIDIA GPU 驱动）以及 MacOS。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#xun-lian) **训练** Unsloth Studio 训练目前可在 NVIDIA GPU 上运行，AMD、MLX、Intel 支持即将推出。你仍然可以使用 [原始的 Unsloth Core](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-requirements) 在 AMD 和 Intel 设备上训练。 **需要 Python 3.11–3.13** 。 要求 Linux / WSL Windows **Git** 通常已预装 由设置脚本安装（`winget`) **CMake** 已预装或 `sudo apt install cmake` 由设置脚本安装（`winget`) **C++ 编译器** `build-essential` Visual Studio Build Tools 2022 **CUDA 工具包** 可选； `nvcc` 自动检测 由设置脚本安装（与驱动程序匹配） [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-core-yao-qiu) Unsloth Core 要求 ----------------------------------------------------------------------------------------------------------------------------------------------- * **操作系统**：可在 Linux 和 [Windowsarrow-up-right](https://docs.unsloth.ai/get-started/install-and-update/windows-installation) * 自 2018+ 起支持 NVIDIA GPU，包括 [Blackwell RTX 50](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) 和 [DGX Spark](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) * 最低 CUDA 能力 7.0（V100、T4、Titan V、RTX 20 和 50、A100、H100、L40 等） [检查你的 GPU！arrow-up-right](https://developer.nvidia.com/cuda-gpus) GTX 1070、1080 可以运行，但速度较慢。 * 官方 [Unsloth Docker 镜像arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` 可在 Docker Hub 上获取 * [Docker](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker) * Unsloth 可在 [AMD](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd) 和 [Intel](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel) GPU 上运行（请参考我们的 [特定指南](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install) ）。Apple/Silicon/MLX 正在开发中 * 你的设备应具备 `xformers`, `torch`, `BitsandBytes` 和 `triton` 支持。 * 如果你有不同版本的 torch、transformers 等， `pip install unsloth` 将自动安装这些库的所有最新版本，因此你无需担心版本兼容性。 circle-info 支持 Python 3.13！ ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#wei-tiao-vram-yao-qiu) 微调 VRAM 要求： 使用 Unsloth 进行 LLM 微调需要多少 GPU 内存？ circle-info 一个常见的问题是当你 OOM 或内存不足时，往往是因为把 batch size 设得太高。将其设为 1、2 或 3 可使用更少的 VRAM。 **有关上下文长度基准，请参见** [**这里**](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/unsloth-benchmarks#context-length-benchmarks) **.** 请查看此表，其中按模型参数和微调方法列出了 VRAM 要求。QLoRA 使用 4 位，LoRA 使用 16 位。请注意，有时根据模型可能需要更多 VRAM，因此这些数字是绝对最低值： 模型参数 QLoRA（4 位）VRAM LoRA（16 位）VRAM 3B 3.5 GB 8 GB 7B 5 GB 19 GB 8B 6 GB 22 GB 9B 6.5 GB 24 GB 11B 7.5 GB 29 GB 14B 8.5 GB 33 GB 27B 22GB 64GB 32B 26 GB 76 GB 40B 30GB 96GB 70B 41 GB 164 GB 81B 48GB 192GB 90B 53GB 212GB 405B 237 GB 950 GB [上一页Beginner? Start here!chevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners) [下一页FAQ + 微调适合我吗？chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) 最后更新于 17天前 这有帮助吗？ * [Unsloth Studio 要求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-studio-yao-qiu) * [Windows](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#windows) * [MacOS](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#macos) * [Linux 和 WSL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#linux-he-wsl) * [仅 CPU](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#jin-cpu) * [训练](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#xun-lian) * [Unsloth Core 要求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#unsloth-core-yao-qiu) * [微调 VRAM 要求：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#wei-tiao-vram-yao-qiu) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 在 MacOS 上安装 Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 要在本地 Apple MacOS 设备上本地安装 Unsloth，请按照以下步骤操作： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#an-zhuang-unsloth) 安装 Unsloth 复制 curl -fsSL https://unsloth.ai/install.sh | sh 使用相同的命令进行更新或使用 `unsloth studio update`. ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#qi-dong) 启动 每次你想再次启动 Unsloth 时： 复制 unsloth studio -H 0.0.0.0 -p 8888 有关详细的 Unsloth Studio 安装说明和要求， [请查看我们的指南](https://unsloth.ai/docs/zh/xin/studio/install) . ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#xie-zai) 卸载 要在 macOs 上卸载 Unsloth Studio，请按照以下 4 个步骤操作： #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#id-1.-yi-chu-ying-yong-cheng-xu) **1\. 移除应用程序** * MacOS、Linux： `rm -rf ~/.unsloth/studio/unsloth ~/.unsloth/studio/studio` 这会移除应用程序，但会保留你的模型检查点、导出、历史记录、缓存和聊天内容。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#id-2.-yi-chu-kuai-jie-fang-shi-he-fu-hao-lian-jie) **2\. 移除快捷方式和符号链接** **macOS：** #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#id-3.-yi-chu-cli-ming-ling) **3\. 移除 CLI 命令** **macOS、Linux：** #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#id-4.-yi-chu-suo-you-nei-rong-ke-xuan) **4\. 移除所有内容（可选）** 如需同时删除历史记录、缓存、聊天内容、模型检查点和模型导出，请删除整个 Unsloth 文件夹： * MacOS、Linux： `rm -rf ~/.unsloth` 请注意，下载的 HF 模型文件是单独存储在 Hugging Face 缓存中的——以上步骤都不会将其删除。请参见 **删除模型文件** 如下，如果你想释放那部分磁盘空间。 circle-exclamation 注意：使用 `rm -rf` 命令将会 **删除所有内容**，包括你的历史记录、缓存、聊天内容等。 如果你仍然遇到与 Unsloth 相关的依赖问题，许多用户通过强制卸载并重新安装 Unsloth 解决了它们： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#shan-chu-mo-xing-wen-jian) **删除模型文件** 你可以从模型搜索中的垃圾桶图标删除旧模型文件，或者从 Hugging Face 缓存目录中移除相关的缓存模型文件夹。 默认缓存位置是： 如果 `HF_HUB_CACHE` 或 `HF_HOME` 已设置，请改用该位置。在 Linux 和 WSL 上， `XDG_CACHE_HOME` 也可以更改默认缓存根目录。你可以通过以下方式检查： 要删除特定模型，请移除其文件夹（例如 `models--unsloth--Llama-3.1-8B-bnb-4bit`）从缓存目录中移除。要清除所有已缓存的模型： [上一页uv, pip install & venvchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install) [下一页Windowschevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation) 最后更新于 3天前 这有帮助吗？ * [安装 Unsloth](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#an-zhuang-unsloth) * [启动](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#qi-dong) * [卸载](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#xie-zai) * [删除模型文件](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac#shan-chu-mo-xing-wen-jian) 这有帮助吗？ sun-brightdesktopmoon 复制 rm -rf ~/Applications/Unsloth\ Studio.app ~/Desktop/Unsloth\ Studio 复制 rm -f ~/.local/bin/unsloth 复制 pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo 复制 ~/.cache/huggingface/hub/ 复制 echo ${HF_HUB_CACHE:-${HF_HOME:-${XDG_CACHE_HOME:-$HOME/.cache}/huggingface}/hub} 复制 rm -rf ~/.cache/huggingface/hub/ sun-brightdesktopmoon --- # Unsloth 安装 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 可通过两种方式使用：通过 [Unsloth Studio](https://unsloth.ai/docs/zh/xin/studio/install) ，即网页界面，或者通过 Unsloth Core，即最初的基于代码版本。查看我们的 [系统要求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements) Unsloth Studio 可在 MacOS、Linux、Windows、NVIDIA 等平台上运行。使用相同的安装命令来更新或 `unsloth studio update`. **MacOS、Linux、WSL：** 复制 curl -fsSL https://unsloth.ai/install.sh | sh **Windows PowerShell：** 复制 irm https://unsloth.ai/install.ps1 | iex **启动 Unsloth Studio：** 复制 unsloth studio -H 0.0.0.0 -p 8888 apple[MacOS](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac) [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install) desktop-arrow-down[uv, pip install & venv](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install) windows[Windows](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation) docker[Docker](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker) [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating) arrow-rotate-right[Updating](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating) square-up-right[AMD](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd) info[Intel](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel) [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/conda-install) snake[Conda](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/conda-install) vscode[VS Code](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/vs-code) [](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/google-colab) google[Google Colab](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/google-colab) [上一页All Our Modelschevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-model-catalog) [下一页uv, pip install & venvchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install) 最后更新于 8天前 这有帮助吗？ 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 通过 pip 和 uv 安装 Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 可以通过两种方式使用：通过 [Unsloth Studio](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-studio) ，网页界面，或通过 [Unsloth Core](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-core) ，基于代码的版本。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-studio) **Unsloth Studio** ----------------------------------------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#macos-linux-wsl) **MacOS、Linux、WSL：** 复制 curl -fsSL https://unsloth.ai/install.sh | sh 使用相同命令进行更新，或使用 `unsloth studio update`. ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#windows-powershell) **Windows PowerShell：** 复制 irm https://unsloth.ai/install.ps1 | iex 使用相同命令进行更新，或使用 `unsloth studio update`. ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#qi-dong) 启动： 复制 unsloth studio -H 0.0.0.0 -p 8888 有关 Unsloth Studio 的详细安装说明和要求， [请查看我们的指南](https://unsloth.ai/docs/zh/xin/studio/install) . ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#xie-zai) 卸载 要卸载 Unsloth Studio，请按照以下 4 个步骤操作： #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#id-1.-shan-chu-ying-yong-cheng-xu) **1\. 删除应用程序** * MacOS、WSL、Linux： `rm -rf ~/.unsloth/studio/unsloth ~/.unsloth/studio/studio` * Windows（PowerShell）： `Remove-Item -Recurse -Force "$HOME\.unsloth\studio\unsloth", "$HOME\.unsloth\studio\studio"` 这将删除应用程序，但会保留您的模型检查点、导出、历史记录、缓存和聊天记录。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#id-2.-shan-chu-kuai-jie-fang-shi-he-fu-hao-lian-jie) **2\. 删除快捷方式和符号链接** **macOS：** **Linux：** **WSL / Windows（PowerShell）：** #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#id-3.-shan-chu-cli-ming-ling) **3\. 删除 CLI 命令** **MacOS、Linux、WSL：** **Windows（PowerShell）：** 安装程序已将虚拟环境的 `Scripts` 目录添加到您的用户 PATH 中。要将其移除，请打开 设置 → 系统 → 关于 → 高级系统设置 → 环境变量，在 `Path` 下的用户变量中，删除指向 `.unsloth\studio\...\Scripts`. #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#id-4.-shan-chu-quan-bu-nei-rong-ke-xuan) **4\. 删除全部内容（可选）** 如需同时删除历史记录、缓存、聊天记录、模型检查点和模型导出，请删除整个 Unsloth 文件夹： * MacOS、WSL、Linux： `rm -rf ~/.unsloth` * Windows（PowerShell）： `Remove-Item -Recurse -Force "$HOME\.unsloth"` 请注意，已下载的 HF 模型文件是单独存储在 Hugging Face 缓存中的——以上步骤都不会将其删除。请参见 **删除模型文件** 以下内容，如果您想回收那部分磁盘空间。 circle-exclamation 注意：使用 `rm -rf` 命令将 **删除一切**，包括您的历史记录、缓存、聊天记录等。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#shan-chu-huan-cun-de-hf-mo-xing-wen-jian) **删除缓存的 HF 模型文件** 您可以通过模型搜索中的垃圾桶图标删除旧模型文件，或者从默认的 Hugging Face 缓存目录中移除相应的缓存模型文件夹。默认情况下，Hugging Face 使用 `~/.cache/huggingface/hub/` 在 macOS/Linux/WSL 上，以及 `C:\Users\\.cache\huggingface\hub\` 在 Windows 上。 * **MacOS、Linux、WSL：** `~/.cache/huggingface/hub/` * **Windows：** `%USERPROFILE%\.cache\huggingface\hub\` 如果 `HF_HUB_CACHE` 或 `HF_HOME` 已设置，请改用该位置。在 Linux 和 WSL 上， `XDG_CACHE_HOME` 也可以更改默认缓存根目录。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-core) **Unsloth Core** ------------------------------------------------------------------------------------------------------------- **使用 uv pip 安装（推荐），以获取最新的 pip 版本：** 或者直接使用 pip： 要将 **vLLM 和 Unsloth** 一起安装，请执行： 要安装 Unsloth 的 **最新主分支** ，请执行： 对于 **venv 和虚拟环境安装** ，为了将安装与系统包隔离，避免对系统造成不可修复的损害，请使用 venv： 如果您是在 Jupyter、Colab 或其他笔记本中安装 Unsloth，请务必在命令前加上 `!`。在终端中使用时则不需要这样做 circle-info 现在已支持 Python 3.13！ ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#xie-zai-unsloth-core) 卸载 Unsloth Core 如果您仍然遇到 Unsloth 的依赖问题，许多用户通过强制卸载并重新安装 Unsloth 已经解决了： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#gao-ji-pip-an-zhuang) 高级 Pip 安装 circle-exclamation 请 **不要** 在您已使用 [Conda](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/conda-install) . 时使用此方法。由于依赖问题，pip 会稍微复杂一些。pip 命令会因以下版本而不同： `torch 2.2,2.3,2.4,2.5` 以及 CUDA 版本。 对于其他 torch 版本，我们支持 `torch211`, `torch212`, `torch220`, `torch230`, `torch240` 对于 CUDA 版本，我们支持 `cu118` 和 `cu121` 和 `cu124`。对于 Ampere 设备（A100、H100、RTX3090）及以上，请使用 `cu118-ampere` 或 `cu121-ampere` 或 `cu124-ampere`. 例如，如果您有 `torch 2.4` 和 `CUDA 12.1`，请使用： 另一个例子，如果您有 `torch 2.5` 和 `CUDA 12.4`，请使用： 其他示例： 或者，在终端中运行下面的命令以获取 **最优的** pip 安装命令： 或者，在 Python REPL 中手动运行下面的命令： [上一页Installationchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install) [下一页MacOSchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac) 最后更新于 3天前 这有帮助吗？ * [Unsloth Studio](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-studio) * [MacOS、Linux、WSL：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#macos-linux-wsl) * [Windows PowerShell：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#windows-powershell) * [启动：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#qi-dong) * [卸载](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#xie-zai) * [删除缓存的 HF 模型文件](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#shan-chu-huan-cun-de-hf-mo-xing-wen-jian) * [Unsloth Core](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#unsloth-core) * [卸载 Unsloth Core](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#xie-zai-unsloth-core) * [高级 Pip 安装](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/pip-install#gao-ji-pip-an-zhuang) 这有帮助吗？ sun-brightdesktopmoon 复制 rm -rf ~/Applications/Unsloth\ Studio.app ~/Desktop/Unsloth\ Studio 复制 rm -f ~/.local/share/applications/unsloth-studio.desktop ~/Desktop/unsloth-studio.desktop 复制 Remove-Item -Force "$HOME\Desktop\Unsloth Studio.lnk" Remove-Item -Force "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk" 复制 rm -f ~/.local/bin/unsloth 复制 curl -LsSf https://astral.sh/uv/install.sh | sh uv venv unsloth_env --python 3.13 source unsloth_env/bin/activate uv pip install unsloth --torch-backend=auto 复制 pip install unsloth 复制 uv pip install unsloth vllm --torch-backend=auto 复制 uv pip install unsloth --torch-backend=auto pip uninstall unsloth unsloth_zoo -y && pip install --no-deps git+https://github.com/unslothai/unsloth_zoo.git && pip install --no-deps git+https://github.com/unslothai/unsloth.git 复制 apt install python3.10-venv python3.11-venv python3.12-venv python3.13-venv -y python -m venv unsloth_env source unsloth_env/bin/activate pip install --upgrade pip && pip install uv uv pip install unsloth --torch-backend=auto 复制 pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo 复制 pip install --upgrade pip pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git" 复制 pip install --upgrade pip pip install "unsloth[cu124-torch250] @ git+https://github.com/unslothai/unsloth.git" 复制 pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu118-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu118-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch250] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu124-ampere-torch250] @ git+https://github.com/unslothai/unsloth.git" 复制 wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python - 复制 # 依据 Apache License, Version 2.0（“许可证”）授权 try: import torch except: raise ImportError('通过 `pip install torch` 安装 torch') from packaging.version import Version as V import re v = V(re.match(r"[0-9\.]{3,}", torch.__version__).group(0)) cuda = str(torch.version.cuda) is_ampere = torch.cuda.get_device_capability()[0] >= 8 USE_ABI = torch._C._GLIBCXX_USE_CXX11_ABI if cuda not in ("11.8", "12.1", "12.4", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} 不受支持！") if v <= V('2.1.0'): raise RuntimeError(f"Torch = {v} 版本过旧！") elif v <= V('2.1.1'): x = 'cu{}{}-torch211' elif v <= V('2.1.2'): x = 'cu{}{}-torch212' elif v < V('2.3.0'): x = 'cu{}{}-torch220' elif v < V('2.4.0'): x = 'cu{}{}-torch230' elif v < V('2.5.0'): x = 'cu{}{}-torch240' elif v < V('2.5.1'): x = 'cu{}{}-torch250' elif v <= V('2.5.1'): x = 'cu{}{}-torch251' elif v < V('2.7.0'): x = 'cu{}{}-torch260' elif v < V('2.7.9'): x = 'cu{}{}-torch270' elif v < V('2.8.0'): x = 'cu{}{}-torch271' elif v < V('2.8.9'): x = 'cu{}{}-torch280' elif v < V('2.9.1'): x = 'cu{}{}-torch290' elif v < V('2.9.2'): x = 'cu{}{}-torch291' else: raise RuntimeError(f"Torch = {v} 版本过新！") if v > V('2.6.9') and cuda not in ("11.8", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} 不受支持！") x = x.format(cuda.replace(".", ""), "-ampere" if False else "") # 由于 flash-attn，is_ampere 有问题 print(f'pip install --upgrade pip && pip install --no-deps git+https://github.com/unslothai/unsloth-zoo.git && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git" --no-build-isolation') sun-brightdesktopmoon --- # 更新 Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#geng-xin-unsloth-studio) **更新 Unsloth Studio** 使用相同的安装命令进行更新。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#macos-linux-wsl) **MacOS、Linux、WSL：** 复制 curl -fsSL https://unsloth.ai/install.sh | sh #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#windows-powershell) **Windows PowerShell：** 复制 irm https://unsloth.ai/install.ps1 | iex #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#huo-zhe-ni-ye-ke-yi-shi-yong-wo-men-de-geng-xin-ming-ling) 或者，你也可以使用我们的更新命令： 复制 unsloth studio update ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#geng-xin-unsloth-core) 更新 Unsloth Core： 复制 pip install --upgrade unsloth unsloth_zoo #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#zai-bu-geng-xin-yi-lai-xiang-de-qing-kuang-xia-geng-xin-unsloth-core) 在不更新依赖项的情况下更新 Unsloth Core： 复制 pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#yao-shi-yong-jiu-ban-unsloth) 要使用旧版 Unsloth： 复制 pip install --force-reinstall --no-cache-dir --no-deps unsloth==2025.1.5 “2025.1.5”是 Unsloth 之前的旧版本之一。请将其改为我们 [Github 上列出的某个特定版本arrow-up-right](https://github.com/unslothai/unsloth/releases) . [上一页Dockerchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker) [下一页AMDchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd) 最后更新于 4天前 这有帮助吗？ * [更新 Unsloth Studio](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#geng-xin-unsloth-studio) * [更新 Unsloth Core：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating#geng-xin-unsloth-core) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 使用 Unsloth 在 AMD GPU 上微调 LLM 指南 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 现在您可以使用 Unsloth 在本地 AMD 环境上微调大型语言模型。Unsloth 支持 AMD Radeon RX、MI300X（192GB）GPU 等。 1 **创建新的隔离环境（可选）** 为了不破坏系统包，您可以创建一个隔离的 pip 环境。提醒检查您使用的 Python 版本！它可能是 `pip3`, `pip3.13`, `python3`, `python.3.13` 等等。 复制 apt install python3.10-venv python3.11-venv python3.12-venv python3.13-venv -y python -m venv unsloth_env source unsloth_env/bin/activate ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FCqOhjYTr4GqQ90ToPEig%252Famd1.png%3Falt%3Dmedia%26token%3Dd8f96a07-90be-4d93-b848-ad182c262d1f&width=768&dpr=3&quality=100&sign=84fb531&sv=2) 2 **安装 PyTorch** 从以下位置安装最新的 PyTorch、TorchAO、Xformers： [https://pytorch.org/arrow-up-right](https://pytorch.org/) 通过以下命令检查您的 ROCM 版本： `amd-smi version` 然后更改 `https://download.pytorch.org/whl/rocm7.0` 以匹配您的版本。 复制 uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm7.0 --upgrade --force-reinstall 我们还编写了一个单行终端命令以提取正确的 ROCM 版本，以便使用时更方便。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJ1VZQ9QhzWFizDceg3ye%252Famd2.png%3Falt%3Dmedia%26token%3D937d1eba-3c7e-4c73-b6a6-9a9450d0e4ac&width=768&dpr=3&quality=100&sign=f2dc7b0a&sv=2) 复制 ROCM_TAG="$({ command -v amd-smi >/dev/null 2>&1 && amd-smi version 2>/dev/null | awk -F'ROCm version: ' 'NF>1{split($2,a,"."); print "rocm"a[1]"."a[2]; ok=1; exit} END{exit !ok}'; } || { [ -r /opt/rocm/.info/version ] && awk -F. '{print "rocm"$1"."$2; exit}' /opt/rocm/.info/version; } || { command -v hipconfig >/dev/null 2>&1 && hipconfig --version 2>/dev/null | awk -F': *' '/HIP version/{split($2,a,"."); print "rocm"a[1]"."a[2]; ok=1; exit} END{exit !ok}'; } || { command -v dpkg-query >/dev/null 2>&1 && ver="$(dpkg-query -W -f="${Version}\n" rocm-core 2>/dev/null)" && [ -n "$ver" ] && awk -F'[.-]' '{print "rocm"$1"."$2; exit}' <<<"$ver"; } || { command -v rpm >/dev/null 2>&1 && ver="$(rpm -q --qf '%{VERSION}\n' rocm-core 2>/dev/null)" && [ -n "$ver" ] && awk -F'[.-]' '{print "rocm"$1"."$2; exit}' <<<"$ver"; })"; [ -n "$ROCM_TAG" ] && uv pip install torch torchvision torchaudio --index-url "https://download.pytorch.org/whl/$ROCM_TAG" --upgrade --force-reinstall 3 **安装 Unsloth** 安装 Unsloth 的专用 AMD 分支： 复制 pip install --no-deps unsloth unsloth-zoo pip install --no-deps git+https://github.com/unslothai/unsloth-zoo.git pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth" ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Frz8GOvVgST7beQ8pmgmC%252Famd3.png%3Falt%3Dmedia%26token%3D03a12c20-af1d-4b98-9aaf-18ccc6a1d4a4&width=768&dpr=3&quality=100&sign=46273bb4&sv=2) 4 **使用 Unsloth 开始微调！** 就是这样。尝试我们的一些示例，位于我们的 [**Unsloth 笔记本**](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) 页面！ 您可以查看我们的专用 [微调](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 或 [强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 指南。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlDpKitaEagbh0Er8wJFC%252Famd4.png%3Falt%3Dmedia%26token%3Df54448fe-0719-464f-bbd1-d73f82aedfc0&width=768&dpr=3&quality=100&sign=a7a4b24f&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd#zai-amd-gpu-shang-de-qiang-hua-xue-xi) 🔢 在 AMD GPU 上的强化学习 您可以使用我们的 📒[gpt-oss RL 自动赢 2048arrow-up-right](https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb) 示例在 MI300X（192GB）GPU 上运行。目标是使用强化学习自动玩并赢得 2048 游戏。LLM（gpt-oss 20b）会自动制定赢得 2048 游戏的策略，我们为获胜策略计算高奖励，为失败策略计算低奖励。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-2bc5a2e25a51781fd945ab9e87e73821ed4eb6c9%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=fa3bef05&sv=2) 在大约 300 步左右后，奖励随时间增加！ 强化学习的目标是最大化平均奖励以赢得 2048 游戏。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-8d7ea897fd57156a796e4f74aa2e3b60afe9d405%252F2048%2520Auto%2520Win%2520Game%2520Reward.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=87f51ec0&sv=2) 我们使用一台 AMD MI300X（192GB）机器运行了带 Unsloth 的 2048 强化学习示例，运行良好！ ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-174890aa5f63632ebe6f3f212f1ced0d0e8dc381%252FScreenshot%25202025-10-17%2520052504.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=9cbe0cf5&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-f907ba596705496515fdfb39b49d649697317ca7%252FScreenshot%25202025-10-17%2520052641.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=fd5951c9&sv=2) 您还可以使用我们的 📒[自动内核生成 强化学习 笔记本arrow-up-right](https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_GRPO_BF16.ipynb) 同样使用 gpt-oss 在 Python 中自动创建矩阵乘法内核。该笔记本还设计了多种方法来对抗奖励操纵。 我们用来自动创建这些内核的提示是： 例如，强化学习过程会学会如何在 Python 内部应用 Strassen 算法以更快地进行矩阵乘法。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ddb993e5d2c986794ede1f2b0d08897469b78506%252Fimage%2520%281%29%2520%281%29%2520%281%29%2520%281%29%2520%281%29%2520%281%29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=58e8d89e&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd#amd-mian-fei-yi-jian-bi-ji-ben) 📚AMD 免费一键笔记本 AMD 提供配备以下资源的一键笔记本： **免费的 192GB VRAM MI300X GPU** 通过他们的开发云。完全免费训练大型模型（无需注册或信用卡）： * [Qwen3（32B）arrow-up-right](https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Qwen3_(32B)_A100-Reasoning-Conversational.ipynb) * [Llama 3.3（70B）arrow-up-right](https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Llama3.3_(70B)_A100-Conversational.ipynb) * [Qwen3（14B）arrow-up-right](http://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Qwen3_%2814B%29-Reasoning-Conversational.ipynb) * [Mistral v0.3（7B）arrow-up-right](http://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Alpaca.ipynb) * [GPT OSS MXFP4（20B）arrow-up-right](http://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Kaggle-GPT_OSS_MXFP4_(20B)-Inference.ipynb) - 推理 * 强化学习 笔记本： [Loading GitHub Notebook - AMD Dev Cloudoneclickamd.aichevron-right](https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb) 您可以通过在任何 Unsloth 笔记本前添加以下前缀来使用： _**https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb**_ 在 [Unsloth 笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) 通过将链接从 [https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3\_(270M).ipynbarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) 更改为 [https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Gemma3\_(270M).ipynbarrow-up-right](https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7NNi4jLKvmZoRnLel9Kg%252Fimage.png%3Falt%3Dmedia%26token%3D0379eda9-569c-4614-afb5-ffec463a7676&width=768&dpr=3&quality=100&sign=b4ce81&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRfKS1GAW7BqL9lGNTcxh%252Fimage.png%3Falt%3Dmedia%26token%3D3a8aeb01-62a7-4d55-89a9-98526052e305&width=768&dpr=3&quality=100&sign=c1e3243&sv=2) [上一页Updatingchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating) [下一页Intelchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel) 最后更新于 1个月前 这有帮助吗？ * [🔢 在 AMD GPU 上的强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd#zai-amd-gpu-shang-de-qiang-hua-xue-xi) * [📚AMD 免费一键笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd#amd-mian-fei-yi-jian-bi-ji-ben) 这有帮助吗？ sun-brightdesktopmoon 复制 仅使用原生 Python 代码创建一个新的快速矩阵乘法函数。 您将获得一个数字的列表的列表。 使用下面的格式将您的新函数用反引号输出： ``` python def matmul(A, B): return ... ``` sun-brightdesktopmoon --- # 通过 Docker 安装 Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 了解如何使用我们的 Docker 容器，所有依赖均已预装，可立即安装。无需设置，只需运行并开始训练！ Unsloth Docker 镜像： [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) circle-check Unsloth Studio 现在与笔记本和脚本共享相同的缓存，以避免不必要的重复下载。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#kuai-su-kai-shi) ⚡ 快速开始 1 **安装 Docker 和 NVIDIA Container Toolkit。** 通过以下方式安装 Docker [Linuxarrow-up-right](https://docs.docker.com/engine/install/) 或 [桌面版arrow-up-right](https://docs.docker.com/desktop/) （其他）。 然后安装 [NVIDIA Container Toolkitarrow-up-right](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation) : 复制 export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1 sudo apt-get update && sudo apt-get install -y \ nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION} ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-41cae231ed4761f844ce9836e03b17aabd7c803c%252Fnvidia%2520toolkit.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c53138dd&sv=2) 2 **运行容器。** [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) 是 Unsloth 唯一的 Docker 镜像。 复制 docker run -d -e JUPYTER_PASSWORD="mypassword" \ -p 8888:8888 -p 8000:8000 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-2b50d78c5d54eaf189c0a40d46c405585ea23082%252Fdocker%2520run.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=1592327c&sv=2) 3 **访问 Jupyter Lab** 前往 [http://localhost:8888arrow-up-right](http://localhost:8888/) 并打开 Unsloth。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-828df0a668fd94025c1193c24a7f09c1d58dcbd8%252Fjupyter.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=6827d4e1&sv=2) 访问 `unsloth-notebooks` 标签页以查看 Unsloth 笔记本。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e7a3f620a3ec5bff335632ff9b0cb422f76528a1%252FScreenshot_from_2025-09-30_21-38-15.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=dba002ad&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-531882c33eb96dec24e2d7673471d6a3928a3951%252FScreenshot_from_2025-09-30_21-39-41.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7b81a0c4&sv=2) 4 **开始使用 Unsloth 训练** 如果你是新手，请按照我们的分步 [微调指南](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) , [RL 指南](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 或者直接保存/复制我们任何预制的 [笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-665f900b008991ddcd8fdabb773b292de3c41e72%252FScreenshot_from_2025-09-30_21-40-29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=28e3e3d3&sv=2) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#rong-qi-jie-gou) 📂 容器结构 * `/workspace/work/` — 你挂载的工作目录 * `/workspace/unsloth-notebooks/` — 示例微调笔记本 * `/home/unsloth/` — 用户主目录 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#shi-yong-shi-li) 📖 使用示例 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#wan-zheng-shi-li) 完整示例 复制 docker run -d -e JUPYTER_PORT=8000 \ -e JUPYTER_PASSWORD="mypassword" \ -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" \ -e USER_PASSWORD="unsloth2024" \ -p 8000:8000 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#she-zhi-ssh-mi-yao) 设置 SSH 密钥 如果你没有 SSH 密钥对： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#wei-shen-me-xuan-ze-unsloth-rong-qi) 🦥为什么选择 Unsloth 容器？ * **可靠**：精心维护的环境，拥有稳定且受维护的软件包版本。压缩后仅 7 GB（而其他地方为 10–11 GB） * **即用型**：预装在中的笔记本 `/workspace/unsloth-notebooks/` * **安全**：以非 root 用户身份安全运行 * **通用**：兼容所有基于 transformer 的模型（TTS、BERT 等） ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#unsloth-mei-you-jian-ce-dao-huo-shi-yong-wo-de-gpu) **Unsloth 没有检测到或使用我的 GPU** 如果模型在 Docker 中没有专门使用你的 GPU，请尝试： 手动拉取最新镜像： * 使用 GPU 访问启动容器： * `docker run`: `--gpus all` * Docker Compose： `capabilities: [gpu]` * 在 Linux 上，请确保已安装 NVIDIA Container Toolkit。 * 在 Windows 上： * 检查 `nvcc --version` 是否与以下内容中显示的 CUDA 版本匹配 `nvidia-smi` * 参照： [https://docs.docker.com/desktop/features/gpu/arrow-up-right](https://docs.docker.com/desktop/features/gpu/) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#gao-ji-she-zhi) ⚙️ 高级设置 变量 描述 默认值 `JUPYTER_PASSWORD` Jupyter Lab 密码 `unsloth` `JUPYTER_PORT` 容器内的 Jupyter Lab 端口 `8888` `SSH_KEY` 用于身份验证的 SSH 公钥 `无` `USER_PASSWORD` 的密码 `unsloth` 用户（sudo） `unsloth` * Jupyter Lab： `-p 8000:8888` * SSH 访问： `-p 2222:22` circle-exclamation **重要**：使用卷挂载以在容器运行之间保留你的工作。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#an-quan-shuo-ming) **🔒 安全说明** * 容器默认以非 root `unsloth` 用户运行 * 使用 `USER_PASSWORD` 在容器内执行 sudo 操作 * SSH 访问需要公钥认证 [上一页Windowschevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation) [下一页Updatingchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/updating) 最后更新于 6天前 这有帮助吗？ * [⚡ 快速开始](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#kuai-su-kai-shi) * [📖 使用示例](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#shi-yong-shi-li) * [🦥为什么选择 Unsloth 容器？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#wei-shen-me-xuan-ze-unsloth-rong-qi) * [Unsloth 没有检测到或使用我的 GPU](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#unsloth-mei-you-jian-ce-dao-huo-shi-yong-wo-de-gpu) * [⚙️ 高级设置](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#gao-ji-she-zhi) * [🔒 安全说明](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker#an-quan-shuo-ming) 这有帮助吗？ sun-brightdesktopmoon 复制 # 生成新的密钥对 ssh-keygen -t rsa -b 4096 -f ~/.ssh/container_key # 在 docker run 中使用公钥 -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" # 通过 SSH 连接 ssh -i ~/.ssh/container_key -p 2222 unsloth@localhost 复制 docker pull unsloth/unsloth:latest 复制 # 生成 SSH 密钥对 ssh-keygen -t rsa -b 4096 -f ~/.ssh/container_key # 连接到容器 ssh -i ~/.ssh/container_key -p 2222 unsloth@localhost 复制 -p : 复制 -v : 复制 docker run -d -e JUPYTER_PORT=8000 \ -e JUPYTER_PASSWORD="mypassword" \ -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" \ -e USER_PASSWORD="unsloth2024" \ -p 8000:8000 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth sun-brightdesktopmoon --- # 我应该用什么模型来微调？ | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#llamaqwenmistralphi-hai-shi) Llama、Qwen、Mistral、Phi 还是？ ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 在准备微调时，你首先要面临的决定之一是选择合适的模型。以下是一份分步指南，帮助你做出选择： 1 **选择与您的使用案例一致的模型** * 例如：对于基于图像的训练，选择诸如视觉模型 _Llama 3.2 视觉_. 对于代码数据集，选择像这样的专用模型 _Qwen Coder 2.5_. * **许可和要求**: 不同模型可能有特定的许可条款和 [系统要求](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#system-requirements) . 请务必仔细审查这些内容以避免兼容性问题。 2 **评估您的存储、计算能力和数据集** * 使用我们的 [显存指导](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/unsloth-requirements#approximate-vram-requirements-based-on-model-parameters) 来确定您正在考虑的模型所需的显存要求。 * 您的数据集将反映您将使用的模型类型以及训练所需的时间 3 **选择模型和参数** * 我们建议使用最新模型以获得最佳性能和功能。例如，截至 2025 年 1 月，领先的 70B 模型是 _Llama 3.3_. * 您可以通过浏览我们的 [模型目录](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-model-catalog) 来查找最新和相关的选项并保持更新。 4 **在 Base 模型和 Instruct 模型之间选择** 更多细节如下： [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#instruct-mo-xing-hai-shi-base-mo-xing) Instruct 模型还是 Base 模型？ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 在准备微调时，你首先要面临的决定之一是使用 instruct 模型还是 base 模型。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#instruct-mo-xing) Instruct 模型 Instruct 模型在预训练时已内置指令，使其无需任何微调即可使用。这些模型（包括 GGUF 等常见格式）针对直接使用进行了优化，能够开箱即用地对提示做出有效响应。Instruct 模型可与 ChatML 或 ShareGPT 等会话聊天模板配合使用。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#base-mo-xing) **Base 模型** 另一方面，Base 模型是未经过指令微调的原始预训练版本。它们专为通过微调进行自定义而设计，允许你将其调整为特定需求。Base 模型兼容像 [Alpaca 或 Vicuna](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/chat-templates) 这样的指令式模板，但通常开箱即用时不支持会话聊天模板。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#wo-ying-gai-xuan-ze-instruct-hai-shi-base) 我应该选择 Instruct 还是 Base？ 决定通常取决于你的数据的数量、质量和类型： * **1000+ 行数据**: 如果你有一个超过 1000 行的大型数据集，通常最好对 Base 模型进行微调。 * **300–1000 行高质量数据**: 对于中等规模的高质量数据集，对 Base 或 Instruct 模型进行微调都是可行的选择。 * **少于 300 行**: 对于较小的数据集，通常选择 Instruct 模型更为合适。对 Instruct 模型进行微调可以使其与特定需求对齐，同时保留其内置的指令能力。这可确保它在不需要额外输入的情况下遵循一般指令，除非你打算显著改变其功能。 * 有关你的数据集应有多大信息， [见此处](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/datasets-guide#how-big-should-my-dataset-be) [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#shi-yong-unsloth-jin-xing-wei-tiao-mo-xing) 使用 Unsloth 进行微调模型 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 你可以通过将模型名称与 Hugging Face 上的模型名称匹配来更改为你喜欢的任何模型，例如 'unsloth/llama-3.1-8b-unsloth-bnb-4bit'。 我们建议从 **Instruct 模型**开始，因为它们允许使用会话聊天模板（ChatML、ShareGPT 等）直接进行微调，并且与 **Base 模型** （使用 Alpaca、Vicuna 等）相比需要更少的数据。了解更多关于 [instruct 与 base 模型之间差异的信息请见此处](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model) . * 以以下后缀结尾的模型名称 `**unsloth-bnb-4bit**` 表示它们是 [**Unsloth 动态 4 位**arrow-up-right](https://unsloth.ai/blog/dynamic-4bit) **量化**。这些模型比标准的 BitsAndBytes 4 位模型消耗稍多的显存，但提供显著更高的准确性。 * 如果模型名称仅以 `**bnb-4bit**`结尾，而没有包含 "unsloth"，则表示它指的是标准的 BitsAndBytes 4 位量化。 * 没有 **后缀** 的模型处于其原始的 **16 位或 8 位格式**。虽然它们是来自官方模型创建者的原始模型，但我们有时会包含重要修复——例如聊天模板或分词器修复。因此建议在可用时使用我们的版本。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#shi-yan-hen-guan-jian) 实验很关键 circle-info 我们建议在可能的情况下对两种模型进行实验。微调每个模型并评估输出，以查看哪个更符合你的目标。 [上一页Hyperparameters Guidechevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/lora-hyperparameters-guide) [下一页Tutorial: Finetune Llama-3 and Use In Ollamachevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/tutorial-how-to-finetune-llama-3-and-use-in-ollama) 最后更新于 3个月前 这有帮助吗？ * [Llama、Qwen、Mistral、Phi 还是？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#llamaqwenmistralphi-hai-shi) * [Instruct 模型还是 Base 模型？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#instruct-mo-xing-hai-shi-base-mo-xing) * [Instruct 模型](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#instruct-mo-xing) * [Base 模型](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#base-mo-xing) * [我应该选择 Instruct 还是 Base？](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#wo-ying-gai-xuan-ze-instruct-hai-shi-base) * [使用 Unsloth 进行微调模型](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#shi-yong-unsloth-jin-xing-wei-tiao-mo-xing) * [实验很关键](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide/what-model-should-i-use#shi-yan-hen-guan-jian) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 如何使用 Unsloth 在 Windows 上微调 LLM（分步指南） | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 现在你可以直接在本地 Windows 设备上微调模型，而无需 WSL，方法是使用 [Unslotharrow-up-right](https://github.com/unslothai/unsloth) 。本指南提供 3 种主要方法供你使用（[Conda](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-1-windows-via-conda) , [Docker](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-2-docker) 和 [WSL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-3-wsl) ）。 如果你已经在 Windows 上安装了 PyTorch， `pip install unsloth` 应该可以正常工作。否则，请按照下面的指南： [Conda 教程](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-1-windows-via-conda) [Docker 教程](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-2-docker) [WSL 教程](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#method-3-wsl) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#unsloth-studio) Unsloth Studio 我们推出了一个名为 [Unsloth Studio](https://unsloth.ai/docs/zh/xin/studio/install) 的新 Web UI，可直接在 Windows 上使用： 复制 irm https://unsloth.ai/install.ps1 | iex 使用相同的命令进行更新，或者使用 `unsloth studio update`. 然后每次启动时： 复制 unsloth studio -H 0.0.0.0 -p 8888 有关 Unsloth Studio 的详细安装说明和要求， [请查看我们的指南](https://unsloth.ai/docs/zh/xin/studio/install) . 下面是原始 **Unsloth 核心**: ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#fang-fa-1-tong-guo-conda-zai-windows-shang-an-zhuang) 方法 #1 - 通过 Conda 在 Windows 上安装： 1 **安装 Miniconda（或 Anaconda）** 下载 Anaconda [这里arrow-up-right](https://www.anaconda.com/download) 。我们的建议是使用 [Minicondaarrow-up-right](https://www.anaconda.com/docs/getting-started/miniconda/install#quickstart-install-instructions) 。要使用它，先打开 Powershell——在开始菜单中搜索“Windows Powershell”： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FvCgJ3hTR5ChVmCR1ndAh%252Fimage.png%3Falt%3Dmedia%26token%3Dbcabe210-793f-40ae-944a-a349dddc8c35&width=768&dpr=3&quality=100&sign=fa6feadb&sv=2) 然后它会打开 Powershell： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fm7J0b8Qz5W2aGktt3KA9%252Fimage.png%3Falt%3Dmedia%26token%3Df84327e4-408f-492c-a909-982ed458f393&width=768&dpr=3&quality=100&sign=acf43ba&sv=2) 然后复制粘贴下面内容：CTRL+C，并将其粘贴到 Powershell 中 CTRL+V： 复制 Invoke-WebRequest -Uri "https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe" -OutFile ".\miniconda.exe" Start-Process -FilePath ".\miniconda.exe" -ArgumentList "/S" -Wait del .\miniconda.exe 接受警告并点击“仍要粘贴”，然后等待。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZCFxO1FrYGk7sV7AmCe8%252Fimage.png%3Falt%3Dmedia%26token%3Df753dbdb-efa9-462c-875b-0a18509a10cf&width=768&dpr=3&quality=100&sign=e59acb3b&sv=2) 它正在下载如下所示的安装程序： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2TzsqlGZyUI4wBT0hXW9%252Fimage.png%3Falt%3Dmedia%26token%3Da680690a-3179-4525-bf83-0163424b5ddc&width=768&dpr=3&quality=100&sign=268f61d&sv=2) 安装完成后，打开 **Anaconda Powershell Prompt** 通过开始菜单 -> 搜索它来使用 Miniconda： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWoQSFFfmB26WT6BdzrvJ%252Fimage.png%3Falt%3Dmedia%26token%3D747c6c4e-f676-4927-abad-cb667e757309&width=768&dpr=3&quality=100&sign=a1c26bc0&sv=2) 然后你会看到： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FNTTeflwW9Gw7lxJdCUcy%252Fimage.png%3Falt%3Dmedia%26token%3D77f093b7-9fd5-47f2-856d-18f238f5a95e&width=768&dpr=3&quality=100&sign=d283d3a1&sv=2) 2 **创建 conda 环境** 复制 conda create --name unsloth_env python==3.12 -y conda activate unsloth_env **你会看到：** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEFYV6IhOeXIDbzxYHzEJ%252Fimage.png%3Falt%3Dmedia%26token%3De3452b73-cfd2-4148-a735-cfe400369c17&width=768&dpr=3&quality=100&sign=e1490a58&sv=2) 3 **检查** `**nvidia-smi**` **以确认你有 GPU，并查看 CUDA 版本** 输入 `nvidia-smi` 到 Powershell 后，你应该会看到类似如下内容。如果你没有 `nvidia-smi` 或者下面的内容无法弹出，你需要重新安装 [NVIDIA 驱动程序arrow-up-right](https://www.nvidia.com/en-us/drivers/) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1CMAzx7LX8LEc8GHy1I9%252Fimage.png%3Falt%3Dmedia%26token%3Da0ad52e3-be17-4dc4-ae97-ba400a639098&width=768&dpr=3&quality=100&sign=943d8e60&sv=2) 4 **安装 PyTorch** 运行 `nvidia-smi` 时，你会在右上角看到：“CUDA Version: 13.0”。在 PowerShell 中安装 PyTorch。将 `130` 更改为你的 CUDA 版本——确保该 [版本存在arrow-up-right](https://pytorch.org/) 并且与您的 CUDA 驱动程序版本匹配。 复制 pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130 你会看到： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fo4aNeIbILGYvfpjYa1X2%252Fimage.png%3Falt%3Dmedia%26token%3D444d6907-04e0-4d5e-8de5-d8cdcaf85364&width=768&dpr=3&quality=100&sign=840104ca&sv=2) 在 Python 中尝试运行此命令： `python` 在安装 PyTorch 后： 复制 import torch print(torch.cuda.is_available()) A = torch.ones((10, 10), device = "cuda") B = torch.ones((10, 10), device = "cuda") A @ B 你应该会看到一个由 10 组成的矩阵。同时检查第一个结果是否为 True。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJkf39Nyfgyt4QmTLGhOx%252Fimage.png%3Falt%3Dmedia%26token%3D2d1a16a6-e524-461a-ac75-a24b6bda333f&width=768&dpr=3&quality=100&sign=d058a76b&sv=2) 5 **安装 Unsloth（仅当 PyTorch 正常工作时！）** triangle-exclamation **确认 PyTorch 能正常工作并运行——如果不能，那就是 PyTorch 出问题了，这意味着你的 Windows 机器可能不幸需要重新安装 CUDA 驱动程序。** 在 Powershell 中（通过以下方式退出 Python 后 `exit()` ，执行并等待： 复制 pip install unsloth 6 **验证 Unsloth 是否正常工作** 现在使用 [Unsloth 笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) 中的任意脚本（保存为 .py 文件），或者使用下面的基础脚本： 你应该会看到： 复制 🦥 Unsloth：将为你的计算机打补丁，以实现快 2 倍的免费微调。 🦥 Unsloth Zoo 现在将为所有内容打补丁，让训练更快！ ==((====))== Unsloth 2026.1.4：快速 Gemma3 打补丁。Transformers: 4.57.6. \\ /| NVIDIA GeForce RTX 3060。GPU 数量 = 1。最大显存：12.0 GB。平台：Windows。 O^O/ \_/ \ Torch: 2.10.0+cu130。CUDA: 8.6。CUDA Toolkit: 13.0。Triton: 3.6.0 \ / Bfloat16 = TRUE。FA [Xformers = 0.0.34. FA2 = False] "-____-" 免费许可：http://github.com/unslothai/unsloth Unsloth：已启用快速下载——请忽略红色的下载进度条！ Unsloth：Gemma3 不支持 SDPA——正在切换到 fast eager。 Unsloth：使 `model.base_model.model.model` 需要梯度 Unsloth：正在对 ["text"] 进行分词（num_proc=1）： 0%| | 0/210289 [00:00 下载 .ipynb，然后加载它们。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FVbqNWsG2CCHKJJjrnU4s%252Funknown.png%3Falt%3Dmedia%26token%3D854a6d0e-fc84-4e44-bf8e-4bf254801692&width=300&dpr=3&quality=100&sign=e388efdd&sv=2) circle-exclamation 如果你正在使用 GRPO 或计划使用 vLLM，目前 vLLM 不直接支持 Windows，但可通过 WSL 或 Linux 使用。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#gu-zhang-pai-cha-gao-ji) **故障排查 /** 高级 对于 **高级安装说明** ，或如果你在安装过程中看到奇怪的错误： 1. 安装 `torch` 和 `triton`。前往 https://pytorch.org 进行安装。例如 `pip install torch torchvision torchaudio triton` 2. 确认 CUDA 是否已正确安装。尝试 `nvcc`。如果失败，你需要安装 `cudatoolkit` 或 CUDA 驱动程序。 3. 如果使用 Intel GPU，你需要遵循我们的 [Intel Windows 指南](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#windows-only-runtime-configurations) 4. 安装 `xformers` 手动安装。你可以尝试安装 `vllm` 并查看是否 `vllm` 成功。检查是否 `xformers` 通过以下命令成功： `python -m xformers.info` 前往 https://github.com/facebookresearch/xformers。另一个选择是为 Ampere GPU 安装 `flash-attn` 。 5. 仔细检查你的 Python、CUDA、CUDNN、 `torch`, `triton`以及 `xformers` 版本彼此兼容。 [PyTorch 兼容性矩阵arrow-up-right](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) 可能会有帮助。 6. 最后，安装 `bitsandbytes` 并通过以下命令检查它 `python -m bitsandbytes` 7. 如果 Unsloth 没有检测到或使用你的 GPU，并且你在 Windows 上使用我们的 Docker 容器，那么你的 CUDA toolkit 版本 `nvcc --version` 应与主机上 nvidia-smi 显示的 CUDA 版本一致；Windows 上的 Docker 容器 GPU 支持不是自动的。 [你需要遵循 Docker 的指南arrow-up-right](https://docs.docker.com/desktop/features/gpu/) . ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#xie-zai-unsloth-studio) 卸载 Unsloth Studio 要在 Windows 上卸载 Unsloth Studio，请按照以下 4 个步骤： #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#id-1.-shan-chu-ying-yong-cheng-xu) **1\. 删除应用程序** * Windows（PowerShell）： `Remove-Item -Recurse -Force "$HOME\.unsloth\studio\unsloth", "$HOME\.unsloth\studio\studio"` 这会删除应用程序，但会保留你的模型检查点、导出、历史记录、缓存和聊天内容。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#id-2.-shan-chu-kuai-jie-fang-shi-he-fu-hao-lian-jie) **2\. 删除快捷方式和符号链接** **WSL / Windows（PowerShell）：** #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#id-3.-shan-chu-cli-ming-ling) **3\. 删除 CLI 命令** **WSL：** **Windows（PowerShell）：** 安装程序已将 venv 的 `Scripts` 目录添加到你的用户 PATH 中。要将其删除，请打开 设置 → 系统 → 关于 → 高级系统设置 → 环境变量，找到 `Path` （位于用户变量下），并删除指向 `.unsloth\studio\...\Scripts`. #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#id-4.-shan-chu-suo-you-nei-rong-ke-xuan) **4\. 删除所有内容（可选）** 如果你还想删除历史记录、缓存、聊天内容、模型检查点和模型导出，请删除整个 Unsloth 文件夹： * WSL，Linux： `rm -rf ~/.unsloth` * Windows（PowerShell）： `Remove-Item -Recurse -Force "$HOME\.unsloth"` 请注意，下载的 HF 模型文件会单独存储在 Hugging Face 缓存中——以上步骤都不会删除它们。请参见 **删除模型文件** 如果你想回收那部分磁盘空间，请查看下面内容。 [上一页MacOSchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/mac) [下一页Dockerchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/docker) 最后更新于 3天前 这有帮助吗？ * [Unsloth Studio](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#unsloth-studio) * [方法 #1 - 通过 Conda 在 Windows 上安装：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#fang-fa-1-tong-guo-conda-zai-windows-shang-an-zhuang) * [方法 #2 - Docker：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#fang-fa-2-docker) * [方法 #3 - WSL：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#fang-fa-3-wsl) * [故障排查 / 高级](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#gu-zhang-pai-cha-gao-ji) * [卸载 Unsloth Studio](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/windows-installation#xie-zai-unsloth-studio) 这有帮助吗？ sun-brightdesktopmoon 复制 from unsloth import FastLanguageModel, FastModel import torch from trl import SFTTrainer, SFTConfig from datasets import load_dataset max_seq_length = 512 url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl" dataset = load_dataset("json", data_files = {"train" : url}, split = "train") model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/gemma-3-270m-it", max_seq_length = max_seq_length, # 长上下文可任选！ load_in_4bit = True, # 4 位量化。False = 16 位 LoRA。 load_in_8bit = False, # 8 位量化 load_in_16bit = False, # 16 位 LoRA full_finetuning = False, # 用于全量微调。 trust_remote_code = False, # 启用以支持新模型 # token = "hf_...", # 如果使用受限模型，请使用一个 ) # 执行模型补丁并添加快速 LoRA 权重 model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # 支持任意值，但 = 0 已优化 bias = "none", # 支持任意值，但 = "none" 已优化 # [NEW] “unsloth” 可减少 30% 显存占用，可容纳 2 倍更大的批量大小！ use_gradient_checkpointing = "unsloth", # 对于超长上下文，可设为 True 或 "unsloth" random_state = 3407, max_seq_length = max_seq_length, use_rslora = False, # 我们支持秩稳定 LoRA loftq_config = None, # 也支持 LoftQ ) trainer = SFTTrainer( model = model, train_dataset = dataset, tokenizer = tokenizer, args = SFTConfig( max_seq_length = max_seq_length, per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 10, max_steps = 60, logging_steps = 1, output_dir = "outputs", optim = "adamw_8bit", seed = 3407, dataset_num_proc = 1, ), ) trainer.train() chevron-down显示全部 54 行 复制 wsl.exe --install Ubuntu-24.04 wsl.exe -d Ubuntu-24.04 复制 wsl 复制 sudo apt update sudo apt install python3 python3-full python3-pip python3-venv -y 复制 pip install torch torchvision --force-reinstall --index-url https://download.pytorch.org/whl/cu130 复制 pip install unsloth jupyter 复制 jupyter notebook 复制 Remove-Item -Force "$HOME\Desktop\Unsloth Studio.lnk" Remove-Item -Force "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk" 复制 rm -f ~/.local/bin/unsloth sun-brightdesktopmoon --- # Unsloth 笔记本 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 使用我们基于免费 GPU 算力的 notebooks 训练你自己的模型。点击“运行全部”（或本地保存），添加你的数据集，进行训练并部署。你可以在 notebooks 中使用任意模型。 [GRPO（RL）](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-reasoning-rl) [文本转语音](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#text-to-speech-tts) [视觉](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#vision-multimodal) [嵌入](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#embedding-models) [Kaggle](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#kaggle-notebooks) 另请查看我们的 GitHub 仓库以获取我们的 notebooks： [github.com/unslothai/notebooksarrow-up-right](https://github.com/unslothai/notebooks/) [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#colab-notebooks) Colab notebooks ------------------------------------------------------------------------------------------------------------- **介绍我们的** [**Unsloth Studio**](https://unsloth.ai/docs/zh/xin/studio) ✨ **notebook。** 训练和运行 22B 参数以下的模型： [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2Fc04527d1807929a628269bb0e1319bf2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=48a6a277&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#biao-zhun-sft-notebooks) 标准 SFT notebooks： * [**Gemma 4**](https://unsloth.ai/docs/zh/mo-xing/gemma-4/train) **:** [E4B **（视觉）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E4B)-Vision.ipynb) **•** [E2B **（文本）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Text.ipynb) **•** [E2B **（音频）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Audio.ipynb) **•** [**31B** （Kaggle）arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) **•** [**推理**arrow-up-right](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) * [**Qwen3.5**](https://unsloth.ai/docs/zh/mo-xing/qwen3.5/fine-tune) **:** [**0.8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(0_8B)_Vision.ipynb) • [**2B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(2B)_Vision.ipynb) • [**4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision.ipynb) * [gpt-oss（20b）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) • [推理arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_(20B)-Inference.ipynb) • [微调arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) * [EmbeddingGemma（300M）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) * [Qwen3（14B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) • [**Qwen3-VL（8B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) * [**Qwen3-2507-4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) • [思考arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Thinking.ipynb) • [指令arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) * [Gemma 3（4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) • [文本arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) • [视觉arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) • [270Marrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) • [**FunctionGemma**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) * [Gemma 3n（E4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) • [文本arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) • [视觉arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) • [音频arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) * [**Mistral Ministral 3**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) * [**DeepSeek-OCR 2**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_2_(3B).ipynb) * [IBM Granite-4.0-Harrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [Phi-4（14B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb) * [Llama 3.1（8B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) • [Llama 3.2（1B + 3B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-tui-li-rl) GRPO（推理 RL）： * [**Qwen3.5（4B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision_GRPO.ipynb) \- 视觉 GRPO - 新 * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) （自动创建 kernel） * [Mistral Ministral 3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb) （解数独）- 新 * [Qwen3-8B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) （L4）- 新 * [Llama-3.2-1B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama_FP8_GRPO.ipynb) （L4）- 新 * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) （自动赢得 2048 游戏） * [Qwen3-VL（8B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - 视觉 GSPO * [Qwen3（4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) - 高级 GRPO LoRA * [Gemma 3（4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) - 视觉 GSPO * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) （2048 OpenEnv 示例） * [DeepSeek-R1-0528-Qwen3（8B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) （用于多语言场景） * [Gemma 3（1B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) * [Llama 3.2（3B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_(3B)_GRPO_LoRA.ipynb) - 高级 GRPO LoRA * [Llama 3.1（8B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb) * [Phi-4（14B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) * [Mistral v0.3（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-GRPO.ipynb) * [NeMo Gym 多智能体环境 arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb) （多个智能体环境） ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#wen-ben-zhuan-yu-yin-tts) 文本转语音（TTS）： * [Sesame-CSM（1B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb) * [Orpheus-TTS（3B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) - 语音转文本（STT） * [Llasa-TTS（1B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_(1B).ipynb) * [Spark-TTS（0.5B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb) * [Oute-TTS（1B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb) **语音转文本（SST）：** * [**Gemma 4（E2B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Audio.ipynb) **- 音频 - 新** * [Whisper-Large-V3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) * [Gemma 3n（E4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) - 音频 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#shi-jue-duo-mo-tai) 视觉（多模态）： * [**Gemma 4**](https://unsloth.ai/docs/zh/mo-xing/gemma-4/train) **:** [E2Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Vision.ipynb) - 新 * [**Qwen3.5**](https://unsloth.ai/docs/zh/mo-xing/qwen3.5/fine-tune) **:** [**0.8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(0_8B)_Vision.ipynb) • [**2B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(2B)_Vision.ipynb) • [**4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision.ipynb) - 新 * [**Qwen3-VL（8B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) * [**Mistral Ministral 3**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) * [**DeepSeek-OCR**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb) * [**Paddle-OCR（1B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Paddle_OCR_(1B)_Vision.ipynb) * [Gemma 3n（E4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) * [Gemma 3（4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) * [Llama 3.2 Vision（11B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb) * [Qwen2.5-VL（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral（12B）2409arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Pixtral_(12B)-Vision.ipynb) * [Qwen3-VLarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - 视觉 GSPO - 新 * [Qwen2.5-VLarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) - 视觉 GSPO * [Gemma 3（4B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) - 视觉 GSPO ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qian-ru-mo-xing) 嵌入模型： * [EmbeddingGemma（300M）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) - 新 * [Qwen3-Embedding 4Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(4B).ipynb) - 新 * [Qwen3-Embedding 0.6Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(0_6B).ipynb) - 新 * [BGE M3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/BGE_M3.ipynb) - 新 * [ModernBERT-largearrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) - 新 * [All-MiniLM-L6-v2arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/All_MiniLM_L6_v2.ipynb) - 新 * [GTE ModernBertarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ModernBert.ipynb) - 新 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#da-xing-llm) 大型 LLM： **用于大型模型的 notebooks：** 这些模型超出了 Colab 免费 15 GB VRAM 档位。使用 Colab 新的 80 GB GPU，你可以微调 120B 参数模型。 circle-info 需要 Colab 订阅或积分。我们 **不** 从这些 notebooks 中赚取任何收益。 * [**Gemma-4-31B** （Kaggle）arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - 新且 **免费** * [Gemma-4-26B-A4Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(26B_A4B)-Vision.ipynb) - 新 * [Gemma-4-31Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(31B)-Vision.ipynb) - 新 * [Qwen3.5-35B-A3Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_MoE.ipynb) * [Qwen3.5‑27Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen_3_5_27B_A100(80GB).ipynb) * [GLM-4.7-Flasharrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100(80GB).ipynb) * [gpt-oss-20b（50万上下文）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) * [Qwen3-30B-A3Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_MoE.ipynb) * [Nemotron-3-Nano-30B-A3B LoRA notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Nemotron-3-Nano-30B-A3B_A100.ipynb) * [NeMo Gym 数独 GRPO notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Sudoku.ipynb) * [NeMo Gym 多环境 GRPO notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb) * [gpt-oss-120barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(120B)_A100-Fine-tuning.ipynb) * [Qwen3（32B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(32B)_A100-Reasoning-Conversational.ipynb) * [Llama 3.3（70B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.3_(70B)_A100-Conversational.ipynb) * [Gemma 3（27B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(27B)_A100-Conversational.ipynb) * [Baidu ERNIE 4.5 VL（28B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ERNIE_4_5_VL_28B_A3B_PT_Vision.ipynb) - 新 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qi-ta-zhong-yao-de-notebooks) 其他重要的 notebooks： * [**客服代理**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [Mistral Ministral 3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb) - 新（解数独） * [在 LM Studio 上部署 arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-LMStudio.ipynb) \- 新 * [量化感知训练arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) （QAT）- 新 * [手机部署 arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(0_6B)-Phone_Deployment.ipynb) \- 新 * [先思考 **工具调用** notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) - 新 * [移动操作 notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb) - 新 * [**自动创建 Kernel**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) 使用 RL * [**ModernBERT-large**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) **\- 新** 8月19日 * [**合成数据生成 Llama 3.2（3B）**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) * [gpt-oss-20b（50万上下文）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) \- 新（A100） * [**工具调用**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) * [Mistral v0.3 Instruct（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Conversational.ipynb) * [Ollamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [ORPOarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-ORPO.ipynb) * [持续预训练arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyrarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [_**仅推理**_arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Inference.ipynb) * [Llama 3（8B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Alpaca.ipynb) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#te-ding-yong-li-notebooks) 特定用例 notebooks： * [手机部署 arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(0_6B)-Phone_Deployment.ipynb) \- 新 * [在 LM Studio 上部署 arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-LMStudio.ipynb) \- 新 * [先思考 **工具调用**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) - 新 * [移动操作arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb) - 新 * [**客服代理**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [量化感知训练arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) （QAT）- 新 * [**自动创建 Kernel**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) 使用 RL **\- 新** * [DPO Zephyrarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [BERT - 文本分类arrow-up-right](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) -（AutoModelForSequenceClassification） * [Ollamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [**工具调用**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) * [持续预训练（CPT）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [多个数据集arrow-up-right](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) 由 Flail 提供 * [KTOarrow-up-right](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) 由 Jeffrey 提供 * [推理聊天 UIarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Unsloth_Studio.ipynb) * [对话式arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [ChatMLarrow-up-right](https://colab.research.google.com/drive/15F1xyn8497_dUbxZP4zWmPZ3PJx1Oymv?usp=sharing) * [文本补全arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qi-yu-notebooks) 其余 notebooks： * [Qwen2.5（3B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) * [Gemma 2（9B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(9B)-Alpaca.ipynb) * [Mistral NeMo（12B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Nemo_(12B)-Alpaca.ipynb) * [Phi-3.5（mini）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3.5_Mini-Conversational.ipynb) * [Phi-3（medium）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3_Medium-Conversational.ipynb) * [Gemma 2（2B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(2B)-Alpaca.ipynb) * [Qwen 2.5 Coder（14B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(14B)-Conversational.ipynb) * [Mistral Small（22B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/TinyLlama_(1.1B)-Alpaca.ipynb) * [CodeGemma（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/CodeGemma_(7B)-Conversational.ipynb) * [Mistral v0.3（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Alpaca.ipynb) * [Qwen2（7B）arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_(7B)-Alpaca.ipynb) [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#kaggle-notebooks) Kaggle notebooks --------------------------------------------------------------------------------------------------------------- #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#biao-zhun-notebooks) 标准 notebooks： * [**Gemma-4-31B** （Kaggle）arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - 新且 **免费** * [**gpt-oss（20B）**arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-gpt-oss-(20B)-Fine-tuning.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3n（E4B）arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma-3n-4b-multimodal-finetuning-inference) * [Qwen3（14B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(14B).ipynb) * [Magistral-2509（24B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_(24B)-Reasoning-Conversational.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3（4B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(4B).ipynb) * [Phi-4（14B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4-Conversational.ipynb) * [Llama 3.1（8B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Alpaca.ipynb) * [Llama 3.2（1B + 3B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(1B_and_3B)-Conversational.ipynb) * [Qwen 2.5（7B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(7B)-Alpaca.ipynb) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-tui-li-notebooks) GRPO（推理）notebooks： * [**Qwen2.5-VL**arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2_5_7B_VL_GRPO.ipynb&accelerator=nvidiaTeslaT4) - 视觉 GRPO - 新 * [Qwen3（4B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(4B)-GRPO.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3（1B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(1B)-GRPO.ipynb) * [Llama 3.1（8B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-GRPO.ipynb) * [Phi-4（14B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4_(14B)-GRPO.ipynb) * [Qwen 2.5（3B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(3B)-GRPO.ipynb) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#wen-ben-zhuan-yu-yin-ttsnotebooks) 文本转语音（TTS）notebooks： * [Sesame-CSM（1B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Sesame_CSM_(1B)-TTS.ipynb) * [Orpheus-TTS（3B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Whisper.ipynb) – 语音转文本 * [Llasa-TTS（1B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llasa_TTS_(1B).ipynb) * [Spark-TTS（0.5B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Spark_TTS_(0_5B).ipynb) * [Oute-TTS（1B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Oute_TTS_(1B).ipynb) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#shi-jue-duo-mo-tai-notebooks) 视觉（多模态）notebooks： * [Llama 3.2 Vision（11B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(11B)-Vision.ipynb) * [Qwen 2.5-VL（7B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral（12B）2409arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Pixtral_(12B)-Vision.ipynb) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#te-ding-yong-li-notebooks-1) 特定用例 notebooks： * [工具调用arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb&accelerator=nvidiaTeslaT4) * [ORPOarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-ORPO.ipynb) * [持续预训练arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyrarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Zephyr_(7B)-DPO.ipynb) * [仅推理arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Inference.ipynb) * [Ollamaarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-Ollama.ipynb) * [文本补全arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_(7B)-Text_Completion.ipynb) * [CodeForces-cot（推理）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeForces-cot-Finetune_for_Reasoning_on_CodeForces.ipynb) * [Unsloth Studio（聊天 UI）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Unsloth_Studio.ipynb) #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qi-yu-notebooks-1) 其余 notebooks： * [Gemma 2（9B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(9B)-Alpaca.ipynb) * [Gemma 2（2B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(2B)-Alpaca.ipynb) * [CodeGemma（7B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeGemma_(7B)-Conversational.ipynb) * [Mistral NeMo（12B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Nemo_(12B)-Alpaca.ipynb) * [Mistral Small（22B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlama（1.1B）arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-TinyLlama_(1.1B)-Alpaca.ipynb) 要查看我们所有 Kaggle notebooks 的完整列表， [点击这里arrow-up-right](https://github.com/unslothai/notebooks#-kaggle-notebooks) . circle-info 欢迎通过访问我们的 [仓库arrow-up-right](https://github.com/unslothai/notebooks) ! [上一页FAQ + 微调适合我吗？chevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) [下一页All Our Modelschevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-model-catalog) 最后更新于 3天前 这有帮助吗？ * [Colab notebooks](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#colab-notebooks) * [标准 SFT notebooks：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#biao-zhun-sft-notebooks) * [GRPO（推理 RL）：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-tui-li-rl) * [文本转语音（TTS）：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#wen-ben-zhuan-yu-yin-tts) * [视觉（多模态）：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#shi-jue-duo-mo-tai) * [嵌入模型：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qian-ru-mo-xing) * [大型 LLM：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#da-xing-llm) * [其他重要的 notebooks：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qi-ta-zhong-yao-de-notebooks) * [特定用例 notebooks：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#te-ding-yong-li-notebooks) * [其余 notebooks：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#qi-yu-notebooks) * [Kaggle notebooks](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#kaggle-notebooks) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # RL 奖励作弊 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 强化学习的最终目标是最大化某种回报（比如速度、收益、某个指标）。但强化学习可能会 **作弊。** 当强化学习算法学会某个技巧或利用某些手段来增加回报，而实际上并没有完成最终任务时，这被称为“**回报操纵**". 这就是模型学会修改单元测试以通过编程挑战的原因，这些问题是现实部署的关键障碍。另一些很好的例子来自于 [维基百科arrow-up-right](https://en.wikipedia.org/wiki/Reward_hacking) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fi.pinimg.com%2Foriginals%2F55%2Fe0%2F1b%2F55e01b94a9c5546b61b59ae300811c83.gif&width=768&dpr=3&quality=100&sign=9247e4cc&sv=2) **可以对抗回报操纵吗？可以！** 在我们的 [免费 gpt-oss 强化学习笔记本arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) 中，我们探讨了如何在代码生成环境中对抗回报操纵，并展示了针对常见错误模式的切实可行的解决方案。我们看到模型会编辑计时函数、外包给其他库、缓存结果，甚至直接作弊。经过对抗之后，结果是我们的模型生成真正优化过的矩阵乘法内核，而不是巧妙的作弊手段。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking#hui-bao-cao-zong-gai-shu) 🏆 回报操纵概述 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 强化学习中一些常见的回报操纵示例包括： #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking#lan-duo) 懒惰 强化学习学会使用 Numpy、Torch 等库，这些库调用了优化过的 CUDA 内核。我们可以通过检查生成的代码是否导入了其他非标准的 Python 库来阻止强化学习算法调用优化代码。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking#huan-cun-yu-zuo-bi) 缓存与作弊 强化学习学会缓存输出结果，并且通过检查 Python 全局变量来找到实际输出。 我们可以通过用一个大的伪矩阵清空缓存来阻止强化学习算法使用缓存数据。我们还必须通过多个循环和轮次来仔细基准测试。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking#zuo-bi) 作弊 强化学习学会编辑计时函数，使其输出为 0 秒。我们可以通过限制其 `局部变量` 和 `全局变量`来阻止强化学习算法使用全局或缓存变量。我们还将使用 `exec` 来创建函数，因此我们必须将输出保存到一个空字典中。我们还通过以下方式禁止全局变量访问 `types.FunctionType(f.__code__, {})`\\ [上一页GSPO RLchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning) [下一页RL 中的 FP16 与 BF16chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl) 最后更新于 2个月前 这有帮助吗？ 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # GSPO 强化学习 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 我们引入了 GSPO，这是一种变体 [GRPO](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr) 由阿里巴巴的 Qwen 团队提出。他们注意到一个观察：当 GRPO 为每个标记采用重要性权重时，尽管优势本质上并不会随每个标记而缩放或改变。这导致了 GSPO 的创建，GSPO 将重要性分配到序列似然性上，而不是标记的单独似然性。 * 使用我们的免费 GSPO 笔记本用于： [**gpt-oss-20b**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) 和 [**Qwen2.5-VL**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) 通过在 Unsloth 中设置来启用 GSPO `importance_sampling_level = "sequence"` 在 GRPO 配置中。下面可以看到这两种算法的差异，来自 Qwen 和阿里巴巴的 GSPO 论文： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-45d743dd5dcd590626777ce09cfab61808aa8c24%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=a9867df0&sv=2) GRPO 算法，来源： [Qwenarrow-up-right](https://arxiv.org/abs/2507.18071) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ee755850cbe17482ce240dde227d55c62e9a3e64%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=29b997d1&sv=2) GSPO 算法，来源： [Qwenarrow-up-right](https://arxiv.org/abs/2507.18071) 在式（1）中可以看到，优势将每一行按比例缩放到标记对数概率上，然后对该张量求和。实质上，每个标记被赋予相同的缩放，尽管该缩放是给整个序列而不是每个单独标记的。下面可以看到一个简单的示意图： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-b3c944808a15dde0a7ff45782f9f074993304bf1%252FCopy%2520of%2520GSPO%2520diagram%2520%281%29.jpg%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d23c70e8&sv=2) GRPO 对数概率比按行用优势缩放 式（2）表明，对数概率比在计算后对每个序列求和并取指数，只对得到的序列比值按行乘以优势。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-62fc5b50921e79cce155d2794201c9b96faf941e%252FGSPO%2520diagram%2520%281%29.jpg%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=55a91f4e&sv=2) GSPO 序列比按行用优势缩放 启用 GSPO 很简单，你只需在 GRPO 配置中设置 `importance_sampling_level = "sequence"` 标志。 [上一页Advanced RL Docschevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation) [下一页RL 奖励作弊chevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking) 最后更新于 2个月前 这有帮助吗？ 这有帮助吗？ sun-brightdesktopmoon 复制 training_args = GRPOConfig( output_dir = "vlm-grpo-unsloth", per_device_train_batch_size = 8, gradient_accumulation_steps = 4, learning_rate = 5e-6, adam_beta1 = 0.9, adam_beta2 = 0.99, weight_decay = 0.1, warmup_ratio = 0.1, lr_scheduler_type = "cosine", optim = "adamw_8bit", # beta = 0.00, epsilon = 3e-4, epsilon_high = 4e-4, num_generations = 8, max_prompt_length = 1024, max_completion_length = 1024, log_completions = False, max_grad_norm = 0.1, temperature = 0.9, # report_to = "none", # 如果你想记录到 Weights & Biases，请设置为 "wandb" num_train_epochs = 2, # 用于快速测试运行，完整训练请增加 report_to = "none" # GSPO 如下： importance_sampling_level = "sequence", # Dr GRPO / GAPO 等 loss_type = "dr_grpo", ) sun-brightdesktopmoon --- # RL 中的 FP16 与 BF16 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#float16-yu-bfloat16) Float16 与 Bfloat16 有一篇论文标题为 "**通过 FP16 克服训练-推理不匹配**" [https://arxiv.org/pdf/2510.26788arrow-up-right](https://arxiv.org/pdf/2510.26788) 展示了在进行强化学习时使用 float16 精度相比使用 bfloat16 可以显著更好。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Frec4qe1aQS0xyMzGvS9c%252Fimage.png%3Falt%3Dmedia%26token%3D2137e766-0f1f-48ec-b25f-2292d6f149f4&width=768&dpr=3&quality=100&sign=bd9d6d10&sv=2) 实际上，生成长度越长，使用 bfloat16 时情况越糟： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWs7ioB2lraTbDbUCOAnn%252Fimage.png%3Falt%3Dmedia%26token%3Dac2b4f8e-210f-4bcc-bcbb-6e68f80781a6&width=768&dpr=3&quality=100&sign=16c9f375&sv=2) 我们进行了调查，且 **确实发现 float16 更稳定** 比 bfloat16 具有更小得多的梯度范数，见 [https://x.com/danielhanchen/status/1985557028295827482arrow-up-right](https://x.com/danielhanchen/status/1985557028295827482) 以及 [https://x.com/danielhanchen/status/1985562902531850472arrow-up-right](https://x.com/danielhanchen/status/1985562902531850472) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhvQ1W5wtV6TTfsetp7y2%252FG44d7ZFbIAANBBd.jpg%3Falt%3Dmedia%26token%3D35181a07-de3e-4321-b54e-4436b4a201ff&width=768&dpr=3&quality=100&sign=f8e75b17&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F62HkxnGcaKvxnSxbZMZu%252FG44c20SbwAAGo8j.jpg%3Falt%3Dmedia%26token%3De0c7ecb8-6f0c-4ecf-b1a0-50f1b2a9a807&width=768&dpr=3&quality=100&sign=12ed7a1b&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fsi18IkGqE4IuUvzroyHh%252FG44ix5FbQAM0L5l.jpg%3Falt%3Dmedia%26token%3Dbc3b97ce-5df4-4b69-aa50-a8e339f21601&width=768&dpr=3&quality=100&sign=a55e799b&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#a100-ji-lian-zhu-yi-li-cuo-wu) 🤯A100 级联注意力错误 根据 [https://x.com/RichardYRLi/status/1984858850143715759arrow-up-right](https://x.com/RichardYRLi/status/1984858850143715759) 以及 [https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15eddaarrow-up-right](https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda) ，较旧的 vLLM 版本（0.11.0 之前）在 A100 和类似 GPU 上存在损坏的注意力机制。请更新 vLLM！如果我们检测到较旧的 vLLM 版本，在 Unsloth 强化学习期间我们默认也会禁用 vLLM 的级联注意力。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FnkCLRVIIGLADXBSCe58e%252Fimage.png%3Falt%3Dmedia%26token%3D6669642f-8690-44bf-b2de-6aa89acf2332&width=768&dpr=3&quality=100&sign=4a5421b9&sv=2) 不同硬件也会改变结果，较新且更昂贵的 GPU 在推理端与训练端之间的 KL 差异更小： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FaroTTz68zzyofy6nagtH%252Fimage.webp%3Falt%3Dmedia%26token%3D3be09506-b8a0-42eb-8d17-af72496a9cd1&width=768&dpr=3&quality=100&sign=7cd915bf&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#zai-unsloth-rl-zhong-shi-yong-float16) 🔥在 Unsloth RL 中使用 float16 要在 Unsloth GRPO 和 RL 中使用 float16 精度，你只需设置 `dtype = torch.float16` 我们会处理剩下的！ [上一页RL 奖励作弊chevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking) [下一页内存高效 RLchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl) 最后更新于 2个月前 这有帮助吗？ * [Float16 与 Bfloat16](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#float16-yu-bfloat16) * [🤯A100 级联注意力错误](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#a100-ji-lian-zhu-yi-li-cuo-wu) * [🔥在 Unsloth RL 中使用 float16](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl#zai-unsloth-rl-zhong-shi-yong-float16) 这有帮助吗？ sun-brightdesktopmoon 复制 pip install unsloth vllm import torch max_seq_length = 2048 # 对于更长的推理轨迹可以增加 lora_rank = 32 # 更大的秩 = 更智能，但更慢 from unsloth import FastLanguageModel model_name = "unsloth/Qwen3-4B-Base", max_seq_length = max_seq_length, load_in_4bit = False, # LoRA 16 位时为 False fast_inference = True, # 启用 vLLM 快速推理 max_lora_rank = lora_rank, gpu_memory_utilization = 0.9, # 若内存不足请降低 dtype = torch.float16, # 使用 torch.float16、torch.bfloat16 ) sun-brightdesktopmoon --- # 使用 Unsloth Studio 导出模型 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 使用 [Unsloth Studio](https://unsloth.ai/docs/zh/xin/studio) 将模型导出、保存或转换为 GGUF、Safetensors 或 LoRA，用于在 Unsloth、llama.cpp、Ollama、vLLM 等环境中部署、共享或进行本地推理。可导出已训练的检查点，或转换任何现有模型。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrrFY8YczW3dDpfYi1k9f%252FScreenshot%25202026-03-15%2520at%25209.28.19%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd2729e16-799f-48f0-8b07-0248b93fa599&width=768&dpr=3&quality=100&sign=27a801cd&sv=2) 1 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/export#xuan-ze-xun-lian-yun-xing) 选择训练运行 首先选择要导出的训练运行。每次运行代表一次完整的训练会话，可能包含多个检查点。 选择运行后，选择要导出的检查点。检查点是在训练过程中创建并保存的模型版本。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzB12XFNP3UjoAT1l9vz3%252Fimage.png%3Falt%3Dmedia%26token%3D021b8864-b2c5-4a92-927e-e23350610036&width=768&dpr=3&quality=100&sign=d934a4fe&sv=2) 2 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/export#xuan-ze-jian-cha-dian) 选择检查点 较后期的检查点通常代表最终训练完成的模型，但你可以根据需要导出任何检查点。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8VfRPUcY3w6zYfNmAIDn%252Fimage.png%3Falt%3Dmedia%26token%3D42565a7d-e62f-4cf0-bd33-90422f1b2194&width=768&dpr=3&quality=100&sign=8b49aef1&sv=2) 3 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/export#dao-chu-fang-shi) 导出方式 根据你的工作流程，你可以导出合并模型、LoRA 适配器权重，或用于本地推理的 GGUF 模型。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh4sPts9rJhHiGqf0UxIs%252Fimage.png%3Falt%3Dmedia%26token%3D4f1d6a76-bd40-4471-ab8d-0b2fe33d0410&width=768&dpr=3&quality=100&sign=290e3e6d&sv=2) 每种导出方式会生成模型的不同版本，具体取决于你计划如何运行或共享它。下表解释了每个选项导出的内容。 导出类型 说明 合并模型 **16 位模型** ，将 LoRA 适配器合并到基础权重中。 仅 LoRA 导出 **仅适配器权重**。需要原始基础模型。 GGUF / llama.cpp 将模型转换为 **GGUF 格式** ，用于 Unsloth / llama.cpp **/** Ollama / LM Studio 推理。 4 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/export#dao-chu-ben-di-bao-cun) 导出 / 本地保存 导出模型时，你可以选择结果文件的保存位置。模型可以直接下载到你的机器，也可以推送到 Hugging Face Hub 进行托管和共享。 将导出的模型文件直接保存到你的机器上。此选项适用于在本地运行模型、手动分发文件或与本地推理工具集成。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfsBaE8V2o69jSyCVGIz4%252Fimage.png%3Falt%3Dmedia%26token%3D4ef3fa06-d25b-424a-91e3-42debd3b6908&width=768&dpr=3&quality=100&sign=ae03a432&sv=2) 5 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/export#tui-song-dao-hub) 推送到 Hub 将导出的模型上传到 Hugging Face Hub。这使你能够从一个中心仓库托管、共享和部署模型。 你需要一个 Hugging Face 写入令牌才能发布模型。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrvVnuVUYQWv2nkrgFxpK%252Fimage.png%3Falt%3Dmedia%26token%3D5e0b91fe-5225-4bff-9fa9-ec1fb3867b1a&width=768&dpr=3&quality=100&sign=168ae7cf&sv=2) circle-check 如果你已经通过 Hugging Face CLI 完成身份验证，则写入令牌可以留空。 [上一页Data Recipeschevron-left](https://unsloth.ai/docs/zh/xin/studio/data-recipe) [下一页Unsloth 更新chevron-right](https://unsloth.ai/docs/zh/xin/changelog) 最后更新于 17天前 这有帮助吗？ * [选择训练运行](https://unsloth.ai/docs/zh/xin/studio/export#xuan-ze-xun-lian-yun-xing) * [选择检查点](https://unsloth.ai/docs/zh/xin/studio/export#xuan-ze-jian-cha-dian) * [导出方式](https://unsloth.ai/docs/zh/xin/studio/export#dao-chu-fang-shi) * [导出 / 本地保存](https://unsloth.ai/docs/zh/xin/studio/export#dao-chu-ben-di-bao-cun) * [推送到 Hub](https://unsloth.ai/docs/zh/xin/studio/export#tui-song-dao-hub) 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 使用 Unsloth 在 Intel GPU 上微调 LLM | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 现在您可以使用 Unsloth 在本地 Intel 设备上微调大型语言模型（LLM）！阅读我们的指南，了解如何开始训练您自己的自定义模型。 在开始之前，请确保您具备： * **Intel GPU：** 数据中心 GPU Max 系列、Arc 系列或 Intel Ultra AIPC * **操作系统：** Linux（建议使用 Ubuntu 22.04+）或 Windows 11（推荐） * **仅限 Windows：** 安装 Intel oneAPI Base Toolkit 2025.2.1（选择版本 2025.2.1） * **Intel 显卡驱动：** Windows/Linux 的最新推荐驱动 * **Python：** 3.10+ ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-yong-dui-intel-de-zhi-chi-gou-jian-unsloth) 使用对 Intel 的支持构建 Unsloth 1 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#chuang-jian-xin-de-conda-huan-jing-ke-xuan) 创建新的 conda 环境（可选） 复制 conda create -n unsloth-xpu python==3.10 conda activate unsloth-xpu 2 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#an-zhuang-unsloth) 安装 Unsloth 复制 git clone https://github.com/unslothai/unsloth.git cd unsloth pip install .[intel-gpu-torch290] circle-info 仅限 Linux：安装 [vLLM](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/vllm-guide) (可选) 您也可以为 [推理](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment) 和 [强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 安装。请遵循 [vLLM 的指南arrow-up-right](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#intel-xpu) . 3 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#yan-zheng-nin-de-huan-jing) 验证您的环境 复制 import torch print(f"PyTorch version: {torch.__version__}") print(f"XPU available: {torch.xpu.is_available()}") print(f"XPU device count: {torch.xpu.device_count()}") print(f"XPU device name: {torch.xpu.get_device_name(0)}") 4 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#kai-shi-wei-tiao) 开始微调。 您可以直接使用我们的 Unsloth [笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks) 或查看我们的专门 [微调](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 或 [强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 指南。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#jin-xian-windows-yun-xing-shi-pei-zhi) 仅限 Windows - 运行时配置 以管理员权限在命令提示符中，在 Windows 注册表中启用长路径支持： 复制 powershell -Command "Set-ItemProperty -Path "HKLM:\\SYSTEM\\CurrentControlSet\\Control\\FileSystem" -Name "LongPathsEnabled" -Value 1 此命令只需在单台机器上设置一次。无需在每次运行前配置。然后： 1. 从以下位置下载 level-zero-win-sdk-1.20.2.zip [GitHubarrow-up-right](https://github.com/oneapi-src/level-zero/releases/tag/v1.20.2) 2. 解压 level-zero-win-sdk-1.20.2.zip 3. 在命令提示符中，在 conda 环境 unsloth-xpu 下： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-li-1-shi-yong-sft-de-qlora-wei-tiao) 示例 1：使用 SFT 的 QLoRA 微调 此示例演示如何在 Intel GPU 上使用 4 位 QLoRA 对 Qwen3-32B 模型进行微调。QLoRA 大幅降低内存需求，使在消费级硬件上微调大型模型成为可能。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-li-2-qiang-hua-xue-xi-grpo) 示例 2：强化学习 GRPO GRPO 是一种 [强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) 用于将语言模型与人类偏好对齐的技术。此示例展示如何使用多个奖励函数训练模型以遵循特定的 XML 输出格式。 #### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shen-me-shi-grpo) 什么是 GRPO？ GRPO 在传统 RLHF 基础上改进： * 使用基于组的归一化以实现更稳定的训练 * 支持多个奖励函数以进行多目标优化 * 比 PPO 更节省内存 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#gu-zhang-pai-chu) 故障排除 ----------------------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#nei-cun-bu-zu-oom-cuo-wu) 内存不足（OOM）错误 如果发生内存不足，请尝试以下解决方案： 1. **减少批量大小：** 降低 `per_device_train_batch_size`. 2. **使用更小的模型：** 从更小的模型入手以减少内存需求。 3. **减少序列长度：** 降低 `max_seq_length`. 4. **降低 LoRA 秩：** 使用 `r=8` 代替 `r=16` 或 `r=32`. 5. **对于 GRPO，减少生成数量：** 降低 `num_generations`. ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#jin-xian-windowsintel-ultra-aipc-igpu-gong-xiang-nei-cun) （仅限 Windows）Intel Ultra AIPC iGPU 共享内存 对于在 Windows 上使用近期 GPU 驱动的 Intel Ultra AIPC，集成 GPU 的共享显存通常默认为系统内存的 **57%** 。对于较大的模型（例如， **Qwen3-32B**）或在使用更长的最大序列长度、更大的批量、具有更大 LoRA 秩的 LoRA 适配器等情况下，在微调期间可以通过提高分配给 iGPU 的系统内存百分比来增加可用显存。 您可以通过修改注册表来调整： * 路径： `Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\MemoryManager` * 要更改的键： `SystemPartitionCommitLimitPercentage` （设置为更大的百分比） [上一页AMDchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/amd) [下一页Fine-tuning Guidechevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/fine-tuning-llms-guide) 最后更新于 1个月前 这有帮助吗？ * [使用对 Intel 的支持构建 Unsloth](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-yong-dui-intel-de-zhi-chi-gou-jian-unsloth) * [仅限 Windows - 运行时配置](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#jin-xian-windows-yun-xing-shi-pei-zhi) * [示例 1：使用 SFT 的 QLoRA 微调](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-li-1-shi-yong-sft-de-qlora-wei-tiao) * [示例 2：强化学习 GRPO](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#shi-li-2-qiang-hua-xue-xi-grpo) * [故障排除](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#gu-zhang-pai-chu) * [内存不足（OOM）错误](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#nei-cun-bu-zu-oom-cuo-wu) * [（仅限 Windows）Intel Ultra AIPC iGPU 共享内存](https://unsloth.ai/docs/zh/kai-shi-shi-yong/install/intel#jin-xian-windowsintel-ultra-aipc-igpu-gong-xiang-nei-cun) 这有帮助吗？ sun-brightdesktopmoon 复制 call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" - set ZE_PATH=path\to\the\unzipped\level-zero-win-sdk-1.20.2 复制 from unsloth import FastLanguageModel, FastModel from trl import SFTTrainer, SFTConfig from datasets import load_dataset max_seq_length = 2048 # 内部支持 RoPE 缩放，因此可自由选择！ # 获取 LAION 数据集 url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl" dataset = load_dataset("json", data_files = {"train" : url}, split = "train") # 我们支持的 4 位预量化模型，便于快速下载且不会 OOM（内存溢出）。 fourbit_models = [\ "unsloth/Qwen3-32B-bnb-4bit",\ "unsloth/Qwen3-14B-bnb-4bit",\ "unsloth/Qwen3-8B-bnb-4bit",\ "unsloth/Qwen3-4B-bnb-4bit",\ "unsloth/Qwen3-1.7B-bnb-4bit",\ "unsloth/Qwen3-0.6B-bnb-4bit",\ # "unsloth/Qwen2.5-32B-bnb-4bit",\ # "unsloth/Qwen2.5-14B-bnb-4bit",\ # "unsloth/Qwen2.5-7B-bnb-4bit",\ # "unsloth/Qwen2.5-3B-bnb-4bit",\ # "unsloth/Qwen2.5-1.5B-bnb-4bit",\ # "unsloth/Qwen2.5-0.5B-bnb-4bit",\ # "unsloth/Llama-3.2-3B-bnb-4bit",\ # "unsloth/Llama-3.2-1B-bnb-4bit",\ # "unsloth/Llama-3.1-8B-bnb-4bit",\ # "unsloth/Llama-3.1-70B-bnb-4bit",\ # "unsloth/mistral-7b-bnb-4bit",\ # "unsloth/Phi-4",\ # "unsloth/Phi-3.5-mini-instruct",\ # "unsloth/Phi-3-medium-4k-instruct",\ # "unsloth/Phi-3-mini-4k-instruct",\ # "unsloth/gemma-2-9b-bnb-4bit",\ # "unsloth/gemma-2-27b-bnb-4bit",\ ] # 更多模型见 https://huggingface.co/unsloth model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Qwen3-32B-bnb-4bit", max_seq_length = max_seq_length, load_in_4bit = True, # token = "hf_...", # 如果使用诸如 meta-llama/Llama-2-7b-hf 之类的受限模型，可使用令牌 ) model = FastLanguageModel.get_peft_model( model, r = 16, # 选择任意大于 0 的数！建议 8、16、32、64、128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",\ ], lora_alpha = 16, lora_dropout = 0, # 支持任意值，但 = 0 已优化 bias = "none", # 支持任意值，但 = "none" 已优化 use_gradient_checkpointing = "unsloth", # 对于非常长的上下文使用 True 或 "unsloth" random_state = 3407, use_rslora = False, # 我们支持秩稳定 LoRA（rank stabilized LoRA） loftq_config = None, # 以及 LoftQ ) trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 1, # 在 Windows 上推荐 packing = False, # 对于短序列可使训练快 5 倍。 args = SFTConfig( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, dataset_num_proc=1, # 在 Windows 上推荐 ), ) trainer.train() chevron-down显示全部 82 行 复制 from unsloth import FastLanguageModel import re from trl import GRPOConfig, GRPOTrainer from datasets import load_dataset, Dataset max_seq_length = 1024 # 可为更长的推理轨迹增加 lora_rank = 32 # 更大的秩＝更聪明，但更慢 max_prompt_length = 256 # 加载并准备数据集 SYSTEM_PROMPT = """ 以以下格式响应： ... ... """ XML_COT_FORMAT = """\ {reasoning} {answer} """ def extract_xml_answer(text: str) -> str: answer = text.split("")[-1] answer = answer.split("")[0] return answer.strip() def extract_hash_answer(text: str) -> str | None: if "####" not in text: return None return text.split("####")[1].strip() # 取消注释中间消息以进行 1-shot 提示 def get_gsm8k_questions(split: str = "train") -> Dataset: data = load_dataset("openai/gsm8k", "main")[split] # type: ignore data = data.map( lambda x: { # type: ignore "prompt": [\ {"role": "system", "content": SYSTEM_PROMPT},\ {"role": "user", "content": x["question"]},\ ], "answer": extract_hash_answer(x["answer"]), } ) # type: ignore return data # type: ignore # 奖励函数 def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]: responses = [completion[0]["content"] for completion in completions] q = prompts[0][-1]["content"] extracted_responses = [extract_xml_answer(r) for r in responses] print( "-" * 20, f"Question:\n{q}", f"\nAnswer:\n{answer[0]}", f"\nResponse:\n{responses[0]}", f"\nExtracted:\n{extracted_responses[0]}", ) return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)] def int_reward_func(completions, **kwargs) -> list[float]: responses = [completion[0]["content"] for completion in completions] extracted_responses = [extract_xml_answer(r) for r in responses] return [0.5 if r.isdigit() else 0.0 for r in extracted_responses] def strict_format_reward_func(completions, **kwargs) -> list[float]: """检查完成内容是否具有特定格式的奖励函数。""" pattern = r"^\n.*?\n\n\n.*?\n\n$" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def soft_format_reward_func(completions, **kwargs) -> list[float]: """检查完成内容是否具有特定格式的奖励函数。""" pattern = r".*?\s*.*?" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def count_xml(text: str) -> float: count = 0.0 if text.count("\n") == 1: count += 0.125 if text.count("\n\n") == 1: count += 0.125 if text.count("\n\n") == 1: count += 0.125 count -= len(text.split("\n\n")[-1]) * 0.001 if text.count("\n") == 1: count += 0.125 count -= (len(text.split("\n")[-1]) - 1) * 0.001 return count def xmlcount_reward_func(completions, **kwargs) -> list[float]: contents = [completion[0]["content"] for completion in completions] return [count_xml(c) for c in contents] if __name__ == "__main__": model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3-0.6B", max_seq_length=max_seq_length, load_in_4bit=False, # LoRA 使用 16 位 时为 False fast_inference=False, # 启用 vLLM 快速推理 max_lora_rank=lora_rank, gpu_memory_utilization=0.7, # 若内存不足请降低 device_map="xpu:0", ) model = FastLanguageModel.get_peft_model( model, r=lora_rank, # 选择任意大于 0 的数！建议 8、16、32、64、128 target_modules=[\ "q_proj",\ "k_proj",\ "v_proj",\ "o_proj",\ "gate_proj",\ "up_proj",\ "down_proj",\ ], # 如果内存不足可移除 QKVO lora_alpha=lora_rank, use_gradient_checkpointing="unsloth", # 启用长上下文微调 random_state=3407, ) dataset = get_gsm8k_questions() training_args = GRPOConfig( learning_rate=5e-6, adam_beta1=0.9, adam_beta2=0.99, weight_decay=0.1, warmup_ratio=0.1, lr_scheduler_type="cosine", optim="adamw_torch", logging_steps=1, per_device_train_batch_size=1, gradient_accumulation_steps=1, # 若要更平滑的训练可增加到 4 num_generations=4, # 若内存不足请减少 max_prompt_length=max_prompt_length, max_completion_length=max_seq_length - max_prompt_length, # num_train_epochs=1, # 对于完整训练运行设为 1 max_steps=20, save_steps=250, max_grad_norm=0.1, report_to="none", # 可使用 Weights & Biases output_dir="outputs", ) trainer = GRPOTrainer( model=model, processing_class=tokenizer, reward_funcs=[\ xmlcount_reward_func,\ soft_format_reward_func,\ strict_format_reward_func,\ int_reward_func,\ correctness_reward_func,\ ], args=training_args, train_dataset=dataset, dataset_num_proc=1, # 在 Windows 上推荐 ) trainer.train() chevron-down显示全部 183 行 sun-brightdesktopmoon --- # 视觉强化学习（VLM RL） | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 现在支持带有视觉/多模态强化学习的 [Qwen3-VL](https://unsloth.ai/docs/zh/mo-xing/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) , [Gemma 3](https://unsloth.ai/docs/zh/mo-xing/tutorials/gemma-3-how-to-run-and-fine-tune) 以及更多。由于 Unsloth 独特的 [权重共享](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide#what-unsloth-offers-for-rl) 和自定义内核，Unsloth 使得 VLM 强化学习 **快 1.5–2×，** 使用 **90% 更少的显存**，并且启用 **比 FA2 配置长 15× 的上下文** 长度，而不会损失准确性。此更新还引入了 Qwen 的 [GSPO](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#gspo-rl) 算法。 Unsloth 可以在免费 Colab T4 GPU 上使用 GSPO/GRPO 训练 Qwen3-VL-8B。其他 VLM 也可行，但可能需要更大的 GPU。Gemma 需要比 T4 更新的 GPU，因为 vLLM [限制为 Bfloat16](https://unsloth.ai/docs/zh/mo-xing/tutorials/gemma-3-how-to-run-and-fine-tune#unsloth-fine-tuning-fixes) ，因此我们建议在 Colab 上使用 NVIDIA L4。我们的笔记本解决涉及图像和图示的数值数学问题： * **Qwen-3 VL-8B** （vLLM 推理）**:** [Colabarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) * **Qwen-2.5 VL-7B** （vLLM 推理）**:** [Colabarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) • [Kagglearrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2_5_7B_VL_GRPO.ipynb&accelerator=nvidiaTeslaT4) * **Gemma-3-4B** （Unsloth 推理）： [Colabarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) 我们还将 vLLM VLM 原生集成到 Unsloth 中，因此使用 vLLM 推理时你只需在初始化模型时启用 `fast_inference=True` 标志即可。特别感谢 [Sinoué GADarrow-up-right](https://github.com/unslothai/unsloth/pull/2752) 提供了 [第一个笔记本arrow-up-right](https://github.com/GAD-cell/vlm-grpo/blob/main/examples/VLM_GRPO_basic_example.ipynb) 使得集成 VLM 强化学习更容易！ 此 VLM 支持还整合了我们最新的更新，以实现更节省内存且更快的强化学习，包括我们的 [待机功能](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl#unsloth-standby) ，它在限制速度退化方面与其他实现相比具有独特优势。 circle-info 你只能对 `fast_inference` 为 vLLM 支持的 VLM 使用。有些模型，例如 Llama 3.2 Vision，因此只能在不使用 vLLM 的情况下运行，但它们仍然可以在 Unsloth 中工作。 复制 os.environ['UNSLOTH_VLLM_STANDBY'] = '1' # 启用与 vLLM 一起的节省内存的 GRPO model, tokenizer = FastVisionModel.from_pretrained( model_name = "Qwen/Qwen2.5-VL-7B-Instruct", max_seq_length = 16384, # 必须这么大以将图像放入上下文 load_in_4bit = True, # LoRA 16 位时为 False fast_inference = True, # 启用 vLLM 快速推理 gpu_memory_utilization = 0.8, # 内存不足时降低该值 ) 同样重要的是要注意，vLLM 不支持视觉/编码器层的 LoRA，因此在加载 LoRA 适配器时将 `finetune_vision_layers = False` 设置为 False。 但是如果你通过 transformers/Unsloth 使用推理，你也可以训练视觉层。 复制 # 将 LoRA 适配器添加到模型以进行参数高效的微调 model = FastVisionModel.get_peft_model( model, finetune_vision_layers = False,# fast_inference 目前还不支持 finetune_vision_layers :( finetune_language_layers = True, # 如果不微调语言层则为 False finetune_attention_modules = True, # 如果不微调注意力层则为 False finetune_mlp_modules = True, # 如果不微调 MLP 层则为 False r = lora_rank, # 选择任意大于 0 的数！建议 8、16、32、64、128 lora_alpha = lora_rank*2, # *2 可加速训练 use_gradient_checkpointing = "unsloth", # 降低内存使用 random_state = 3407, ) [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#qwen-2.5-vl-shi-jue-qiang-hua-xue-xi-de-wen-ti-yu-guai-yi-xing-wei) 🦋Qwen 2.5 VL 视觉强化学习的问题与怪异行为 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 在 Qwen 2.5 VL 的强化学习过程中，你可能会看到以下推理输出： 这被 [报告arrow-up-right](https://github.com/QwenLM/Qwen2.5-VL/issues/759) 在 Qwen2.5-VL-7B-Instruct 输出意外结果 “addCriterion” 中也有提及。事实上我们也看到了这个！我们尝试了非 Unsloth、bfloat16 和 float16 的机器以及其他方法，但它似乎仍然存在。例如第 165 项 即 `train_dataset[165]` 来自 [AI4Math/MathVistaarrow-up-right](https://huggingface.co/datasets/AI4Math/MathVista) 数据集如下： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-61a659529171fcc10ed6398a15912b21d6b1a076%252FUntitled.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=a538adc5&sv=2) 然后我们得到上面的乱码输出。可以添加一个奖励函数以惩罚出现 addCriterion 的行为，或惩罚乱码输出。然而，另一种方法是让模型训练更长时间。例如大约在 60 步之后我们才看到模型通过强化学习实际学到东西： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-5f34f66f0ac6508fd28343b16592c59b889ec5ca%252Fimage.webp%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=6ee880d3&sv=2) circle-check 强制 `<|assistant|>` 在生成期间将减少这些乱码结果的出现，这符合预期因为这是一个 Instruct 模型，不过最好还是如下一节所述添加奖励函数来惩罚不良生成。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#yong-yu-jian-shao-luan-ma-de-jiang-li-han-shu) 🏅用于减少乱码的奖励函数 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 为了惩罚 `addCriterion` 和乱码输出，我们修改了奖励函数以惩罚过多的 `addCriterion` 和换行符。 [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#gspo-qiang-hua-xue-xi) 🏁GSPO 强化学习 ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 此更新另外添加了 GSPO（[组序列策略优化arrow-up-right](https://arxiv.org/abs/2507.18071) ）这是由阿里巴巴 Qwen 团队提出的 GRPO 变体。他们注意到 GRPO 隐式地为每个标记导致了重要性权重，尽管显式的优势并不会随每个标记缩放或改变。 这导致了 GSPO 的创建，GSPO 现在将重要性分配给序列似然而不是各个标记的个体似然。这两种算法之间的差异可以在下面看到，均来自 Qwen 和阿里巴巴的 GSPO 论文： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-45d743dd5dcd590626777ce09cfab61808aa8c24%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=a9867df0&sv=2) GRPO 算法，来源： [Qwenarrow-up-right](https://arxiv.org/abs/2507.18071) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ee755850cbe17482ce240dde227d55c62e9a3e64%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=29b997d1&sv=2) GSPO 算法，来源： [Qwenarrow-up-right](https://arxiv.org/abs/2507.18071) 在公式 1 中，可以看出优势在该张量求和之前将每一行缩放到标记的对数概率上。本质上，尽管该缩放是赋予整个序列而非每个单独标记，但每个标记都得到了相同的缩放。下面可以看到一个简单示意图： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-b3c944808a15dde0a7ff45782f9f074993304bf1%252FCopy%2520of%2520GSPO%2520diagram%2520%281%29.jpg%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d23c70e8&sv=2) 按行用优势缩放的 GRPO 对数概率比 公式 2 显示了每个序列的对数概率比在计算完成后被求和并取指数，只有得到的序列比随后被按行乘以优势。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-62fc5b50921e79cce155d2794201c9b96faf941e%252FGSPO%2520diagram%2520%281%29.jpg%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=55a91f4e&sv=2) 按行用优势缩放的 GSPO 序列比 启用 GSPO 很简单，你只需在 GRPO 配置中设置 `importance_sampling_level = "sequence"` 标志即可。 总体而言，Unsloth 现在通过 VLM vLLM 快速推理既实现了 90% 的内存使用减少，又在 GRPO 和 GSPO 下实现了 1.5-2x 的速度提升！ 如果你想了解更多关于强化学习的内容，请查看我们的 RL 指南： [Reinforcement Learning Guide](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) _**作者：**_ _特别感谢_ [_Keith_arrow-up-right](https://www.linkedin.com/in/keith-truongcao-7bb84a23b/) _和_ [_Datta_arrow-up-right](https://www.linkedin.com/in/datta0/) _为本文做出的贡献！_ [上一页7x Longer Context RLchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context) [下一页FP8 RLchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/fp8-reinforcement-learning) 最后更新于 3个月前 这有帮助吗？ * [🦋Qwen 2.5 VL 视觉强化学习的问题与怪异行为](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#qwen-2.5-vl-shi-jue-qiang-hua-xue-xi-de-wen-ti-yu-guai-yi-xing-wei) * [🏅用于减少乱码的奖励函数](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#yong-yu-jian-shao-luan-ma-de-jiang-li-han-shu) * [🏁GSPO 强化学习](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl#gspo-qiang-hua-xue-xi) 这有帮助吗？ sun-brightdesktopmoon 复制 addCriterion \n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n\n addCriterion\n\n 自动生成\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n\n addCriterion\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n 复制 图为赛车手行驶路径的俯视图，在他的赛车与赛道护栏碰撞时。碰撞前，他以速度 v_i=70 m/s 沿一条与护栏成 30° 的直线行驶。碰撞后，他以速度 v_f=50 m/s 沿一条与护栏成 10° 的直线行驶。他的质量 m 为 80 kg。碰撞持续 14 ms。在碰撞期间，作用在车手身上的平均力的大小是多少？ 复制 def formatting_reward_func(completions,**kwargs): import re thinking_pattern = f'{REASONING_START}(.*?){REASONING_END}' answer_pattern = f'{SOLUTION_START}(.*?){SOLUTION_END}' scores = [] for completion in completions: score = 0 thinking_matches = re.findall(thinking_pattern, completion, re.DOTALL) answer_matches = re.findall(answer_pattern, completion, re.DOTALL) if len(thinking_matches) == 1: score += 1.0 if len(answer_matches) == 1: score += 1.0 # 修复 addCriterion 问题 # 参见 https://docs.unsloth.ai/new/vision-reinforcement-learning-vlm-rl#qwen-2.5-vl-vision-rl-issues-and-quirks # 对过多的 addCriterion 和换行符进行惩罚 if len(completion) != 0: removal = completion.replace("addCriterion", "").replace("\n", "") if (len(completion)-len(removal))/len(completion) >= 0.5: score -= 2.0 scores.append(score) return scores 复制 training_args = GRPOConfig( output_dir = "vlm-grpo-unsloth", per_device_train_batch_size = 8, gradient_accumulation_steps = 4, learning_rate = 5e-6, adam_beta1 = 0.9, adam_beta2 = 0.99, weight_decay = 0.1, warmup_ratio = 0.1, lr_scheduler_type = "cosine", optim = "adamw_8bit", # beta = 0.00, epsilon = 3e-4, epsilon_high = 4e-4, num_generations = 8, max_prompt_length = 1024, max_completion_length = 1024, log_completions = False, max_grad_norm = 0.1, temperature = 0.9, # report_to = "none", # 如果你想将日志记录到 Weights & Biases，请设置为 "wandb" num_train_epochs = 2, # 用于快速测试运行，完整训练请增加 report_to = "none" # GSPO 如下： importance_sampling_level = "sequence", # Dr GRPO / GAPO 等 loss_type = "dr_grpo", ) sun-brightdesktopmoon --- # 如何使用 Unsloth Studio 运行模型 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close [Unsloth Studio](https://unsloth.ai/docs/zh/xin/studio) 让你能在电脑上 100% 离线运行 AI 模型。可运行 GGUF 和 safetensors 等模型格式，来源可以是 Hugging Face 或本地文件。 * **适用于所有 MacOS、CPU、Windows、Linux、WSL 环境！无需 GPU** * **搜索 + 下载 + 运行** 任何模型，如 GGUF、LoRA 适配器、safetensors 等。 * [**对比**](https://unsloth.ai/docs/zh/xin/studio/chat#model-arena) 将两个不同模型的输出并排比较 * [**自我修复式工具调用**](https://unsloth.ai/docs/zh/xin/studio/chat#auto-healing-tool-calling) / 网页搜索， [**代码执行**](https://unsloth.ai/docs/zh/xin/studio/chat#code-execution) 并调用与 OpenAI 兼容的 API * [**自动推理参数**](https://unsloth.ai/docs/zh/xin/studio/chat#auto-parameter-tuning) 调优（temp、top-p 等）并编辑聊天模板 * 上传图片、音频、PDF、代码、DOCX 以及更多文件类型来聊天。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Ft1WkYzHmOVMXumiz71N0%252Ftoolcalling%2520chat%2520preview.png%3Falt%3Dmedia%26token%3Da1741a6c-bf24-4df8-9f27-ce21b868dbdf&width=768&dpr=3&quality=100&sign=53f64fba&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#shi-yong-unsloth-studio-chat) 使用 Unsloth Studio Chat #### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#sou-suo-bing-yun-xing-mo-xing) 搜索并运行模型 你可以通过 Hugging Face 搜索并下载任何模型，或者使用本地文件。 Studio 支持多种模型类型，包括 **GGUF**、视觉语言和文本转语音模型。运行最新模型，例如 [Qwen3.5](https://unsloth.ai/docs/zh/mo-xing/qwen3.5) 或 NVIDIA [Nemotron 3](https://unsloth.ai/docs/zh/mo-xing/nemotron-3) . 上传图片、音频、PDF、代码、DOCX 以及更多文件类型来聊天。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBf3UDywdNSlvCBhUuVsp%252FScreenshot%25202026-03-17%2520at%252012.34.23%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3Db6127cbf-76f7-48da-b869-3760ed5e9b42&width=768&dpr=3&quality=100&sign=dbdcbea4&sv=2) circle-check Unsloth Studio Chat 会自动适用于 **多 GPU 配置** 进行推理。 #### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#dai-ma-zhi-xing) 代码执行 Unsloth Studio 让 LLM 运行 Bash 和 Python，而不只是 JavaScript。它还会像 Claude Artifacts 一样对程序进行沙盒隔离，因此模型可以测试代码、生成文件，并用真实计算验证答案。 这使得模型给出的答案更可靠、更准确。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fel6jjv4rUTRCRwcRpIr7%252Flong%2520code%2520exec.png%3Falt%3Dmedia%26token%3D9d3d5930-0fdc-4d97-941c-983e5629296d&width=768&dpr=3&quality=100&sign=5c98bec3&sv=2) #### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#zi-dong-xiu-fu-shi-gong-ju-diao-yong) 自动修复式工具调用 Unsloth Studio 不仅支持工具调用和网页搜索，还能自动修复可能发生的任何错误。 这意味着你总能得到推理输出 **不会出现** 损坏的工具调用。 例如，Qwen3.5-4B 搜索了 20 多个网站并引用了来源，网页搜索发生在它的思考轨迹内部。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FXPQGEEr1YoKofrTatAKK%252Ftoolcallingif.gif%3Falt%3Dmedia%26token%3D25d68698-fb13-4c46-99b2-d39fb025df08&width=768&dpr=3&quality=100&sign=349d48e4&sv=2) #### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#zi-dong-can-shu-tiao-you) 自动参数调优 推理参数，例如 **temperature**, **top-p**, **top-k** 会自动为 Qwen3.5 等新模型预设，这样你无需担心设置就能获得最佳输出。你也可以手动调整参数并编辑系统提示。 借助 llama.cpp 的智能自动上下文，已经无需再调整上下文长度，它只会使用你需要的上下文，而不会额外加载任何内容。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FAQKsjtynvCXKtadvKhq1%252FRecording%25202026-03-13%2520114257.gif%3Falt%3Dmedia%26token%3Db5bfff0c-8189-4358-9344-08d0ae17782a&width=768&dpr=3&quality=100&sign=a1ffc7c5&sv=2) #### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#liao-tian-gong-zuo-qu) 聊天工作区 输入提示，附加任何文档、图片（webp、png）、代码文件、txt 或音频作为额外上下文，并实时查看模型的回复。 开关：思考 + 网页搜索。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHlOKWnSB6slhE1EXgAeZ%252Fimage.png%3Falt%3Dmedia%26token%3Db5bdfe4e-fe0e-4a2a-9eba-b04b15a79018&width=768&dpr=3&quality=100&sign=2b451969&sv=2) ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#mo-xing-jing-ji-chang) 模型竞技场 Studio Chat 让你使用同一个提示并排比较任意两个模型。例如比较基础模型和 LoRa 适配器。推理会先加载一个模型，然后再加载第二个（并行推理正在开发中）。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FC3xjqlunbpUr7nx6sQ4j%252Fimage.png%3Falt%3Dmedia%26token%3D65501d63-1346-4a1e-b055-c94294a24305&width=768&dpr=3&quality=100&sign=8bb80911&sv=2) 训练完成后，你可以使用相同的提示并排比较基础模型和微调后的模型，查看发生了什么变化以及结果是否有所提升。 这种工作流能让你轻松看出微调如何改变模型的回复，以及它是否改善了你的使用场景结果。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FVgnE7eMPQk2vaFboJ4BU%252Fmodel%2520arena%2520closeup.png%3Falt%3Dmedia%26token%3D8b0a910b-440c-4859-a846-0060e61e157b&width=768&dpr=3&quality=100&sign=dcfff0ef&sv=2) circle-check Unsloth Studio Chat 自动适用于 **多 GPU 配置** 进行推理。 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#shi-yong-jiu-de-xian-you-de-gguf-mo-xing) 使用旧的 / 现有的 GGUF 模型 **4 月 1 日更新：** 现在你可以选择一个现有文件夹，让 Unsloth 从中检测。 **3 月 27 日更新：** Unsloth Studio 现在 **会自动检测旧的 / 预先存在的模型** 这些模型是从 Hugging Face、LM Studio 等下载的。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBn3Fs1cchFchl328wSOs%252FScreenshot%25202026-04-05%2520at%25205.43.57%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3Dcc57ec6e-653a-4824-8e8d-a6bfbcd27493&width=768&dpr=3&quality=100&sign=fd9b21be&sv=2) **手动说明：** Unsloth Studio 会检测下载到你 Hugging Face Hub 缓存中的模型 `(C:\Users{your_username}.cache\huggingface\hub)`。如果你有通过 LM Studio 下载的 GGUF 模型，请注意这些模型存储在 `C:\Users\{your_username}.cache\lm-studio\models` _**或**_ `C:\Users{your_username}\lm-studio\models` 中，并且默认情况下 llama.cpp 无法看到它们——你需要将这些 .gguf 文件移动或复制到你的 Hugging Face Hub 缓存目录（或 llama.cpp 可访问的其他路径）中，Unsloth Studio 才能加载它们。 在 Studio 中微调模型或适配器后，你可以将其导出为 GGUF，并通过 **llama.cpp** 直接在 Studio Chat 中运行本地推理。Unsloth Studio 由 llama.cpp 和 Hugging Face 提供支持。 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#jiang-wen-jian-zuo-wei-shang-xia-wen-tian-jia) 将文件作为上下文添加 Studio Chat 直接支持对话中的多模态输入。你可以将文档、图片或音频作为提示的额外上下文附加进去。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FSitddQpGkOwUvirMem5P%252Fimage.png%3Falt%3Dmedia%26token%3D43b7af91-ea86-4279-a787-b4b444640d82&width=768&dpr=3&quality=100&sign=a916b7b&sv=2) 这让测试模型如何处理 PDF、截图或参考资料等真实输入变得很容易。文件会在本地处理，并作为上下文提供给模型。 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#shan-chu-mo-xing-wen-jian) **删除模型文件** 你可以通过模型搜索中的垃圾桶图标删除旧模型文件，或者从默认的 Hugging Face 缓存目录中移除相关的缓存模型文件夹。默认情况下，Hugging Face 使用 `~/.cache/huggingface/hub/` 在 macOS/Linux/WSL 上，以及 `C:\Users\\.cache\huggingface\hub\` 在 Windows 上。 * **MacOS、Linux、WSL：** `~/.cache/huggingface/hub/` * **Windows：** `%USERPROFILE%\.cache\huggingface\hub\` 如果 `HF_HUB_CACHE` 或 `HF_HOME` 已设置，则改用该位置。在 Linux 和 WSL 上， `XDG_CACHE_HOME` 也可以更改默认缓存根目录。 ### [hashtag](https://unsloth.ai/docs/zh/xin/studio/chat#unsloth-mei-you-jian-ce-dao-huo-shi-yong-wo-de-gpu) **Unsloth 没有检测到或使用我的 GPU** 如果模型没有使用你的 GPU，尤其是在 Docker 中，请尝试： 手动拉取最新镜像： * 使用 GPU 访问启动容器： * `docker run`: `--gpus all` * Docker Compose： `capabilities: [gpu]` * 在 Linux 上，请确保已安装 NVIDIA Container Toolkit。 * 在 Windows 上： * 检查 `nvcc --version` 是否与 `nvidia-smi` * 中显示的 CUDA 版本一致。按以下说明操作： [https://docs.docker.com/desktop/features/gpu/arrow-up-right](https://docs.docker.com/desktop/features/gpu/) [上一页Get Startedchevron-left](https://unsloth.ai/docs/zh/xin/studio/start) [下一页Installationchevron-right](https://unsloth.ai/docs/zh/xin/studio/install) 最后更新于 4天前 这有帮助吗？ * [使用 Unsloth Studio Chat](https://unsloth.ai/docs/zh/xin/studio/chat#shi-yong-unsloth-studio-chat) * [模型竞技场](https://unsloth.ai/docs/zh/xin/studio/chat#mo-xing-jing-ji-chang) * [使用旧的 / 现有的 GGUF 模型](https://unsloth.ai/docs/zh/xin/studio/chat#shi-yong-jiu-de-xian-you-de-gguf-mo-xing) * [将文件作为上下文添加](https://unsloth.ai/docs/zh/xin/studio/chat#jiang-wen-jian-zuo-wei-shang-xia-wen-tian-jia) * [删除模型文件](https://unsloth.ai/docs/zh/xin/studio/chat#shan-chu-mo-xing-wen-jian) * [Unsloth 没有检测到或使用我的 GPU](https://unsloth.ai/docs/zh/xin/studio/chat#unsloth-mei-you-jian-ce-dao-huo-shi-yong-wo-de-gpu) 这有帮助吗？ sun-brightdesktopmoon 复制 docker pull unsloth/unsloth:latest sun-brightdesktopmoon --- # 偏好优化训练 - DPO、ORPO 和 KTO | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close DPO（直接偏好优化）、ORPO（赔率比偏好优化）、PPO、KTO 奖励建模都可以与 Unsloth 一起使用。 我们有用于重现 GRPO、ORPO、DPO Zephyr、KTO 和 SimPO 的 Google Colab 笔记本： * [GRPO 笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-reasoning-rl-notebooks) * [ORPO 笔记本arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-ORPO.ipynb) * [DPO Zephyr 笔记本arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [KTO 笔记本arrow-up-right](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) * [SimPO 笔记本arrow-up-right](https://colab.research.google.com/drive/1Hs5oQDovOay4mFA6Y9lQhVJ8TnbFLFh2?usp=sharing) 我们也出现在 🤗Hugging Face 的官方文档中！我们在 [SFT 文档arrow-up-right](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) 和 [DPO 文档arrow-up-right](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth) . [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/preference-dpo-orpo-and-kto#dpo-dai-ma) DPO 代码 ----------------------------------------------------------------------------------------------------------------------------------------- 复制 import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" # 可选：设置 GPU 设备 ID from unsloth import FastLanguageModel, PatchDPOTrainer from unsloth import is_bfloat16_supported PatchDPOTrainer() import torch from trl import DPOTrainer, DPOConfig # 已从 TrainingArguments 更改 model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/zephyr-sft-bnb-4bit", max_seq_length = max_seq_length, dtype = None, load_in_4bit = True, ) # 对模型进行补丁并添加快速 LoRA 权重 model = FastLanguageModel.get_peft_model( model, r = 64, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 64, lora_dropout = 0, # 支持任意值，但 = 0 已优化 bias = "none", # 支持任意值，但 = "none" 已优化 # [新] "unsloth" 使用 30% 更少的显存，支持 2 倍更大的批次！ use_gradient_checkpointing = "unsloth", # 对于非常长的上下文，可设置为 True 或 "unsloth" random_state = 3407, max_seq_length = max_seq_length, ) dpo_trainer = DPOTrainer( model = model, ref_model = None, args = DPOConfig( # 使用 DPOConfig per_device_train_batch_size = 4, gradient_accumulation_steps = 8, warmup_ratio = 0.1, num_train_epochs = 3, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", seed = 42, output_dir = "outputs", ), beta = 0.1, train_dataset = YOUR_DATASET_HERE, # eval_dataset = YOUR_DATASET_HERE, tokenizer = tokenizer, max_length = 1024, max_prompt_length = 512, ) dpo_trainer.train() [上一页内存高效 RLchevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl) [下一页Introducing Unsloth Studiochevron-right](https://unsloth.ai/docs/zh/xin/studio) 最后更新于 1个月前 这有帮助吗？ 这有帮助吗？ sun-brightdesktopmoon sun-brightdesktopmoon --- # 具有 7 倍更长上下文的强化学习 GRPO | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 强化学习（RL）最大的挑战是支持长推理轨迹。我们正在引入新的批处理算法以实现约**7 倍更长的上下文** （可能超过 12 倍） RL 训练在准确性或速度上不劣于使用 FA3、内核和分块损失的其他优化设置。 * Unsloth 现在使用以下配置训练 gpt-oss QLoRA： **380K 上下文** 在单个 192GB 的 NVIDIA B200 GPU 上 * [Qwen3](https://unsloth.ai/docs/zh/mo-xing/tutorials/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) \-8B GRPO 达到 **110K 上下文** 在 80GB VRAM 的 H100 上通过 [vLLM](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#vllm-for-rl) 和 QLoRA，并且 **65K** 用于 [gpt-oss](https://unsloth.ai/docs/zh/mo-xing/gpt-oss-how-to-run-and-fine-tune/gpt-oss-reinforcement-learning) 使用 BF16 LoRA。 * 在 24GB VRAM 上，gpt-oss 达到 20K 上下文，Qwen3-VL 可达 32K， [Qwen3-VL](https://unsloth.ai/docs/zh/mo-xing/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) \-8B QLoRA * Unsloth GRPO RL 可与 Llama、Gemma 及所有模型自动支持更长上下文一起运行 我们新的数据移动和批处理内核与算法解锁了更多 上下文 通过： * 动态 [扁平化序列分块](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#flattened-sequence-length-chunking) 以避免物化巨大的 logits 张量并且 * [卸载 log softmax](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#offloading-activations-for-log-softmax) 激活，这可以防止随时间静默增长的内存占用。 circle-info **您可以在 Unsloth 中将所有特性结合使用：** 1. Unsloth 的 [权重共享](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl) 功能与 [vLLMarrow-up-right](https://github.com/vllm-project/vllm) 以及我们在 [内存高效 RL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl) 2. Unsloth 的 [Flex Attention](https://unsloth.ai/docs/zh/mo-xing/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) 中的备用（Standby）特性，适用于长上下文的 gpt-oss，以及我们的 [500K Context Training](https://unsloth.ai/docs/zh/bo-ke/500k-context-length-fine-tuning) 3. 中的 Float8 训练， [FP8 RL](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/fp8-reinforcement-learning) 以及 Unsloth 的 [异步梯度检查点（async gradient checkpointing）arrow-up-right](https://unsloth.ai/blog/long-context) 以及更多功能 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#ru-men) 🎉入门 要开始，您可以使用任何现有的 [GRPO 笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-reasoning-rl-notebooks) （或在本地更新 Unsloth）： [**gpt-oss-20b**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) GSPO [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2f65290ed198da6677931a6fbe6443d2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=e2fb64e1&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) [**Qwen3-VL-8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) 视觉 RL [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2f65290ed198da6677931a6fbe6443d2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=e2fb64e1&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) [Qwen3-8B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) L4 GPU [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2679fbdeac28beb748693be7b214bce0%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=afee7e5f&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) 将 Unsloth 应用于您的 RL 任务可为高效管理大规模模型提供稳健的框架。为了有效利用 Unsloth 的增强功能： * **硬件建议**：建议使用 NVIDIA H100 或等效设备以实现最佳 VRAM 利用率。 * **配置提示**：请确保 `batch_size` 和 `gradient_accumulation_steps` 设置与您的计算资源对齐以获得最佳性能。 circle-check 将 Unsloth 更新到最新的 Pypi 版本以获取最新更新： 我们的基准测试突出了与早期版本相比在 GPT OSS 和 Qwen3-8B 上实现的内存节省。下面两个图（不含 [备用（standby）](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/memory-efficient-rl) ）是在 `batch_size = 4` 和 `gradient_accumulation_steps=2` 的情况下运行的，因为 standby 设计上会使用所有 VRAM。 在我们的基准中，我们将 BF16 GRPO 与在所有优化启用情况下的 Hugging Face 进行比较（kernels 库中的所有内核、Flash Attention 3、分块损失内核等）： ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#bian-ping-hua-xu-lie-chang-du-fen-kuai) 🔢扁平化序列长度分块 以前，Unsloth 通过在批次维度上分块来避免 logits 张量的完全物化，从而减少了 RL 的内存使用。前向传递期间物化 logits 所需 VRAM 的粗略估计如公式（1）所示。 Equation 1: Logit Memory (GB)\=batch size×context length×vocab dim10243\\text{Equation 1: } \\text{Logit Memory (GB)} = \\frac{\\text{batch size} \\times\\text{context length} \\times \\text{vocab dim}}{1024^3} Equation 1: Logit Memory (GB)\=10243batch size×context length×vocab dim​ 使用此公式，配置为 `batch_size = 4`, `context_length = 8192`，并且 `vocab_dim = 128,000` 将大约需要 **3.3 GB 的 VRAM** 来存储 logits 张量。 通过 [Long Context gpt-oss](https://unsloth.ai/docs/zh/mo-xing/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) 去年，我们随后为 GRPO 引入了融合损失方法。该方法确保一次仅处理单个批样本，从而显著降低峰值内存使用。在相同配置下，VRAM 使用降至约 **0.83 GB**，如公式（2）所示。 Equation 2: Logit Memory (GB)\=context length×vocab dim10243\\text{Equation 2: }\\text{Logit Memory (GB)} = \\frac{\\text{context length} \\times \\text{vocab dim}}{1024^3} Equation 2: Logit Memory (GB)\=10243context length×vocab dim​ ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fy1TkzxySrNAeeSWJSVLU%252Funsloth_vs_trl_gpt_oss.png%3Falt%3Dmedia%26token%3D0303423d-1454-4410-8be8-7d6110ac1df0&width=768&dpr=3&quality=100&sign=fbd30f5c&sv=2) 图 1：gpt-oss BF16 GRPO LoRA（Unsloth vs. HF 在所有优化开启的情况） ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FmfKhenN0TGRDlMcuxob6%252Fqwen38b%2520long%2520context%2520grpo.png%3Falt%3Dmedia%26token%3D22883f90-5bf0-4478-91a9-6a191c920f12&width=768&dpr=3&quality=100&sign=89cc8f48&sv=2) 图 2：Qwen3-8B QLoRA GRPO LoRA（Unsloth vs. HF 在所有优化开启的情况） 在本次更新中，我们通过引入沿 **序列维度** 的分块进一步扩展了相同思路。我们不再一次性为整个 `（batch_size × context_length）` 空间物化 logits，而是将这些维度扁平化并使用可配置的乘数按较小块处理。这使 Unsloth 在不增加峰值内存使用的情况下支持显著更长的上下文。 在下面的图 5 中，我们使用的乘数为 `max(4, context_length // 4096)`，尽管可以根据所需的内存-性能权衡指定任意乘数。使用此设置，相同示例配置（`batch_size = 4`, `context_length = 8192`, `vocab_dim = 128,000`）现在仅需要 **0.207 GB 的 VRAM** 用于 logits 的物化。 Equation 3: Logit Memory (GB)\=context lengthmultiplier×vocab dim10243\\text{Equation 3: }\\text{Logit Memory (GB)} = \\frac{\\frac{\\text{context length}}{\\text{multiplier}} \\times \\text{vocab dim}}{1024^3} Equation 3: Logit Memory (GB)\=10243multipliercontext length​×vocab dim​ ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWFTbejdIn3T6E6yHgF1Z%252FCode_Generated_Image%2520%282%29.png%3Falt%3Dmedia%26token%3D790a1ee4-2814-4b29-afcb-bb9ffd1eb729&width=768&dpr=3&quality=100&sign=3cb757ed&sv=2) 图 3：gpt-oss-20b（H100）Unsloth 新版 vs. 旧版 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fi4QipufoavtPKyeRU0Vv%252FCode_Generated_Image%2520%283%29.png%3Falt%3Dmedia%26token%3D226c5a3c-a0a4-458d-a0df-8c84523b04b5&width=768&dpr=3&quality=100&sign=6ec9b4b8&sv=2) 图 4：Qwen3-8B（H100）Unsloth 新版 vs. 旧版 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkSIh5DIWvKGemnNHowPs%252FCode_Generated_Image_4.png%3Falt%3Dmedia%26token%3D0a3dfe85-ae8c-4280-bc0a-6c1f1523c90e&width=768&dpr=3&quality=100&sign=a384275d&sv=2) 图 5：gpt-oss-20b（H100） ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRP1RPiOeIYOt82L1Ifkc%252FCode_Generated_Image_5.png%3Falt%3Dmedia%26token%3D4ce06b0f-2464-41fd-8795-e5bf0dbf4327&width=768&dpr=3&quality=100&sign=e508508d&sv=2) 图 6：Qwen3-8B（B200） 此更新反映在下方编译的 `chunked_hidden_states_selective_log_softmax` 中，该实现现在支持跨批次和序列两个维度的分块。为了保留 logits 张量（ `[batch_size, context_length, vocab_dim]` ），它始终在批次维度上进行分块。额外的序列分块由 GRPO 配置中的 `unsloth_logit_chunk_multiplier` 控制；如果未设置，则默认为 `max(4, context_length // 4096)`。在下面的示例中， `input_ids_chunk[0]` 对应于优化 2 中隐藏状态小批次的大小。 1. 我们使用带有自定义编译选项的 torch.compile 以减少 VRAM 并提高速度。 2. 所有分块的 logits 都会被提升为 float32 以保留精度。 3. 我们支持 logit 软上限、温度缩放以及所有其他功能。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#yin-cang-zhuang-tai-fen-kuai) 👻隐藏状态分块 我们还观察到，在更长的上下文长度下，隐藏状态可能成为内存使用的重要来源。为演示起见，我们假设 `hidden_states_dim=4096`。相应的内存使用遵循与 logits 情况类似的公式，如下所示。 Hidden States Memory (GB)\=batch size×context length×hidden states dim10243\\text{Hidden States Memory (GB)} = \\frac{\\text{batch size} \\times\\text{context length} \\times \\text{hidden states dim}}{1024^3} Hidden States Memory (GB)\=10243batch size×context length×hidden states dim​ 在 `batch_size = 8` 和 `context_length = 64000`的情况下，这将导致大约 **2 GB**的 VRAM 使用。在此版本中，我们引入了在计算对数概率时对隐藏状态张量在批次维度上的可选分块。这将使 VRAM 使用按批次大小划分，在本例中为 **0.244 GB**。这减少了物化隐藏状态所需的峰值 VRAM，如下更新的公式所示： Hidden States Memory (GB)\=context length×hidden states dim10243\\text{Hidden States Memory (GB)} = \\frac{\\text{context length} \\times \\text{hidden states dim}}{1024^3} Hidden States Memory (GB)\=10243context length×hidden states dim​ 类似于我们在 [500K Context Training](https://unsloth.ai/docs/zh/bo-ke/500k-context-length-fine-tuning) 版本中对交叉熵损失所做的工作，新的实现 **会自动调整隐藏状态的批处理大小**。用户也可以通过 `unsloth_grpo_mini_batch`来控制此行为。然而，将 `unsloth_grpo_mini_batch` 增加到超过最佳值可能会引入轻微的性能提升或变慢（通常是更快），与之前的损失函数相比。 然而，在一次 GPT-OSS 运行中（`context_length = 8192, batch_size = 4, gradient_accumulation_steps = 2`），设置 `unsloth_grpo_mini_batch = 1` 和 `unsloth_logit_chunk_multiplier = 4` 会导致 **几乎不影响速度，同时将 VRAM 使用大约减少 5 GB** 与旧版本的 Unsloth 相比。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYZsgoKpZKyJKrNmbvehR%252FCode_Generated_Image%2520%284%29.png%3Falt%3Dmedia%26token%3D5d3c0605-9ed8-4d4a-a722-a85132510222&width=768&dpr=3&quality=100&sign=31e21587&sv=2) circle-check **注意：** 在图 3 和图 4 中，我们使用了最大的有效批次大小，在此设置中为 8。有效批次大小计算为 `batch_size × gradient_accumulation_steps`，得到 `4 × 2 = 8`。有关有效批次大小在 RL 中如何工作的更深入解释，请参见我们的 [高级 RL 文档](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/advanced-rl-documentation) . ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#wei-log-softmax-xie-zai-ji-huo) 🌵为 log softmax 卸载激活 在本次发布的开发过程中，我们发现当在隐藏状态的批次维度上进行铺瓦（tiling）时，激活在融合的 logits 和 logprobs 计算之后并未被卸载。由于 logits 是使用 `hidden_states[i] @ lm_head`逐批次计算的，因此现有的激活卸载和梯度检查点逻辑（设计为在模型的前向传递内工作）在这种情况下并不适用。 为了解决此问题，我们添加了明确的逻辑以在模型前向传递之外卸载这些激活，如下面的 Python 伪代码所示： circle-check **注意：** 仅当在批次维度上进行分块或当 `unsloth_grpo_mini_batch > 1`时，此特性才有效。如果在前向传递期间一次性物化所有隐藏状态（即 `unsloth_grpo_mini_batch = 1`），则无论是否卸载激活，反向传递都需要相同量的 GPU 内存。由于在这种情况下激活卸载会引入轻微的性能减慢且并不减少内存使用，因此并无益处。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#pei-zhi-can-shu) ✨配置参数： 如果您不配置 `unsloth_grpo_mini_batch` 和 `unsloth_logit_chunk_multiplier`，我们将为您 **基于您可用的 VRAM 并根据上下文长度的大小自动调整这两个参数。** 下面是如何在您的 GRPO 运行中更改这些变量： 下面的示意图展示了这些优化和 `unsloth_grpo_mini_batch` 和 `unsloth_logit_chunk_multiplier` 的可视化效果。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2OmZA297HzG3CdRzi3X5%252FLogit%2520Chunking%2520%281%29.png%3Falt%3Dmedia%26token%3Db953a62b-fefa-43f2-a9ce-108675b8735f&width=768&dpr=3&quality=100&sign=958f8a9b&sv=2) 这 3 个矩阵代表总体上更大的批次或 `unsloth_grpo_mini_batch` （由黑色方括号的数量表示），每个矩阵的行表示该 `unsloth_logit_chunk_multiplier` 通过（由红色方括号的数量表示）对序列长度进行分块的数量。 ### [hashtag](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#yong-yu-rl-de-vllm) 📼用于 RL 的 vLLM **对于 RL 工作流，推理/生成阶段是主要瓶颈**。为了解决这一问题，我们使用了 [vLLMarrow-up-right](https://github.com/vllm-project/vllm) ，与普通生成相比，它将生成速度提高了最多 11 倍。自从去年 GRPO 普及以来，vLLM 已成为包括 Unsloth 在内的大多数 RL 框架的核心组件。我们要向 vLLM 团队及所有贡献者表示感谢，因为他们在提升 Unsloth 的 RL 表现方面起到了关键作用！ 要尝试更长上下文的 RL，您可以使用任何现有的 [GRPO 笔记本](https://unsloth.ai/docs/zh/kai-shi-shi-yong/unsloth-notebooks#grpo-reasoning-rl-notebooks) （或在本地更新 Unsloth）： [**gpt-oss-20b**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) - GSPO [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2f65290ed198da6677931a6fbe6443d2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=e2fb64e1&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) [**Qwen3-VL-8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) 视觉 RL [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2f65290ed198da6677931a6fbe6443d2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=e2fb64e1&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) [Qwen3-8B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) L4 GPU [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2F2679fbdeac28beb748693be7b214bce0%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=afee7e5f&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) 致谢：非常感谢 Hugging Face 团队和其库为 Unsloth 提供支持并使之成为可能。 [上一页Reinforcement Learning Guidechevron-left](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide) [下一页Vision RLchevron-right](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) 最后更新于 2个月前 这有帮助吗？ * [🎉入门](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#ru-men) * [🔢扁平化序列长度分块](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#bian-ping-hua-xu-lie-chang-du-fen-kuai) * [👻隐藏状态分块](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#yin-cang-zhuang-tai-fen-kuai) * [🌵为 log softmax 卸载激活](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#wei-log-softmax-xie-zai-ji-huo) * [✨配置参数：](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#pei-zhi-can-shu) * [📼用于 RL 的 vLLM](https://unsloth.ai/docs/zh/kai-shi-shi-yong/reinforcement-learning-rl-guide/grpo-long-context#yong-yu-rl-de-vllm) 这有帮助吗？ sun-brightdesktopmoon 复制 pip install --upgrade --no-cache-dir unsloth unsloth_zoo 复制 logprobs_chunk = chunked_hidden_states_selective_log_softmax( new_hidden_states_chunk, lm_head, completion_ids, chunks=input_ids_chunk.shape[0]*multiplier, logit_scale_multiply=logit_scale_multiply, logit_scale_divide=logit_scale_divide, logit_softcapping=logit_softcapping, temperature=temperature, ) 复制 class Unsloth_Offloaded_Log_Softmax(torch.autograd.Function): def forward(...): with torch.no_grad(): output = chunked_hidden_states_selective_log_softmax(hidden_states, lm_head, ...) return output def backward(ctx, grad_output): hidden_states = ctx.saved_hidden_states hidden_states.requires_grad_(True) with torch.enable_grad(): output = chunked_hidden_states_selective_log_softmax(hidden_states, lm_head, ...) torch.autograd.backward(output, grad_output) return ... 复制 training_args = GRPOConfig( ... unsloth_grpo_mini_batch = 3 unsloth_logit_chunk_multiplier = 2 ... ) sun-brightdesktopmoon --- # 使用 Blackwell、RTX 50 系列和 Unsloth 微调 LLM | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 现在支持 NVIDIA 的 Blackwell 架构 GPU，包括 RTX 50 系列 GPU（5060–5090）、RTX PRO 6000，以及 B200、B40、GB100、GB102 等 GPU！您可以阅读官方的 [NVIDIA 博客文章在这里arrow-up-right](https://developer.nvidia.com/blog/train-an-llm-on-an-nvidia-blackwell-desktop-with-unsloth-and-scale-it/) . Unsloth 现在兼容自 2018 年以来的所有 NVIDIA GPU，包括 [DGX Spark](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) . > **我们的新** [**Docker 镜像**](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) > **支持 Blackwell。运行该 Docker 镜像并开始训练！** [**指南**](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pip-an-zhuang) Pip 安装 只需安装 Unsloth： 复制 pip install unsloth 如果遇到问题，另一种选择是创建一个独立的隔离环境： 复制 python -m venv unsloth source unsloth/bin/activate pip install unsloth 注意 可能是 `pip3` 或 `pip3.13` 并且也可能是 `python3` 或 `python3.13` 您可能会遇到一些 Xformers 问题，在这种情况下您应当从源码构建： 复制 # 首先卸载先前库安装的 xformers pip uninstall xformers -y # 克隆并构建 pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) Docker [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) 是 Unsloth 的唯一 Docker 镜像。对于 Blackwell 和 50 系列 GPU，请使用相同的镜像——无需单独的镜像。 有关安装说明，请参阅我们的 [Unsloth Docker 指南](https://unsloth.ai/docs/zh/bo-ke/how-to-fine-tune-llms-with-unsloth-and-docker) . ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) uv #### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv-gao-ji) uv（高级） 安装顺序很重要，因为我们希望用特定版本覆盖捆绑的依赖项（即， `xformers` 和 `triton`). 1. 我更喜欢使用 `uv` 而不是 `pip` 因为它更快并且在解析依赖项时更好，尤其是对于依赖 `torch` 但在此场景下需要特定的 `CUDA` 版本。 安装 `uv` 创建项目目录和虚拟环境： 2. 安装 `vllm` 注意我们必须指定 `cu128`，否则 `vllm` 将安装 `torch==2.7.0` 但会使用 `cu126`. 3. 安装 `unsloth` 依赖项 如果您注意到由于 Xformers 导致的奇怪解析问题，您也可以从源码安装 Unsloth 而不安装 Xformers： 4. 下载并构建 `xformers` （可选） Xformers 是可选的，但它确实更快且占用更少内存。如果您不想使用 Xformers，我们将使用 PyTorch 的原生 SDPA。注意，从源码构建 Xformers 可能会很慢，请注意！ 注意我们必须显式设置 `TORCH_CUDA_ARCH_LIST=12.0`. 5. `transformers` 安装任何版本的 transformers，但最好获取最新版本。 ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#conda-huo-mamba-gao-ji) Conda 或 mamba（高级） 1. 安装 `conda/mamba` 运行安装脚本 创建 conda 或 mamba 环境 激活新创建的环境 2. 安装 `vllm` 确保您处于已激活的 conda/mamba 环境中。您应该在终端提示符前看到环境名称作为前缀，如下所示，您的 `(unsloth-blackwell)user@machine:` 注意我们必须指定 `cu128`，否则 `vllm` 将安装 `torch==2.7.0` 但会使用 `cu126`. 3. 安装 `unsloth` 依赖项 确保您处于已激活的 conda/mamba 环境中。您应该在终端提示符前看到环境名称作为前缀，如下所示，您的 `(unsloth-blackwell)user@machine:` 4. 下载并构建 `xformers` （可选） Xformers 是可选的，但它确实更快且占用更少内存。如果您不想使用 Xformers，我们将使用 PyTorch 的原生 SDPA。注意，从源码构建 Xformers 可能会很慢，请注意！ 您应该在终端提示符前看到环境名称作为前缀，如下所示，您的 `(unsloth-blackwell)user@machine:` 注意我们必须显式设置 `TORCH_CUDA_ARCH_LIST=12.0`. 5. 更新 `triton` 确保您处于已激活的 conda/mamba 环境中。您应该在终端提示符前看到环境名称作为前缀，如下所示，您的 `(unsloth-blackwell)user@machine:` `triton>=3.3.1` 是所需的 `Blackwell` 支持。 6. `Transformers` 安装任何版本的 transformers，但最好获取最新版本。 如果您使用 mamba 作为包管理器，只需将上面所有命令中的 conda 替换为 mamba。 ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wsl-te-ding-zhu-yi-shi-xiang) WSL 特定注意事项 如果您在使用 WSL（Windows 子系统 Linux）并在 xformers 编译过程中遇到问题（提醒：Xformers 是可选的，但对训练更快），请遵循这些额外步骤： 1. **增加 WSL 内存限制** 创建或编辑 WSL 配置文件： 在进行这些更改后，重启 WSL： 2. **安装 xformers** 使用以下命令在 WSL 上以优化的编译方式安装 xformers： 该 `--no-build-isolation` 标志有助于避免在 WSL 环境中可能出现的构建问题。 [上一页DGX Spark and Unslothchevron-left](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) 最后更新于 2个月前 这有帮助吗？ * [Pip 安装](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pip-an-zhuang) * [Docker](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) * [uv](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) * [Conda 或 mamba（高级）](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#conda-huo-mamba-gao-ji) * [WSL 特定注意事项](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wsl-te-ding-zhu-yi-shi-xiang) 这有帮助吗？ sun-brightdesktopmoon 复制 uv pip install unsloth 复制 curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env 复制 mkdir 'unsloth-blackwell' && cd 'unsloth-blackwell' uv venv .venv --python=3.12 --seed source .venv/bin/activate 复制 uv pip install -U vllm --torch-backend=cu128 复制 uv pip install unsloth unsloth_zoo bitsandbytes 复制 uv pip install -qqq \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" 复制 # 首先卸载先前库安装的 xformers pip uninstall xformers -y # 克隆并构建 pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. 复制 uv pip install -U transformers 复制 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" 复制 bash Miniforge3-$(uname)-$(uname -m).sh 复制 conda create --name unsloth-blackwell python==3.12 -y 复制 conda activate unsloth-blackwell 复制 pip install -U vllm --extra-index-url https://download.pytorch.org/whl/cu128 复制 pip install unsloth unsloth_zoo bitsandbytes 复制 # 首先卸载先前库安装的 xformers pip uninstall xformers -y # 克隆并构建 pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. 复制 pip install -U triton>=3.3.1 复制 uv pip install -U transformers 复制 # 在您的 Windows 用户目录中创建或编辑 .wslconfig #（通常位于 C:\Users\YourUsername\.wslconfig） # 将这些行添加到文件中 [wsl2] memory=16GB # 建议至少 16GB 以进行 xformers 编译 processors=4 # 根据您的 CPU 核心调整 swap=2GB localhostForwarding=true 复制 wsl --shutdown 复制 # 为 Blackwell GPU 设置 CUDA 架构 export TORCH_CUDA_ARCH_LIST="12.0" # 使用优化的构建标志从源码安装 xformers pip install -v --no-build-isolation -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers sun-brightdesktopmoon --- # 使用 NVIDIA DGX Spark 和 Unsloth 微调 LLM | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth 使得在 NVIDIA DGX™ Spark 上对最大 **200B 参数** 的大型语言模型进行本地微调成为可能。借助 128 GB 的统一内存，您可以训练诸如 **gpt-oss-120b**等大型模型，并在 DGX Spark 上直接运行或部署推理。 如在 [OpenAI DevDayarrow-up-right](https://x.com/UnslothAI/status/1976284209842118714) 所示，gpt-oss-20b 曾在 DGX Spark 上使用 RL 和 Unsloth 训练以自动赢得 2048。您可以在 DGX Spark 的 Docker 容器或虚拟环境中使用 Unsloth 进行训练。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ff5c4752dccb8f922b937f8e3b0db58e2d836507%252Funsloth%2520nvidia%2520dgx%2520spark.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=e43f58f6&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a8472482c49e1763378b609f8f537ca89df60260%252FNotebooks%2520on%2520dgx.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=17cf53c9&sv=2) 在本教程中，我们将在 DGX Spark 上安装 Unsloth 后，使用 Unsloth 笔记本通过 RL 训练 gpt-oss-20b。gpt-oss-120b 将使用大约 **68GB** 的统一内存。 在 1,000 步和 4 小时的 RL 训练后，gpt-oss 模型在 2048 上大大优于原始模型，且更长时间的训练会进一步提升结果。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-3bdcb0fda2ad188142e58f04c855b6dcfbd5ba94%252Fopenai%2520devday%2520unsloth%2520feature.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=f18b7a3a&sv=2) 您可以观看 Unsloth 在 OpenAI DevDay 2025 上的展示 [此处arrow-up-right](https://youtu.be/1HL2YHRj270?si=8SR6EChF34B1g-5r&t=1080) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-4a8bd4ecc7ee3d123c19158df5dfdcec35df8532%252FScreenshot%25202025-10-13%2520at%25204.22.32%25E2%2580%25AFPM.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=a1f5a982&sv=2) 使用 RL 训练的 gpt-oss 在 2048 上持续表现更佳。 ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#zhu-bu-jiao-cheng) ⚡ 逐步教程 1 **从 DGX Spark 的 Unsloth Docker 镜像开始** 首先，使用 DGX Spark Dockerfile 构建 Docker 镜像，该文件可以 [在此找到arrow-up-right](https://raw.githubusercontent.com/unslothai/notebooks/main/Dockerfile_DGX_Spark) 。您也可以在 DGX Spark 的终端中运行以下命令： 然后，使用保存的 Dockerfile 构建训练用 Docker 镜像： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-7ebcf195c154b0e569115e1f9513cf002ee57b16%252Fdgx1.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=a0c4a5a3&sv=2) chevron-right您也可以点击查看完整的 DGX Spark Dockerfile[hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#nin-ye-ke-yi-dian-ji-cha-kan-wan-zheng-de-dgx-spark-dockerfile) 2 **启动容器** 以 GPU 访问和卷挂载启动训练容器： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a67c36494f5c4ab4017748d490fb258655cd2378%252Fdgx2.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=ebd52043&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-b7758db087ab8b724049361781952b5ed154dfe8%252Fdgx5.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=6f6d3014&sv=2) 3 **启动 Jupyter 并运行笔记本** 在容器内，启动 Jupyter 并运行所需的笔记本。您可以使用“强化学习 gpt-oss 20b 赢取 2048” [笔记本在此arrow-up-right](https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_DGX_Spark.ipynb) 。实际上，所有 [Unsloth 笔记本arrow-up-right](https://docs.unsloth.ai/get-started/unsloth-notebooks) 都可在 DGX Spark 中运行，包括 **120b** 笔记本！只需移除安装单元格即可。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a8472482c49e1763378b609f8f537ca89df60260%252FNotebooks%2520on%2520dgx.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=17cf53c9&sv=2) 下面的命令也可用于运行 RL 笔记本。Jupyter Notebook 启动后，打开“`gpt_oss_20B_RL_2048_Game.ipynb`” ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-0862eed0acf0656ff0cb802b6aebc30892997e3b%252Fdgx6.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7840508b&sv=2) 别忘了 Unsloth 还允许您 [保存并运行](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment) 微调后的模型，以便您之后可以在本地直接将它们部署到 DGX Spark 上。 非常感谢 [Lakshmi Ramesharrow-up-right](https://www.linkedin.com/in/rlakshmi24/) 和 [Barath Anandanarrow-up-right](https://www.linkedin.com/in/barathsa/) 来自 NVIDIA 的帮助，他们协助 Unsloth 在 DGX Spark 上发布并构建了该 Docker 镜像。 ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#tong-yi-nei-cun-shi-yong-qing-kuang) 统一内存使用情况 gpt-oss-120b QLoRA 4-bit 微调将使用大约 **68GB** 的统一内存。您的统一内存使用情况在 **之前** （左）和 **之后** （右）训练时应如下所示： ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e079a9aa8d853b319520fe0f0fbcca2e85b31ea6%252Fdgx7.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=35b513e1&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F2657992854-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-c389a73a48ad059bbb92121b328fa7ccc61bee95%252Fdgx8.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=eac75d97&sv=2) 就是这样！祝您在 NVIDIA DGX Spark 上完全本地训练和运行 LLM 玩得开心！ ### [hashtag](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#shi-pin-jiao-cheng) 视频教程 感谢来自 [AnythingLLMarrow-up-right](https://github.com/Mintplex-Labs/anything-llm) 的 Tim 提供了在 DGX Spark 上使用 Unsloth 进行微调的精彩教程： [上一页Unsloth Docker Guidechevron-left](https://unsloth.ai/docs/zh/bo-ke/how-to-fine-tune-llms-with-unsloth-and-docker) [下一页Blackwell, RTX 50 and Unslothchevron-right](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) 最后更新于 3个月前 这有帮助吗？ * [⚡ 逐步教程](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#zhu-bu-jiao-cheng) * [统一内存使用情况](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#tong-yi-nei-cun-shi-yong-qing-kuang) * [视频教程](https://unsloth.ai/docs/zh/bo-ke/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#shi-pin-jiao-cheng) 这有帮助吗？ sun-brightdesktopmoon 复制 sudo apt update && sudo apt install -y wget wget -O Dockerfile "https://raw.githubusercontent.com/unslothai/notebooks/main/Dockerfile_DGX_Spark" 复制 docker build -f Dockerfile -t unsloth-dgx-spark . 复制 FROM nvcr.io/nvidia/pytorch:25.09-py3 # 设置 CUDA 环境变量 ENV CUDA_HOME=/usr/local/cuda-13.0/ ENV CUDA_PATH=$CUDA_HOME ENV PATH=$CUDA_HOME/bin:$PATH ENV LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH ENV C_INCLUDE_PATH=$CUDA_HOME/include:$C_INCLUDE_PATH ENV CPLUS_INCLUDE_PATH=$CUDA_HOME/include:$CPLUS_INCLUDE_PATH # 从源代码安装 triton 以支持最新的 blackwell RUN git clone https://github.com/triton-lang/triton.git && \ cd triton && \ git checkout c5d671f91d90f40900027382f98b17a3e04045f6 && \ pip install -r python/requirements.txt && \ pip install . && \ cd .. # 从源代码安装 xformers 以支持 blackwell RUN git clone --depth=1 https://github.com/facebookresearch/xformers --recursive && \ cd xformers && \ export TORCH_CUDA_ARCH_LIST="12.1" && \ python setup.py install && \ cd .. # 安装 unsloth 及其他依赖 RUN pip install unsloth unsloth_zoo bitsandbytes==0.48.0 transformers==4.56.2 trl==0.22.2 # 启动 shell CMD ["/bin/bash"] 复制 docker run -it \ --gpus=all \ --net=host \ --ipc=host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ -v $(pwd):$(pwd) \ -v $HOME/.cache/huggingface:/root/.cache/huggingface \ -w $(pwd) \ unsloth-dgx-spark 复制 NOTEBOOK_URL="https://raw.githubusercontent.com/unslothai/notebooks/refs/heads/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_DGX_Spark.ipynb" wget -O "gpt_oss_20B_RL_2048_Game.ipynb" "$NOTEBOOK_URL" jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root sun-brightdesktopmoon --- # Blackwell、RTX 50 シリーズと Unsloth を使った LLM のファインチューニング | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unslothは現在、RTX 50シリーズGPU（5060–5090）、RTX PRO 6000、およびB200、B40、GB100、GB102などのGPUを含むNVIDIAのBlackwellアーキテクチャGPUをサポートしています！公式の [NVIDIAブログ投稿はこちらarrow-up-right](https://developer.nvidia.com/blog/train-an-llm-on-an-nvidia-blackwell-desktop-with-unsloth-and-scale-it/) . Unslothは2018年以降のすべてのNVIDIA GPU（以下を含む）と互換性があります [DGX Spark](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) . > **私たちの新しい** [**を使って Unsloth をインストールすることもできます。**](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) > **はBlackwellをサポートします。Dockerイメージを実行してトレーニングを開始してください！** [**ガイド**](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) ### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pipinsutru) Pipインストール Unslothを簡単にインストール： コピー pip install unsloth 問題が発生する場合、別のオプションとして分離された環境を作成する方法があります： コピー python -m venv unsloth source unsloth/bin/activate pip install unsloth 注意：それは `pip3` または `pip3.13` また `python3` または `python3.13` Xformersに関する問題が発生する可能性があり、その場合はソースからビルドするべきです： コピー # まず以前のライブラリによってインストールされた xformers をアンインストール pip uninstall xformers -y # クローンしてビルド pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. ### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) Docker [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) はUnslothの唯一のDockerイメージです。Blackwellおよび50シリーズGPUについては、この同じイメージを使用してください—別途イメージは不要です。 インストール手順については、以下に従ってください [Unsloth Dockerガイド](https://unsloth.ai/docs/jp/burogu/how-to-fine-tune-llms-with-unsloth-and-docker) . ### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) uv #### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uvke) uv（上級者向け） インストールの順序は重要です。バンドルされた依存関係を特定のバージョンで上書きしたいため（具体的には `xformers` および `triton`). 1. 私は次を使うのが好みです： `uv` よりも `pip` はより高速で依存関係の解決に優れており、特に `torch` に依存するライブラリに対して有利ですが、 `CUDA` 特定のバージョンがこのシナリオでは必要になる場合があります。 インストールしてください `uv` プロジェクトディレクトリと仮想環境を作成： 2. インストールしてください `vllm` 注意：指定する必要があります `cu128`、そうしないと `vllm` がインストールされます `torch==2.7.0` しかし `cu126`. 3. インストールしてください `unsloth` の依存関係で Xformersによる解決の問題に気づいた場合、XformersなしでソースからUnslothをインストールすることもできます： 4. ダウンロードしてビルド `xformers` （オプション） Xformersはオプションですが、確かに高速でメモリ使用量が少なくなります。Xformersを使用したくない場合はPyTorchのネイティブなSDPAを使用します。Xformersをソースからビルドするのは遅くなる可能性があるので注意してください！ 明示的に設定する必要があることに注意してください： `TORCH_CUDA_ARCH_LIST=12.0`. 5. `transformers` 任意のtransformersバージョンをインストールできますが、最新を取得するのが最良です。 ### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#condamatahamambake) Condaまたはmamba（上級者向け） 1. インストールしてください `conda/mamba` インストールスクリプトを実行 condaまたはmamba環境を作成 新しく作成した環境を有効化 2. インストールしてください `vllm` 有効化されたconda/mamba環境の中にいることを確認してください。ターミナルのプロンプトに環境名が接頭辞として表示されるはずです（例えば） `(unsloth-blackwell)user@machine:` 注意：指定する必要があります `cu128`、そうしないと `vllm` がインストールされます `torch==2.7.0` しかし `cu126`. 3. インストールしてください `unsloth` の依存関係で 有効化されたconda/mamba環境の中にいることを確認してください。ターミナルのプロンプトに環境名が接頭辞として表示されるはずです（例えば） `(unsloth-blackwell)user@machine:` 4. ダウンロードしてビルド `xformers` （オプション） Xformersはオプションですが、確かに高速でメモリ使用量が少なくなります。Xformersを使用したくない場合はPyTorchのネイティブなSDPAを使用します。Xformersをソースからビルドするのは遅くなる可能性があるので注意してください！ ターミナルのプロンプトに環境名が接頭辞として表示されるはずです（例えば） `(unsloth-blackwell)user@machine:` 明示的に設定する必要があることに注意してください： `TORCH_CUDA_ARCH_LIST=12.0`. 5. 更新 `triton` 有効化されたconda/mamba環境の中にいることを確認してください。ターミナルのプロンプトに環境名が接頭辞として表示されるはずです（例えば） `(unsloth-blackwell)user@machine:` `triton>=3.3.1` はサポートに必要です。 `Blackwell` サポート。 6. `Transformers` 任意のtransformersバージョンをインストールできますが、最新を取得するのが最良です。 パッケージマネージャとしてmambaを使用している場合は、上記のすべてのコマンドでcondaをmambaに置き換えてください。 ### [hashtag](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wslno) WSL固有の注意点 WSL（Windows Subsystem for Linux）を使用していてxformersのコンパイル中に問題が発生した場合（補足：Xformersはオプションですがトレーニングが速くなります）、以下の追加手順に従ってください： 1. **WSLのメモリ上限を増やす** WSLの設定ファイルを作成または編集： これらの変更を行った後、WSLを再起動： 2. **xformersをインストール** WSL向けに最適化されたコンパイルでxformersをインストールするには、次のコマンドを使用： その `--no-build-isolation` フラグはWSL環境での潜在的なビルド問題を回避するのに役立ちます。 [前へDGX Spark and Unslothchevron-left](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) 最終更新 2 か月前 役に立ちましたか？ * [Pipインストール](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pipinsutru) * [Docker](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) * [uv](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) * [Condaまたはmamba（上級者向け）](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#condamatahamambake) * [WSL固有の注意点](https://unsloth.ai/docs/jp/burogu/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wslno) 役に立ちましたか？ sun-brightdesktopmoon コピー uv pip install unsloth コピー curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env コピー mkdir 'unsloth-blackwell' && cd 'unsloth-blackwell' uv venv .venv --python=3.12 --seed source .venv/bin/activate コピー uv pip install -U vllm --torch-backend=cu128 コピー uv pip install unsloth unsloth_zoo bitsandbytes コピー uv pip install -qqq \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" コピー # まず以前のライブラリによってインストールされた xformers をアンインストール pip uninstall xformers -y # クローンしてビルド pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. コピー uv pip install -U transformers コピー curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" コピー bash Miniforge3-$(uname)-$(uname -m).sh コピー conda create --name unsloth-blackwell python==3.12 -y コピー conda activate unsloth-blackwell コピー pip install -U vllm --extra-index-url https://download.pytorch.org/whl/cu128 コピー pip install unsloth unsloth_zoo bitsandbytes コピー # まず以前のライブラリによってインストールされた xformers をアンインストール pip uninstall xformers -y # クローンしてビルド pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. コピー pip install -U triton>=3.3.1 コピー uv pip install -U transformers コピー # Windowsのユーザーディレクトリに .wslconfig を作成または編集 # （通常は C:\Users\YourUsername\.wslconfig） # 次の行をファイルに追加 [wsl2] memory=16GB # xformersのコンパイルには最低16GBを推奨 processors=4 # CPUコアに応じて調整 swap=2GB localhostForwarding=true コピー wsl --shutdown コピー # Blackwell GPU用のCUDAアーキテクチャを設定 export TORCH_CUDA_ARCH_LIST="12.0" # 最適化されたビルドフラグでソースからxformersをインストール pip install -v --no-build-isolation -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers sun-brightdesktopmoon --- # Conda でインストール | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close circle-exclamation Conda をお持ちの場合のみ使用してください。お持ちでない場合は、 [Pip](https://unsloth.ai/docs/jp/meru/install/pip-install) . いずれかを選択してください `pytorch-cuda=11.8,12.1` は CUDA 11.8 または CUDA 12.1 用です。私たちは以下をサポートしています `python=3.10,3.11,3.12`. コピー conda create --name unsloth_env python=3.11 -y conda activate unsloth_env pip install unsloth Linux 環境に Conda をインストールしたい場合は、 [ここを読むarrow-up-right](https://docs.anaconda.com/miniconda/) 、または以下を実行してください： コピー mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh ~/miniconda3/bin/conda init bash ~/miniconda3/bin/conda init zsh 最終更新 2 か月前 役に立ちましたか？ 役に立ちましたか？ sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth と Colab GPU を使って VS Code で LLM をファインチューニングする方法 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close これで Visual Studio Code (VS Code) から直接、ローカルまたは Google Colab の拡張機能を使って LLM をファインチューニングできるようになりました。このガイドでは、オープンソースのトレーニング [リポジトリ: Unslotharrow-up-right](https://github.com/unslothai/unsloth) を使用して、任意の [ファインチューニングノートブック](https://unsloth.ai/docs/jp/meru/unsloth-notebooks) を VS Code で Colab ランタイムに接続し、ローカルや無料の Colab GPU で学習できるようにする方法を説明します。ビデオチュートリアルも [こちら](https://unsloth.ai/docs/jp/meru/install/vs-code#video-tutorial) . 1 ### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#vs-code-to-colab-chtoriaru) VS Code と Colab チュートリアル： 始めるにあたり、次が必要です： * インストール済み [VS Codearrow-up-right](https://code.visualstudio.com/) 。ノートブックのリポジトリをクローンするための Git は通常デフォルトでインストールされています。 * 1つの **Google アカウント** （Colab に認証するため） * 推奨： **Jupyter** 拡張（ほとんどの VS Code セットアップには既に入っています） 2 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#vs-code-ni-colab-woinsutrusuru) VS Code に Colab 拡張をインストールする 1. 開く **拡張機能** を VS Code で（`Ctrl+Shift+X` / `Cmd+Shift+X`) 2. 検索する **「Colab」** をインストールし、 **Google Colab** 拡張機能 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJ1f5BS4EK0ok5QLy6vz5%252Fcolab_img_1.png%3Falt%3Dmedia%26token%3D1faa7aac-c016-4c31-90ba-ccad655244e1&width=768&dpr=3&quality=100&sign=91d259c6&sv=2) 3 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#unsloth-nontobukkuwoku) Unsloth のノートブックを開く 1. Unsloth の [notebooks リポジトリをクローンするarrow-up-right](https://github.com/unslothai/notebooks) : コピー git clone https://github.com/unslothai/notebooks cd notebooks/nb ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FwRL8pvgtLRpz5rcmS6vy%252FScreenshot%25202026-02-18%2520at%25206.29.16%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3D7a72c2ec-6f94-4015-8355-27e76ba9b974&width=768&dpr=3&quality=100&sign=979b388a&sv=2) 1. 目的のノートブックを開きます。Unsloth は埋め込み（embedding）や音声合成（TTS）などのほとんどのモデルをサポートしています。 [埋め込み](https://unsloth.ai/docs/jp/ji-ben/embedding-finetuning) , [TTS](https://unsloth.ai/docs/jp/ji-ben/text-to-speech-tts-fine-tuning) 。例えば、Qwen3-4B RL を使用します： `nb/Qwen3_(4B)-GRPO.ipynb` ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FleZtIzyflSgi1o1QcZDU%252Fcolab_img_2.png%3Falt%3Dmedia%26token%3D805ebe8b-cb50-49a4-83d4-0daf96ad21c1&width=768&dpr=3&quality=100&sign=93e5c909&sv=2) 4 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#kneruwoshite-colab-wobu) カーネルを選択して Colab を選ぶ ノートブックのツールバーで、 **Select Kernel**をクリックし、次に **Colab** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FleGtAKSVehrlspPGvDy2%252Fcolab_img_3.png%3Falt%3Dmedia%26token%3Dba3d8f1d-9cbb-44ed-8e52-8d0e52280607&width=768&dpr=3&quality=100&sign=a2a4164a&sv=2) 5 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#shii-colab-sbwosuru) 新しい Colab サーバーを追加する を選ぶと、サーバーオプションのドロップダウンが表示されます。 **Colab**最初は Google 認証のためにブラウザウィンドウが開くことがあります 1. クリック **\+ Add New Colab Server** 2. ログインしてアクセスを許可し、VS Code に戻ってください * ログインし、アクセスを許可してから VS Code に戻ります ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZcDsB2i0aUStIKEeFTvF%252Fcolab_img_4.png%3Falt%3Dmedia%26token%3Da7c3642f-db35-4215-933f-99c17ebc92a2&width=768&dpr=3&quality=100&sign=a1910cad&sv=2) 6 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#gpu-woshitesbniwokeru) GPU を選択してサーバーに名前を付ける 1. に設定する **Hardware accelerator** を **GPU** 2. GPU タイプを選択します（例えば利用可能であれば **T4**など） 3. サーバーに名前を付けます（任意の名前で構いません） ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FLuK8oheihdCNn9MAZSlk%252Fcolab_img_5.png%3Falt%3Dmedia%26token%3De5ed396f-afe2-4c09-b1b8-120c7301793c&width=768&dpr=3&quality=100&sign=c3ad76c0&sv=2) circle-info 注意: GPU の利用可否は Colab プランと現在のキャパシティに依存します。GPU オプションが見つからない場合は、以下のトラブルシューティングを参照してください。 7 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#python-kneruwobu) Python カーネルを選ぶ Colab サーバーに接続したら、そのランタイムに表示される **Python** カーネル（通常は Python 3 のカーネル）を選択してください。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBexUiXJ7LRFBXs95tX2S%252Fcolab_img_6.png%3Falt%3Dmedia%26token%3D391e343d-d2fa-4a77-8e3f-ef244adfcb4c&width=768&dpr=3&quality=100&sign=bfb28a27&sv=2) 8 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#ntobukkuwosuru) ノートブックを実行する * クリック **Run All** をノートブックのツールバーで実行する（またはセルを上から順に実行） * セットアップ用セルが依存関係をインストールし、その後 Unsloth ワークフローが開始されるのを確認します * より詳しく始め方を知りたい場合は、専用の [ファインチューニング](https://unsloth.ai/docs/jp/meru/fine-tuning-llms-guide) または [強化学習](https://unsloth.ai/docs/jp/meru/reinforcement-learning-rl-guide) ガイドを参照してください。 ### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#bideochtoriaru) ビデオチュートリアル ### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#toraburushtingu) トラブルシューティング #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#colab-sbgasaretantobukkugashiisbdekanai) Colab サーバーが切断された後、ノートブックが新しいサーバーで動かない **何が起きているか：** ノートブックが開いたまま Colab サーバーが切断されると、再接続後に VS Code が不正なカーネル／ランタイム状態のまま固まることがあります。関連する [GitHub issuearrow-up-right](https://github.com/googlecolab/colab-vscode/issues/200) . **修正方法：** ノートブックタブを完全に閉じてから、ノートブックを再度開いてください。 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#gpu-wodekinaicpu-nomisareru) GPU を選択できない（CPU のみ表示される） 考えられる原因と対処法： * **Colab 無料枠のキャパシティ：** GPU が一時的に利用できない可能性があります → 後でもう一度試してください。 * **実際には Colab ランタイムに接続されていない：** 再確認してください **Select Kernel → Colab** を選び、Colab サーバーがアクティブであることを確認してください。 * **アカウント／地域の制限や上限に達している：** 待つか、別の Google アカウント／プランを使用する必要があるかもしれません。 #### [hashtag](https://unsloth.ai/docs/jp/meru/install/vs-code#subetehanishitaganipakkjigaeta) すべては正常に動作したが、再接続後にパッケージが「消えた」 Colab ランタイムは **エフェメラル（短命）**です。サーバーが再起動したときは、通常セットアップ／インストールセル（多くの場合ノートブックの最初の数セル）を再実行する必要があります。 最終更新 1 か月前 役に立ちましたか？ * [VS Code と Colab チュートリアル：](https://unsloth.ai/docs/jp/meru/install/vs-code#vs-code-to-colab-chtoriaru) * [ビデオチュートリアル](https://unsloth.ai/docs/jp/meru/install/vs-code#bideochtoriaru) * [トラブルシューティング](https://unsloth.ai/docs/jp/meru/install/vs-code#toraburushtingu) 役に立ちましたか？ sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth 推論 | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth はネイティブで推論を2x高速化します。推論専用ノートブックについては、をクリックしてください [ここarrow-up-right](https://colab.research.google.com/drive/1aqlNQi7MMJbynFDyOQteD2t0yVfjb9Zh?usp=sharing) . すべての QLoRA、LoRA、および非 LoRA の推論経路が2x高速になります。これはコードの変更や新しい依存関係を必要としません。 コピー from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "lora_model", # トレーニングに使用したあなたのモデル max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) FastLanguageModel.for_inference(model) # ネイティブな2x高速推論を有効化 text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 64) #### [hashtag](https://unsloth.ai/docs/jp/ji-ben/inference-and-deployment/unsloth-inference#notimplementederror-utf-8-rokrugadesuansi-gasaremashita) NotImplementedError: UTF-8 ロケールが必要です。ANSI が検出されました セルを実行するときに [このエラーがarrow-up-right](https://github.com/googlecolab/colabtools/issues/3409) 発生することがあります。これを解決するには、新しいセルで次を実行してください： コピー import locale locale.getpreferredencoding = lambda: "UTF-8" 最終更新 3 か月前 役に立ちましたか？ 役に立ちましたか？ sun-brightdesktopmoon sun-brightdesktopmoon --- # Google Colab | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-4d1b1778f3c8bde62a40130d7b4395b8bb1ce90f%252FColab%2520Options.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=342ad1b2&sv=2) Colabノートブックを使ったことがない場合、ノートブック自体の簡単な説明： 1. **各「セル」の再生ボタン。** これをクリックするとそのセルのコードが実行されます。セルをスキップしてはいけませんし、すべてのセルを時系列に沿って実行する必要があります。エラーが発生した場合は、実行していないセルを再度実行してください。再生ボタンをクリックしたくない場合は、CTRL + ENTER を押すという選択肢もあります。 2. **上部ツールバーの Runtime ボタン。** このボタンを使って「Run all」をクリックすると、ノートブック全体を一度に実行できます。これによりすべてのカスタマイズ手順はスキップされますが、最初の試みとしては良い方法です。 3. **Connect / Reconnect T4 ボタン。** T4 は Google が提供している無料の GPU です。かなり強力です！ 最初のインストール用セルは以下のようになっています：角かっこ \[ \] 内の再生ボタンをクリックするのを忘れないでください。オープンソースのGithubパッケージを取得し、いくつかの他のパッケージをインストールします。 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F735611837-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-135cd796b01420cc4d5ce3ca243e9065154070a5%252Fimage%2520%2813%29%2520%281%29%2520%281%29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=970d6f79&sv=2) ### [hashtag](https://unsloth.ai/docs/jp/meru/install/google-colab#colab-nosanpurukdo) Colab のサンプルコード gpt-oss-20b をファインチューニングする Unsloth のサンプルコード： 最終更新 3 か月前 役に立ちましたか？ 役に立ちましたか？ sun-brightdesktopmoon コピー from unsloth import FastLanguageModel, FastModel import torch from trl import SFTTrainer, SFTConfig from datasets import load_dataset max_seq_length = 2048 # 内部で RoPE スケーリングをサポートしているので、任意の値を選んでください！ # LAION データセットを取得 url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl" dataset = load_dataset("json", data_files = {"train" : url}, split = "train") # 4bit 事前量子化モデルはダウンロードが4倍高速で OOM を防ぎます。 fourbit_models = [\ "unsloth/gpt-oss-20b-unsloth-bnb-4bit", # または任意のモデルを選択\ \ ] # 詳細は https://huggingface.co/unsloth を参照 model, tokenizer = FastModel.from_pretrained( model_name = "unsloth/gpt-oss-20b", max_seq_length = 2048, # 長いコンテキストには任意の値を選択！ load_in_4bit = True, # 4ビット量子化。False = 16ビットLoRA。 load_in_8bit = False, # 8ビット量子化 load_in_16bit = False, # [新機能！] 16 ビット LoRA full_finetuning = False, # フルファインチューニングに使用します。 # token = "hf_...", # ゲート付きモデルを使う場合は指定してください ) # モデルパッチ適用と高速LoRA重みの追加を行う model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # 任意をサポートしますが = 0 が最適化されています bias = "none", # 任意をサポートしますが = "none" が最適化されています # [NEW] "unsloth" は VRAM を30%節約し、2倍大きなバッチサイズに対応します！ use_gradient_checkpointing = "unsloth", # 非常に長いコンテキストには True または "unsloth" random_state = 3407, max_seq_length = max_seq_length, use_rslora = False, # ランク安定化LoRAをサポートします loftq_config = None, # および LoftQ ) trainer = SFTTrainer( model = model, train_dataset = dataset, tokenizer = tokenizer, args = SFTConfig( max_seq_length = max_seq_length, per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 10, max_steps = 60, logging_steps = 1, output_dir = "outputs", optim = "adamw_8bit", seed = 3407, ), ) trainer.train() # 詳細なヒントは https://docs.unsloth.ai を参照してください。例えば # (1) GGUF に保存 / vLLM 用に 16bit にマージする方法 # (2) 保存した LoRA アダプターからの継続トレーニング # (3) 評価ループの追加 / OOM（メモリ不足）対策 # (4) カスタマイズされたチャットテンプレート sun-brightdesktopmoon --- # 使用 Hugging Face Jobs 部署 LLM | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close 本指南介绍如何使用 [Unslotharrow-up-right](https://github.com/unslothai/unsloth) 和 [Liquid LFM2.5](https://unsloth.ai/docs/zh/mo-xing/tutorials/lfm2.5) 通过像 [Claude Code](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/claude-code) 这样的编码代理进行快速 LLM 微调。与标准方法相比，Unsloth 提供约 2 倍更快的训练速度和约 60% 的显存使用减少。 ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#nin-xu-yao) 您需要 * 一个 [Hugging Facearrow-up-right](https://huggingface.co/) 账户（用于 HF Jobs 必需） * 具有写权限的 Hugging Face 令牌 * 一个编码代理（Open Code、Claude Code、Codex） * 阅读我们的 [Claude Code](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/claude-code) 关于如何设置它们的指南。 ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#an-zhuang-ji-neng) 安装技能 #### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#claude-code) Claude Code Claude Code 通过其 [插件系统arrow-up-right](https://code.claude.com/docs/en/discover-plugins) . 1. 添加市场： 复制 /plugin marketplace add huggingface/skills 1. 在以下位置浏览可用的技能： **发现** 选项卡： 复制 /plugin 1. 安装模型训练技能： 复制 /plugin install hugging-face-model-trainer@huggingface-skills 欲了解更多详情，请参阅 [Claude Code 插件 文档arrow-up-right](https://code.claude.com/docs/en/discover-plugins) 以及 [技能 文档arrow-up-right](https://code.claude.com/docs/en/skills) . #### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#codex) Codex Codex 通过 [`AGENTS.md`arrow-up-right](https://developers.openai.com/codex/guides/agents-md) 文件和 [`.agents/skills/`arrow-up-right](https://developers.openai.com/codex/skills) 目录来发现技能。 **使用以下命令安装单个技能** `**$skill-installer**` 欲了解更多详情，请参阅 [Codex 技能 文档arrow-up-right](https://developers.openai.com/codex/skills) 以及 [AGENTS.md 指南arrow-up-right](https://developers.openai.com/codex/guides/agents-md) . ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#kuai-su-kai-shi) 快速开始 技能安装完成后，请让您的编码代理训练一个模型。我们正在使用 [Liquid LFM2.5](https://unsloth.ai/docs/zh/mo-xing/tutorials/lfm2.5) 代理将基于技能中的一个 [示例生成训练脚本arrow-up-right](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/unsloth_sft_example.py) ，将训练提交到 HF Jobs，并通过 Trackio 提供监控链接。 ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#shi-yong-hugging-face-jobs) 使用 Hugging Face Jobs 训练作业将在 [Hugging Face Jobsarrow-up-right](https://huggingface.co/docs/huggingface_hub/guides/jobs) 上运行 — 完全托管的云 GPU。如果您熟悉 Google Colab 积分，Hugging Face Jobs 也提供类似的积分系统。它是按使用付费的结构，或者您可以事先获得积分。代理会： 1. 生成带内联依赖项的 UV 脚本 2. 通过 `hf` CLI 3. 将其提交到 HF Jobs 4. 报告作业 ID 和监控 URL #### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#xun-lian-wan-cheng-de-mo-xing-hui-bei-tui-song-dao-nin-de-hugging-face-hub-cang-ku) 训练完成的模型会被推送到您的 Hugging Face Hub 仓库 示例训练脚本 trainer.push\_to\_hub() 使用 Hugging Face Jobs 进行训练的费用如下： 模型规模 推荐 GPU 大约 费用/小时 `<1B 参数` ~$0.40 t4-small `1-3B 参数` ~$0.60 t4-medium `3-7B 参数` ~$1.00 a10g-small `7-13B 参数` ~$3.00 a10g-large [有关 Hugging Face 空间定价的完整概述，请查看本指南arrow-up-right](https://huggingface.co/docs/hub/en/spaces-overview#hardware-resources) . ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#ci-chu) 此处 * 与编码代理协作的提示 `明确指定要使用的模型和数据集并包含 Hub ID（例如，`, `Qwen/Qwen2.5-0.5B`trl-lib/Capybara * ）。代理将搜索并验证这些组合。 * 如果您希望使用 Unsloth，请明确提及。否则代理会根据模型和预算自行决定框架。 * 在启动大型作业前请先询问费用估算 * 请求 Trackio 监控以获得实时损失曲线 ### [hashtag](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#ti-jiao-hou-qing-rang-dai-li-jian-cha-ri-zhi-yi-cha-kan-zuo-ye-zhuang-tai) 提交后请让代理检查日志以查看作业状态 * [资源：Hugging Face Skills 仓库arrow-up-right](https://github.com/huggingface/skills) 最后更新于 1个月前 这有帮助吗？ * [您需要](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#nin-xu-yao) * [安装技能](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#an-zhuang-ji-neng) * [快速开始](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#kuai-su-kai-shi) * [使用 Hugging Face Jobs](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#shi-yong-hugging-face-jobs) * [此处](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#ci-chu) * [提交后请让代理检查日志以查看作业状态](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/deploying-llms-with-hugging-face-jobs#ti-jiao-hou-qing-rang-dai-li-jian-cha-ri-zhi-yi-cha-kan-zuo-ye-zhuang-tai) 这有帮助吗？ sun-brightdesktopmoon 复制 $skill-installer install https://github.com/huggingface/skills/tree/main/skills/hugging-face-model-trainer 复制 在 HF Jobs 上使用 Unsloth 对 trl-lib/Capybara 上的 LiquidAI/LFM2.5-1.2B-Instruct 进行训练 复制 该技能会生成如下脚本： # /// script # /// # dependencies = ["unsloth", "trl>=0.12.0", "datasets", "trackio"] from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig from datasets import load_dataset model, tokenizer = FastLanguageModel.from_pretrained( "Qwen/Qwen2.5-0.5B", load_in_4bit=True, ) max_seq_length=2048, model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=32, lora_dropout=0, target_modules=["q_proj", "k_proj", "v_proj", "o_proj",\ )\ \ "gate_proj", "up_proj", "down_proj"], dataset = load_dataset("trl-lib/Capybara", split="train") trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, args=SFTConfig( output_dir="./output", push_to_hub=True, hub_model_id="username/my-model", per_device_train_batch_size=4, gradient_accumulation_steps=4, num_train_epochs=1, learning_rate=2e-4, ), ) report_to="trackio", trainer.train() chevron-down显示全部 43 行 sun-brightdesktopmoon --- # Text-to-Speech (TTS) Fine-tuning Guide | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Fine-tuning TTS models allows them to adapt to your specific dataset, use case, or desired style and tone. The goal is to customize these models to clone voices, adapt speaking styles and tones, support new languages, handle specific tasks and more. We also support **Speech-to-Text (STT)** models like OpenAI's Whisper. With [Unslotharrow-up-right](https://github.com/unslothai/unsloth) , you can fine-tune **any** TTS model (`transformers` compatible) 1.5x faster with 50% less memory than other implementations with Flash Attention 2. ⭐ **Unsloth supports any** `**transformers**` **compatible TTS model.** Even if we don’t have a notebook or upload for it yet, it’s still supported e.g., try fine-tuning Dia-TTS or Moshi. circle-info Zero-shot cloning captures tone but misses pacing and expression, often sounding robotic and unnatural. Fine-tuning delivers far more accurate and realistic voice replication. [Read more here](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-voice-models-vs.-zero-shot-voice-cloning) . ### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-notebooks) Fine-tuning Notebooks: We've also uploaded TTS models (original and quantized) to our [Hugging Face pagearrow-up-right](https://huggingface.co/collections/unsloth/text-to-speech-tts-models-68007ab12522e96be1e02155) . [Sesame-CSM (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb) [Orpheus-TTS (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) [Whisper Large V3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) (STT) [Spark-TTS (0.5B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb) [Llasa-TTS (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_(1B).ipynb) [Oute-TTS (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb) circle-check If you notice that the output duration reaches a maximum of 10 seconds, increase`max_new_tokens = 125` from its default value of 125. Since 125 tokens corresponds to 10 seconds of audio, you'll need to set a higher value for longer outputs. ### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#choosing-and-loading-a-tts-model) Choosing and Loading a TTS Model For TTS, smaller models are often preferred due to lower latency and faster inference for end users. Fine-tuning a model under 3B parameters is often ideal, and our primary examples uses Sesame-CSM (1B) and Orpheus-TTS (3B), a Llama-based speech model. #### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#sesame-csm-1b-details) Sesame-CSM (1B) Details **CSM-1B** is a base model, while **Orpheus-ft** is fine-tuned on 8 professional voice actors, making voice consistency the key difference. CSM requires audio context for each speaker to perform well, whereas Orpheus-ft has this consistency built in. Fine-tuning from a base model like CSM generally needs more compute, while starting from a fine-tuned model like Orpheus-ft offers better results out of the box. To help with CSM, we’ve added new sampling options and an example showing how to use audio context for improved voice consistency. #### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#orpheus-tts-3b-details) Orpheus-TTS (3B) Details Orpheus is pre-trained on a large speech corpus and excels at generating realistic speech with built-in support for emotional cues like laughs and sighs. Its architecture makes it one of the easiest TTS models to utilize and train as it can be exported via llama.cpp meaning it has great compatibility across all inference engines. For unsupported models, you'll only be able to save the LoRA adapter safetensors. #### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#loading-the-models) Loading the models Because voice models are usually small in size, you can train the models using LoRA 16-bit or full fine-tuning FFT which may provide higher quality results. To load it in LoRA 16-bit: When this runs, Unsloth will download the model weights if you prefer 8-bit, you could use `load_in_8bit = True`, or for full fine-tuning set `full_finetuning = True` (ensure you have enough VRAM). You can also replace the model name with other TTS models. circle-info **Note:** Orpheus’s tokenizer already includes special tokens for audio output (more on this later). You do _not_ need a separate vocoder – Orpheus will output audio tokens directly, which can be decoded to a waveform. ### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#preparing-your-dataset) Preparing Your Dataset At minimum, a TTS fine-tuning dataset consists of **audio clips and their corresponding transcripts** (text). Let’s use the [_Elise_ datasetarrow-up-right](https://huggingface.co/datasets/MrDragonFox/Elise) which is ~3 hour single-speaker English speech corpus. There are two variants: * [`MrDragonFox/Elise`arrow-up-right](https://huggingface.co/datasets/MrDragonFox/Elise) – an augmented version with **emotion tags** (e.g. , ) embedded in the transcripts. These tags in angle brackets indicate expressions (laughter, sighs, etc.) and are treated as special tokens by Orpheus’s tokenizer * [`Jinsaryko/Elise`arrow-up-right](https://huggingface.co/datasets/Jinsaryko/Elise) – base version with transcripts without special tags. The dataset is organized with one audio and transcript per entry. On Hugging Face, these datasets have fields such as `audio` (the waveform), `text` (the transcription), and some metadata (speaker name, pitch stats, etc.). We need to feed Unsloth a dataset of audio-text pairs. circle-check Instead of solely focusing on tone, cadence, and pitch, the priority should be ensuring your dataset is fully annotated and properly normalized. circle-info With some models like **Sesame-CSM-1B**, you might notice voice variation across generations using speaker ID 0 because it's a **base model**—it doesn’t have fixed voice identities. Speaker ID tokens mainly help maintain **consistency within a conversation**, not across separate generations. To get a consistent voice, provide **contextual examples**, like a few reference audio clips or prior utterances. This helps the model mimic the desired voice more reliably. Without this, variation is expected, even with the same speaker ID. **Option 1: Using Hugging Face Datasets library** – We can load the Elise dataset using Hugging Face’s `datasets` library: This will download the dataset (~328 MB for ~1.2k samples). Each item in `dataset` is a dictionary with at least: * `"audio"`: the audio clip (waveform array and metadata like sampling rate), and * `"text"`: the transcript string Orpheus supports tags like ``, ``, ``, ``, ``, ``, ``, ``, etc. For example: `"I missed you so much!"`. These tags are enclosed in angle brackets and will be treated as special tokens by the model (they match [Orpheus’s expected tagsarrow-up-right](https://github.com/canopyai/Orpheus-TTS) like `` and ``. During training, the model will learn to associate these tags with the corresponding audio patterns. The Elise dataset with tags already has many of these (e.g., 336 occurrences of “laughs”, 156 of “sighs”, etc. as listed in its card). If your dataset lacks such tags but you want to incorporate them, you can manually annotate the transcripts where the audio contains those expressions. **Option 2: Preparing a custom dataset** – If you have your own audio files and transcripts: * Organize audio clips (WAV/FLAC files) in a folder. * Create a CSV or TSV file with columns for file path and transcript. For example: * Use `load_dataset("csv", data_files="mydata.csv", split="train")` to load it. You might need to tell the dataset loader how to handle audio paths. An alternative is using the `datasets.Audio` feature to load audio data on the fly: Then `dataset[i]["audio"]` will contain the audio array. * **Ensure transcripts are normalized** (no unusual characters that the tokenizer might not know, except the emotion tags if used). Also ensure all audio have a consistent sampling rate (resample them if necessary to the target rate the model expects, e.g. 24kHz for Orpheus). In summary, for **dataset preparation**: * You need a **list of (audio, text)** pairs. * Use the HF `datasets` library to handle loading and optional preprocessing (like resampling). * Include any **special tags** in the text that you want the model to learn (ensure they are in `` format so the model treats them as distinct tokens). * (Optional) If multi-speaker, you could include a speaker ID token in the text or use a separate speaker embedding approach, but that’s beyond this basic guide (Elise is single-speaker). ### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-tts-with-unsloth) Fine-Tuning TTS with Unsloth Now, let’s start fine-tuning! We’ll illustrate using Python code (which you can run in a Jupyter notebook, Colab, etc.). **Step 1: Load the Model and Dataset** In all our TTS notebooks, we enable LoRA (16-bit) training and disable QLoRA (4-bit) training with: `load_in_4bit = False`. This is so the model can usually learn your dataset better and have higher accuracy. circle-info If memory is very limited or if dataset is large, you can stream or load in chunks. Here, 3h of audio easily fits in RAM. If using your own dataset CSV, load it similarly. **Step 2: Advanced - Preprocess the data for training (Optional)** We need to prepare inputs for the Trainer. For text-to-speech, one approach is to train the model in a causal manner: concatenate text and audio token IDs as the target sequence. However, since Orpheus is a decoder-only LLM that outputs audio, we can feed the text as input (context) and have the audio token ids as labels. In practice, Unsloth’s integration might do this automatically if the model’s config identifies it as text-to-speech. If not, we can do something like: circle-info The above is a simplification. In reality, to fine-tune Orpheus properly, you would need the _audio tokens as part of the training labels_. Orpheus’s pre-training likely involved converting audio to discrete tokens (via an audio codec) and training the model to predict those given the preceding text. For fine-tuning on new voice data, you would similarly need to obtain the audio tokens for each clip (using Orpheus’s audio codec). The Orpheus GitHub provides a script for data processing – it encodes audio into sequences of `` tokens. However, **Unsloth may abstract this away**: if the model is a FastModel with an associated processor that knows how to handle audio, it might automatically encode the audio in the dataset to tokens. If not, you’d have to manually encode each audio clip to token IDs (using Orpheus’s codebook). This is an advanced step beyond this guide, but keep in mind that simply using text tokens won’t teach the model the actual audio – it needs to match the audio patterns. Let's assume Unsloth provides a way to feed audio directly (for example, by setting `processor` and passing the audio array). If Unsloth does not yet support automatic audio tokenization, you might need to use the Orpheus repository’s `encode_audio` function to get token sequences for the audio, then use those as labels. (The dataset entries do have `phonemes` and some acoustic features which suggests a pipeline.) **Step 3: Set up training arguments and Trainer** We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. Using a per\_device\_train\_batch\_size >1 may lead to errors if multi-GPU setup to avoid issues, ensure CUDA\_VISIBLE\_DEVICES is set to a single GPU (e.g., CUDA\_VISIBLE\_DEVICES=0). Adjust as needed. **Step 4: Begin fine-tuning** This will start the training loop. You should see logs of loss every 50 steps (as set by `logging_steps`). The training might take some time depending on GPU – for example, on a Colab T4 GPU, a few epochs on 3h of data may take 1-2 hours. Unsloth’s optimizations will make it faster than standard HF training. **Step 5: Save the fine-tuned model** After training completes (or if you stop it mid-way when you feel it’s sufficient), save the model. This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down! This saves the model weights (for LoRA, it might save only adapter weights if the base is not fully fine-tuned). If you used `--push_model` in CLI or `trainer.push_to_hub()`, you could upload it to Hugging Face Hub directly. Now you should have a fine-tuned TTS model in the directory. The next step is to test it out and if supported, you can use llama.cpp to convert it into a GGUF file. ### [hashtag](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-voice-models-vs.-zero-shot-voice-cloning) Fine-tuning Voice models vs. Zero-shot voice cloning People say you can clone a voice with just 30 seconds of audio using models like XTTS - no training required. That’s technically true, but it misses the point. Zero-shot voice cloning, which is also available in models like Orpheus and CSM, is an approximation. It captures the general **tone and timbre** of a speaker’s voice, but it doesn’t reproduce the full expressive range. You lose details like speaking speed, phrasing, vocal quirks, and the subtleties of prosody - things that give a voice its **personality and uniqueness**. If you just want a different voice and are fine with the same delivery patterns, zero-shot is usually good enough. But the speech will still follow the **model’s style**, not the speaker’s. For anything more personalized or expressive, you need training with methods like LoRA to truly capture how someone speaks. [PreviousFaster MoE Trainingchevron-left](https://unsloth.ai/docs/basics/faster-moe) [NextDynamic 2.0 GGUFschevron-right](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) Last updated 2 months ago Was this helpful? * [Fine-tuning Notebooks:](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-notebooks) * [Choosing and Loading a TTS Model](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#choosing-and-loading-a-tts-model) * [Preparing Your Dataset](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#preparing-your-dataset) * [Fine-Tuning TTS with Unsloth](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-tts-with-unsloth) * [Fine-tuning Voice models vs. Zero-shot voice cloning](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning#fine-tuning-voice-models-vs.-zero-shot-voice-cloning) Was this helpful? sun-brightdesktopmoon Copy from unsloth import FastModel model_name = "unsloth/orpheus-3b-0.1-pretrained" model, tokenizer = FastModel.from_pretrained( model_name, load_in_4bit=False # use 4-bit precision (QLoRA) ) Copy from datasets import load_dataset, Audio # Load the Elise dataset (e.g., the version with emotion tags) dataset = load_dataset("MrDragonFox/Elise", split="train") print(len(dataset), "samples") # ~1200 samples in Elise # Ensure all audio is at 24 kHz sampling rate (Orpheus’s expected rate) dataset = dataset.cast_column("audio", Audio(sampling_rate=24000)) Copy filename,text 0001.wav,Hello there! 0002.wav, I am very tired. Copy from datasets import Audio dataset = load_dataset("csv", data_files="mydata.csv", split="train") dataset = dataset.cast_column("filename", Audio(sampling_rate=24000)) Copy from unsloth import FastLanguageModel import torch dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False. model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/orpheus-3b-0.1-ft", max_seq_length= 2048, # Choose any for long context! dtype = dtype, load_in_4bit = load_in_4bit, #token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf ) from datasets import load_dataset dataset = load_dataset("MrDragonFox/Elise", split = "train") Copy # Tokenize the text transcripts def preprocess_function(example): # Tokenize the text (keep the special tokens like intact) tokens = tokenizer(example["text"], return_tensors="pt") # Flatten to list of token IDs input_ids = tokens["input_ids"].squeeze(0) # The model will generate audio tokens after these text tokens. # For training, we can set labels equal to input_ids (so it learns to predict next token). # But that only covers text tokens predicting the next text token (which might be an audio token or end). # A more sophisticated approach: append a special token indicating start of audio, and let the model generate the rest. # For simplicity, use the same input as labels (the model will learn to output the sequence given itself). return {"input_ids": input_ids, "labels": input_ids} train_data = dataset.map(preprocess_function, remove_columns=dataset.column_names) Copy from transformers import TrainingArguments,Trainer,DataCollatorForSeq2Seq from unsloth import is_bfloat16_supported trainer = Trainer( model = model, train_dataset = dataset, args = TrainingArguments( per_device_train_batch_size = 1, gradient_accumulation_steps = 4, warmup_steps = 5, # num_train_epochs = 1, # Set this for 1 full training run. max_steps = 60, learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # Use this for WandB etc ), ) Copy model.save_pretrained("lora_model") # Local saving tokenizer.save_pretrained("lora_model") # model.push_to_hub("your_name/lora_model", token = "...") # Online saving # tokenizer.push_to_hub("your_name/lora_model", token = "...") # Online saving sun-brightdesktopmoon --- # Fine-tuning Embedding Models with Unsloth Guide | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Fine-tuning embedding models can largely improve retrieval and RAG performance on specific tasks. It aligns the model's vectors with your domain and the kind of 'similarity' that matters for your use case, which improves search, RAG, clustering, and recommendations on your data. Example: The headlines “Google launches Pixel 10” and “Qwen releases Qwen3” might be embedded as similar if you’re just labeling both as 'Tech,' but not similar if you’re doing semantic search because they’re about different things. Fine-tuning helps the model make the 'right' kind of similarity for your use case, reducing errors and improving results. [**Unsloth**arrow-up-right](https://github.com/unslothai/unsloth) now supports training embedding, **classifier**, **BERT**, **reranker** models [**~1.8-3.3x faster**](https://unsloth.ai/docs/basics/embedding-finetuning#unsloth-benchmarks) with 20% less memory and 2x longer context than other Flash Attention 2 implementations - no accuracy degradation. EmbeddingGemma-300M works on just **3GB VRAM**. You can use your trained **model anywhere**: transformers, LangChain, Ollama, vLLM, llama.cpp etc. Unsloth uses [SentenceTransformersarrow-up-right](https://github.com/huggingface/sentence-transformers) to support compatible models like Qwen3-Embedding, BERT and more. **Even if there's no notebook or upload, it’s still supported.** **We created free fine-tuning notebooks, with 3 main use-cases:** [EmbeddingGemma (300M)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) [Qwen3-Embedding 4Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(4B).ipynb) • [0.6Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(0_6B).ipynb) [BGE M3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/BGE_M3.ipynb) [ModernBERTarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) - classification [All-MiniLM-L6-v2arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/All_MiniLM_L6_v2.ipynb) [ModernBERT-largearrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) * `All-MiniLM-L6-v2`: produce compact, domain-specific sentence embeddings for semantic search, retrieval, and clustering, tuned on your own data. * `tomaarsen/miriad-4.4M-split`: embed medical questions and biomedical papers for high-quality medical semantic search and RAG. * `electroglyph/technical`: better capture meaning and semantic similarity in technical text (docs, specs, and engineering discussions). You can view the rest of our uploaded models in [our collection herearrow-up-right](https://huggingface.co/collections/unsloth/embedding-models) . > A huge thanks to Unsloth contributor [**electroglyph**arrow-up-right](https://github.com/unslothai/unsloth/pull/3719) > , whose work was significant to support this. You can check out electroglyph’s custom models on Hugging Face [herearrow-up-right](https://huggingface.co/electroglyph) > . ### [hashtag](https://unsloth.ai/docs/basics/embedding-finetuning#unsloth-features) 🦥 Unsloth Features * LoRA/QLoRA or full fine-tuning for embeddings, without needing to rewrite your pipeline * Best support for encoder-only `SentenceTransformer` models (with a `modules.json`) * Cross-encoder models are confirmed to train properly even under the fallback path * This release also supports `transformers v5` There is limited support for models without `modules.json` (we’ll auto-assign default `SentenceTransformers` pooling modules). If you’re doing something custom (custom heads, nonstandard pooling), double-check outputs like the pooled embedding behavior. Some models needed custom additions such as MPNet or DistilBERT were enabled by patching gradient checkpointing into the `transformers` models. ### [hashtag](https://unsloth.ai/docs/basics/embedding-finetuning#fine-tuning-workflow) 🛠️ Fine-tuning Workflow The new fine-tuning flow is centered around `FastSentenceTransformer`. Main save/push methods: * `save_pretrained()` Saves **LoRA adapters** to a local folder * `save_pretrained_merged()` Saves the **merged model** to a local folder * `push_to_hub()` Pushes **LoRA adapters** to Hugging Face * `push_to_hub_merged()` Pushes the **merged model** to Hugging Face **And one very important detail: Inference loading requires** `**for_inference=True**` `from_pretrained()` is similar to Lacker’s other fast classes, with **one exception**: * To load a model for **inference** using `FastSentenceTransformer`, you **must** pass: `for_inference=True` So your inference loads should look like: For Hugging Face authorization, if you run: inside the same virtualenv before calling the hub methods, then: * `push_to_hub()` and `push_to_hub_merged()` **don’t require a token argument**. ### [hashtag](https://unsloth.ai/docs/basics/embedding-finetuning#docs-internal-guid-c10bfa80-7fff-446e-714d-732eebcd72d6) ✅ Inference and Deploy Anywhere! Your fine-tuned Unsloth model can be used and deployed with all major tools: transformers, LangChain, Weaviate, sentence-transformers, Text Embeddings Inference (TEI), vLLM, and llama.cpp, custom embedding API, pgvector, FAISS/vector databases, and any RAG framework. There is no lock in as the fine-tuned model can later be downloaded locally on your own device. ### [hashtag](https://unsloth.ai/docs/basics/embedding-finetuning#unsloth-benchmarks) 📊 Unsloth Benchmarks Unsloth's advantages include speed for embedding fine-tuning! We show we are consistently **1.8 to 3.3x faster** on a wide variety of embedding models and on different sequence lengths from 128 to 2048 and longer. EmbeddingGemma-300M QLoRA works on just **3GB VRAM** and LoRA works on 6GB VRAM. Below are our Unsloth benchmarks in a heatmap vs. `SentenceTransformers` + Flash Attention 2 (FA2) for 4bit QLoRA. **For 4bit QLoRA, Unsloth is 1.8x to 2.6x faster:** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQqagyYR6DebgX768A0HV%252Foutput%2816%29.png%3Falt%3Dmedia%26token%3De3ea6510-b129-401a-83ae-301d01865547&width=768&dpr=3&quality=100&sign=df622c31&sv=2) Below are our Unsloth benchmarks in a heatmap vs. `SentenceTransformers` + Flash Attention 2 (FA2) for 16bit LoRA. **For 16bit LoRA, Unsloth is 1.2x to 3.3x faster:** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FTl12zuBg68ZPSyOC9hUe%252Foutput%2815%29.png%3Falt%3Dmedia%26token%3D47d7cade-7eac-4366-8011-7034de087431&width=768&dpr=3&quality=100&sign=2247dd61&sv=2) ### [hashtag](https://unsloth.ai/docs/basics/embedding-finetuning#model-support) 🔮 Model Support Here are some popular embedding models Unsloth supports (not all models are listed here): Most [common modelsarrow-up-right](https://huggingface.co/models?library=sentence-transformers) are already supported. If there’s an encoder-only model you’d like that isn’t, feel free to open a [GitHub issuearrow-up-right](https://github.com/unslothai/unsloth/issues) requesting it. [PreviousDistributed Data Parallel (DDP)chevron-left](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp) [NextFaster MoE Trainingchevron-right](https://unsloth.ai/docs/basics/faster-moe) Last updated 19 days ago Was this helpful? * [🦥 Unsloth Features](https://unsloth.ai/docs/basics/embedding-finetuning#unsloth-features) * [🛠️ Fine-tuning Workflow](https://unsloth.ai/docs/basics/embedding-finetuning#fine-tuning-workflow) * [✅ Inference and Deploy Anywhere!](https://unsloth.ai/docs/basics/embedding-finetuning#docs-internal-guid-c10bfa80-7fff-446e-714d-732eebcd72d6) * [📊 Unsloth Benchmarks](https://unsloth.ai/docs/basics/embedding-finetuning#unsloth-benchmarks) * [🔮 Model Support](https://unsloth.ai/docs/basics/embedding-finetuning#model-support) Was this helpful? sun-brightdesktopmoon Copy model = FastSentenceTransformer.from_pretrained( "sentence-transformers/all-MiniLM-L6-v2", for_inference=True, ) Copy hf auth login Copy # 1. Load a pretrained Sentence Transformer model model = SentenceTransformer("}]},\ {'role': 'assistant',\ 'content': [{'type': 'text',\ 'text': 'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'}]}]} Copy FastVisionModel.for_inference(model) # Enable for inference! image = dataset[0]["image"] instruction = "You are an expert radiographer. Describe accurately what you see in this image." messages = [\ {"role": "user", "content": [\ {"type": "image"},\ {"type": "text", "text": instruction}\ ]}\ ] input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True) inputs = tokenizer( image, input_text, add_special_tokens = False, return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer, skip_prompt = True) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True, temperature = 1.5, min_p = 0.1) chevron-downShow all 23 lines Copy This radiograph appears to be a panoramic view of the upper and lower dentition, specifically an Orthopantomogram (OPG). * The panoramic radiograph demonstrates normal dental structures. * There is an abnormal area on the upper right, represented by an area of radiolucent bone, corresponding to the antrum. **Key Observations** * The bone between the left upper teeth is relatively radiopaque. * There are two large arrows above the image, suggesting the need for a closer examination of this area. One of the arrows is in a left-sided position, and the other is in the right-sided position. However, only Copy class UnslothVisionDataCollator: def __init__( self, ... # from unsloth.chat_templates import train_on_responses_only # trainer = train_on_responses_only( # trainer, # instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", # response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", # ) train_on_responses_only = False, # EQUIVALENT to train_on_responses_only for LLMs instruction_part = None, # EQUIVALENT to train_on_responses_only(instruction_part = ...) response_part = None, # EQUIVALENT to train_on_responses_only(response_part = ...) force_match = True, # Match newlines as well! ) Copy UnslothVisionDataCollator( model, tokenizer, ... train_on_responses_only = True, instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", ... ) sun-brightdesktopmoon --- # Fine-tuning for Beginners | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close If you're a beginner, here might be the first questions you'll ask before your first fine-tune. You can also always ask our community by joining our [Reddit pagearrow-up-right](https://www.reddit.com/r/unsloth/) . [](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) 🧬[Fine-tuning Guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) Step-by-step on how to fine-tune! Learn the core basics of training. [](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use) ❓[What Model Should I Use?](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use) Instruct or Base Model? How big should my dataset be? [](https://unsloth.ai/docs/models/tutorials) 🚀[Complete LLM Directory](https://unsloth.ai/docs/models/tutorials) How to Run & Fine-tune DeepSeek? What settings should I set when running Gemma 3? [](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) 🤔[FAQ + Is Fine-tuning Right For Me?](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) What can fine-tuning do for me? RAG vs. Fine-tuning? [](https://unsloth.ai/docs/get-started/install) 📥[Installation](https://unsloth.ai/docs/get-started/install) How do I install Unsloth locally? How to update Unsloth? 📈[Datasets Guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide) How do I structure/prepare my dataset? How do I collect data? [](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) 🛠️[Unsloth Requirements](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) Does Unsloth work on my GPU? How much VRAM will I need? [](https://unsloth.ai/docs/basics/inference-and-deployment) 🖥️[Inference & Deployment](https://unsloth.ai/docs/basics/inference-and-deployment) How do I save my model locally? How do I run my model via Ollama or vLLM? 🧠[Hyperparameters Guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) What happens when I change a parameter? What parameters should I change? ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-559e7890f607e34fd6004517296e65e942c93b41%252FLarge%2520sloth%2520Question%2520mark.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=47c4ca62&sv=2) [PreviousHomepagechevron-left](https://unsloth.ai/docs) [NextUnsloth Requirementschevron-right](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) Last updated 3 months ago Was this helpful? Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth Notebooks | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Train your own model with our notebooks, powered by free GPU compute. Click Run all (or save locally), add your dataset, train, and deploy. You can use any model in the notebooks. [GRPO (RL)](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl) [Text-to-speech](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts) [Vision](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal) [Embedding](https://unsloth.ai/docs/get-started/unsloth-notebooks#embedding-models) [Kaggle](https://unsloth.ai/docs/get-started/unsloth-notebooks#kaggle-notebooks) Also see our GitHub repo for our notebooks: [github.com/unslothai/notebooksarrow-up-right](https://github.com/unslothai/notebooks/) [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#colab-notebooks) Colab notebooks ----------------------------------------------------------------------------------------------------- **Introducing our** [**Unsloth Studio**](https://unsloth.ai/docs/new/studio) ✨ **notebook.** Train and run models under 22B parameters: [![Logo](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2Fssl.gstatic.com%2Fcolaboratory-static%2Fcommon%2Fc04527d1807929a628269bb0e1319bf2%2Fimg%2Ffavicon.ico&width=20&dpr=3&quality=100&sign=48a6a277&sv=2)Google Colabcolab.research.google.comchevron-right](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#standard-sft-notebooks) Standard SFT notebooks: * [**Gemma 4**](https://unsloth.ai/docs/models/gemma-4/train) **:** [E4B **(Vision)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E4B)-Vision.ipynb) **•** [E2B **(Text)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Text.ipynb) **•** [E2B **(Audio)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Audio.ipynb) **•** [**31B** (Kaggle)arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) **•** [**Inference**arrow-up-right](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) * [**Qwen3.5**](https://unsloth.ai/docs/models/qwen3.5/fine-tune) **:** [**0.8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(0_8B)_Vision.ipynb) • [**2B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(2B)_Vision.ipynb) • [**4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision.ipynb) * [gpt-oss (20b)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) • [Inferencearrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_(20B)-Inference.ipynb) • [Fine-tuningarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) * [EmbeddingGemma (300M)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) * [Qwen3 (14B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) • [**Qwen3-VL (8B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) * [**Qwen3-2507-4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) • [Thinkingarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Thinking.ipynb) • [Instructarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) * [Gemma 3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) • [Textarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) • [Visionarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) • [270Marrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) • [**FunctionGemma**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) * [Gemma 3n (E4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) • [Textarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) • [Visionarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) • [Audioarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) * [**Mistral Ministral 3**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) * [**DeepSeek-OCR 2**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_2_(3B).ipynb) * [IBM Granite-4.0-Harrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [Phi-4 (14B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb) * [Llama 3.1 (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) • [Llama 3.2 (1B + 3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl) GRPO (Reasoning RL): * [**Qwen3.5 (4B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision_GRPO.ipynb) \- Vision GRPO - new * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) (automatic kernels creation) * [Mistral Ministral 3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb) (solving sodoku) - new * [Qwen3-8B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) (L4) - new * [Llama-3.2-1B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama_FP8_GRPO.ipynb) (L4) - new * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) (auto win 2048 game) * [Qwen3-VL (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - Vision GSPO * [Qwen3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) - Advanced GRPO LoRA * [Gemma 3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) - Vision GSPO * [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) (2048 OpenEnv example) * [DeepSeek-R1-0528-Qwen3 (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) (for multilingual usecase) * [Gemma 3 (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) * [Llama 3.2 (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_(3B)_GRPO_LoRA.ipynb) - Advanced GRPO LoRA * [Llama 3.1 (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb) * [Phi-4 (14B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) * [Mistral v0.3 (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-GRPO.ipynb) * [NeMo Gym Multi Agents Environment arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb) (Multiple agentic environments) ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts) Text-to-Speech (TTS): * [Sesame-CSM (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb) * [Orpheus-TTS (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) - Speech-to-Text (STT) * [Llasa-TTS (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_(1B).ipynb) * [Spark-TTS (0.5B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb) * [Oute-TTS (1B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb) **Speech-to-Text (SST):** * [**Gemma 4 (E2B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Audio.ipynb) **- Audio - new** * [Whisper-Large-V3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) * [Gemma 3n (E4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) - Audio ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal) Vision (Multimodal): * [**Gemma 4**](https://unsloth.ai/docs/models/gemma-4/train) **:** [E2Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Vision.ipynb) - new * [**Qwen3.5**](https://unsloth.ai/docs/models/qwen3.5/fine-tune) **:** [**0.8B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(0_8B)_Vision.ipynb) • [**2B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(2B)_Vision.ipynb) • [**4B**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision.ipynb) - new * [**Qwen3-VL (8B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) * [**Mistral Ministral 3**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) * [**DeepSeek-OCR**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb) * [**Paddle-OCR (1B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Paddle_OCR_(1B)_Vision.ipynb) * [Gemma 3n (E4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) * [Gemma 3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) * [Llama 3.2 Vision (11B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb) * [Qwen2.5-VL (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral (12B) 2409arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Pixtral_(12B)-Vision.ipynb) * [Qwen3-VLarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - Vision GSPO - new * [Qwen2.5-VLarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) - Vision GSPO * [Gemma 3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) - Vision GSPO ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#embedding-models) Embedding models: * [EmbeddingGemma (300M)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) - new * [Qwen3-Embedding 4Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(4B).ipynb) - new * [Qwen3-Embedding 0.6Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_(0_6B).ipynb) - new * [BGE M3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/BGE_M3.ipynb) - new * [ModernBERT-largearrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) - new * [All-MiniLM-L6-v2arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/All_MiniLM_L6_v2.ipynb) - new * [GTE ModernBertarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ModernBert.ipynb) - new ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#large-llms) Large LLMs: **Notebooks for large models:** These exceed Colab’s free 15 GB VRAM tier. With Colab’s new 80 GB GPUs, you can fine-tune 120B parameter models. circle-info Colab subscription or credits are required. We **don't** earn anything from these notebooks. * [**Gemma-4-31B** (Kaggle)arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - new and **FREE** * [Gemma-4-26B-A4Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(26B_A4B)-Vision.ipynb) - new * [Gemma-4-31Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(31B)-Vision.ipynb) - new * [Qwen3.5-35B-A3Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_MoE.ipynb) * [Qwen3.5‑27Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen_3_5_27B_A100(80GB).ipynb) * [GLM-4.7-Flasharrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100(80GB).ipynb) * [gpt-oss-20b (500K context)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) * [Qwen3-30B-A3Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_MoE.ipynb) * [Nemotron-3-Nano-30B-A3B LoRA notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Nemotron-3-Nano-30B-A3B_A100.ipynb) * [NeMo Gym Sudoku GRPO notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Sudoku.ipynb) * [NeMo Gym Multi Environment GRPO Notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb) * [gpt-oss-120barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(120B)_A100-Fine-tuning.ipynb) * [Qwen3 (32B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(32B)_A100-Reasoning-Conversational.ipynb) * [Llama 3.3 (70B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.3_(70B)_A100-Conversational.ipynb) * [Gemma 3 (27B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(27B)_A100-Conversational.ipynb) * [Baidu ERNIE 4.5 VL (28B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ERNIE_4_5_VL_28B_A3B_PT_Vision.ipynb) - new ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#other-important-notebooks) Other important notebooks: * [**Customer support agent**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [Mistral Ministral 3arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb) - new (solving sodoku) * [Deploy on LM Studio arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-LMStudio.ipynb) \- new * [Quantization-Aware Trainingarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) (QAT) - new * [Phone Deployment arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(0_6B)-Phone_Deployment.ipynb) \- new * [Reason before **Tool Calling** notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) - new * [Mobile Actions notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb) - new * [**Automatic Kernel Creation**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) with RL * [**ModernBERT-large**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) **\- new** Aug 19 * [**Synthetic Data Generation Llama 3.2 (3B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) * [gpt-oss-20b (500K context)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) \- new (A100) * [**Tool Calling**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) * [Mistral v0.3 Instruct (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Conversational.ipynb) * [Ollamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [ORPOarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-ORPO.ipynb) * [Continued Pretrainingarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyrarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [_**Inference only**_arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Inference.ipynb) * [Llama 3 (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Alpaca.ipynb) ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#specific-use-case-notebooks) Specific use-case notebooks: * [Phone Deployment arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(0_6B)-Phone_Deployment.ipynb) \- new * [Deploy on LM Studio arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-LMStudio.ipynb) \- new * [Reason before **Tool Calling**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb) - new * [Mobile Actionsarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb) - new * [**Customer support agent**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) * [Quantization-Aware Trainingarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) (QAT) - new * [**Automatic Kernel Creation**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) with RL **\- new** * [DPO Zephyrarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [BERT - Text Classificationarrow-up-right](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) - (AutoModelForSequenceClassification) * [Ollamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [**Tool Calling**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) * [Continued Pretraining (CPT)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [Multiple Datasetsarrow-up-right](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) by Flail * [KTOarrow-up-right](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) by Jeffrey * [Inference chat UIarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Unsloth_Studio.ipynb) * [Conversationalarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [ChatMLarrow-up-right](https://colab.research.google.com/drive/15F1xyn8497_dUbxZP4zWmPZ3PJx1Oymv?usp=sharing) * [Text Completionarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) ### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#rest-of-notebooks) Rest of notebooks: * [Qwen2.5 (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) * [Gemma 2 (9B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(9B)-Alpaca.ipynb) * [Mistral NeMo (12B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Nemo_(12B)-Alpaca.ipynb) * [Phi-3.5 (mini)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3.5_Mini-Conversational.ipynb) * [Phi-3 (medium)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3_Medium-Conversational.ipynb) * [Gemma 2 (2B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(2B)-Alpaca.ipynb) * [Qwen 2.5 Coder (14B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(14B)-Conversational.ipynb) * [Mistral Small (22B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlamaarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/TinyLlama_(1.1B)-Alpaca.ipynb) * [CodeGemma (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/CodeGemma_(7B)-Conversational.ipynb) * [Mistral v0.3 (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Alpaca.ipynb) * [Qwen2 (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_(7B)-Alpaca.ipynb) [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#kaggle-notebooks) Kaggle notebooks ------------------------------------------------------------------------------------------------------- #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#standard-notebooks) Standard notebooks: * [**Gemma-4-31B** (Kaggle)arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - new and **FREE** * [**gpt-oss (20B)**arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-gpt-oss-(20B)-Fine-tuning.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3n (E4B)arrow-up-right](https://www.kaggle.com/code/danielhanchen/gemma-3n-4b-multimodal-finetuning-inference) * [Qwen3 (14B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(14B).ipynb) * [Magistral-2509 (24B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_(24B)-Reasoning-Conversational.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3 (4B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(4B).ipynb) * [Phi-4 (14B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4-Conversational.ipynb) * [Llama 3.1 (8B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Alpaca.ipynb) * [Llama 3.2 (1B + 3B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(1B_and_3B)-Conversational.ipynb) * [Qwen 2.5 (7B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(7B)-Alpaca.ipynb) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-notebooks) GRPO (Reasoning) notebooks: * [**Qwen2.5-VL**arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2_5_7B_VL_GRPO.ipynb&accelerator=nvidiaTeslaT4) - Vision GRPO - new * [Qwen3 (4B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(4B)-GRPO.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3 (1B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(1B)-GRPO.ipynb) * [Llama 3.1 (8B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-GRPO.ipynb) * [Phi-4 (14B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4_(14B)-GRPO.ipynb) * [Qwen 2.5 (3B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(3B)-GRPO.ipynb) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts-notebooks) Text-to-Speech (TTS) notebooks: * [Sesame-CSM (1B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Sesame_CSM_(1B)-TTS.ipynb) * [Orpheus-TTS (3B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Whisper.ipynb) – Speech-to-Text * [Llasa-TTS (1B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llasa_TTS_(1B).ipynb) * [Spark-TTS (0.5B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Spark_TTS_(0_5B).ipynb) * [Oute-TTS (1B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Oute_TTS_(1B).ipynb) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal-notebooks) Vision (Multimodal) notebooks: * [Llama 3.2 Vision (11B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(11B)-Vision.ipynb) * [Qwen 2.5-VL (7B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral (12B) 2409arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Pixtral_(12B)-Vision.ipynb) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#specific-use-case-notebooks-1) Specific use-case notebooks: * [Tool Callingarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb&accelerator=nvidiaTeslaT4) * [ORPOarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-ORPO.ipynb) * [Continued Pretrainingarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyrarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Zephyr_(7B)-DPO.ipynb) * [Inference onlyarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Inference.ipynb) * [Ollamaarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-Ollama.ipynb) * [Text Completionarrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_(7B)-Text_Completion.ipynb) * [CodeForces-cot (Reasoning)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeForces-cot-Finetune_for_Reasoning_on_CodeForces.ipynb) * [Unsloth Studio (chat UI)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Unsloth_Studio.ipynb) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-notebooks#rest-of-notebooks-1) Rest of notebooks: * [Gemma 2 (9B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(9B)-Alpaca.ipynb) * [Gemma 2 (2B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(2B)-Alpaca.ipynb) * [CodeGemma (7B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeGemma_(7B)-Conversational.ipynb) * [Mistral NeMo (12B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Nemo_(12B)-Alpaca.ipynb) * [Mistral Small (22B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlama (1.1B)arrow-up-right](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-TinyLlama_(1.1B)-Alpaca.ipynb) To view a complete list of all our Kaggle notebooks, [click herearrow-up-right](https://github.com/unslothai/notebooks#-kaggle-notebooks) . circle-info Feel free to contribute to the notebooks by visiting our [repoarrow-up-right](https://github.com/unslothai/notebooks) ! [PreviousFAQ + Is Fine-tuning Right For Me?chevron-left](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) [NextAll Our Modelschevron-right](https://unsloth.ai/docs/get-started/unsloth-model-catalog) Last updated 3 days ago Was this helpful? * [Colab notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks#colab-notebooks) * [Standard SFT notebooks:](https://unsloth.ai/docs/get-started/unsloth-notebooks#standard-sft-notebooks) * [GRPO (Reasoning RL):](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl) * [Text-to-Speech (TTS):](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts) * [Vision (Multimodal):](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal) * [Embedding models:](https://unsloth.ai/docs/get-started/unsloth-notebooks#embedding-models) * [Large LLMs:](https://unsloth.ai/docs/get-started/unsloth-notebooks#large-llms) * [Other important notebooks:](https://unsloth.ai/docs/get-started/unsloth-notebooks#other-important-notebooks) * [Specific use-case notebooks:](https://unsloth.ai/docs/get-started/unsloth-notebooks#specific-use-case-notebooks) * [Rest of notebooks:](https://unsloth.ai/docs/get-started/unsloth-notebooks#rest-of-notebooks) * [Kaggle notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks#kaggle-notebooks) Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Get started with Unsloth Studio | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth Studio is a local, browser-based GUI for fine-tuning LLMs without writing any code. It wraps the training pipeline in a clean interface that handles model loading, dataset formatting, hyperparameter configuration, and live training monitoring. [boltStudio](https://unsloth.ai/docs/new/studio/start#studio-quickstart) [hat-chefData Recipe](https://unsloth.ai/docs/new/studio/start#data-recipes-quickstart) [box-isometricExport](https://unsloth.ai/docs/new/studio/start#export-quickstart) [comment-dotsChat](https://unsloth.ai/docs/new/studio/start#chat-quickstart) [videoVideo](https://unsloth.ai/docs/new/studio/start#video-tutorial) #### [hashtag](https://unsloth.ai/docs/new/studio/start#setup-unsloth-studio) Setup Unsloth Studio First, launch Unsloth Studio using either a local install or a cloud option. Follow the [install instructions](https://unsloth.ai/docs/new/studio/install) for your setup, or use our [free Colab](https://unsloth.ai/docs/new/studio#google-colab-notebook) notebook. For a local setup, run: Copy unsloth studio -H 0.0.0.0 -p 8888 Then open `http://localhost:8888` in your browser. On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time and configure everything manually. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZPtRrwafzmVX54HhyyBD%252FScreenshot%25202026-03-16%2520at%252011.25.22%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3D9153c153-ec61-4fff-b1b9-db7f70ac2936&width=768&dpr=3&quality=100&sign=87f4bffb&sv=2) [hashtag](https://unsloth.ai/docs/new/studio/start#studio-quickstart) bolt Studio - Quickstart --------------------------------------------------------------------------------------------------- Unsloth Studio homepage has 4 main areas: [Model](https://unsloth.ai/docs/new/studio/start#id-1.-select-model-and-method) , [Dataset](https://unsloth.ai/docs/new/studio/start#id-2.-dataset) , [Parameters](https://unsloth.ai/docs/new/studio/start#id-3.-hyperparameters) , and [Training/Config](https://unsloth.ai/docs/new/studio/start#id-4.-training-and-config) * **Easy setup for models and data** from Hugging Face or local files * **Flexible training choices** like QLoRA, LoRA, or full fine-tuning, with defaults filled in * **Helpful config tools** for splits, column mapping, hyperparameters and YAML configs * **Great training visibility** with live progress, GPU stats, charts, startup status ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxV1PO5DbF3ksB51nE2Tw%252Fmore%2520cropped%2520ui%2520for%2520homepage.png%3Falt%3Dmedia%26token%3Df75942c9-3d8d-4b59-8ba2-1a4a38de1b86&width=768&dpr=3&quality=100&sign=a663c397&sv=2) ### [hashtag](https://unsloth.ai/docs/new/studio/start#id-1.-select-model-and-method) 1\. Select model and method #### [hashtag](https://unsloth.ai/docs/new/studio/start#model-type) **Model Type** Select the modality that matches your use-case: Type Use case **Text** Chat, instruction following, completion **Vision** Image + text (VLMs) **Audio** Speech / audio understanding **Embeddings** Sentence embeddings, retrieval #### [hashtag](https://unsloth.ai/docs/new/studio/start#training-method) **Training Method** Three methods are available, toggled with a pill selector: Method Description VRAM **QLoRA** 4-bit quantized base model + LoRA adapter Lowest **LoRA** Full-precision base model + LoRA adapter Medium **Full Fine-tuning** All weights are trained Highest Type any Hugging Face model name or search the Hub directly from the combobox. Local models stored in `~/.unsloth/studio/models` and your Hugging Face cache also appear in the list. circle-exclamation GGUF format models are excluded from training - they are inference only. When you pick a model the Studio automatically fetches its configuration from the backend and pre-fills sensible defaults for all hyperparameters. **HuggingFace Token** Paste your Hugging Face access token here if the model is gated (e.g. Llama, Gemma). The token is validated in real-time and an error is shown inline if it is invalid. ### [hashtag](https://unsloth.ai/docs/new/studio/start#id-2.-dataset) 2\. Dataset Switch between two tabs to choose where your data comes from: * **HuggingFace Hub** - live search against the Hub. The last-updated date is shown for each result. * **Local** - drag-and-drop or click to upload a file unstructured or structured files like: `PDF`, `DOCX`, `JSONL`, `JSON`, `CSV`, or `Parquet` format. Previously uploaded datasets appear in a list that refreshes automatically. You can view our detailed [Datasets Guide here](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide) . Prompt Studio how to interpret and format your data: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FCtWUm7GdHnKbe14fUQyT%252Fupdated_dataset.webp%3Falt%3Dmedia%26token%3D3fcefe8d-b6a4-44c2-be9b-6dc282166095&width=768&dpr=3&quality=100&sign=96190e78&sv=2) Format When to use `auto` Let Unsloth detect the format automatically `alpaca` `instruction` / `input` / `output` columns `chatml` OpenAI-style `messages` array `sharegpt` ShareGPT-style conversations **Splits and Slicing** * **Subset** - automatically populated from the dataset card. * **Train split / Eval split** - choose which splits to use. Setting an eval split enables the **Eval Loss** chart during training. * **Dataset slice** - optionally restrict training to a row range (start index / end index) for quick experiments. **Column Mapping** If the Studio cannot automatically map your dataset columns to the correct roles a **Dataset Preview dialog** opens. It shows sample rows and lets you assign each column to `instruction`, `input`, `output`, `image`, etc. Suggested mappings are pre-filled where possible. ### [hashtag](https://unsloth.ai/docs/new/studio/start#id-3.-hyperparameters) 3\. Hyperparameters Parameters are grouped into collapsible sections. You can view our detailed [LoRA hyperparameters guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) here: [🧠Hyperparameters Guidechevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) Parameter Default Notes **Max Steps** `0` `0` means use Epochs instead **Context Length** `2048` Options: 512 → 32768 **Learning Rate** `2e-4` **LoRA Settings** _(Hidden when Full Fine-tuning is selected)_ Parameter Default Notes **Rank** `16` Slider 4–128 **Alpha** `32` Slider 4–256 **Dropout** `0.05` **LoRA Variant** `LoRA` `LoRA` / `RS-LoRA` / `LoftQ` **Target Modules** All on `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` For **Vision** models with an image dataset, four additional checkboxes appear. Fine-tune: Vision Layers Language Layers Attention Modules MLP Modules **Training Hyperparameters** Organized into three tabs: Optimization Schedule Logging Parameter Default Epochs 3 Batch Size 4 Gradient Accumulation 8 Weight Decay 0.01 Optimizer AdamW 8-bit Parameter Default LR Scheduler linear Warmup Steps 5 Gradient Checkpointing unsloth Random Seed 3407 Save Steps 0 Eval Steps 0 Packing false Train on Completions false Parameter Default Enable W&B false W&B Project llm-finetuning Enable TensorBoard false TensorBoard Dir runs Log Frequency 10 circle-info [**Unsloth Gradient Checkpointing**](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhancements) **:** `**unsloth**` uses Unsloth's custom memory-efficient implementation, which can reduce VRAM usage significantly compared to the standard PyTorch option. It is the recommended default. ### [hashtag](https://unsloth.ai/docs/new/studio/start#id-4.-training-and-config) 4\. Training and Config The bottom-right card has three config management buttons and the **Start Training** button. Button Action **Upload** Load a previously saved `.yaml` config file **Save** Export the current config to YAML **Reset** Revert all parameters to the model's defaults The Start Training button stays disabled until a model and dataset are both configured. Validation errors appear inline - for example, setting eval steps without choosing an eval split, or pairing a text-only model with a vision dataset. #### [hashtag](https://unsloth.ai/docs/new/studio/start#loading-screen) Loading Screen After you click **Start Training**, a full-page overlay appears while the backend prepares everything. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYtsUxHI0szGw8ZPxCHep%252Fimage.png%3Falt%3Dmedia%26token%3D1701f4af-ef35-48da-80e7-4aba4e80f4d4&width=768&dpr=3&quality=100&sign=496c6938&sv=2) The overlay shows an animated terminal with live phase updates: * Blue: Downloading model / dataset * Amber: Loading model / dataset * Blue: Configuring * Green: Training You can cancel at any time using the **×** button in the corner. A confirmation dialog will appear before anything is stopped. ### [hashtag](https://unsloth.ai/docs/new/studio/start#training-progress-and-observability) Training Progress and Observability Once the first training step arrives the overlay dismisses and the live training view is revealed. The fine-tuning process is complete when steps reach 100% on the progress bar. You can view the elapsed time and tokens. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fah3G1rYgRaDNY8Ay6Uw7%252Fimage.png%3Falt%3Dmedia%26token%3D0528c15e-7a4b-4028-8070-95dc0871da5d&width=768&dpr=3&quality=100&sign=50415dd8&sv=2) #### [hashtag](https://unsloth.ai/docs/new/studio/start#status-panel) Status Panel The left column shows: * **Epoch** - current fractional epoch (e.g. `Epoch 1.23`) * **Progress bar** - step-based, with percentage * **Key metrics**: * **Loss** - training loss to 4 decimal places * **LR** - current learning rate in scientific notation * **Grad Norm** - gradient norm * **Model** - the model being trained * **Method** - `QLoRA` / `LoRA` / `Full` * **Timing row** - elapsed time, ETA, steps per second, and total tokens processed #### [hashtag](https://unsloth.ai/docs/new/studio/start#gpu-monitor) GPU Monitor The right column shows live GPU stats polled every few seconds: * **Utilization** - percentage bar * **Temperature** - °C bar * **VRAM** - used / total GB * **Power** - draw / limit in watts #### [hashtag](https://unsloth.ai/docs/new/studio/start#stopping-training) Stopping Training Use the **Stop Training** button in the top-right of the progress card. A dialog gives you two choices: * **Stop & Save** - saves a checkpoint before stopping * **Cancel** - stops immediately with no checkpoint #### [hashtag](https://unsloth.ai/docs/new/studio/start#charts) Charts Four live charts update as training progresses: 1. **Training Loss** - raw values plus an EMA-smoothed line and a running average reference line 2. **Learning Rate** - the LR schedule curve 3. **Gradient Norm** - gradient norm over steps 4. **Eval Loss** - only shown when you configured an eval split ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRgXfe3sobdQWxha8yslr%252Fimage.png%3Falt%3Dmedia%26token%3Db3aa9004-778b-4e3d-85b1-40a205ad0602&width=768&dpr=3&quality=100&sign=38b8a3f7&sv=2) Each chart has settings (gear icon) with: Option Default Viewing window Last N steps slider EMA Smoothing `0.6` Show Raw On Show Smoothed On Show Average line On Scale (per series) Linear / Log Outlier clipping No clip / p99 / p95 ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFJtjQpAgOFaieyQCYhkq%252Fimage.png%3Falt%3Dmedia%26token%3D4da9cdc2-c088-4ab8-8d0d-40d8d392ee03&width=768&dpr=3&quality=100&sign=a27f33de&sv=2) #### [hashtag](https://unsloth.ai/docs/new/studio/start#config-files) Config Files All training configurations can be saved and reloaded as YAML files. Files are named automatically as: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FuGAKdGkANbh2wIENA9X7%252Fimage.png%3Falt%3Dmedia%26token%3D9553db5b-5c88-4556-be49-fe61035edf11&width=768&dpr=3&quality=100&sign=2c03dfd5&sv=2) The YAML is structured into three sections: This makes it easy to reproduce runs, share configurations, or version-control your experiments. [hashtag](https://unsloth.ai/docs/new/studio/start#data-recipes-quickstart) hat-chef Data Recipes - Quickstart ------------------------------------------------------------------------------------------------------------------- [Unsloth Data Recipes](https://unsloth.ai/docs/new/studio/data-recipe) lets you upload documents like PDFs or CSVs files and transforms them into useable datasets. Create and edit datasets visually via a graph-node workflow. The recipes page is the main entry point. Recipes are stored locally in the browser, so you come back to saved work later. From here, you can create a blank recipe or open a guided learning recipe. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQ6e19jESrJg0VjHnX58c%252Fdata%2520recipes%2520final.png%3Falt%3Dmedia%26token%3D8d74e453-815d-4790-83d1-76d0bc80a3ce&width=768&dpr=3&quality=100&sign=d8ec1d11&sv=2) Data Recipes follows the same basic path. You open the recipes page, create or pick a recipe, build the workflow in the editor, validate it run a preview, then run the full dataset once the output looks right. Add seed data and generation blocks, validate the workflow, preview sample output, then run a full dataset build. Unsloth Data Recipes is powered by NVIDIA [DataDesignerarrow-up-right](https://github.com/NVIDIA-NeMo/DataDesigner) . At a glance a usual workflow should look like this: 1. Open the recipes page. 2. Create a new recipe or open an existing one. 3. Add blocks to define your dataset workflow. 4. Click **Validate** to catch configuration issues early. 5. Run a preview to inspect sample rows quickly. 6. Run a full dataset build when the recipe is ready. 7. Review progress and output live in graph or in **Executions** view for mode details. 8. Select the resulting dataset in **Studio** and fine tune a model. [hashtag](https://unsloth.ai/docs/new/studio/start#export-quickstart) box-isometric Export - Quickstart ------------------------------------------------------------------------------------------------------------ Use Unsloth Studio 'Export' to export, save, or convert models to GGUF, Safetensors, or LoRA for deployment, sharing, or local inference in Unsloth, llama.cpp, Ollama, vLLM, and more. Export a trained checkpoint or convert any existing model. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrrFY8YczW3dDpfYi1k9f%252FScreenshot%25202026-03-15%2520at%25209.28.19%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd2729e16-799f-48f0-8b07-0248b93fa599&width=768&dpr=3&quality=100&sign=caee257&sv=2) You can read our detailed tutorial / guide about exporting models with Unsloth Studio here: [box-isometricModel Exportchevron-right](https://unsloth.ai/docs/new/studio/export) [hashtag](https://unsloth.ai/docs/new/studio/start#chat-quickstart) comment-dots Chat - Quickstart ------------------------------------------------------------------------------------------------------- [Unsloth Studio Chat](https://unsloth.ai/docs/new/studio/chat) lets you run models 100% offline on your computer. Run model formats like GGUF and safetensors from Hugging Face or from your local files. * **Download + Run** any model like GGUFs, fine-tuned adapters, safetensors etc. * [**Compare** different model](https://unsloth.ai/docs/new/studio/start#model-arena) outputs side-by-side * **Upload** documents, images, and audio in your prompts * [**Tune** inference](https://unsloth.ai/docs/new/studio/start#generation-settings) settings like: temperature, top-p, top-k and system prompt ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRCnTAZ6Uh88DIlU3g0Ij%252Fmainpage%2520unsloth.png%3Falt%3Dmedia%26token%3D837c96b6-bd09-4e81-bc76-fa50421e9bfb&width=768&dpr=3&quality=100&sign=c1a39da1&sv=2) You can read our detailed tutorial / guide about running models with Unsloth Studio here: [comment-dotsStudio Chatchevron-right](https://unsloth.ai/docs/new/studio/chat) [hashtag](https://unsloth.ai/docs/new/studio/start#video-tutorial) video Video Tutorial -------------------------------------------------------------------------------------------- circle-exclamation The Unsloth Studio versions shown in the videos are old and are not reflective of the current version. Here is a video tutorial created by NVIDIA to get you started with Studio: How to Install Unsloth Studio Video Tutorial [hashtag](https://unsloth.ai/docs/new/studio/start#advanced-settings) Advanced Settings -------------------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/new/studio/start#cli-commands) CLI Commands The Unsloth CLI (`cli.py`) provides the following commands: ### [hashtag](https://unsloth.ai/docs/new/studio/start#project-structure) Project Structure ### [hashtag](https://unsloth.ai/docs/new/studio/start#api-reference) API Reference All endpoints require a valid JWT `Authorization: Bearer ` header (except `/api/auth/*` and `/api/health`). Method Endpoint Description `GET` `/api/health` Health check `GET` `/api/system` System info (GPU, CPU, memory) `POST` `/api/auth/signup` Create account (requires setup token on first run) `POST` `/api/auth/login` Login and receive JWT tokens `POST` `/api/auth/refresh` Refresh an expired access token `GET` `/api/auth/status` Check if auth is initialized `POST` `/api/train/start` Start a training job `POST` `/api/train/stop` Stop a running training job `POST` `/api/train/reset` Reset training state `GET` `/api/train/status` Get current training status `GET` `/api/train/metrics` Get training metrics (loss, LR, steps) `GET` `/api/train/stream` SSE stream of real-time training progress `GET` `/api/models/` List available models `POST` `/api/inference/chat` Send a chat message for inference `GET` `/api/datasets/` List / manage datasets [PreviousIntroducing Unsloth Studiochevron-left](https://unsloth.ai/docs/new/studio) [NextStudio Chatchevron-right](https://unsloth.ai/docs/new/studio/chat) Last updated 8 days ago Was this helpful? * [Studio - Quickstart](https://unsloth.ai/docs/new/studio/start#studio-quickstart) * [1\. Select model and method](https://unsloth.ai/docs/new/studio/start#id-1.-select-model-and-method) * [2\. Dataset](https://unsloth.ai/docs/new/studio/start#id-2.-dataset) * [3\. Hyperparameters](https://unsloth.ai/docs/new/studio/start#id-3.-hyperparameters) * [4\. Training and Config](https://unsloth.ai/docs/new/studio/start#id-4.-training-and-config) * [Training Progress and Observability](https://unsloth.ai/docs/new/studio/start#training-progress-and-observability) * [Data Recipes - Quickstart](https://unsloth.ai/docs/new/studio/start#data-recipes-quickstart) * [Export - Quickstart](https://unsloth.ai/docs/new/studio/start#export-quickstart) * [Chat - Quickstart](https://unsloth.ai/docs/new/studio/start#chat-quickstart) * [Video Tutorial](https://unsloth.ai/docs/new/studio/start#video-tutorial) * [Advanced Settings](https://unsloth.ai/docs/new/studio/start#advanced-settings) * [CLI Commands](https://unsloth.ai/docs/new/studio/start#cli-commands) * [Project Structure](https://unsloth.ai/docs/new/studio/start#project-structure) * [API Reference](https://unsloth.ai/docs/new/studio/start#api-reference) Was this helpful? sun-brightdesktopmoon Copy {model}_{method}_{dataset}_{timestamp}.yaml Copy training: max_steps: 0 num_train_epochs: 3 per_device_train_batch_size: 4 ... lora: r: 16 lora_alpha: 32 ... logging: report_to: none ... chevron-downShow all 14 lines Copy Usage: cli.py [COMMAND] Commands: train Fine-tune a model inference Run inference on a trained model export Export a trained adapter list-checkpoints List saved checkpoints ui Launch the Unsloth Studio web UI studio Launch the studio (alias) Copy new-ui-prototype/ ├── cli.py # CLI entry point ├── cli/ # Typer CLI commands │ └── commands/ │ ├── train.py │ ├── inference.py │ ├── export.py │ ├── ui.py │ └── studio.py ├── setup.sh # Bootstrap script (Linux / WSL / Colab) ├── setup.ps1 # Bootstrap script (Windows native) ├── setup.bat # Wrapper to launch setup.ps1 via double-click ├── install_python_stack.py # Cross-platform Python dependency installer └── studio/ ├── backend/ │ ├── main.py # FastAPI app & middleware │ ├── run.py # Server launcher (uvicorn) │ ├── auth/ # Auth storage & JWT logic │ ├── routes/ # API route handlers │ │ ├── training.py │ │ ├── models.py │ │ ├── inference.py │ │ ├── datasets.py │ │ └── auth.py │ ├── models/ # Pydantic request/response schemas │ ├── core/ # Training engine & config │ ├── utils/ # Hardware detection, helpers │ └── requirements.txt ├── frontend/ │ ├── src/ │ │ ├── features/ # Feature modules │ │ │ ├── auth/ # Login / signup flow │ │ │ ├── training/ # Training config & monitoring │ │ │ ├── studio/ # Main studio workspace │ │ │ ├── chat/ # Inference chat UI │ │ │ ├── export/ # Model export flow │ │ │ └── onboarding/# Onboarding wizard │ │ ├── components/ # Shared UI components (shadcn) │ │ ├── hooks/ # Custom React hooks │ │ ├── stores/ # Zustand state stores │ │ └── types/ # TypeScript type definitions │ ├── package.json │ └── vite.config.ts └── tests/ # Backend test suite chevron-downShow all 44 lines sun-brightdesktopmoon --- # Unsloth Updates | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close To use the latest changes, update Unsloth via `unsloth studio update`. April 11, 2026 Model release [hashtag](https://unsloth.ai/docs/new/changelog#gemma-4-update--minimax-m2.7) **Gemma 4 Update + MiniMax-M2.7** -------------------------------------------------------------------------------------------------------------------- [Gemma 4 GGUFsarrow-up-right](https://huggingface.co/collections/unsloth/gemma-4) are now updated with Google's official chat template fixes (which fixed/improved tool-calling), along with the latest llama.cpp fixes. Update to the latest llama.cpp, re-download quants and you shouldn't see `unused token` issues anymore. [MiniMax-M2.7](https://unsloth.ai/docs/models/minimax-m27) is out now! You can run the model locally with our GGUFs in 4-bit quantization on 128GB RAM / unified memory. [**MiniMax-M2.7 GGUF**arrow-up-right](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF) April 8, 2026 rocket-launchNew releasesv0.1.36-beta [hashtag](https://unsloth.ai/docs/new/changelog#gemma-4-fixes) **Gemma 4 Fixes** ------------------------------------------------------------------------------------- We’ve updated Gemma 4 [with many fixes](https://unsloth.ai/docs/models/gemma-4/train) . These bugs are universal and affected all training packages and implementations and **did not originate from Unsloth**. We identified the bugs, fixed them, and Gemma 4 training now works properly in Unsloth. You only need **8GB VRAM** to train **Gemma-4-E2B** locally. Unsloth trains Gemma 4 **~1.5x faster while using ~60% less VRAM** than FA2 setups. For the full guide and notebooks on Gemma 4 training, [see our blog](https://unsloth.ai/docs/models/gemma-4/train) . #### [hashtag](https://unsloth.ai/docs/new/changelog#gemma-4-training-fixes) Gemma 4 Training Fixes 1. **Gradient accumulation** no longer causes loss explosions. Previously, losses could spike to **300–400**; expected loss is around **10–15**. 2. Fixed the **IndexError** affecting **26B** and **31B** inference in `transformers`. 3. Fixed gibberish outputs for **E2B/E4B** when `use_cache=False`. See [issue #45242arrow-up-right](https://github.com/huggingface/transformers/issues/45242) . 4. Fixed **float16 audio** overflow from `-1e9` values. If you see losses above **13–15,** for example **100** or **300** - gradient accumulation is likely being handled incorrectly. This is fixed in both **Unsloth** and **Unsloth Studio**. #### [hashtag](https://unsloth.ai/docs/new/changelog#gemma-4-quant-re-uploads) Gemma 4 Quant Re-uploads We also updated our Gemma 4 GGUFs so you will need to re-download. Again, these quant issues are **not related to or caused by Unsloth**: 1. CUDA: check for buffer overlap before fusing - critical fix for `` tokens - [PR #21566arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21566) 2. `kv-cache`: support attention rotation for heterogeneous iSWA - [PR #21513arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21513) 3. `vocab`: add byte token handling to BPE detokenizer for Gemma 4 - [PR #21488arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21488) 4. `convert`: set `"add bos" == True` for Gemma 4 - [PR #21500arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21500) 5. `common`: add Gemma 4 specialized parser - [PR #21418arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21418) 6. `llama-model`: read `final_logit_softcapping` for Gemma 4 - [PR #21390arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21390) 7. `llama`: add custom newline split for Gemma 4 - [PR #21406arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21406) #### [hashtag](https://unsloth.ai/docs/new/changelog#unsloth-studio-updates) Unsloth Studio Updates * Add **speculative decoding** support (ngram-mod, on by default) * Llama.cpp updated to use latest version with all Gemma 4 Fixes * Fix Qwen3.5 and Gemma 4 training issues * Enable exporting and saving of Gemma 4 models * Harden sandbox security for terminal and python tools * Let recipes use the model loaded in Chat * Fix empty chat threads on navigation (and whenever switching tabs) and stabilize new chat flow * Allow non-LLM recipes to run and move Data tab first in executions * Reuse HF cached repo casing to prevent duplicate downloads April 3, 2026 rocket-launchNew releasesv0.1.36-beta [hashtag](https://unsloth.ai/docs/new/changelog#google-gemma-4) **Google - Gemma 4** ----------------------------------------------------------------------------------------- * You can now run and train the [Gemma 4](https://unsloth.ai/docs/models/gemma-4) models in Unsloth. * Intel Mac now works * Pre-compiled binaries for llama.cpp for 2 Gemma-4 fixes: * vocab: fix Gemma4 tokenizer ([#21343arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21343) ) * fix: gemma 4 template ([#21326arrow-up-right](https://github.com/ggml-org/llama.cpp/pull/21326) ) * Tool calls for smaller models are now more stable and don't cut off anymore * Pre-compiled binaries for Windows, Linux, Mac, WSL devices - CPU and GPU * Speculative Decoding added for non vision models (Gemma-4 is vision sadly and Qwen3.5) * Context length is now properly applied. * Web search now actually gets web content and not just summaries * 90% reduced HF API calls - less rate limits March 31, 2026 rocket-launchNew releaseslayer-plusImprovements [hashtag](https://unsloth.ai/docs/new/changelog#id-50-tool-call-accuracy--more-support) **+50% tool call accuracy + more support** --------------------------------------------------------------------------------------------------------------------------------------- * Tool calls for all models are now **+30% to +80% more accurate.** * Web search now actually gets web content and not just summaries * Number of tool calls allowed are increased to 25 from 10 * Tool calls now terminate much better, so looping / repetitions will be reduced * More **tool call healing** and de-duplication logic to stop tool callings from leaking XML as well * Tested with `unsloth/Qwen3.5-4B-GGUF` (`UD-Q4_K_XL`), web search + code execution + thinking enabled. Metric Before After XML leaks in response 10/10 0/10 URL fetches used 0 4/10 runs Runs with correct song names 0/10 2/10 Avg tool calls 5.5 3.8 Avg response time 12.3s 9.8s #### [hashtag](https://unsloth.ai/docs/new/changelog#new-features) New features * Added **custom folders** so you can use any GGUFs in any folder - for now access in Advanced Settings in Chat and Custom Folders * **Update button** now visible * Install script styling all updated! * Preliminary **Automatic Multi GPU support for inference and training** - useful for large models that don't fit on 1 GPU - Studio auto will allocate GPU resources * Intel Macs should work out of the box ### [hashtag](https://unsloth.ai/docs/new/changelog#much-smoother-and-faster-studio) Much smoother and faster Studio * **Fixed timeouts of downloads of large models** - no more timeouts seen. * **Fixed Hugging Face rate limiting - HF API calls reduced by 90%** * Fixed bun on Windows and faster installs March 27, 2026 rocket-launchNew releasesscrewdriver-wrenchFixeslayer-plusImprovements [hashtag](https://unsloth.ai/docs/new/changelog#new-important-updates) **New Important Updates** ----------------------------------------------------------------------------------------------------- It’s only been 2 days since our previous release, but we’ve got a more important updates: * **Inference is now 20–30% faster.** Previously, tool-calling and repeat penalty could slow inference below normal speeds. Inference tokens/s should now perform the same as `llama-server` / `llama.cpp`. * **Now Auto-detects older or pre-existing models** downloaded from **LM Studio, Hugging Face,** and similar sources. * **Inference token/s speed is now calculated correctly.** Previously, tokens/s included startup time, which made the displayed speed look slower than it actually was. It should now reflect 'true' inference speed. * **CPU usage no longer spikes.** Previously, inline querier identity changed every render, causing `useLiveQuery` to resubscribe continuously. * **Unsloth Studio now has a shutdown x button and shuts down properly.** Previously, closing it after opening from the desktop icon would not close it properly. Now, launching from the shortcut also opens the terminal, and closing that terminal fully exits Unsloth Studio. If you still have it open from a previous session you can restart your computer or run `lsof -i :8888` then `kill -9 `. * **Even better tool-calling and websearch** with reduced errors. * Updated documentation with lots of new info on [deleting models, uninstalling](https://unsloth.ai/docs/new/studio/install#uninstall) etc. * **Cleaner, smarter install and setup logging across Windows and Linux.** Output is now easier to read with consistent formatting, quieter by default for a smoother experience, and supports richer `--verbose` diagnostics when you want full technical detail. * You can now view your training history! March 25, 2026 rocket-launchNew releasesscrewdriver-wrenchFixeslayer-plusImprovements [hashtag](https://unsloth.ai/docs/new/changelog#first-release-post-unsloth-studio) First Release post Unsloth Studio ------------------------------------------------------------------------------------------------------------------------- Hey guys, this is our first release since we launched Unsloth Studio. Lots of new features and fixes: * **You can now update Unsloth Studio!** Please update via: `unsloth studio update` * **Windows** CPU or GPU now works seamlessly. Please reinstall! * **App shortcuts**. Once installed, you can now launch in Windows, MacOS and Linux via a shortcut icon in the Start / Launch and Desktop. * **Pre-compiled** `**llama.cpp**` **binaries** and `mamba_ssm` - 6x faster installs! Also <300MB in size for binaries. * **50% reduced installation sizes** (-7GB or more savings), 2x faster installs and faster resolving. 50% smaller pypi sizes. * **Tool calling improved.** Better llama.cpp parsing, no raw tool markup in chat, faster inference, a new Tool Outputs panel, timers. * MacOS and CPU now have [Data Recipes](https://unsloth.ai/docs/new/studio/data-recipe) enabled with multi-file uploading. * **AMD support preliminary for Linux** only machines - auto detects. * **Settings sidebar redesign.** Settings are now grouped into **Model, Sampling, Tools, and Preferences** * **Context length** now adjustable. Keep in mind this is not needed as llama.cpp smartly uses the exact context you need via `--fit on` * **Multi-file upload.** Data recipes now support multiple drag-and-drop uploads for PDF, DOCX, TXT, and MD, with backend extraction, saved uploads, and improved previews. * **Colab** with free T4 GPUs with Unsloth Studio now fixed! [Try it herearrow-up-right](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) . Due to pre-compiled binaries, it's also 20x faster! * **Better chat observability.** Studio now shows `llama-server` timings and usage, a context-window usage bar, and richer source hover cards. * **Better UX overall** - clickable links, better LaTeX parsing, tool / code / web tooltips for default cards and much more! * **LiteLLM -** Unsloth Studio and Unsloth were **NOT** affected by the recent LiteLLM compromise. Nemo Data Designer used LiteLLM only up to `1.80`, not the affected `1.82.7` or `1.82.8`, and has since removed it entirely. * We now have a new one line install command, just run: #### [hashtag](https://unsloth.ai/docs/new/changelog#fixes) **Fixes:** * **Windows/setup improvements.** Fixed silent Windows exits, Anaconda/conda-forge startup crashes, broken non-NVIDIA Windows installs, and missing early CUDA/stale-venv setup checks. * **System prompts fixed.** They work again for non-GGUF text and vision inference. * **Persistent system prompts and presets.** Custom system prompts and chat presets now persist across reloads and page changes. * **GGUF export expanded.** Full fine-tunes, not just LoRA/PEFT, can now export to GGUF. Base model resolution is more reliable, and unsupported export options are disabled in the UI. * **Chat scroll/layout fixes.** Fixed scroll-position issues during generation, thinking-panel layout shift, and viewport jumps when collapsing reasoning panels. * **Smarter port conflict detection.** Studio now detects loopback conflicts, can identify the blocking process when possible, and gives clearer fallback-port messages. March 17, 2026 screwdriver-wrenchFixeslayer-plusImprovements [hashtag](https://unsloth.ai/docs/new/changelog#new-tool-calling--windows-stability) New Tool calling + Windows Stability ------------------------------------------------------------------------------------------------------------------------------ * Claude Artifacts works so HTML can be executed like a snake game inside the chat * +30% more accurate tool calls esp for small models + Timer for tool calls * Tool + Web Search outputs can be saved + Toggle auto healing tool on/off * Many bug fixes - Windows CPU works, Mac more seamless, faster and smaller installs [PreviousModel Exportchevron-left](https://unsloth.ai/docs/new/studio/export) [NextGemma 4chevron-right](https://unsloth.ai/docs/models/gemma-4) Last updated 1 day ago Was this helpful? Was this helpful? sun-brightdesktopmoon Copy curl -fsSL https://unsloth.ai/install.sh | sh sun-brightdesktopmoon --- # Fine-tuning LLMs Guide | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-1.-what-is-fine-tuning) 1\. What Is Fine-tuning? ----------------------------------------------------------------------------------------------------------------------------- Fine-tuning / training / post-training models customizes its behavior, enhances + injects knowledge, and optimizes performance for domains and specific tasks. For example: * OpenAI’s **GPT-5** was post-trained to improve instruction following and helpful chat behavior. * The standard method of post training is called Supervised Fine-Tuning (SFT). Other methods include preference optimization (DPO, ORPO), distillation and [Reinforcement Learning (RL)](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) (GRPO, GSPO), where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**. With [Unslotharrow-up-right](https://github.com/unslothai/unsloth) , you can fine-tune or do RL for free on Colab, Kaggle, or locally with just 3GB VRAM by using our [notebooksarrow-up-right](https://docs.unsloth.ai/get-started/unsloth-notebooks) . By fine-tuning a pre-trained model on a dataset, you can: * **Update + Learn New Knowledge**: Inject and learn new domain-specific information. * **Customize Behavior**: Adjust the model’s tone, personality, or response style. * **Optimize for Tasks**: Improve accuracy and relevance for specific use cases. **Example fine-tuning or RL use-cases**: * Enables LLMs to predict if a headline impacts a company positively or negatively. * Can use historical customer interactions for more accurate and custom responses. * Fine-tune LLM on legal texts for contract analysis, case law research, and compliance. You can think of a fine-tuned model as a specialized agent designed to do specific tasks more effectively and efficiently. **Fine-tuning can replicate all of RAG's capabilities**, but not vice versa. #### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#what-is-lora-qlora) ❓What is LoRA/QLoRA? In LLMs, we have model weights. Llama 70B has 70 billion numbers. Instead of changing all 70B numbers, we instead add thin matrices A and B to each weight, and optimize those. This means we only optimize 1% of weights. LoRA is when the original model is 16-bit unquantized while QLoRA quantizes to 4-bit to save 75% memory. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-715b6260aae497f160d7f9a1019bcfa472675dcf%252Fimage%2520%287%29%2520%281%29%2520%281%29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=20322e39&sv=2) Instead of optimizing Model Weights (yellow), we optimize 2 thin matrices A and B. #### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#fine-tuning-misconceptions) Fine-tuning misconceptions: You may have heard that fine-tuning does not make a model learn new knowledge or RAG performs better than fine-tuning. That is **false**. You can train a specialized coding model with fine-tuning and RL while RAG can’t change the model’s weights and only augments what the model sees at inference time. Read more FAQ + misconceptions [here](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me#fine-tuning-vs.-rag-whats-the-difference) : [🤔FAQ + Is Fine-tuning Right For Me?chevron-right](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me) > [**Introducing Unsloth Studio:**](https://unsloth.ai/docs/new/studio) > Our new open-source web UI for training and running models. This means you can now fine-tune models with no-code and have observability and automatic dataset creation features. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxV1PO5DbF3ksB51nE2Tw%252Fmore%2520cropped%2520ui%2520for%2520homepage.png%3Falt%3Dmedia%26token%3Df75942c9-3d8d-4b59-8ba2-1a4a38de1b86&width=768&dpr=3&quality=100&sign=a663c397&sv=2) [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-2.-choose-the-right-model--method) 2\. Choose the Right Model + Method --------------------------------------------------------------------------------------------------------------------------------------------------- If you're a beginner, it is best to start with a small instruct model like Llama 3.1 (8B) and experiment from there. You'll also need to decide between normal fine-tuning, RL, QLoRA or LoRA training: * **Reinforcement Learning (RL)** is used when you need a model to excel at a specific behavior (e.g., tool-calling) using an environment and reward function rather than labeled data. We have several [notebook examples](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) , but for most use-cases, standard SFT is sufficient. * **LoRA** is a parameter efficient training method that typically keeps the base model’s weights frozen and trains a small set of added low-rank adapter weights (in 16-bit precision). * **QLoRA** combines LoRA with 4-bit precision to handle very large models with minimal resources. * Unsloth also supports full fine-tuning (FFT) and pretraining, which require significantly more resources, but FFT is usually unnecessary. When done correctly, LoRA can match FFT. * Unsloth **all types models**: [text-to-speech](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [embedding](https://unsloth.ai/docs/basics/embedding-finetuning) , GRPO, RL, [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) , multimodal and more. circle-info Research shows that **training and serving in the same precision** helps preserve accuracy. This means if you want to serve in 4-bit, train in 4-bit and vice versa. We recommend starting with QLoRA, as it is one of the most accessible and effective methods for training models. Our [dynamic 4-bitarrow-up-right](https://unsloth.ai/blog/dynamic-4bit) quants, the accuracy loss for QLoRA compared to LoRA is now largely recovered. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-cfc51c261e6d24df3aa967d9b9a482313465cbc1%252Fmodel%2520name%2520change.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c9d42ca6&sv=2) You can change the model name to whichever model you like by matching it with model's name on Hugging Face e.g. '`unsloth/llama-3.1-8b-unsloth-bnb-4bit`'. We recommend starting with **Instruct models**, as they allow direct fine-tuning using conversational chat templates (ChatML, ShareGPT etc.) and require less data compared to **Base models** (which uses Alpaca, Vicuna etc). Learn more about the differences between [instruct and base models here](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model) . * Model names ending in `**unsloth-bnb-4bit**` indicate they are [**Unsloth dynamic 4-bit**arrow-up-right](https://unsloth.ai/blog/dynamic-4bit) **quants**. These models consume slightly more VRAM than standard BitsAndBytes 4-bit models but offer significantly higher accuracy. * If a model name ends with just `**bnb-4bit**`, without "unsloth", it refers to a standard BitsAndBytes 4-bit quantization. * Models with **no suffix** are in their original **16-bit or 8-bit formats**. While they are the original models from the official model creators, we sometimes include important fixes - such as chat template or tokenizer fixes. So it's recommended to use our versions when available. There are other settings which you can toggle: * `**max_seq_length = 2048**` – Controls context length. While Llama-3 supports 8192, we recommend 2048 for testing. Unsloth enables 4× longer context fine-tuning. * `**dtype = None**` – Defaults to None; use `torch.float16` or `torch.bfloat16` for newer GPUs. * `**load_in_4bit = True**` – Enables 4-bit quantization, reducing memory use 4× for fine-tuning. Disabling it enables LoRA 16-bit fine-tuning. You can also enable 16-bit LoRA with `load_in_16bit = True` * To enable full fine-tuning (FFT), set `full_finetuning = True`. For 8-bit fine-tuning, set `load_in_8bit = True`. * **Note:** Only one training method can be set to `True` at a time. circle-info A common mistake is jumping straight into full fine-tuning (FFT), which is compute-heavy. Start by testing with LoRA or QLoRA first, if it won’t work there, it almost certainly won’t work with FFT. And if LoRA fails, don’t assume FFT will magically fix it. You can also do [Text-to-speech (TTS)](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [reasoning (GRPO)](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) , [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) , [RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/preference-dpo-orpo-and-kto) (GRPO, DPO), [continued pretraining](https://unsloth.ai/docs/basics/continued-pretraining) , text completion and other training methodologies with Unsloth. Read our guide on choosing models: [❓What Model Should I Use?chevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use) For inidivudal tutorials on models: [🚀Complete LLM Directorychevron-right](https://unsloth.ai/docs/models/tutorials) [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-3.-your-dataset) 3\. Your Dataset -------------------------------------------------------------------------------------------------------------- For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized. * You will need to create a dataset usually with 2 columns - question and answer. The quality and amount will largely reflect the end result of your fine-tune so it's imperative to get this part right. * You can [synthetically generate data](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-data-generation) and structure your dataset (into QA pairs) using ChatGPT or local LLMs. * You can also use our new Synthetic Dataset notebook which automatically parses documents (PDFs, videos etc.), generates QA pairs and auto cleans data using local models like Llama 3.2. [Access the notebook here.arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) * Fine-tuning can learn from an existing repository of documents and continuously expand its knowledge base, but just dumping data alone won’t work as well. For optimal results, curate a well-structured dataset, ideally as question-answer pairs. This enhances learning, understanding, and response accuracy. * But, that's not always the case, e.g. if you are fine-tuning a LLM for code, just dumping all your code data can actually enable your model to yield significant performance improvements, even without structured formatting. So it really depends on your use case. _**Read more about creating your dataset:**_ [📈Datasets Guidechevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide) For most of our notebook examples, we utilize the [Alpaca datasetarrow-up-right](https://docs.unsloth.ai/basics/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset) however other notebooks like Vision will use different datasets which may need images in the answer ouput as well. ### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-4.-understand-training-hyperparameters) 4\. Understand Training Hyperparameters Learn how to choose the right [hyperparameters](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) using best practices from research and real-world experiments - and understand how each one affects your model's performance. **For a complete guide on how hyperparameters affect training, see:** [🧠Hyperparameters Guidechevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-5.-install--requirements) 5\. Install + Requirements --------------------------------------------------------------------------------------------------------------------------------- You can use Unsloth via two main ways, our free notebooks or locally. ### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#unsloth-notebooks) Unsloth Notebooks We would recommend beginners to utilise our pre-made [notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks) first as it's the easiest way to get started with guided steps. You can later export the notebooks to use locally. Unsloth has step-by-step notebooks for [text-to-speech](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning) , [embedding](https://unsloth.ai/docs/basics/embedding-finetuning) , GRPO, RL, [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) , multimodal, different use-cases and more. ### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#local-installation) Local Installation You can also install Unsloth locally via [Docker](https://unsloth.ai/docs/get-started/install/docker) or `pip install unsloth` (with Linux, WSL or [Windows](https://unsloth.ai/docs/get-started/install/windows-installation) ). Also depending on the model you're using, you'll need enough VRAM and resources. Installing Unsloth will require a Windows or Linux device. Once you install Unsloth, you can copy and paste our notebooks and use them in your own local environment. See: [🛠️Unsloth Requirementschevron-right](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) [📥Installationchevron-right](https://unsloth.ai/docs/get-started/install) [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-6.-training--evaluation) 6\. Training + Evaluation ------------------------------------------------------------------------------------------------------------------------------- Once you have everything set, it's time to train! If something's not working, remember you can always change hyperparameters, your dataset etc. You’ll see a log of numbers during training. This is the training loss, which shows how well the model is learning from your dataset. For many cases, a loss around 0.5 to 1.0 is a good sign, but it depends on your dataset and task. If the loss is not going down, you might need to adjust your settings. If the loss goes to 0, that could mean overfitting, so it's important to check validation too. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-feb9b0f5763d41cecaec9a3a9cd227ad918f0ca7%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=cecca0ca&sv=2) The training loss will appear as numbers We generally recommend keeping the default settings unless you need longer training or larger batch sizes. * `**per_device_train_batch_size = 2**` – Increase for better GPU utilization but beware of slower training due to padding. Instead, increase `gradient_accumulation_steps` for smoother training. * `**gradient_accumulation_steps = 4**` – Simulates a larger batch size without increasing memory usage. * `**max_steps = 60**` – Speeds up training. For full runs, replace with `num_train_epochs = 1` (1–3 epochs recommended to avoid overfitting). * `**learning_rate = 2e-4**` – Lower for slower but more precise fine-tuning. Try values like `1e-4`, `5e-5`, or `2e-5`. #### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#evaluation) Evaluation In order to evaluate, you could do manually evaluation by just chatting with the model and see if it's to your liking. You can also enable evaluation for Unsloth, but keep in mind it can be time-consuming depending on the dataset size. To speed up evaluation you can: reduce the evaluation dataset size or set `evaluation_steps = 100`. For testing, you can also take 20% of your training data and use that for testing. If you already used all of the training data, then you have to manually evaluate it. You can also use automatic eval tools but keep in mind that automated tools may not perfectly align with your evaluation criteria. [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-7.-running--deploying-the-model) 7\. Running + Deploying the model ----------------------------------------------------------------------------------------------------------------------------------------------- Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-f2d5f23fa62ec89e06bf20fea433f9a1e42a2fe3%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7c1328f7&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-cdf5d779635901dce7793df92531dbf3caf0fb0a%252Fimage%2520%2847%29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=361a912f&sv=2) Reminder Unsloth itself provides **2x faster inference** natively as well, so always do not forget to call `FastLanguageModel.for_inference(model)`. If you want the model to output longer responses, set `max_new_tokens = 128` to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well! ### [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#saving--deployment) Saving + Deployment For saving and deploying your model in desired inference engines like Ollama, vLLM, Open WebUI, you will need to use the LoRA adapter on top of the base model. We have designated guides for each framework: [🖥️Inference & Deploymentchevron-right](https://unsloth.ai/docs/basics/inference-and-deployment) If you’re running inference on a single device (like a laptop or Mac), use llama.cpp to convert to GGUF format to use in Ollama, llama.cpp, LM Studio etc: [GGUF & llama.cppchevron-right](https://unsloth.ai/docs/basics/inference-and-deployment/saving-to-gguf) If you’re deploying an LLM for enterprise or multi-user inference for FP8, AWQ, use vLLM: [vLLMchevron-right](https://unsloth.ai/docs/basics/inference-and-deployment/vllm-guide) We can now save the fine-tuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face [tokenarrow-up-right](https://huggingface.co/settings/tokens) and add your token! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-8c577103f7c4fe883cabaf35c8437307c6501686%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7edca8a8&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-1a1be852ca551240bdce47cf99e6ccd7d31c1326%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d862cc0&sv=2) After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference! [hashtag](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-8.-were-done) 8\. We're done! ---------------------------------------------------------------------------------------------------------- You've successfully fine-tuned a language model and exported it to your desired inference engine with Unsloth! To learn more about fine-tuning tips and tricks, head over to our blogs which provide tremendous and educational value: [https://unsloth.ai/blog/arrow-up-right](https://unsloth.ai/blog/) If you need any help on fine-tuning, you can also join our Discord server [herearrow-up-right](https://discord.gg/unsloth) or [Reddit r/unslotharrow-up-right](https://www.reddit.com/r/unsloth/) . Thanks for reading and hopefully this was helpful! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-69482ba90d417f7bf98dddaf83795cdd3eb20efc%252Fsloth%2520sparkling%2520square.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=9008d40d&sv=2) [PreviousIntelchevron-left](https://unsloth.ai/docs/get-started/install/intel) [NextDatasets Guidechevron-right](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide) Last updated 27 days ago Was this helpful? * [1\. What Is Fine-tuning?](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-1.-what-is-fine-tuning) * [2\. Choose the Right Model + Method](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-2.-choose-the-right-model--method) * [3\. Your Dataset](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-3.-your-dataset) * [4\. Understand Training Hyperparameters](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-4.-understand-training-hyperparameters) * [5\. Install + Requirements](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-5.-install--requirements) * [Unsloth Notebooks](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#unsloth-notebooks) * [Local Installation](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#local-installation) * [6\. Training + Evaluation](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-6.-training--evaluation) * [7\. Running + Deploying the model](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-7.-running--deploying-the-model) * [Saving + Deployment](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#saving--deployment) * [8\. We're done!](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide#id-8.-were-done) Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Reinforcement Learning (RL) Guide | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Reinforcement Learning is where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**. * **Action:** What the model generates (e.g. a sentence). * **Reward:** A signal indicating how good or bad the model's action was (e.g. did the response follow instructions? was it helpful?). * **Environment:** The scenario or task the model is working on (e.g. answering a user’s question). ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-you-will-learn) 🦥What you will learn 1. What is RL? RLVR? PPO? GRPO? RLHF? RFT? Is **"Luck is All You Need?"** for RL? 2. What is an environment? Agent? Action? Reward function? Rewards? This article covers everything (from beginner to advanced) you need to know about GRPO, Reinforcement Learning (RL) and reward functions, along with tips, and the basics of using GRPO with [Unslotharrow-up-right](https://github.com/unslothai/unsloth) . If you're looking for a step-by-step tutorial for using GRPO, see our guide [here](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) . circle-check **Jan 15, 2026 update:** [Ultra long context RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/grpo-long-context) is here! Train gpt-oss with a 380K context window. **Nov 26, 2025 update:** We're introducing FP8 precision RL and GRPO in Unsloth! [Read blog](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-is-reinforcement-learning-rl) ❓What is Reinforcement Learning (RL)? ----------------------------------------------------------------------------------------------------------------------------------------------------------- The goal of RL is to: 1. **Increase the chance of seeing** **"good"** **outcomes.** 2. **Decrease the chance of seeing** **"bad"** **outcomes.** **That's it!** There are intricacies on what "good" and "bad" means, or how do we go about "increasing" or "decreasing" it, or what even "outcomes" means. For example, in the **Pacman game**: 1. The **environment** is the game world. 2. The **actions** you can take are UP, LEFT, RIGHT and DOWN. 3. The **rewards** are good if you eat a cookie, or bad if you hit one of the squiggly enemies. 4. In RL, you can't know the "best action" you can take, but you can observe intermediate steps, or the final game state (win or lose) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e853f7e6da505ee587642314b98180ebf840252c%252FRL%2520Game.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=374ada54&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-30bade1550c877bb7f79075c80ac79476b0ecd76%252FMath%2520RL.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=f56986b4&sv=2) Another example is imagine you are given the question: **"What is 2 + 2?"** (4) An unaligned language model will spit out 3, 4, C, D, -10, literally anything. 1. Numbers are better than C or D right? 2. Getting 3 is better than say 8 right? 3. Getting 4 is definitely correct. We just designed a **reward function**! ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr) 🏃From RLHF, PPO to GRPO and RLVR ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-5d0c90e4b45507d3e12c8b938cbd1679cd38f4f9%252FRLHF.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=f602f1f4&sv=2) OpenAI popularized the concept of [RLHFarrow-up-right](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback) (Reinforcement Learning from Human Feedback), where we train an **"agent"** to produce outputs to a question (the **state**) that are rated more useful by human beings. The thumbs up and down in ChatGPT for example can be used in the RLHF process. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-1e1dff9c921e787e669dee79c41a76db89e882e7%252FPPO.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=f3add6e7&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-f6156f2c519baf81e6ef286476f4092037303799%252FPPO%2520formula.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b984721d&sv=2) PPO formula The clip(..., 1-e, 1+e) term is used to force PPO not to take too large changes. There is also a KL term with beta set to > 0 to force the model not to deviate too much away. In order to do RLHF, [**PPO**arrow-up-right](https://en.wikipedia.org/wiki/Proximal_policy_optimization) (Proximal policy optimization) was developed. The **agent** is the language model in this case. In fact it's composed of 3 systems: 1. The **Generating Policy (current trained model)** 2. The **Reference Policy (original model)** 3. The **Value Model (average reward estimator)** We use the **Reward Model** to calculate the reward for the current environment, and our goal is to **maximize this**! The formula for PPO looks quite complicated because it was designed to be stable. Visit our [AI Engineer talkarrow-up-right](https://docs.unsloth.ai/ai-engineers-2025) we gave in 2025 about RL for more in depth maths derivations about PPO. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-4f4e188edbcad4f53aaa4a626bc5b2fd01334574%252FGRPO%2520%252B%2520RLVR.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7b61ba93&sv=2) DeepSeek developed [**GRPO**arrow-up-right](https://unsloth.ai/blog/grpo) (Group Relative Policy Optimization) to train their R1 reasoning models. The key differences to PPO are: 1. The **Value Model is removed,** replaced with statistics from calling the reward model multiple times. 2. The **Reward Model is removed** and replaced with just custom reward function which **RLVR** can be used. This means GRPO is extremely efficient. Previously PPO needed to train multiple models - now with the reward model and value model removed, we can save memory and speed up everything. **RLVR (Reinforcement Learning with Verifiable Rewards)** allows us to reward the model based on tasks with easy to verify solutions. For example: 1. Maths equations can be easily verified. Eg 2+2 = 4. 2. Code output can be verified as having executed correctly or not. 3. Designing verifiable reward functions can be tough, and so most examples are math or code. 4. Use-cases for GRPO isn’t just for code or math—its reasoning process can enhance tasks like email automation, database retrieval, law, and medicine, greatly improving accuracy based on your dataset and reward function - the trick is to define a **rubric - ie a list of smaller verifiable rewards, and not a final all consuming singular reward.** OpenAI popularized this in their [reinforcement learning finetuning (RFT)arrow-up-right](https://platform.openai.com/docs/guides/reinforcement-fine-tuning) offering for example. **Why "Group Relative"?** GRPO removes the value model entirely, but we still need to estimate the **"average reward"** given the current state. The **trick is to sample the LLM**! We then calculate the average reward through statistics of the sampling process across multiple different questions. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-29e188e5adc6de1e62c841e6cd9e34a2dae4994a%252FGroup%2520Relative.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b20b2c73&sv=2) For example for "What is 2+2?" we sample 4 times. We might get 4, 3, D, C. We then calculate the reward for each of these answers, then calculate the **average reward** and **standard deviation**, then **Z-score standardize** this! This creates the **advantages A**, which we will use in replacement of the value model. This saves a lot of memory! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-d40a73cd48b05b9205810a1946f4fc1dce81ae7d%252FStatistics.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=fbe3553c&sv=2) GRPO advantage calculation ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#luck-well-patience-is-all-you-need) 🤞Luck (well Patience) Is All You Need The trick of RL is you need 2 things only: 1. A question or instruction eg "What is 2+2?" "Create a Flappy Bird game in Python" 2. A reward function and verifier to verify if the output is good or bad. With only these 2, we can essentially **call a language model an infinite times** until we get a good answer. For example for "What is 2+2?", an untrained bad language model will output: _**0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31\*\*\*\***_ _**then suddenly 4**__**.**_ _**The reward signal was 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0\*\*\*\***_ _**then suddenly 1.**_ So by luck and by chance, RL managed to find the correct answer across multiple **rollouts**. Our goal is we want to see the good answer 4 more, and the rest (the bad answers) much less. **So the goal of RL is to be patient - in the limit, if the probability of the correct answer is at least a small number (not zero), it's just a waiting game - you will 100% for sure encounter the correct answer in the limit.** **So I like to call it as "Luck Is All You Need" for RL.** **Well a better phrase is "Patience is All You Need" for RL.** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-4f0cb4803aa22583e88dfa8de8061b66bbe6a6b1%252FLuck%2520is%2520all%2520you%2520need.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=cd17a32a&sv=2) RL essentially provides us a trick - instead of simply waiting for infinity, we do get "bad signals" ie bad answers, and we can essentially "guide" the model to already try not generating bad solutions. This means although you waited very long for a "good" answer to pop up, the model already has been changed to try its best not to output bad answers. In the "What is 2+2?" example - _**0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31\*\*\*\***_ _**then suddenly 4**__**.**_ Since we got bad answers, RL will influence the model to try NOT to output bad answers. This means over time, we are carefully "pruning" or moving the model's output distribution away from bad answers. This means RL is **efficient**, since we are NOT just waiting for infinity, but we are actively trying to "push" the model to go as much as possible to the "correct answer space". triangle-exclamation **If the probability is always 0, then RL will never work**. This is also why people like to do RL from an already instruction finetuned model, which can partially follow instructions reasonably well - this boosts the probability most likely above 0. [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-unsloth-offers-for-rl) 🦥What Unsloth offers for RL ------------------------------------------------------------------------------------------------------------------------------------------- * With 15GB VRAM, Unsloth allows you to transform any model up to 17B parameters like Llama 3.1 (8B), Phi-4 (14B), Mistral (7B) or Qwen2.5 (7B) into a reasoning model * **Unsloth now supports** [**RL for Vision/multimodal**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) **models!** * **Minimum requirement:** Just  5GB VRAM is enough to train your own reasoning model locally (for any model with 1.5B parameters or less) [⚡Tutorial: GRPO Trainingchevron-right](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) [👁️‍🗨️Vision RLchevron-right](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) circle-info For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation) ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-notebooks) GRPO notebooks: [**Qwen3.5 (4B)**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision_GRPO.ipynb) **- Vision -** _**new**_ [Qwen3-VL-8Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - Vision GSPO [Gemma 3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision-GRPO.ipynb) - Vision GSPO [gpt-oss-20barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) GSPO [DeepSeek-R1-0528-Qwen3-8Barrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) [Llama 3.2 (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_(3B)_GRPO_LoRA.ipynb) - Advanced [Qwen3 (4B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) - Advanced [Phi-4 (14B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) [Qwen2.5 (3B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) [Mistral v0.3 (7B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-GRPO.ipynb) [Llama 3.1 (8B)arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb) [Qwen3-8B - **FP8**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) (L4) circle-check We support [**GSPO**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning) and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable: * If you're not getting any reasoning, make sure you have enough training steps and ensure your [reward function/verifier](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-functions-verifier) is working. We provide examples for reward functions [here](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-function-examples) . * Previous demonstrations show that you could achieve your own "aha" moment with Qwen2.5 (3B) - but it required 2xA100 GPUs (160GB VRAM). Now, with Unsloth, you can achieve the same "aha" moment using just a single 5GB VRAM GPU. * Previously, GRPO was only supported for full fine-tuning, but we've made it work with QLoRA and LoRA * On [**20K context lengths**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-requirement-guidelines) for example with 8 generations per prompt, Unsloth uses only 54.3GB of VRAM for Llama 3.1 (8B), whilst standard implementations (+ Flash Attention 2) take **510.8GB (90% less for Unsloth)**. * Please note, this isn’t fine-tuning DeepSeek’s R1 distilled models or using distilled data from R1 for tuning which Unsloth already supported. This is converting a standard model into a full-fledged reasoning model using GRPO. In a test example, even though we only trained Phi-4 with 100 steps using GRPO, the results are already clear. The model without GRPO does not have the thinking token, whilst the one trained with GRPO does and also has the correct answer. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-5ae836156344a7c22241d0f76dbea09d58e04f8f%252Fprompt%2520only%2520example.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c649c9cf&sv=2) [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#training-with-grpo) 💻Training with GRPO --------------------------------------------------------------------------------------------------------------------------- For a tutorial on how to transform any open LLM into a reasoning model using Unsloth & GRPO, [see here](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) . circle-check For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation) ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#how-grpo-trains-a-model) **How GRPO Trains a Model** 1. For each question-answer pair, the model generates multiple possible responses (e.g., 8 variations). 2. Each response is evaluated using reward functions. 3. Training Steps: * If you have 300 rows of data, that's 300 training steps (or 900 steps if trained for 3 epochs). * You can increase the number of generated responses per question (e.g., from 8 to 16). 4. The model learns by updating its weights every step. circle-exclamation If you're having issues with your GRPO model not learning, we'd highly recommend to use our [Advanced GRPO notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-notebooks) as it has a much better reward function and you should see results much faster and frequently. ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#basics-tips) Basics/Tips * Wait for at least **300 steps** for the reward to actually increase. In order to get decent results, you may need to trade for a minimum of 12 hours (this is how GRPO works), but keep in mind this isn't compulsory as you can stop at anytime. * For optimal results have at least **500 rows of data**. You can try with even 10 rows of data but it's better to have more. * Each training run will always be different depending on your model, data, reward function/verifier etc. so though 300 steps is what we wrote as the minimum, sometimes it might be 1000 steps or more. So, it depends on various factors. * If you're using GRPO with Unsloth locally, please "pip install diffusers" as well if you get an error. Please also use the latest version of vLLM. * It’s advised to apply GRPO to a model at least **1.5B in parameters** to correctly generate thinking tokens as smaller models may not. * For GRPO's [**GPU VRAM requirements**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-requirement-guidelines) **for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM. * **Continuous fine-tuning is** possible and you can just leave GRPO running in the background. * In the example notebooks, we use the [**GSM8K dataset**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#gsm8k-reward-functions) , the current most popular choice for R1-style training. * If you’re using a base model, ensure you have a chat template. * The more you train with GRPO the better. The best part of GRPO is you don't even need that much data. All you need is a great reward function/verifier and the more time spent training, the better your model will get. Expect your reward vs step to increase as time progresses like this: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e44683faa4765a3b803edd4c02c4b468e45cc91d%252Funnamed.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b9bf85f2&sv=2) * Training loss tracking for GRPO is now built directly into Unsloth, eliminating the need for external tools like wandb etc. It contains full logging details for all reward functions now including the total aggregated reward function itself. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-58d958e1a3bfd968f1b1a4995a28261aa6413337%252FScreenshot%25202025-02-20%2520at%252004-52-52%2520Copy%2520of%2520Yet%2520another%2520copy%2520of%2520Llama3.1_%288B%29-GRPO.ipynb%2520-%2520Colab.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=9a8553b6&sv=2) ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#rl-on-unsupported-models) RL on unsupported models: You can also run RL with Unsloth on models that are not supported by vLLM, such as [Qwen3.5](https://unsloth.ai/docs/models/qwen3.5/fine-tune) . Simply set `fast_inference=False` when loading the model. [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-functions-verifiers) 📋Reward Functions / Verifiers --------------------------------------------------------------------------------------------------------------------------------------------- In Reinforcement Learning a **Reward Function** and a **Verifier** serve distinct roles in evaluating a model’s output. In general, you could interpret them as the same thing however, technically they're not but it does not matter as much as they are usually used in conjunction with each other. **Verifier**: * Determines whether the generated response is correct or incorrect. * It does not assign a numerical score—it simply verifies correctness. * Example: If a model generates "5" for "2+2", the verifier checks and labels it as "wrong" (since the correct answer is 4). * Verifiers can also execute code (e.g., in Python) to validate logic, syntax, and correctness without needing manual evaluation. **Reward Function**: * Converts verification results (or other criteria) into a numerical score. * Example: If an answer is wrong, it might assign a penalty (-1, -2, etc.), while a correct answer could get a positive score (+1, +2). * It can also penalize based on criteria beyond correctness, such as excessive length or poor readability. **Key Differences**: * A **Verifier** checks correctness but doesn’t score. * A **Reward Function** assigns a score but doesn’t necessarily verify correctness itself. * A Reward Function _can_ use a Verifier, but they are technically not the same. ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#understanding-reward-functions) **Understanding Reward Functions** GRPO's primary goal is to maximize reward and learn how an answer was derived, rather than simply memorizing and reproducing responses from its training data. * With every training step, GRPO **adjusts model weights** to maximize the reward. This process fine-tunes the model incrementally. * **Regular fine-tuning** (without GRPO) only **maximizes next-word prediction probability** but does not optimize for a reward. GRPO **optimizes for a reward function** rather than just predicting the next word. * You can **reuse data** across multiple epochs. * **Default reward functions** can be predefined to be used on a wide array of use cases or you can ask ChatGPT/local model to generate them for you. * There’s no single correct way to design reward functions or verifiers - the possibilities are endless. However, they must be well-designed and meaningful, as poorly crafted rewards can unintentionally degrade model performance. ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-function-examples) 🪙Reward Function Examples You can refer to the examples below. You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria #### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#example-1-simple-arithmetic-task) **Example #1: Simple Arithmetic Task** * **Question:** `"2 + 2"` * **Answer:** `"4"` * **Reward Function 1:** * If a number is detected → **+1** * If no number is detected → **\-1** * **Reward Function 2:** * If the number matches the correct answer → **+3** * If incorrect → **\-3** * **Total Reward:** _Sum of all reward functions_ #### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#example-2-email-automation-task) **Example #2: Email Automation Task** * **Question:** Inbound email * **Answer:** Outbound email * **Reward Functions:** * If the answer contains a required keyword → **+1** * If the answer exactly matches the ideal response → **+1** * If the response is too long → **\-1** * If the recipient's name is included → **+1** * If a signature block (phone, email, address) is present → **+1** ### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#unsloth-proximity-based-reward-function) Unsloth Proximity-Based Reward Function If you’ve checked out our [**Advanced GRPO Colab Notebook**](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-notebooks) , you’ll notice we’ve created a **custom proximity-based reward function** built completely from scratch, which is designed to reward answers that are closer to the correct one. This flexible function can be applied across a wide range of tasks. * In our examples, we enable reasoning in Qwen3 (Base) and guide it toward specific tasks * Apply Pre-finetuning strategies to avoid GRPO’s default tendency to just learn formatting * Boost evaluation accuracy with regex-based matching * Create custom GRPO templates beyond generic prompts like `think`, e.g., `` * Apply proximity-based scoring — models get more reward for closer answers (e.g., predicting 9 instead of 10 is better than 3) while outliers are penalized #### [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#gsm8k-reward-functions) GSM8K Reward Functions In our other examples, we use existing GSM8K reward functions by [@willccbbarrow-up-right](https://x.com/willccbb) which is popular and shown to be quite effective: * **correctness\_reward\_func** – Rewards exact label matches. * **int\_reward\_func** – Encourages integer-only answers. * **soft\_format\_reward\_func** – Checks structure but allows minor newline mismatches. * **strict\_format\_reward\_func** – Ensures response structure matches the prompt, including newlines. * **xmlcount\_reward\_func** – Ensures exactly one of each XML tag in the response. [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#using-vllm) 🧮Using vLLM ----------------------------------------------------------------------------------------------------------- You can now use [vLLMarrow-up-right](https://github.com/vllm-project/vllm/) directly in your finetuning stack, which allows for much more throughput and allows you to finetune and do inference on the model at the same time! On 1x A100 40GB, expect 4000 tokens / s or so with Unsloth’s dynamic 4bit quant of Llama 3.2 3B Instruct. On a 16GB Tesla T4 (free Colab GPU), you can get 300 tokens / s. We also magically removed double memory usage when loading vLLM and Unsloth together, allowing for savings of 5GB or so for Llama 3.1 8B and 3GB for Llama 3.2 3B. Unsloth could originally finetune Llama 3.3 70B Instruct in 1x 48GB GPU with Llama 3.3 70B weights taking 40GB of VRAM. If we do not remove double memory usage, then we’ll need >= 80GB of VRAM when loading Unsloth and vLLM together. But with Unsloth, you can still finetune and get the benefits of fast inference in one package in under 48GB of VRAM! To use fast inference, first install vllm, and instantiate Unsloth with fast\_inference: [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-requirement-guidelines) ✅GRPO Requirement Guidelines -------------------------------------------------------------------------------------------------------------------------------------------- When you’re using Unsloth to do GRPO, we smartly reduce VRAM usage by over 90% when compared to standard implementations with Flash Attention 2 by using multiple tricks! On 20K context lengths for example with 8 generations per prompt, Unsloth uses only **54.3GB of VRAM for Llama 3.1 8B**, whilst standard implementations take **510.8GB (90% less for Unsloth)**. 1. For GRPO's **GPU VRAM requirements for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM. 2. Our new memory efficient linear kernels for GRPO slashes memory usage by 8x or more. This shaves 68.5GB of memory, whilst being actually faster through the help of torch.compile! 3. We leverage our smart [Unsloth gradient checkpointingarrow-up-right](https://unsloth.ai/blog/long-context) algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves 52GB of memory. 4. Unsloth also uses the same GPU / CUDA memory space as the underlying inference engine (vLLM), unlike implementations in other packages. This shaves 16GB of memory. Metrics Unsloth Standard + FA2 Training Memory Cost (GB) 42GB 414GB GRPO Memory Cost (GB) 9.8GB 78.3GB Inference Cost (GB) 0GB 16GB Inference KV Cache for 20K context length (GB) 2.5GB 2.5GB Total Memory Usage 54.33GB (90% less) 510.8GB In typical standard GRPO implementations, you need to create 2 logits of size (8. 20K) to calculate the GRPO loss. This takes 2 \* 2 bytes \* 8 (num generations) \* 20K (context length) \* 128256 (vocabulary size) = 78.3GB in VRAM. Unsloth shaves 8x memory usage for long context GRPO, so we need only an extra 9.8GB in extra VRAM for 20K context lengths! We also need to from the KV Cache in 16bit. Llama 3.1 8B has 32 layers, and both K and V are 1024 in size. So memory usage for 20K context length = 2 \* 2 bytes \* 32 layers \* 20K context length \* 1024 = 2.5GB per batch. We would set the batch size for vLLM to 8, but we shall leave it at 1 for our calculations to save VRAM. Otherwise you will need 20GB for the KV cache. [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#unsloth-rl-3-hour-workshop-video) 🎥 Unsloth RL 3 hour Workshop Video -------------------------------------------------------------------------------------------------------------------------------------------------------- [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#further-reading) 🎓Further Reading --------------------------------------------------------------------------------------------------------------------- 1. Nathan Lambert's RLHF Book is a must! [https://rlhfbook.com/c/11-policy-gradients.htmlarrow-up-right](https://rlhfbook.com/c/11-policy-gradients.html) 2. Yannic Kilcher's GRPO Youtube video is also a must! [https://www.youtube.com/watch?v=bAWV\_yrqx4warrow-up-right](https://www.youtube.com/watch?v=bAWV_yrqx4w) 3. We did a 3 hour workshop at AI Engineer World's Fair 2025. Slides are other material are at [https://docs.unsloth.ai/ai-engineers-2025arrow-up-right](https://docs.unsloth.ai/ai-engineers-2025) 4. Advanced GRPO notebook via Unsloth. [https://docs.unsloth.ai/basics/reinforcement-learning-guide/tutorial-train-your-own-reasoning-model-with-grpoarrow-up-right](https://docs.unsloth.ai/basics/reinforcement-learning-guide/tutorial-train-your-own-reasoning-model-with-grpo) 5. GRPO from a base model notebook: [https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(4B)-GRPO.ipynbarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) [hashtag](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#video-tutorials) Video Tutorials ------------------------------------------------------------------------------------------------------------------- Here are some video tutorials created by amazing YouTubers who we think are fantastic! Great to learn about how to prep your dataset and explanations behind Reinforcement Learning + GRPO basics Local GRPO on your own device [PreviousTutorial: Finetune Llama-3 and Use In Ollamachevron-left](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/tutorial-how-to-finetune-llama-3-and-use-in-ollama) [Next7x Longer Context RLchevron-right](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/grpo-long-context) Last updated 11 days ago Was this helpful? * [🦥What you will learn](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-you-will-learn) * [❓What is Reinforcement Learning (RL)?](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-is-reinforcement-learning-rl) * [🏃From RLHF, PPO to GRPO and RLVR](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr) * [🤞Luck (well Patience) Is All You Need](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#luck-well-patience-is-all-you-need) * [🦥What Unsloth offers for RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#what-unsloth-offers-for-rl) * [GRPO notebooks:](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-notebooks) * [💻Training with GRPO](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#training-with-grpo) * [How GRPO Trains a Model](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#how-grpo-trains-a-model) * [Basics/Tips](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#basics-tips) * [RL on unsupported models:](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#rl-on-unsupported-models) * [📋Reward Functions / Verifiers](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-functions-verifiers) * [Understanding Reward Functions](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#understanding-reward-functions) * [🪙Reward Function Examples](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#reward-function-examples) * [Unsloth Proximity-Based Reward Function](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#unsloth-proximity-based-reward-function) * [🧮Using vLLM](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#using-vllm) * [✅GRPO Requirement Guidelines](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#grpo-requirement-guidelines) * [🎥 Unsloth RL 3 hour Workshop Video](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#unsloth-rl-3-hour-workshop-video) * [🎓Further Reading](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#further-reading) * [Video Tutorials](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide#video-tutorials) Was this helpful? sun-brightdesktopmoon Copy epsilon=0.2, epsilon_high=0.28, # one sided delta=1.5 # two sided loss_type='gspo', # or: loss_type='grpo', # or: loss_type='dr_grpo', mask_truncated_completions=True, Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3.5-4B", fast_inference=False, ) Copy # pip install unsloth vllm from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-3B-Instruct", fast_inference = True, ) sun-brightdesktopmoon --- # Unsloth Installation | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth can be used in two ways: through [Unsloth Studio](https://unsloth.ai/docs/new/studio/install) , the web UI, or through Unsloth Core, the original code-based version. See our [system requirements](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) Unsloth Studio works on MacOS, Linux, Windows, NVIDIA, and more. Use the same install commands to update or `unsloth studio update`. **MacOS, Linux, WSL:** Copy curl -fsSL https://unsloth.ai/install.sh | sh **Windows PowerShell:** Copy irm https://unsloth.ai/install.ps1 | iex **Launch Unsloth Studio:** Copy unsloth studio -H 0.0.0.0 -p 8888 apple[MacOS](https://unsloth.ai/docs/get-started/install/mac) [](https://unsloth.ai/docs/get-started/install/pip-install) desktop-arrow-down[uv, pip install & venv](https://unsloth.ai/docs/get-started/install/pip-install) windows[Windows](https://unsloth.ai/docs/get-started/install/windows-installation) docker[Docker](https://unsloth.ai/docs/get-started/install/docker) [](https://unsloth.ai/docs/get-started/install/updating) arrow-rotate-right[Updating](https://unsloth.ai/docs/get-started/install/updating) square-up-right[AMD](https://unsloth.ai/docs/get-started/install/amd) info[Intel](https://unsloth.ai/docs/get-started/install/intel) [](https://unsloth.ai/docs/get-started/install/conda-install) snake[Conda](https://unsloth.ai/docs/get-started/install/conda-install) vscode[VS Code](https://unsloth.ai/docs/get-started/install/vs-code) [](https://unsloth.ai/docs/get-started/install/google-colab) google[Google Colab](https://unsloth.ai/docs/get-started/install/google-colab) [PreviousAll Our Modelschevron-left](https://unsloth.ai/docs/get-started/unsloth-model-catalog) [Nextuv, pip install & venvchevron-right](https://unsloth.ai/docs/get-started/install/pip-install) Last updated 8 days ago Was this helpful? Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Unsloth Environment Flags | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Environment variable Purpose `os.environ["UNSLOTH_RETURN_LOGITS"] = "1"` Forcibly returns logits - useful for evaluation if logits are needed. `os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"` Disables auto compiler. Could be useful to debug incorrect finetune results. `os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"` Disables fast generation for generic models. `os.environ["UNSLOTH_ENABLE_LOGGING"] = "1"` Enables auto compiler logging - useful to see which functions are compiled or not. `os.environ["UNSLOTH_FORCE_FLOAT32"] = "1"` On float16 machines, use float32 and not float16 mixed precision. Useful for Gemma 3. `os.environ["UNSLOTH_STUDIO_DISABLED"] = "1"` Disables extra features. `os.environ["UNSLOTH_COMPILE_DEBUG"] = "1"` Turns on extremely verbose `torch.compile`logs. `os.environ["UNSLOTH_COMPILE_MAXIMUM"] = "0"` Enables maximum `torch.compile`optimizations - not recommended. `os.environ["UNSLOTH_COMPILE_IGNORE_ERRORS"] = "1"` Can turn this off to enable fullgraph parsing. `os.environ["UNSLOTH_FULLGRAPH"] = "0"` Enable `torch.compile` fullgraph mode `os.environ["UNSLOTH_DISABLE_AUTO_UPDATES"] = "1"` Forces no updates to `unsloth-zoo` Another possiblity is maybe the model uploads we uploaded are corrupted, but unlikely. Try the following: Copy model, tokenizer = FastVisionModel.from_pretrained( "Qwen/Qwen2-VL-7B-Instruct", use_exact_model_name = True, ) [PreviousChat Templateschevron-left](https://unsloth.ai/docs/basics/chat-templates) [NextContinued Pretrainingchevron-right](https://unsloth.ai/docs/basics/continued-pretraining) Last updated 2 months ago Was this helpful? Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Continued Pretraining | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close * The [text completion notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) is for continued pretraining/raw text. * The [continued pretraining notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) is for learning another language. You can read more about continued pretraining and our release in our [blog postarrow-up-right](https://unsloth.ai/blog/contpretraining) . [hashtag](https://unsloth.ai/docs/basics/continued-pretraining#what-is-continued-pretraining) What is Continued Pretraining? --------------------------------------------------------------------------------------------------------------------------------- Continued or continual pretraining (CPT) is necessary to “steer” the language model to understand new domains of knowledge, or out of distribution domains. Base models like Llama-3 8b or Mistral 7b are first pretrained on gigantic datasets of trillions of tokens (Llama-3 for e.g. is 15 trillion). But sometimes these models have not been well trained on other languages, or text specific domains, like law, medicine or other areas. So continued pretraining (CPT) is necessary to make the language model learn new tokens or datasets. [hashtag](https://unsloth.ai/docs/basics/continued-pretraining#advanced-features) Advanced Features: --------------------------------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/basics/continued-pretraining#loading-lora-adapters-for-continued-finetuning) Loading LoRA adapters for continued finetuning If you saved a LoRA adapter through Unsloth, you can also continue training using your LoRA weights. The optimizer state will be reset as well. To load even optimizer states to continue finetuning, see the next section. Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "LORA_MODEL_NAME", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) trainer = Trainer(...) trainer.train() ### [hashtag](https://unsloth.ai/docs/basics/continued-pretraining#continued-pretraining-and-finetuning-the-lm_head-and-embed_tokens-matrices) Continued Pretraining & Finetuning the `lm_head` and `embed_tokens` matrices Add `lm_head` and `embed_tokens`. For Colab, sometimes you will go out of memory for Llama-3 8b. If so, just add `lm_head`. Then use 2 different learning rates - a 2-10x smaller one for the `lm_head` or `embed_tokens` like so: [PreviousUnsloth Environment Flagschevron-left](https://unsloth.ai/docs/basics/unsloth-environment-flags) [NextLast Checkpointchevron-right](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint) Last updated 5 months ago Was this helpful? * [What is Continued Pretraining?](https://unsloth.ai/docs/basics/continued-pretraining#what-is-continued-pretraining) * [Advanced Features:](https://unsloth.ai/docs/basics/continued-pretraining#advanced-features) * [Loading LoRA adapters for continued finetuning](https://unsloth.ai/docs/basics/continued-pretraining#loading-lora-adapters-for-continued-finetuning) * [Continued Pretraining & Finetuning the lm\_head and embed\_tokens matrices](https://unsloth.ai/docs/basics/continued-pretraining#continued-pretraining-and-finetuning-the-lm_head-and-embed_tokens-matrices) Was this helpful? sun-brightdesktopmoon Copy model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",\ "lm_head", "embed_tokens",], lora_alpha = 16, ) Copy from unsloth import UnslothTrainer, UnslothTrainingArguments trainer = UnslothTrainer( .... args = UnslothTrainingArguments( .... learning_rate = 5e-5, embedding_learning_rate = 5e-6, # 2-10x smaller than learning_rate ), ) sun-brightdesktopmoon --- # Multi-GPU Fine-tuning with Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed. This means you can already leverage parallelism methods such as **FSDP** and **DDP** with Unsloth. #### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth#see-our-new-distributed-data-parallel-ddp-multi-gpu-guide-here) **See our new Distributed Data Parallel** [**(DDP) multi-GPU Guide here**](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp) **.** We know that the process can be complex and requires manual setup. We’re working hard to make multi-GPU support much simpler and more user-friendly, and we’ll be announcing official multi-GPU support for Unsloth soon. For now, you can use our [Magistral-2509 Kaggle notebook](https://unsloth.ai/docs/models/tutorials/magistral-how-to-run-and-fine-tune#fine-tuning-magistral-with-unsloth) as an example which utilizes multi-GPU Unsloth to fit the 24B parameter model or our [DDP guide](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp) . **In the meantime**, to enable multi GPU for DDP, do the following: 1. Create your training script as `train.py` (or similar). For example, you can use one of our [training scriptsarrow-up-right](https://github.com/unslothai/notebooks/tree/main/python_scripts) created from our various notebooks! 2. Run `accelerate launch train.py` or `torchrun --nproc_per_node N_GPUS train.py` where `N_GPUS` is the number of GPUs you have. #### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth#pipeline-model-splitting-loading) **Pipeline / model splitting loading** If you do not have enough VRAM for 1 GPU to load say Llama 70B, no worries - we will split the model for you on each GPU! To enable this, use the `device_map = "balanced"` flag: Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( "unsloth/Llama-3.3-70B-Instruct", load_in_4bit = True, device_map = "balanced", ) **Stay tuned for our official announcement!** For more details, check out our ongoing [Pull Requestarrow-up-right](https://github.com/unslothai/unsloth/issues/2435) discussing multi-GPU support. [PreviousOpenAI Codexchevron-left](https://unsloth.ai/docs/basics/codex) [NextDistributed Data Parallel (DDP)chevron-right](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp) Last updated 1 month ago Was this helpful? Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Finetuning from Last Checkpoint | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close You must edit the `Trainer` first to add `save_strategy` and `save_steps`. Below saves a checkpoint every 50 steps to the folder `outputs`. Copy trainer = SFTTrainer( .... args = TrainingArguments( .... output_dir = "outputs", save_strategy = "steps", save_steps = 50, ), ) Then in the trainer do: Copy trainer_stats = trainer.train(resume_from_checkpoint = True) Which will start from the latest checkpoint and continue training. ### [hashtag](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint#wandb-integration) Wandb Integration Copy # Install library !pip install wandb --upgrade # Setting up Wandb !wandb login import os os.environ["WANDB_PROJECT"] = "" os.environ["WANDB_LOG_MODEL"] = "checkpoint" Then in `TrainingArguments()` set To train the model, do `trainer.train()`; to resume training, do [hashtag](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint#how-do-i-do-early-stopping) ❓How do I do Early Stopping? -------------------------------------------------------------------------------------------------------------------------------------- If you want to stop or pause the finetuning / training run since the evaluation loss is not decreasing, then you can use early stopping which stops the training process. Use `EarlyStoppingCallback`. As usual, set up your trainer and your evaluation dataset. The below is used to stop the training run if the `eval_loss` (the evaluation loss) is not decreasing after 3 steps or so. We then add the callback which can also be customized: Then train the model as usual via `trainer.train() .` [PreviousContinued Pretrainingchevron-left](https://unsloth.ai/docs/basics/continued-pretraining) [NextUnsloth Benchmarkschevron-right](https://unsloth.ai/docs/basics/unsloth-benchmarks) Last updated 5 months ago Was this helpful? * [Wandb Integration](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint#wandb-integration) * [❓How do I do Early Stopping?](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint#how-do-i-do-early-stopping) Was this helpful? sun-brightdesktopmoon Copy report_to = "wandb", logging_steps = 1, # Change if needed save_steps = 100 # Change if needed run_name = "" # (Optional) Copy import wandb run = wandb.init() artifact = run.use_artifact('//', type='model') artifact_dir = artifact.download() trainer.train(resume_from_checkpoint=artifact_dir) Copy from trl import SFTConfig, SFTTrainer trainer = SFTTrainer( args = SFTConfig( fp16_full_eval = True, per_device_eval_batch_size = 2, eval_accumulation_steps = 4, output_dir = "training_checkpoints", # location of saved checkpoints for early stopping save_strategy = "steps", # save model every N steps save_steps = 10, # how many steps until we save the model save_total_limit = 3, # keep ony 3 saved checkpoints to save disk space eval_strategy = "steps", # evaluate every N steps eval_steps = 10, # how many steps until we do evaluation load_best_model_at_end = True, # MUST USE for early stopping metric_for_best_model = "eval_loss", # metric we want to early stop on greater_is_better = False, # the lower the eval loss, the better ), model = model, tokenizer = tokenizer, train_dataset = new_dataset["train"], eval_dataset = new_dataset["test"], ) Copy from transformers import EarlyStoppingCallback early_stopping_callback = EarlyStoppingCallback( early_stopping_patience = 3, # How many steps we will wait if the eval loss doesn't decrease # For example the loss might increase, but decrease after 3 steps early_stopping_threshold = 0.0, # Can set higher - sets how much loss should decrease by until # we consider early stopping. For eg 0.01 means if loss was # 0.02 then 0.01, we consider to early stop the run. ) trainer.add_callback(early_stopping_callback) sun-brightdesktopmoon --- # Unsloth Benchmarks | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close * For more detailed benchmarks, read our [Llama 3.3 Blogarrow-up-right](https://unsloth.ai/blog/llama3-3) . * Benchmarking of Unsloth was also conducted by [🤗Hugging Facearrow-up-right](https://huggingface.co/blog/unsloth-trl) . circle-exclamation If your speed seems slower at first, it’s likely because `torch.compile` typically takes ~5 minutes (or longer) to warm up and finish compiling. Make sure you measure throughput **after** it’s fully loaded as over longer runs, Unsloth should be much faster. Tested on H100 and [Blackwell](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) GPUs. We tested using the Alpaca Dataset, a batch size of 2, gradient accumulation steps of 4, rank = 32, and applied QLoRA on all linear layers (q, k, v, o, gate, up, down): Model VRAM 🦥Unsloth speed 🦥VRAM reduction 🦥Longer context 😊Hugging Face + FA2 Llama 3.3 (70B) 80GB 2x \>75% 13x longer 1x Llama 3.1 (8B) 80GB 2x \>70% 12x longer 1x [hashtag](https://unsloth.ai/docs/basics/unsloth-benchmarks#context-length-benchmarks) Context length benchmarks --------------------------------------------------------------------------------------------------------------------- circle-info The more data you have, the less VRAM Unsloth uses due to our [gradient checkpointingarrow-up-right](https://unsloth.ai/blog/long-context) algorithm + Apple's CCE algorithm! ### [hashtag](https://unsloth.ai/docs/basics/unsloth-benchmarks#llama-3.1-8b-max.-context-length) **Llama 3.1 (8B) max. context length** We tested Llama 3.1 (8B) Instruct and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads. GPU VRAM 🦥Unsloth context length Hugging Face + FA2 8 GB 2,972 OOM 12 GB 21,848 932 16 GB 40,724 2,551 24 GB 78,475 5,789 40 GB 153,977 12,264 48 GB 191,728 15,502 80 GB 342,733 28,454 ### [hashtag](https://unsloth.ai/docs/basics/unsloth-benchmarks#llama-3.3-70b-max.-context-length) **Llama 3.3 (70B) max. context length** We tested Llama 3.3 (70B) Instruct on a 80GB A100 and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads. GPU VRAM 🦥Unsloth context length Hugging Face + FA2 48 GB 12,106 OOM 80 GB 89,389 6,916 [PreviousLast Checkpointchevron-left](https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint) [NextNew 3x Faster Trainingchevron-right](https://unsloth.ai/docs/blog/3x-faster-training-packing) Last updated 2 months ago Was this helpful? * [Context length benchmarks](https://unsloth.ai/docs/basics/unsloth-benchmarks#context-length-benchmarks) * [Llama 3.1 (8B) max. context length](https://unsloth.ai/docs/basics/unsloth-benchmarks#llama-3.1-8b-max.-context-length) * [Llama 3.3 (70B) max. context length](https://unsloth.ai/docs/basics/unsloth-benchmarks#llama-3.3-70b-max.-context-length) Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # Quantization-Aware Training (QAT) | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close In collaboration with PyTorch, we're introducing QAT (Quantization-Aware Training) in Unsloth to enable **trainable quantization** that recovers as much accuracy as possible. This results in significantly better model quality compared to standard 4-bit naive quantization. QAT can recover up to **70% of the lost accuracy** and achieve a **1–3%** model performance improvement on benchmarks such as GPQA and MMLU Pro. > **Try QAT with our free** [**Qwen3 (4B) notebook**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantization) 📚Quantization Naively quantizing a model is called **post-training quantization** (PTQ). For example, assume we want to quantize to 8bit integers: 1. Find `max(abs(W))` 2. Find `a = 127/max(abs(W))` where a is int8's maximum range which is 127 3. Quantize via `qW = int8(round(W * a))` ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-f3e1cee8e4047dcbbbace7548694ad63af9869de%252Fquant-freeze.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=27520418&sv=2) Dequantizing back to 16bits simply does the reverse operation by `float16(qW) / a` . Post-training quantization (PTQ) can greatly reduce storage and inference costs, but quite often degrades accuracy when representing high-precision values with fewer bits - especially at 4-bit or lower. One way to solve this to utilize our [**dynamic GGUF quants**](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) , which uses a calibration dataset to change the quantization procedure to allocate more importance to important weights. The other way is to make **quantization smarter, by making it trainable or learnable**! ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#smarter-quantization) 🔥Smarter Quantization ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-1f6260ef5c041ada2f8b1fb4c6aad114f61061d4%252F4bit_QAT_recovery_sideways_clipped75_bigtext_all%281%29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=db52d2bd&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ad1ac9d29482ea07cbabb6efa18a0d1f06b297e9%252FQLoRA_QAT_Accuracy_Boosts_v7_bigaxes_nogrid_600dpi.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c4c4a3d3&sv=2) To enable smarter quantization, we collaborated with the [TorchAOarrow-up-right](https://github.com/pytorch/ao) team to add **Quantization-Aware Training (QAT)** directly inside of Unsloth - so now you can fine-tune models in Unsloth and then export them to 4-bit QAT format directly with accuracy improvements! In fact, **QAT recovers 66.9%** of Gemma3-4B on GPQA, and increasing the raw accuracy by +1.0%. Gemma3-12B on BBH recovers 45.5%, and **increased the raw accuracy by +2.1%**. QAT has no extra overhead during inference, and uses the same disk and memory usage as normal naive quantization! So you get all the benefits of low-bit quantization, but with much increased accuracy! ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantization-aware-training) 🔍Quantization-Aware Training QAT simulates the true quantization procedure by "**fake quantizing**" weights and optionally activations during training, which typically means rounding high precision values to quantized ones (while staying in high precision dtype, e.g. bfloat16) and then immediately dequantizing them. TorchAO enables QAT by first (1) inserting fake quantize operations into linear layers, and (2) transforms the fake quantize operations to actual quantize and dequantize operations after training to make it inference ready. Step 1 enables us to train a more accurate quantization representation. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-3d990e2bf19ef1aa7e65a8dd07e4b71cf8882a2a%252Fqat_diagram.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=899f09a1&sv=2) ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#qat--lora-finetuning) ✨QAT + LoRA finetuning QAT in Unsloth can additionally be combined with LoRA fine-tuning to enable the benefits of both worlds: significantly reducing storage and compute requirements during training while mitigating quantization degradation! We support multiple methods via `qat_scheme` including `fp8-int4`, `fp8-fp8`, `int8-int4`, `int4` . We also plan to add custom definitions for QAT in a follow up release! ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#exporting-qat-models) 🫖Exporting QAT models After fine-tuning in Unsloth, you can call `model.save_pretrained_torchao` to save your trained model using TorchAO’s PTQ format. You can also upload these to the HuggingFace hub! We support any config, and we plan to make text based methods as well, and to make the process more simpler for everyone! But first, we have to prepare the QAT model for the final conversion step via: And now we can select which QAT style you want: You can then run the merged QAT lower precision model in vLLM, Unsloth and other systems for inference! These are all in the [Qwen3-4B QAT Colab notebookarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) we have as well! ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantizing-models-without-training) 🫖Quantizing models without training You can also call `model.save_pretrained_torchao` directly without doing any QAT as well! This is simply PTQ or native quantization. For example, saving to Dynamic float8 format is below: ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#executorch-qat-for-mobile-deployment) 📱ExecuTorch - QAT for mobile deployment With Unsloth and TorchAO’s QAT support, you can also fine-tune a model in Unsloth and seamlessly export it to [ExecuTorcharrow-up-right](https://github.com/pytorch/executorch) (PyTorch’s solution for on-device inference) and deploy it directly on mobile. See an example in action [herearrow-up-right](https://huggingface.co/metascroy/Qwen3-4B-int8-int4-unsloth) with more detailed workflows on the way! **Announcement coming soon!** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-53631bae5588644d2c64cec18f371f0a7e2688c6%252Fswiftpm_xcode.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d725d0ee&sv=2) ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#how-to-enable-qat) 🌻How to enable QAT Update Unsloth to the latest version, and also install the latest TorchAO! Then **try QAT with our free** [**Qwen3 (4B) notebook**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)_Instruct-QAT.ipynb) ### [hashtag](https://unsloth.ai/docs/blog/quantization-aware-training-qat#acknowledgements) 💁Acknowledgements Huge thanks to the entire PyTorch and TorchAO team for their help and collaboration! Extreme thanks to Andrew Or, Jerry Zhang, Supriya Rao, Scott Roy and Mergen Nachin for helping on many discussions on QAT, and on helping to integrate it into Unsloth! Also thanks to the Executorch team as well! [Previous500K Context Trainingchevron-left](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning) [NextDGX Stationchevron-right](https://unsloth.ai/docs/blog/dgx-station) Last updated 3 months ago Was this helpful? * [📚Quantization](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantization) * [🔥Smarter Quantization](https://unsloth.ai/docs/blog/quantization-aware-training-qat#smarter-quantization) * [🔍Quantization-Aware Training](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantization-aware-training) * [✨QAT + LoRA finetuning](https://unsloth.ai/docs/blog/quantization-aware-training-qat#qat--lora-finetuning) * [🫖Exporting QAT models](https://unsloth.ai/docs/blog/quantization-aware-training-qat#exporting-qat-models) * [🫖Quantizing models without training](https://unsloth.ai/docs/blog/quantization-aware-training-qat#quantizing-models-without-training) * [📱ExecuTorch - QAT for mobile deployment](https://unsloth.ai/docs/blog/quantization-aware-training-qat#executorch-qat-for-mobile-deployment) * [🌻How to enable QAT](https://unsloth.ai/docs/blog/quantization-aware-training-qat#how-to-enable-qat) * [💁Acknowledgements](https://unsloth.ai/docs/blog/quantization-aware-training-qat#acknowledgements) Was this helpful? sun-brightdesktopmoon Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Qwen3-4B-Instruct-2507", max_seq_length = 2048, load_in_16bit = True, ) model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 32, # We support fp8-int4, fp8-fp8, int8-int4, int4 qat_scheme = "int4", ) Copy from torchao.quantization import quantize_ from torchao.quantization.qat import QATConfig quantize_(model, QATConfig(step = "convert")) Copy # Use the exact same config as QAT (convenient function) model.save_pretrained_torchao( model, "tokenizer", torchao_config = model._torchao_config.base_config, ) # Int4 QAT from torchao.quantization import Int4WeightOnlyConfig model.save_pretrained_torchao( model, "tokenizer", torchao_config = Int4WeightOnlyConfig(), ) # Int8 QAT from torchao.quantization import Int8DynamicActivationInt8WeightConfig model.save_pretrained_torchao( model, "tokenizer", torchao_config = Int8DynamicActivationInt8WeightConfig(), ) Copy # Float8 from torchao.quantization import PerRow from torchao.quantization import Float8DynamicActivationFloat8WeightConfig torchao_config = Float8DynamicActivationFloat8WeightConfig(granularity = PerRow()) model.save_pretrained_torchao(torchao_config = torchao_config) Copy pip install --upgrade --no-cache-dir --force-reinstall unsloth unsloth_zoo pip install torchao==0.14.0 fbgemm-gpu-genai==1.3.0 sun-brightdesktopmoon --- # Fine-tuning LLMs with NVIDIA DGX Spark and Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth enables local fine-tuning of LLMs with up to **200B parameters** on the NVIDIA DGX™ Spark. With 128 GB of unified memory, you can train massive models such as **gpt-oss-120b**, and run or deploy inference directly on DGX Spark. As shown at [OpenAI DevDayarrow-up-right](https://x.com/UnslothAI/status/1976284209842118714) , gpt-oss-20b was trained with RL and Unsloth on DGX Spark to auto-win 2048. You can train using Unsloth in a Docker container or virtual environment on DGX Spark. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-ff5c4752dccb8f922b937f8e3b0db58e2d836507%252Funsloth%2520nvidia%2520dgx%2520spark.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d11731a0&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a8472482c49e1763378b609f8f537ca89df60260%252FNotebooks%2520on%2520dgx.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=288d9ddb&sv=2) In this tutorial, we’ll train gpt-oss-20b with RL using Unsloth notebooks after installing Unsloth on your DGX Spark. gpt-oss-120b will use around **68GB** of unified memory. After 1,000 steps and 4 hours of RL training, the gpt-oss model greatly outperforms the original on 2048, and longer training would further improve results. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-3bdcb0fda2ad188142e58f04c855b6dcfbd5ba94%252Fopenai%2520devday%2520unsloth%2520feature.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=cc268910&sv=2) You can watch Unsloth featured on OpenAI DevDay 2025 [herearrow-up-right](https://youtu.be/1HL2YHRj270?si=8SR6EChF34B1g-5r&t=1080) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-4a8bd4ecc7ee3d123c19158df5dfdcec35df8532%252FScreenshot%25202025-10-13%2520at%25204.22.32%25E2%2580%25AFPM.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=417fc5f8&sv=2) gpt-oss trained with RL consistently outperforms on 2048. ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#step-by-step-tutorial) ⚡ Step-by-Step Tutorial 1 **Start with Unsloth Docker image for DGX Spark** First, build the Docker image using the DGX Spark Dockerfile which can be [found herearrow-up-right](https://raw.githubusercontent.com/unslothai/notebooks/main/Dockerfile_DGX_Spark) . You can also run the below in a Terminal in the DGX Spark: Then, build the training Docker image using saved Dockerfile: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-7ebcf195c154b0e569115e1f9513cf002ee57b16%252Fdgx1.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=7f6afe05&sv=2) chevron-rightYou can also click to see the full DGX Spark Dockerfile[hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#you-can-also-click-to-see-the-full-dgx-spark-dockerfile) 2 **Launch container** Launch the training container with GPU access and volume mounts: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a67c36494f5c4ab4017748d490fb258655cd2378%252Fdgx2.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=14e268c5&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-b7758db087ab8b724049361781952b5ed154dfe8%252Fdgx5.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=6ee6379e&sv=2) 3 **Start Jupyter and Run Notebooks** Inside the container, start Jupyter and run the required notebook. You can use the Reinforcement Learning gpt-oss 20b to win 2048 [notebook herearrow-up-right](https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_DGX_Spark.ipynb) . In fact all [Unsloth notebooksarrow-up-right](https://docs.unsloth.ai/get-started/unsloth-notebooks) work in DGX Spark including the **120b** notebook! Just remove the installation cells. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-a8472482c49e1763378b609f8f537ca89df60260%252FNotebooks%2520on%2520dgx.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=288d9ddb&sv=2) The below commands can be used to run the RL notebook as well. After Jupyter Notebook is launched, open up the “`gpt_oss_20B_RL_2048_Game.ipynb`” ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-0862eed0acf0656ff0cb802b6aebc30892997e3b%252Fdgx6.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d309acbd&sv=2) Don't forget Unsloth also allows you to [save and run](https://unsloth.ai/docs/basics/inference-and-deployment) your models after fine-tuning so you can locally deploy them directly on your DGX Spark after. Many thanks to [Lakshmi Ramesharrow-up-right](https://www.linkedin.com/in/rlakshmi24/) and [Barath Anandanarrow-up-right](https://www.linkedin.com/in/barathsa/) from NVIDIA for helping Unsloth’s DGX Spark launch and building the Docker image. ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#unified-memory-usage) Unified Memory Usage gpt-oss-120b QLoRA 4-bit fine-tuning will use around **68GB** of unified memory. How your unified memory usage should look **before** (left) and **after** (right) training: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e079a9aa8d853b319520fe0f0fbcca2e85b31ea6%252Fdgx7.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=bd11c1ff&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-c389a73a48ad059bbb92121b328fa7ccc61bee95%252Fdgx8.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b3e1e4e9&sv=2) And that's it! Have fun training and running LLMs completely locally on your NVIDIA DGX Spark! ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#video-tutorials) Video Tutorials Thanks to Tim from [AnythingLLMarrow-up-right](https://github.com/Mintplex-Labs/anything-llm) for providing a great fine-tuning tutorial with Unsloth on DGX Spark: [PreviousUnsloth Docker Guidechevron-left](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker) [NextBlackwell, RTX 50 and Unslothchevron-right](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) Last updated 3 months ago Was this helpful? * [⚡ Step-by-Step Tutorial](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#step-by-step-tutorial) * [Unified Memory Usage](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#unified-memory-usage) * [Video Tutorials](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth#video-tutorials) Was this helpful? sun-brightdesktopmoon Copy sudo apt update && sudo apt install -y wget wget -O Dockerfile "https://raw.githubusercontent.com/unslothai/notebooks/main/Dockerfile_DGX_Spark" Copy docker build -f Dockerfile -t unsloth-dgx-spark . Copy FROM nvcr.io/nvidia/pytorch:25.09-py3 # Set CUDA environment variables ENV CUDA_HOME=/usr/local/cuda-13.0/ ENV CUDA_PATH=$CUDA_HOME ENV PATH=$CUDA_HOME/bin:$PATH ENV LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH ENV C_INCLUDE_PATH=$CUDA_HOME/include:$C_INCLUDE_PATH ENV CPLUS_INCLUDE_PATH=$CUDA_HOME/include:$CPLUS_INCLUDE_PATH # Install triton from source for latest blackwell support RUN git clone https://github.com/triton-lang/triton.git && \ cd triton && \ git checkout c5d671f91d90f40900027382f98b17a3e04045f6 && \ pip install -r python/requirements.txt && \ pip install . && \ cd .. # Install xformers from source for blackwell support RUN git clone --depth=1 https://github.com/facebookresearch/xformers --recursive && \ cd xformers && \ export TORCH_CUDA_ARCH_LIST="12.1" && \ python setup.py install && \ cd .. # Install unsloth and other dependencies RUN pip install unsloth unsloth_zoo bitsandbytes==0.48.0 transformers==4.56.2 trl==0.22.2 # Launch the shell CMD ["/bin/bash"] Copy docker run -it \ --gpus=all \ --net=host \ --ipc=host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ -v $(pwd):$(pwd) \ -v $HOME/.cache/huggingface:/root/.cache/huggingface \ -w $(pwd) \ unsloth-dgx-spark Copy NOTEBOOK_URL="https://raw.githubusercontent.com/unslothai/notebooks/refs/heads/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_DGX_Spark.ipynb" wget -O "gpt_oss_20B_RL_2048_Game.ipynb" "$NOTEBOOK_URL" jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root sun-brightdesktopmoon --- # Fine-Tuning LLMs on NVIDIA DGX Station with Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close You can now train LLMs locally on your NVIDIA DGX Station with [Unslotharrow-up-right](https://github.com/unslothai/unsloth) . DGX Station has more than **~200GB VRAM** and over **700GB of unified GPU / CPU memory** and combines a Grace CPU and a Blackwell GPU in a tightly connected system designed for large-scale AI workloads. Linked by NVLink-C2C, the CPU and GPU remain distinct but work together far more efficiently than in a traditional CPU-GPU setup. In this guide, we’ll use Unsloth notebooks train [Qwen3.5](https://unsloth.ai/docs/blog/dgx-station#qwen3.5-35b-a3b-fine-tuning) and [gpt-oss-120b](https://unsloth.ai/docs/blog/dgx-station#gpt-oss-120b-fine-tuning) on DGX Station. Thank you to NVIDIA for providing some early access DGX Station hardware to test Unsloth on! ### [hashtag](https://unsloth.ai/docs/blog/dgx-station#quickstart) Quickstart You will need `python3` installed and in particular the dev headers are needed. On our system we have `python 3.12` so we will install the 3.12 dev headers. Copy sudo apt update sudo apt install python3.12-dev Then create a fresh virtual environment to install [Unslotharrow-up-right](https://github.com/unslothai/unsloth) . This way we minimize dependency conflicts and preserve the state of the current working environment. Copy python3 -m venv .unsloth source .unsloth/bin/activate pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130 circle-exclamation First install `torch` from the `cuda 13` index otherwise we could get the CPU version or a mismatch in architecture and capabilities! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fw04Su0JZriUaQxD31wf0%252Funknown.png%3Falt%3Dmedia%26token%3D83e61cdb-74c3-42c4-a1ff-18cec3752c9e&width=768&dpr=3&quality=100&sign=73a9ac77&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9bs6h6YxI2hqnqOz1bU0%252Funknown.png%3Falt%3Dmedia%26token%3De3e261b5-be18-4d49-9f38-526012add332&width=768&dpr=3&quality=100&sign=4bd7cb89&sv=2) Now we can install Unsloth: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhQZznPQ8O9Wh3At6FclO%252Funknown.png%3Falt%3Dmedia%26token%3D34c8de6e-bef8-414c-8e1b-2913589c4b10&width=768&dpr=3&quality=100&sign=4f1bf758&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdLZCFmln5LaUWtO6eC4A%252Funknown.png%3Falt%3Dmedia%26token%3Dce04e025-32c7-4847-ac35-bee1baf6259f&width=768&dpr=3&quality=100&sign=ccb6bc91&sv=2) Now lets install `xformers` and optionally build `flash-attention` from source. Both packages take time so please be patient while they build. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FnyczIn3YvXAPx5oIfZQQ%252Funknown.png%3Falt%3Dmedia%26token%3D1a2c5f7b-13c5-4f5e-b4c4-61df8d5fc653&width=768&dpr=3&quality=100&sign=7857435d&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FoupUFzx2pOG6l5B91Pw4%252Funknown.png%3Falt%3Dmedia%26token%3D009d2c73-5992-4593-8fd0-e7d813eda3ff&width=768&dpr=3&quality=100&sign=fb07e33f&sv=2) For Qwen 3.5 MoE we’ll want to download two kernel packages `flash-linear-attention` and `causal-conv1d` to make it fast. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4xEY8k3jzxfOgMWAgJD7%252Funknown.png%3Falt%3Dmedia%26token%3D2b8bd62e-23cd-4bcf-a0af-6d161d1ec1a1&width=768&dpr=3&quality=100&sign=be483f7a&sv=2) If you don’t already have a notebook client, install one. For this guide we will use Jupyter Notebook: Finally we download the actual Unsloth notebooks to run. There are 250+ notebooks for LLM Training as well as Python scripts. ### [hashtag](https://unsloth.ai/docs/blog/dgx-station#training-tutorials) Training Tutorials Now we can launch Jupyter Notebook and navigate to the UI on a browser. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FP2seywdWvLHHQkdP8DGy%252Funknown.png%3Falt%3Dmedia%26token%3Dca1b5390-5eb8-416b-a3e9-d9df9b27fb0b&width=768&dpr=3&quality=100&sign=fd1facf4&sv=2) Copy and paste the `localhost` site with token parameter and paste into your browser. You should see something like: The `nb` folder has all the notebooks to run. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FSxN976oDM4WaG5EtpSc9%252Funknown.png%3Falt%3Dmedia%26token%3D7113ba12-5bcc-4bc6-9777-b9d4c440d0bf&width=768&dpr=3&quality=100&sign=181360ba&sv=2) #### [hashtag](https://unsloth.ai/docs/blog/dgx-station#qwen3.5-35b-a3b-training) Qwen3.5-35B-A3B Training Open the file `nb/Qwen3_5_MoE.ipynb`. Skip past the installation section since we already installed everything we need beforehand. Navigate to the Unsloth section and start executing cells from there. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fif8mAvc1au9Hl83IZzNm%252FDGX%2520Station.png%3Falt%3Dmedia%26token%3D1011c8a9-c6ba-48df-a726-d3bc3bc8e947&width=768&dpr=3&quality=100&sign=8809e641&sv=2) The notebook covers model setup, dataset preparation, and trainer configuration. Each step can take some time as we are downloading a very large model, initializing billions of weights, and further optimizing to make it run fast. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4v0NmHdhiYCHFll8U8OD%252Funknown.png%3Falt%3Dmedia%26token%3D69e3d279-4d59-4439-802f-11bd02fe39d3&width=768&dpr=3&quality=100&sign=2d91fe9c&sv=2) Training is very fast with the default setting. On the DGX Station there is plenty of memory so you can play with the default training hyper parameters to really push the memory and compute. Once done training you can save the model for later, push the model to Hugging Face Hub to share with others, or export to a quantized format. #### [hashtag](https://unsloth.ai/docs/blog/dgx-station#gpt-oss-120b-training) gpt-oss-120b Training Open the file `nb/gpt-oss-(120B)_A100-Fine-tuning.ipynb`. Skip past the installation section since we already installed the prerequisites and navigate to the Unsloth section. We can start running the notebook from there. The notebook will use around 72 GB of GPU memory and take about 10 minutes. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8jYYievlemxDJBatevNV%252FDGX%2520Station%25202.png%3Falt%3Dmedia%26token%3Defef1a26-a170-4690-972f-1a7cde67e9ea&width=768&dpr=3&quality=100&sign=963c9c14&sv=2) Each cell can take some time to run as we need to download the model, initialize the weights, and further optimize for a fast experience. The notebook goes through dataset preprocessing and trainer setup. Once we get to the `trainer.train()` cell and execute training begins. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FOxuma3ZZeEbxZrAWIgnq%252FDGX%2520Station%25203.png%3Falt%3Dmedia%26token%3D17beb84e-eb56-4357-aee2-078c4db3eb84&width=768&dpr=3&quality=100&sign=d0a8d479&sv=2) Now that it’s complete we can save the model for later use, push to Hugging Face Hub to share with the world, or export it to GGUF format. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fy1UxtQ01avFK5BIkofwt%252Fimage.png%3Falt%3Dmedia%26token%3D8d137818-a3a6-4d00-a9fd-1e41ed0483a5&width=768&dpr=3&quality=100&sign=76c758e8&sv=2) Read more about NVIDIA's DGX Station at [https://www.nvidia.com/en-us/products/workstations/dgx-station/arrow-up-right](https://www.nvidia.com/en-us/products/workstations/dgx-station/) [PreviousQuantization-Aware Trainingchevron-left](https://unsloth.ai/docs/blog/quantization-aware-training-qat) [NextUnsloth Docker Guidechevron-right](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker) Last updated 1 month ago Was this helpful? * [Quickstart](https://unsloth.ai/docs/blog/dgx-station#quickstart) * [Training Tutorials](https://unsloth.ai/docs/blog/dgx-station#training-tutorials) Was this helpful? sun-brightdesktopmoon Copy pip install unsloth Copy pip install --no-deps --no-build-isolation xformers==0.0.33.post1 # Optionally flash-attn # Clone and build (targets sm_100 for B300) git clone https://github.com/Dao-AILab/flash-attention cd flash-attention # B300 = sm_100, set arch explicitly TORCH_CUDA_ARCH_LIST="10.0" MAX_JOBS=8 pip install . --no-build-isolation cd .. Copy pip install --no-build-isolation flash-linear-attention causal_conv1d==1.6.0 Copy cd .. pip install notebook pip install ipywidgets Copy git clone https://github.com/unslothai/notebooks.git cd notebooks Copy jupyter notebook sun-brightdesktopmoon --- # Multi-GPU Fine-tuning with Distributed Data Parallel (DDP) | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Let’s assume we have multiple GPUs, and we want to fine-tune a model using all of them! To do so, the most straightforward strategy is to use Distributed Data Parallel (DDP), which creates one copy of the model on each GPU device, feeding each copy distinct samples from the dataset during training and aggregating their contributions to weight updates per optimizer step. Why would we want to do this? Well, as we add more GPUs into the training process, we scale the number of samples our models train on per step, making each gradient update more stable and increasing our training throughput dramatically with each added GPU. Here’s a step-by-step guide on how to do this using Unsloth’s command-line interface (CLI)! **Note:** Unsloth DDP will work with any of your training scripts, not just via our CLI! More details below. #### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#install-unsloth-from-source) Install Unsloth from source We’ll clone Unsloth from GitHub and install it. Please consider using a [virtual environmentarrow-up-right](https://docs.python.org/3/tutorial/venv.html) ; we like to use `uv venv –python 3.12 && source .venv/bin/activate`, but any virtual environment creation tooling will do. Copy git clone https://github.com/unslothai/unsloth.git cd unsloth pip install . #### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#choose-target-model-and-dataset-for-finetuning) Choose target model and dataset for finetuning In this demo, we will fine-tune [Qwen/Qwen3-8Barrow-up-right](https://huggingface.co/Qwen/Qwen3-8B) on the [yahma/alpaca-cleanedarrow-up-right](https://huggingface.co/datasets/yahma/alpaca-cleaned) chat dataset. This is a Supervised Fine-Tuning (SFT) workload that is commonly used when attempting to adapt a base model to a desired conversational style, or improve the model’s performance on a downstream task. ### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#use-the-unsloth-cli) Use the Unsloth CLI! First, let’s take a look at the help message built-in to the CLI (we’ve abbreviated here with “...” in various places for brevity): This should give you a sense of what options are available for you to pass into the CLI for training your model! For multi-GPU training (DDP in this case), we will use the [torchrunarrow-up-right](https://docs.pytorch.org/docs/stable/elastic/run.html) launcher, which allows you to spin up multiple distributed training processes in single-node or multi-node settings. In our case, we will focus on the single-node (i.e., one machine) case with two H100 GPUs. Let’s also check our GPUs’ status by using the `nvidia-smi` command-line tool: Great! We have two H100 GPUs, as expected. Both are sitting at 0MiB memory usage as we’re currently not training anything, or have any model loaded into memory. To start your training run, issue a command like the following: If you have more GPUs, you may set `--nproc_per_node` accordingly to utilize them. **Note:** You can use the `torchrun` launcher with any of your Unsloth training scripts, including the [scriptsarrow-up-right](https://github.com/unslothai/notebooks/tree/main/python_scripts) converted from our free Colab notebooks, and DDP will be auto-enabled when training with >1 GPU! Taking a look again at `nvidia-smi` while training is in-flight: We can see that both GPUs are now using ~19GB of VRAM per H100 GPU! Inspecting the training logs, we see that we’re able to train at a rate of ~1.1 iterations/s. This training speed is ~constant even as we add more GPUs, so our training throughput increases ~linearly with the number of GPUs! ### [hashtag](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#training-metrics) Training metrics We ran a few short rank-16 LoRA fine-tunes on [unsloth/Llama-3.2-1B-Instructarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct) on the [yahma/alpaca-cleanedarrow-up-right](https://huggingface.co/datasets/yahma/alpaca-cleaned) dataset to demonstrate the improved training throughput when using DDP training with multiple GPUs. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdySJnhNUzVD3gsWmPqHR%252Funknown.png%3Falt%3Dmedia%26token%3D9905cccb-04c8-45b1-bfb1-680823713319&width=768&dpr=3&quality=100&sign=94c96e05&sv=2) The above figure compares training loss between two Llama-3.2-1B-Instruct LoRA fine-tunes over 500 training steps, with single GPU training (pink) vs. multi-GPU DDP training (blue). Notice that the loss curves match in scale and trend, but otherwise are a _bit_ different, since _the multi-GPU training processes twice as much training data per step_. This results in a slightly different training curve with less variability on a step-by-step basis. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fz4XgknzMgljaFInMEzHc%252Funknown.png%3Falt%3Dmedia%26token%3D4e28e2b1-8bc8-4049-983d-e4f980f3f4cf&width=768&dpr=3&quality=100&sign=caff1467&sv=2) The above figure plots training progress for the same two fine-tunes. Notice that the multi-GPU DDP training progresses through an epoch of the training data in half as many steps as single GPU training. This is because each GPU can process a distinct batch (of size `per_device_train_batch_size`) per step. However, the per-step timing for DDP training is slightly slower due to distributed communication for the model weight updates. As you increase the number of GPUs, the training throughput will continue to increase ~linearly (but with a small, but increasing penalty for the distributed comms). These same loss and training epoch progress behaviors hold for QLoRA fine-tunes, in which we loaded the base models in 4-bit precision in order to save additional GPU memory. This is particularly useful for training large models on limited amounts of GPU VRAM: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FUrCEgA7OBVhc8ICkMaP6%252Funknown.png%3Falt%3Dmedia%26token%3D0f5de3df-77df-4ee5-bf7a-68dead857c9a&width=768&dpr=3&quality=100&sign=52789357&sv=2) Training loss comparison between two Llama-3.2-1B-Instruct QLoRA fine-tunes over 500 training steps, with single GPU training (orange) vs. multi-GPU DDP training (purple). ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8cG6rjmjeznNfgWrYdnG%252Funknown.png%3Falt%3Dmedia%26token%3Dd1c2c1fe-c117-49b5-8e9d-fdc01154cc01&width=768&dpr=3&quality=100&sign=9053ad9b&sv=2) Training progress comparison for the same two fine-tunes. [PreviousMulti-GPU Training Unslothchevron-left](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) [NextEmbedding Fine-tuningchevron-right](https://unsloth.ai/docs/basics/embedding-finetuning) Last updated 3 months ago Was this helpful? * [Use the Unsloth CLI!](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#use-the-unsloth-cli) * [Training metrics](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp#training-metrics) Was this helpful? sun-brightdesktopmoon Copy $ python unsloth-cli.py --help usage: unsloth-cli.py [-h] [--model_name MODEL_NAME] [--max_seq_length MAX_SEQ_LENGTH] [--dtype DTYPE] [--load_in_4bit] [--dataset DATASET] [--r R] [--lora_alpha LORA_ALPHA] [--lora_dropout LORA_DROPOUT] [--bias BIAS] [--use_gradient_checkpointing USE_GRADIENT_CHECKPOINTING] … 🦥 Fine-tune your llm faster using unsloth! options: -h, --help show this help message and exit 🤖 Model Options: --model_name MODEL_NAME Model name to load --max_seq_length MAX_SEQ_LENGTH Maximum sequence length, default is 2048. We auto support RoPE Scaling internally! … 🧠 LoRA Options: These options are used to configure the LoRA model. --r R Rank for Lora model, default is 16. (common values: 8, 16, 32, 64, 128) --lora_alpha LORA_ALPHA LoRA alpha parameter, default is 16. (common values: 8, 16, 32, 64, 128) … 🎓 Training Options: --per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE Batch size per device during training, default is 2. --per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE Batch size per device during evaluation, default is 4. --gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS Number of gradient accumulation steps, default is 4. … chevron-downShow all 36 lines Copy $ nvidia-smi Mon Nov 24 12:53:00 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 80GB HBM3 On | 00000000:04:00.0 Off | 0 | | N/A 32C P0 69W / 700W | 0MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA H100 80GB HBM3 On | 00000000:05:00.0 Off | 0 | | N/A 30C P0 68W / 700W | 0MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ chevron-downShow all 25 lines Copy # required: # --model_name # --dataset # optional; experiment with these: # --learning_rate, --max_seq_length, --per_device_train_batch_size, --gradient_accumulation_steps, --max_steps # to save the model at the end of training: # --save_model torchrun --nproc_per_node=2 unsloth-cli.py \ --model_name=Qwen/Qwen3-8B \ --dataset=yahma/alpaca-cleaned \ --learning_rate=2e-5 \ --max_seq_length=2048 \ --per_device_train_batch_size=1 \ --gradient_accumulation_steps=4 \ --max_steps=1000 \ --save_model chevron-downShow all 17 lines Copy $ nvidia-smi Mon Nov 24 12:58:42 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 80GB HBM3 On | 00000000:04:00.0 Off | 0 | | N/A 38C P0 193W / 700W | 18903MiB / 81559MiB | 25% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA H100 80GB HBM3 On | 00000000:05:00.0 Off | 0 | | N/A 37C P0 199W / 700W | 18905MiB / 81559MiB | 28% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 4935 C ...und/unsloth/.venv/bin/python3 18256MiB | | 0 N/A N/A 4936 C ...und/unsloth/.venv/bin/python3 630MiB | | 1 N/A N/A 4935 C ...und/unsloth/.venv/bin/python3 630MiB | | 1 N/A N/A 4936 C ...und/unsloth/.venv/bin/python3 18258MiB | +-----------------------------------------------------------------------------------------+ chevron-downShow all 28 lines sun-brightdesktopmoon --- # Fine-tuning LLMs with Blackwell, RTX 50 series & Unsloth | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth now supports NVIDIA’s Blackwell architecture GPUs, including RTX 50-series GPUs (5060–5090), RTX PRO 6000, and GPUS such as B200, B40, GB100, GB102 and more! You can read the official [NVIDIA blogpost herearrow-up-right](https://developer.nvidia.com/blog/train-an-llm-on-an-nvidia-blackwell-desktop-with-unsloth-and-scale-it/) . Unsloth is now compatible with every NVIDIA GPU from 2018+ including the [DGX Spark](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) . > **Our new** [**Docker image**](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) > **supports Blackwell. Run the Docker image and start training!** [**Guide**](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pip-install) Pip install Simply install Unsloth: Copy pip install unsloth If you see issues, another option is to create a separate isolated environment: Copy python -m venv unsloth source unsloth/bin/activate pip install unsloth Note it might be `pip3` or `pip3.13` and also `python3` or `python3.13` You might encounter some Xformers issues, in which cause you should build from source: Copy # First uninstall xformers installed by previous libraries pip uninstall xformers -y # Clone and build pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) Docker [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) is Unsloth's only Docker image. For Blackwell and 50-series GPUs, use this same image - no separate image needed. For installation instructions, please follow our [Unsloth Docker guide](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker) . ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) uv #### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv-advanced) uv (Advanced) The installation order is important, since we want the overwrite bundled dependencies with specific versions (namely, `xformers` and `triton`). 1. I prefer to use `uv` over `pip` as it's faster and better for resolving dependencies, especially for libraries which depend on `torch` but for which a specific `CUDA` version is required per this scenario. Install `uv` Create a project dir and venv: 2. Install `vllm` Note that we have to specify `cu128`, otherwise `vllm` will install `torch==2.7.0` but with `cu126`. 3. Install `unsloth` dependencies If you notice weird resolving issues due to Xformers, you can also install Unsloth from source without Xformers: 4. Download and build `xformers` (Optional) Xformers is optional, but it is definitely faster and uses less memory. We'll use PyTorch's native SDPA if you do not want Xformers. Building Xformers from source might be slow, so beware! Note that we have to explicitly set `TORCH_CUDA_ARCH_LIST=12.0`. 5. `transformers` Install any transformers version, but best to get the latest. ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#conda-or-mamba-advanced) Conda or mamba (Advanced) 1. Install `conda/mamba` Run the installation script Create a conda or mamba environment Activate newly created environment 2. Install `vllm` Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Note that we have to specify `cu128`, otherwise `vllm` will install `torch==2.7.0` but with `cu126`. 3. Install `unsloth` dependencies Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` 4. Download and build `xformers` (Optional) Xformers is optional, but it is definitely faster and uses less memory. We'll use PyTorch's native SDPA if you do not want Xformers. Building Xformers from source might be slow, so beware! You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Note that we have to explicitly set `TORCH_CUDA_ARCH_LIST=12.0`. 5. Update `triton` Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` `triton>=3.3.1` is required for `Blackwell` support. 6. `Transformers` Install any transformers version, but best to get the latest. If you are using mamba as your package just replace conda with mamba for all commands shown above. ### [hashtag](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wsl-specific-notes) WSL-Specific Notes If you're using WSL (Windows Subsystem for Linux) and encounter issues during xformers compilation (reminder Xformers is optional, but faster for training) follow these additional steps: 1. **Increase WSL Memory Limit** Create or edit the WSL configuration file: After making these changes, restart WSL: 2. **Install xformers** Use the following command to install xformers with optimized compilation for WSL: The `--no-build-isolation` flag helps avoid potential build issues in WSL environments. [PreviousDGX Spark and Unslothchevron-left](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) Last updated 2 months ago Was this helpful? * [Pip install](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#pip-install) * [Docker](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#docker) * [uv](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#uv) * [Conda or mamba (Advanced)](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#conda-or-mamba-advanced) * [WSL-Specific Notes](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth#wsl-specific-notes) Was this helpful? sun-brightdesktopmoon Copy uv pip install unsloth Copy curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env Copy mkdir 'unsloth-blackwell' && cd 'unsloth-blackwell' uv venv .venv --python=3.12 --seed source .venv/bin/activate Copy uv pip install -U vllm --torch-backend=cu128 Copy uv pip install unsloth unsloth_zoo bitsandbytes Copy uv pip install -qqq \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" Copy # First uninstall xformers installed by previous libraries pip uninstall xformers -y # Clone and build pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. Copy uv pip install -U transformers Copy curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Copy bash Miniforge3-$(uname)-$(uname -m).sh Copy conda create --name unsloth-blackwell python==3.12 -y Copy conda activate unsloth-blackwell Copy pip install -U vllm --extra-index-url https://download.pytorch.org/whl/cu128 Copy pip install unsloth unsloth_zoo bitsandbytes Copy # First uninstall xformers installed by previous libraries pip uninstall xformers -y # Clone and build pip install ninja export TORCH_CUDA_ARCH_LIST="12.0" git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers && python setup.py install && cd .. Copy pip install -U triton>=3.3.1 Copy uv pip install -U transformers Copy # Create or edit .wslconfig in your Windows user directory # (typically C:\Users\YourUsername\.wslconfig) # Add these lines to the file [wsl2] memory=16GB # Minimum 16GB recommended for xformers compilation processors=4 # Adjust based on your CPU cores swap=2GB localhostForwarding=true Copy wsl --shutdown Copy # Set CUDA architecture for Blackwell GPUs export TORCH_CUDA_ARCH_LIST="12.0" # Install xformers from source with optimized build flags pip install -v --no-build-isolation -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers sun-brightdesktopmoon --- # Unsloth Model Catalog | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Unsloth LLMs directory for all our [Dynamicarrow-up-right](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) GGUF, 4-bit, 16-bit models on Hugging Face. • GGUF + 4-bit • Instruct 16-bit • Base 4 & 16-bit • FP8 [Qwen](https://unsloth.ai/docs/get-started/unsloth-model-catalog#qwen-models) [DeepSeek](https://unsloth.ai/docs/get-started/unsloth-model-catalog#deepseek-models) [Gemma](https://unsloth.ai/docs/get-started/unsloth-model-catalog#gemma-models) [Llama](https://unsloth.ai/docs/get-started/unsloth-model-catalog#llama-models) [Mistral](https://unsloth.ai/docs/get-started/unsloth-model-catalog#mistral-models) [GLM](https://unsloth.ai/docs/get-started/unsloth-model-catalog#glm-models) **GGUFs** let you run models in tools like [**Unsloth Studio**](https://unsloth.ai/docs/new/studio) ✨, Ollama and llama.cpp. **Instruct (4-bit)** safetensors can be used for inference or fine-tuning via Unsloth. #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#new-and-recommended-models) **New & recommended models:** Model Variant GGUF Instruct (4-bit) [**Gemma 4**](https://unsloth.ai/docs/models/gemma-4) 26B-A4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) — 31B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-31B-it-unsloth-bnb-4bit) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E4B-it-unsloth-bnb-4bit) E2B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E2B-it-unsloth-bnb-4bit) [**Qwen3.5**arrow-up-right](https://github.com/unslothai/docs/blob/main/models/qwen3.5) 35B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF) — 27B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF) — 122B-A10B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF) — 0.8B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF) — 2B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) — 4B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF) — 9B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) — 397B-A17B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF) — **Qwen3** [Coder-Next](https://unsloth.ai/docs/models/qwen3-coder-next) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF) — NVIDIA Nemotron 3 [Super-120B-A12B](https://unsloth.ai/docs/models/nemotron-3/nemotron-3-super) [linkarrow-up-right](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) [Nano-4B](https://unsloth.ai/docs/models/nemotron-3) [linkarrow-up-right](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF) — **GLM** [4.7-Flash](https://unsloth.ai/docs/models/glm-4.7-flash) [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF) — [5](https://unsloth.ai/docs/models/tutorials/glm-5) [linkarrow-up-right](https://huggingface.co/unsloth/GLM-5-GGUF) — **Kimi** [K2.5](https://unsloth.ai/docs/models/kimi-k2.5) [linkarrow-up-right](https://huggingface.co/unsloth/Kimi-K2.5-GGUF) — [**gpt-oss**](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune) 120B [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-120b-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-120b-unsloth-bnb-4bit) 20B [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-20b-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-20b-unsloth-bnb-4bit) **MiniMax** [M2.5](https://unsloth.ai/docs/models/tutorials/minimax-m25) [linkarrow-up-right](https://huggingface.co/unsloth/MiniMax-M2.5-GGUF) — NVIDIA [Nemotron 3](https://unsloth.ai/docs/models/nemotron-3) 30B [linkarrow-up-right](https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF) — [**Qwen-Image**](https://unsloth.ai/docs/models/tutorials/qwen-image-2512) 2512 [linkarrow-up-right](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF) — Edit-2511 [linkarrow-up-right](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) — [**Ministral 3**](https://unsloth.ai/docs/models/tutorials/ministral-3) 3B [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-GGUF) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-3B-Reasoning-2512-GGUF) [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-unsloth-bnb-4bit) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-3B-Reasoning-2512-GGUF) 8B [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-GGUF) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-8B-Reasoning-2512-GGUF) [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-unsloth-bnb-4bit) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-8B-Reasoning-2512-unsloth-bnb-4bit) 14B [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-GGUF) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-14B-Reasoning-2512-GGUF) [Instructarrow-up-right](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-unsloth-bnb-4bit) • [Reasoningarrow-up-right](https://huggingface.co/unsloth/Ministral-3-14B-Reasoning-2512-unsloth-bnb-4bit) [**Devstral 2**](https://unsloth.ai/docs/models/tutorials/devstral-2) 24B [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF) — 123B [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF) — **Mistral Large 3** 675B [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Large-3-675B-Instruct-2512-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Large-3-675B-Instruct-2512-NVFP4) [**Qwen3-Next**](https://unsloth.ai/docs/models/tutorials/qwen3-next) 80B-A3B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-bnb-4bit/) 80B-A3B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF) — [**Qwen3-VL**](https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) 2B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit) 2B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-unsloth-bnb-4bit) 4B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit) 4B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-unsloth-bnb-4bit) 8B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit) 8B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-unsloth-bnb-4bit) 30B-A3B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Instruct-GGUF) — 30B-A3B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Thinking-GGUF) — 32B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-unsloth-bnb-4bit) 32B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-unsloth-bnb-4bit) 235B-A22B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF) — 235B-A22B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Thinking-GGUF) — [**Qwen3-2507**](https://unsloth.ai/docs/models/tutorials/qwen3-next) 30B-A3B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF) — 30B-A3B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF) — 235B-A22B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/) — [**Qwen3-Coder**](https://unsloth.ai/docs/models/tutorials/qwen3-coder-how-to-run-locally) 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) — [**GLM**](https://unsloth.ai/docs/models/tutorials/glm-4.6-how-to-run-locally) 4.7 [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.7-GGUF) — 4.6V-Flash [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF) — [**DeepSeek-V3.1**](https://unsloth.ai/docs/models/tutorials/deepseek-v3.1-how-to-run-locally) Terminus [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3.1-Terminus-GGUF) — V3.1 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF) — **Granite-4.0** H-Small [linkarrow-up-right](https://huggingface.co/unsloth/granite-4.0-h-small-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/granite-4.0-h-small-unsloth-bnb-4bit) **Kimi-K2** Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF) — 0905 [linkarrow-up-right](https://huggingface.co/unsloth/Kimi-K2-Instruct-0905-GGUF) — #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#deepseek-models) **DeepSeek models:** Model Variant GGUF Instruct (4-bit) **DeepSeek-V3.1** Terminus [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3.1-Terminus-GGUF) V3.1 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF) **DeepSeek-V3** V3-0324 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF) — V3 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3-GGUF) — **DeepSeek-R1** R1-0528 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF) — R1-0528-Qwen3-8B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-unsloth-bnb-4bit) R1 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-GGUF) — R1 Zero [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Zero-GGUF) — Distill Llama 3 8 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit) Distill Llama 3.3 70 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-bnb-4bit) Distill Qwen 2.5 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-unsloth-bnb-4bit) Distill Qwen 2.5 7 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-7B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-7B-unsloth-bnb-4bit) Distill Qwen 2.5 14 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit) Distill Qwen 2.5 32 B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#llama-models) **Llama models:** Model Variant GGUF Instruct (4-bit) **Llama 4** Scout 17 B-16 E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit) Maverick 17 B-128 E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF) — **Llama 3.3** 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-bnb-4bit) **Llama 3.2** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-bnb-4bit) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) 11 B Vision — [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-11B-Vision-Instruct-unsloth-bnb-4bit) 90 B Vision — [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-90B-Vision-Instruct-bnb-4bit) **Llama 3.1** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit) 70 B — [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-70B-Instruct-bnb-4bit) 405 B — [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-405B-Instruct-bnb-4bit) **Llama 3** 8 B — [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit) 70 B — [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-70b-bnb-4bit) **Llama 2** 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b-chat-bnb-4bit) 13 B — [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-13b-bnb-4bit) **CodeLlama** 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/codellama-7b-bnb-4bit) 13 B — [linkarrow-up-right](https://huggingface.co/unsloth/codellama-13b-bnb-4bit) 34 B — [linkarrow-up-right](https://huggingface.co/unsloth/codellama-34b-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#gemma-models) **Gemma models:** Model Variant GGUF Instruct (4-bit) **Gemma 4** E2B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E2B-it-unsloth-bnb-4bit) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-E4B-it-unsloth-bnb-4bit) 26B-A4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) — 31B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-4-31B-it-unsloth-bnb-4bit) **FunctionGemma** 270M [linkarrow-up-right](https://huggingface.co/unsloth/functiongemma-270m-it-GGUF) — **Gemma 3n** E2B ​[linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B-it-unsloth-bnb-4bit) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit) **Gemma 3** 270M [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-270m-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-270m-it) 1 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-it-unsloth-bnb-4bit) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-it-unsloth-bnb-4bit) 12 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-12b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-12b-it-unsloth-bnb-4bit) 27 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-it-unsloth-bnb-4bit) **MedGemma** 4 B (vision) [linkarrow-up-right](https://huggingface.co/unsloth/medgemma-4b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/medgemma-4b-it-unsloth-bnb-4bit) 27 B (vision) [linkarrow-up-right](https://huggingface.co/unsloth/medgemma-27b-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/medgemma-27b-text-it-unsloth-bnb-4bit) **Gemma 2** 2 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-it-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-2b-it-bnb-4bit) 9 B — [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-9b-it-bnb-4bit) 27 B — [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-27b-it-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#qwen-models) **Qwen models:** Model Variant GGUF Instruct (4-bit) [**Qwen3.5**arrow-up-right](https://github.com/unslothai/docs/blob/main/models/qwen3.5) 35B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF) — 27B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF) — 122B-A10B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF) — 0.8B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF) — 2B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) — 4B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF) — 9B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) — 397B-A17B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF) — **Qwen3** [Coder-Next](https://unsloth.ai/docs/models/qwen3-coder-next) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF) — [**Qwen-Image**](https://unsloth.ai/docs/models/tutorials/qwen-image-2512) 2512 [linkarrow-up-right](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF) — Edit-2511 [linkarrow-up-right](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) — [**Qwen3-VL**](https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) 2B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit) 2B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-unsloth-bnb-4bit) 4B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit) 4B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-unsloth-bnb-4bit) 8B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit) 8B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-unsloth-bnb-4bit) **Qwen3-Coder** 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) — 480B-A35B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF) — [**Qwen3-2507**](https://unsloth.ai/docs/models/tutorials/qwen3-next) 30B-A3B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF) — 30B-A3B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF) — 235B-A22B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF/) — 235B-A22B-Instruct [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/) — **Qwen 3** 0.6 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-unsloth-bnb-4bit) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-unsloth-bnb-4bit) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-unsloth-bnb-4bit) 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-unsloth-bnb-4bit) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-unsloth-bnb-4bit) 30 B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-bnb-4bit) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-32B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-32B-unsloth-bnb-4bit) 235 B-A22B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF) — **Qwen 2.5 Omni** 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-3B-GGUF) — 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-7B-GGUF) — **Qwen 2.5 VL** 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-unsloth-bnb-4bit) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-unsloth-bnb-4bit) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-32B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-32B-Instruct-unsloth-bnb-4bit) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct-unsloth-bnb-4bit) **Qwen 2.5** 0.5 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit) 1.5 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit) 3 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct-bnb-4bit) 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-7B-Instruct-bnb-4bit) 14 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-14B-Instruct-bnb-4bit) 32 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-32B-Instruct-bnb-4bit) 72 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-72B-Instruct-bnb-4bit) **Qwen 2.5 Coder (128 K)** 0.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-0.5B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-0.5B-Instruct-bnb-4bit) 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-3B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-14B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-14B-Instruct-bnb-4bit) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-bnb-4bit) **QwQ** 32 B [linkarrow-up-right](https://huggingface.co/unsloth/QwQ-32B-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit) **QVQ (preview)** 72 B — [linkarrow-up-right](https://huggingface.co/unsloth/QVQ-72B-Preview-bnb-4bit) **Qwen 2 (chat)** 1.5 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-1.5B-Instruct-bnb-4bit) 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-7B-Instruct-bnb-4bit) 72 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-72B-Instruct-bnb-4bit) **Qwen 2 VL** 2 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-2B-Instruct-unsloth-bnb-4bit) 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit) 72 B — [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-72B-Instruct-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#glm-models) **GLM models:** Model Variant GGUF Instruct (4-bit) **GLM** [4.7-Flash](https://unsloth.ai/docs/models/glm-4.7-flash) [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF) — [5](https://unsloth.ai/docs/models/tutorials/glm-5) [linkarrow-up-right](https://huggingface.co/unsloth/GLM-5-GGUF) — 4.6V-Flash [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF) — 4.6 [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.6-GGUF) — 4.5-Air [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.5-Air-GGUF) — #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#mistral-models) **Mistral models:** Model Variant GGUF Instruct (4-bit) **Magistral** Small (2506) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2506-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2506-unsloth-bnb-4bit) Small (2509) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2509-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2509-unsloth-bnb-4bit) Small (2507) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2507-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2507-unsloth-bnb-4bit) **Mistral Small** 3.2-24 B (2506) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-unsloth-bnb-4bit) 3.1-24 B (2503) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-unsloth-bnb-4bit) 3-24 B (2501) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-24B-Instruct-2501-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-24B-Instruct-2501-unsloth-bnb-4bit) 2409-22 B — [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-Instruct-2409-bnb-4bit) **Devstral** Small-24 B (2507) [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2507-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2507-unsloth-bnb-4bit) Small-24 B (2505) [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2505-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2505-unsloth-bnb-4bit) **Pixtral** 12 B (2409) — [linkarrow-up-right](https://huggingface.co/unsloth/Pixtral-12B-2409-bnb-4bit) **Mistral NeMo** 12 B (2407) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit) **Mistral Large** 2407 — [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Large-Instruct-2407-bnb-4bit) **Mistral 7 B** v0.3 — [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-instruct-v0.3-bnb-4bit) v0.2 — [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-instruct-v0.2-bnb-4bit) **Mixtral** 8 × 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/Mixtral-8x7B-Instruct-v0.1-unsloth-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#phi-models) **Phi models:** Model Variant GGUF Instruct (4-bit) **Phi-4** Reasoning-plus [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning-plus-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning-plus-unsloth-bnb-4bit) Reasoning [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/phi-4-reasoning-unsloth-bnb-4bit) Mini-Reasoning [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini-reasoning-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini-reasoning-unsloth-bnb-4bit) Phi-4 (instruct) [linkarrow-up-right](https://huggingface.co/unsloth/phi-4-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/phi-4-unsloth-bnb-4bit) mini (instruct) [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini-instruct-unsloth-bnb-4bit) **Phi-3.5** mini — [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3.5-mini-instruct-bnb-4bit) **Phi-3** mini — [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3-mini-4k-instruct-bnb-4bit) medium — [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct-bnb-4bit) #### [hashtag](https://unsloth.ai/docs/get-started/unsloth-model-catalog#other-glm-orpheus-smol-llava-etc.-models) **Other (GLM, Orpheus, Smol, Llava etc.) models:** Model Variant GGUF Instruct (4-bit) GLM 4.5-Air [linkarrow-up-right](https://huggingface.co/unsloth/GLM-4.5-Air-GGUF) — 4.5 [4.5arrow-up-right](https://huggingface.co/unsloth/GLM-4.5-GGUF) — 4-32B-0414 [4-32B-0414arrow-up-right](https://huggingface.co/unsloth/GLM-4-32B-0414-GGUF) — **Grok 2** 270B [linkarrow-up-right](https://huggingface.co/unsloth/grok-2-GGUF) — **Baidu-ERNIE** 4.5-21B-A3B-Thinking [linkarrow-up-right](https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF) — Hunyuan A13B [linkarrow-up-right](https://huggingface.co/unsloth/Hunyuan-A13B-Instruct-GGUF) — Orpheus 0.1-ft (3B) [link](https://unsloth.ai/docs/) [linkarrow-up-right](https://huggingface.co/unsloth/orpheus-3b-0.1-ft-unsloth-bnb-4bit) **LLava** 1.5 (7 B) — [linkarrow-up-right](https://huggingface.co/unsloth/llava-1.5-7b-hf-bnb-4bit) 1.6 Mistral (7 B) — [linkarrow-up-right](https://huggingface.co/unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit) **TinyLlama** Chat — [linkarrow-up-right](https://huggingface.co/unsloth/tinyllama-chat-bnb-4bit) **SmolLM 2** 135 M [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-135M-Instruct-bnb-4bit) 360 M [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-360M-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-360M-Instruct-bnb-4bit) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-1.7B-Instruct-GGUF) [linkarrow-up-right](https://huggingface.co/unsloth/SmolLM2-1.7B-Instruct-bnb-4bit) **Zephyr-SFT** 7 B — [linkarrow-up-right](https://huggingface.co/unsloth/zephyr-sft-bnb-4bit) **Yi** 6 B (v1.5) — [linkarrow-up-right](https://huggingface.co/unsloth/Yi-1.5-6B-bnb-4bit) 6 B (v1.0) — [linkarrow-up-right](https://huggingface.co/unsloth/yi-6b-bnb-4bit) 34 B (chat) — [linkarrow-up-right](https://huggingface.co/unsloth/yi-34b-chat-bnb-4bit) 34 B (base) — [linkarrow-up-right](https://huggingface.co/unsloth/yi-34b-bnb-4bit) 16-bit and 8-bit Instruct models are used for inference or fine-tuning in [**Unsloth Studio**](https://unsloth.ai/docs/new/studio) : **New models:** Model Variant Instruct (16-bit) **gpt-oss** (new) 20b [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-20b) 120b [linkarrow-up-right](https://huggingface.co/unsloth/gpt-oss-120b) **Gemma 3n** E2B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B-it) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B-it) **DeepSeek-R1-0528** R1-0528-Qwen3-8B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B) R1-0528 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528) **Mistral** Small 3.2 24B (2506) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506) Small 3.1 24B (2503) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503) Small 3.0 24B (2501) [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-24B-Instruct-2501) Magistral Small (2506) [linkarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2506) **Qwen 3** 0.6 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B) 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B) 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-32B) 235B-A22B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B) **Llama 4** Scout 17B-16E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct) Maverick 17B-128E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct) **Qwen 2.5 Omni** 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-3B) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-7B) **Phi-4** Reasoning-plus [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning-plus) Reasoning [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning) **DeepSeek models** Model Variant Instruct (16-bit) **DeepSeek-V3** V3-0324 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3-0324) V3 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-V3) **DeepSeek-R1** R1-0528 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528) R1-0528-Qwen3-8B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B) R1 [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1) R1 Zero [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Zero) Distill Llama 3 8B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B) Distill Llama 3.3 70B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B) Distill Qwen 2.5 1.5B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B) Distill Qwen 2.5 7B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-7B) Distill Qwen 2.5 14B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14B) Distill Qwen 2.5 32B [linkarrow-up-right](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B) **Llama models** Family Variant Instruct (16-bit) **Llama 4** Scout 17B-16E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct) Maverick 17B-128E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct) **Llama 3.3** 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct) **Llama 3.2** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct) 11 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-11B-Vision-Instruct) 90 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-90B-Vision-Instruct) **Llama 3.1** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct) 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-70B-Instruct) 405 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-405B-Instruct) **Llama 3** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b-Instruct) 70 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-70b-Instruct) **Llama 2** 7 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b-chat) **Gemma models:** Model Variant Instruct (16-bit) **Gemma 3n** E2B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B-it) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B-it) **Gemma 3** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-it) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-it) 12 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-12b-it) 27 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-it) **Gemma 2** 2 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2b-it) 9 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-9b-it) 27 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-27b-it) **Qwen models:** Family Variant Instruct (16-bit) **Qwen 3** 0.6 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B) 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B) 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-32B) 235B-A22B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B) **Qwen 2.5 Omni** 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-3B) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Omni-7B) **Qwen 2.5 VL** 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-32B-Instruct) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct) **Qwen 2.5** 0.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-0.5B-Instruct) 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-1.5B-Instruct) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-7B-Instruct) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-14B-Instruct) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-32B-Instruct) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-72B-Instruct) **Qwen 2.5 Coder 128 K** 0.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-0.5B-Instruct-128K) 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct-128K) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-3B-Instruct-128K) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-128K) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-14B-Instruct-128K) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-128K) **QwQ** 32 B [linkarrow-up-right](https://huggingface.co/unsloth/QwQ-32B) **QVQ (preview)** 72 B — **Qwen 2 (Chat)** 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-1.5B-Instruct) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-7B-Instruct) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-72B-Instruct) **Qwen 2 VL** 2 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-2B-Instruct) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-7B-Instruct) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-VL-72B-Instruct) **Mistral models:** Model Variant Instruct (16-bit) **Mistral** Small 2409-22B [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-Instruct-2409) **Mistral** Large 2407 [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Large-Instruct-2407) **Mistral** 7B v0.3 [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-instruct-v0.3) **Mistral** 7B v0.2 [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-instruct-v0.2) **Pixtral** 12B 2409 [linkarrow-up-right](https://huggingface.co/unsloth/Pixtral-12B-2409) **Mixtral** 8×7B [linkarrow-up-right](https://huggingface.co/unsloth/Mixtral-8x7B-Instruct-v0.1) **Mistral NeMo** 12B 2407 [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407) **Devstral** Small 2505 [linkarrow-up-right](https://huggingface.co/unsloth/Devstral-Small-2505) **Phi models:** Model Variant Instruct (16-bit) **Phi-4** Reasoning-plus [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning-plus) Reasoning [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-reasoning) Phi-4 (core) [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4) Mini-Reasoning [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini-reasoning) Mini [linkarrow-up-right](https://huggingface.co/unsloth/Phi-4-mini) **Phi-3.5** Mini [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) **Phi-3** Mini [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3-mini-4k-instruct) Medium [linkarrow-up-right](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct) **Text-to-Speech (TTS) models:** Model Instruct (16-bit) Orpheus-3B (v0.1 ft) [linkarrow-up-right](https://huggingface.co/unsloth/orpheus-3b-0.1-ft) Orpheus-3B (v0.1 pt) [linkarrow-up-right](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained) Sesame-CSM 1B [linkarrow-up-right](https://huggingface.co/unsloth/csm-1b) Whisper Large V3 (STT) [linkarrow-up-right](https://huggingface.co/unsloth/whisper-large-v3) Llasa-TTS 1B [linkarrow-up-right](https://huggingface.co/unsloth/Llasa-1B) Spark-TTS 0.5B [linkarrow-up-right](https://huggingface.co/unsloth/Spark-TTS-0.5B) Oute-TTS 1B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-OuteTTS-1.0-1B) Base models are usually used for fine-tuning purposes: **New models:** Model Variant Base (16-bit) Base (4-bit) **Gemma 3n** E2B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E2B-unsloth-bnb-4bit) E4B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3n-E4B-unsloth-bnb-4bit) **Qwen 3** 0.6 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-Base-unsloth-bnb-4bit) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-Base-unsloth-bnb-4bit) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Base-unsloth-bnb-4bit) 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-Base-unsloth-bnb-4bit) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-Base-unsloth-bnb-4bit) 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Base-bnb-4bit) **Llama 4** Scout 17B 16E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E) [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit) Maverick 17B 128E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E) — **Llama models:** Model Variant Base (16-bit) Base (4-bit) **Llama 4** Scout 17B 16E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E) — Maverick 17B 128E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E) — **Llama 3.3** 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B) — **Llama 3.2** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B) — 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B) — 11 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-11B-Vision) — 90 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-90B-Vision) — **Llama 3.1** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-8B) — 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-70B) — **Llama 3** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b-bnb-4bit) **Llama 2** 7 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b-bnb-4bit) 13 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-13b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-13b-bnb-4bit) **Qwen models:** Model Variant Base (16-bit) Base (4-bit) **Qwen 3** 0.6 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-Base-unsloth-bnb-4bit) 1.7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-Base-unsloth-bnb-4bit) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Base-unsloth-bnb-4bit) 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-Base-unsloth-bnb-4bit) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-Base-unsloth-bnb-4bit) 30B-A3B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Base) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Base-unsloth-bnb-4bit) **Qwen 2.5** 0.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-0.5B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-0.5B-bnb-4bit) 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-1.5B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-1.5B-bnb-4bit) 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-3B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-3B-bnb-4bit) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-7B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-7B-bnb-4bit) 14 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-14B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-14B-bnb-4bit) 32 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-32B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-32B-bnb-4bit) 72 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-72B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2.5-72B-bnb-4bit) **Qwen 2** 1.5 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-1.5B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-1.5B-bnb-4bit) 7 B [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-7B) [linkarrow-up-right](https://huggingface.co/unsloth/Qwen2-7B-bnb-4bit) **Llama models:** Model Variant Base (16-bit) Base (4-bit) **Llama 4** Scout 17B 16E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E) — Maverick 17B 128E [linkarrow-up-right](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E) — **Llama 3.3** 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B) — **Llama 3.2** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B) — 3 B [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B) — 11 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-11B-Vision) — 90 B Vision [linkarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-90B-Vision) — **Llama 3.1** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-8B) — 70 B [linkarrow-up-right](https://huggingface.co/unsloth/Meta-Llama-3.1-70B) — **Llama 3** 8 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-3-8b-bnb-4bit) **Llama 2** 7 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-7b-bnb-4bit) 13 B [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-13b) [linkarrow-up-right](https://huggingface.co/unsloth/llama-2-13b-bnb-4bit) **Gemma models** Model Variant Base (16-bit) Base (4-bit) **Gemma 3** 1 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-pt) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-pt-unsloth-bnb-4bit) 4 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-pt) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-pt-unsloth-bnb-4bit) 12 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-12b-pt) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-12b-pt-unsloth-bnb-4bit) 27 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-pt) [linkarrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-pt-unsloth-bnb-4bit) **Gemma 2** 2 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-2b) — 9 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-9b) — 27 B [linkarrow-up-right](https://huggingface.co/unsloth/gemma-2-27b) — **Mistral models:** Model Variant Base (16-bit) Base (4-bit) **Mistral** Small 24B 2501 [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Small-24B-Base-2501) — NeMo 12B 2407 [linkarrow-up-right](https://huggingface.co/unsloth/Mistral-Nemo-Base-2407) — 7B v0.3 [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-v0.3) [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-v0.3-bnb-4bit) 7B v0.2 [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-v0.2) [linkarrow-up-right](https://huggingface.co/unsloth/mistral-7b-v0.2-bnb-4bit) Pixtral 12B 2409 [linkarrow-up-right](https://huggingface.co/unsloth/Pixtral-12B-Base-2409) — **Other (TTS, TinyLlama) models:** Model Variant Base (16-bit) Base (4-bit) **TinyLlama** 1.1 B (Base) [linkarrow-up-right](https://huggingface.co/unsloth/tinyllama) [linkarrow-up-right](https://huggingface.co/unsloth/tinyllama-bnb-4bit) **Orpheus-3b** 0.1-pretrained [linkarrow-up-right](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained) [linkarrow-up-right](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained-unsloth-bnb-4bit) You can use our FP8 uploads for training or serving/deployment. FP8 Dynamic offers slightly faster training and lower VRAM usage than FP8 Block, but with a small trade-off in accuracy. Model Variant FP8 (Dynamic / Block) Qwen3 Coder-Next [Dynamicarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8) GLM 4.7-Flash [Dynamicarrow-up-right](https://huggingface.co/unsloth/GLM-4.7-Flash-FP8-Dynamic) **Llama 3.3** 70B Instruct [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-FP8-Block) **Llama 3.2** 1B Base [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-FP8-Block) 1B Instruct [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-FP8-Block) 3B Base [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-FP8-Block) 3B Instruct [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-FP8-Block) **Llama 3.1** 8B Base [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-8B-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-8B-FP8-Block) 8B Instruct [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-FP8-Block) 70B Base [Dynamicarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-70B-FP8-Dynamic) · [Blockarrow-up-right](https://huggingface.co/unsloth/Llama-3.1-70B-FP8-Block) **Qwen3** 0.6B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-0.6B-FP8) 1.7B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-1.7B-FP8) 4B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-FP8) 8B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-8B-FP8) 14B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-14B-FP8) 32B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-32B-FP8) 235B-A22B [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-FP8) **Qwen3 (2507)** 4B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-FP8) 4B Thinking [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-4B-Thinking-2507-FP8) 30B-A3B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-FP8) 30B-A3B Thinking [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-FP8) 235B-A22B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-FP8) 235B-A22B Thinking [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-FP8) **Qwen3-VL** 4B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-FP8) 4B Thinking [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-FP8) 8B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-FP8) 8B Thinking [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-FP8) **Qwen3-Coder** 480B-A35B Instruct [FP8arrow-up-right](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-FP8) **Granite 4.0** h-tiny [FP8 Dynamicarrow-up-right](https://huggingface.co/unsloth/granite-4.0-h-tiny-FP8-Dynamic) h-small [FP8 Dynamicarrow-up-right](https://huggingface.co/unsloth/granite-4.0-h-small-FP8-Dynamic) **Magistral Small** 2509 [FP8 Dynamicarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-Dynamic) · [FP8 torchaoarrow-up-right](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-torchao) **Mistral Small 3.2** 24B Instruct-2506 [FP8arrow-up-right](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-FP8) **Gemma 3** 270M-it torchao 270m — [FP8arrow-up-right](https://huggingface.co/unsloth/gemma-3-270m-it-FP8-Dynamic) 1B — [FP8arrow-up-right](https://huggingface.co/unsloth/gemma-3-1b-it-FP8-Dynamic) 4B — [FP8arrow-up-right](https://huggingface.co/unsloth/gemma-3-4b-it-FP8-Dynamic) 12B — [FP8arrow-up-right](https://huggingface.co/unsloth/gemma-3-12B-it-FP8-Dynamic) 27B — [FP8arrow-up-right](https://huggingface.co/unsloth/gemma-3-27b-it-FP8-Dynamic) [FP8 torchaoarrow-up-right](https://huggingface.co/unsloth/gemma-3-270m-it-torchao-FP8) [PreviousUnsloth Notebookschevron-left](https://unsloth.ai/docs/get-started/unsloth-notebooks) [NextInstallationchevron-right](https://unsloth.ai/docs/get-started/install) Last updated 9 days ago Was this helpful? Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # How to Fine-tune LLMs with Unsloth & Docker | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Local training can be complex due to dependency hell or breaking environments. Unsloth’s [Docker imagearrow-up-right](https://hub.docker.com/r/unsloth/unsloth) can bypass these issues. No setup is needed: pull and run the image and start training. * **Unsloth official Docker image:** [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) **Why Use Unsloth & Docker?** Unsloth’s Docker image is stable, up-to-date and works in [supported setups](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements#system-requirements) like Windows. * Fully contained dependencies keep your system clean. Runs safely without root. * Use locally or on any platform with pre-installed notebooks. circle-check You can now use our main Docker image `unsloth/unsloth` for Blackwell and 50-series GPUs - no separate image needed. ### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#step-by-step-tutorial) ⚡ Step-by-Step Tutorial 1 **Install Docker and NVIDIA Container Toolkit.** Install Docker via [Linuxarrow-up-right](https://docs.docker.com/engine/install/) or [Desktoparrow-up-right](https://docs.docker.com/desktop/) (other). Then install [NVIDIA Container Toolkitarrow-up-right](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation) : Copy export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1 sudo apt-get update && sudo apt-get install -y \ nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION} ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-41cae231ed4761f844ce9836e03b17aabd7c803c%252Fnvidia%2520toolkit.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=87cec663&sv=2) 2 **Run the container.** [`**unsloth/unsloth**`arrow-up-right](https://hub.docker.com/r/unsloth/unsloth) is Unsloth's only Docker image. For [Blackwell](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) and 50-series GPUs, use this same image - no separate image needed. If using DGX Spark, you'll need to follow our [DGX guide](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) . Copy docker run -d -e JUPYTER_PASSWORD="mypassword" \ -p 8888:8888 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-2b50d78c5d54eaf189c0a40d46c405585ea23082%252Fdocker%2520run.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=73c3ccf6&sv=2) 3 **Access Jupyter Lab** Go to [http://localhost:8888arrow-up-right](http://localhost:8888/) and open Unsloth. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-828df0a668fd94025c1193c24a7f09c1d58dcbd8%252Fjupyter.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=921d75db&sv=2) Access the `unsloth-notebooks` tabs to see Unsloth notebooks. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-e7a3f620a3ec5bff335632ff9b0cb422f76528a1%252FScreenshot_from_2025-09-30_21-38-15.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=9fb12cef&sv=2) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-531882c33eb96dec24e2d7673471d6a3928a3951%252FScreenshot_from_2025-09-30_21-39-41.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=50528e9a&sv=2) 4 **Start training with Unsloth** If you're new, follow our step-by-step [Fine-tuning Guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) , [RL Guide](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) or just save/copy any of our premade [notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks) . ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-665f900b008991ddcd8fdabb773b292de3c41e72%252FScreenshot_from_2025-09-30_21-40-29.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=8366645d&sv=2) #### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#container-structure) 📂 Container Structure * `/workspace/work/` — Your mounted work directory * `/workspace/unsloth-notebooks/` — Example fine-tuning notebooks * `/home/unsloth/` — User home directory ### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#usage-example) 📖 Usage Example #### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#full-example) Full Example #### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#setting-up-ssh-key) Setting up SSH Key If you don't have an SSH key pair: ### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#advanced-settings) ⚙️ Advanced Settings Variable Description Default `JUPYTER_PASSWORD` Jupyter Lab password `unsloth` `JUPYTER_PORT` Jupyter Lab port inside container `8888` `SSH_KEY` SSH public key for authentication `None` `USER_PASSWORD` Password for `unsloth` user (sudo) `unsloth` * Jupyter Lab: `-p 8000:8888` * SSH access: `-p 2222:22` circle-exclamation **Important**: Use volume mounts to preserve your work between container runs. ### [hashtag](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#security-notes) **🔒 Security Notes** * Container runs as non-root `unsloth` user by default * Use `USER_PASSWORD` for sudo operations inside container * SSH access requires public key authentication [PreviousDGX Stationchevron-left](https://unsloth.ai/docs/blog/dgx-station) [NextDGX Spark and Unslothchevron-right](https://unsloth.ai/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) Last updated 3 months ago Was this helpful? * [⚡ Step-by-Step Tutorial](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#step-by-step-tutorial) * [📖 Usage Example](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#usage-example) * [⚙️ Advanced Settings](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#advanced-settings) * [🔒 Security Notes](https://unsloth.ai/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker#security-notes) Was this helpful? sun-brightdesktopmoon Copy docker run -d -e JUPYTER_PORT=8000 \ -e JUPYTER_PASSWORD="mypassword" \ -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" \ -e USER_PASSWORD="unsloth2024" \ -p 8000:8000 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth Copy # Generate new key pair ssh-keygen -t rsa -b 4096 -f ~/.ssh/container_key # Use the public key in docker run -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" # Connect via SSH ssh -i ~/.ssh/container_key -p 2222 unsloth@localhost Copy -p : Copy -v : Copy docker run -d -e JUPYTER_PORT=8000 \ -e JUPYTER_PASSWORD="mypassword" \ -e "SSH_KEY=$(cat ~/.ssh/container_key.pub)" \ -e USER_PASSWORD="unsloth2024" \ -p 8000:8000 -p 2222:22 \ -v $(pwd)/work:/workspace/work \ --gpus all \ unsloth/unsloth sun-brightdesktopmoon --- # Chat Templates | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close In our GitHub, we have a list of every chat template Unsloth uses including for Llama, Mistral, Phi-4 etc. So if you need any pointers on the formatting or use case, you can view them here: [github.com/unslothai/unsloth/blob/main/unsloth/chat\_templates.pyarrow-up-right](https://github.com/unslothai/unsloth/blob/main/unsloth/chat_templates.py) #### [hashtag](https://unsloth.ai/docs/basics/chat-templates#list-of-colab-chat-template-notebooks) List of Colab chat template notebooks: * [Conversationalarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [ChatMLarrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [Ollamaarrow-up-right](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing) * [Text Classificationarrow-up-right](https://github.com/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) by Timotheeee * [Multiple Datasetsarrow-up-right](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) by Flail ### [hashtag](https://unsloth.ai/docs/basics/chat-templates#adding-new-tokens) Adding new tokens Unsloth has a function called `add_new_tokens` which allows you to add new tokens to your finetune. For example if you want to add ``, `` and `` we can do the following: Copy model, tokenizer = FastLanguageModel.from_pretrained(...) from unsloth import add_new_tokens add_new_tokens(model, tokenizer, new_tokens = ["", "", ""]) model = FastLanguageModel.get_peft_model(...) circle-exclamation Note - you MUST always call `add_new_tokens` before `FastLanguageModel.get_peft_model`! [hashtag](https://unsloth.ai/docs/basics/chat-templates#multi-turn-conversations) Multi turn conversations --------------------------------------------------------------------------------------------------------------- An issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-2a65cd74ddd03a6bcbbc9827d9d034e4879a8e6a%252Fdiff.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d776164b&sv=2) So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-2b1b3494b260f1102942d86143a885225c6a06f2%252Fcombine.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=f26d5bca&sv=2) Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column. We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this! ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-7bf83bf802191bda9e417bbe45afa181e7f24f38%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=936b7eae&sv=2) [hashtag](https://unsloth.ai/docs/basics/chat-templates#customizable-chat-templates) Customizable Chat Templates --------------------------------------------------------------------------------------------------------------------- We can now specify the chat template for finetuning itself. The very famous Alpaca format is below: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-59737e6dcb09fed15487d5a57c69f07cb40bb8e7%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=81f68351&sv=2) But remember we said this was a bad idea because ChatGPT style finetunes require only 1 prompt? Since we successfully merged all dataset columns into 1 using Unsloth, we essentially can create the below style chat template with 1 input column (instruction) and 1 output: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-d54582ae98c396d51bfb85628b46c54f2517d030%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=68b8c20&sv=2) We just require you must put a `{INPUT}` field for the instruction and an `{OUTPUT}` field for the model's output field. We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. For example, below are some cool examples which you can customize the chat template to be: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-cc455dc380d3d44ef136e485754964159dc773d8%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=82deed73&sv=2) For the ChatML format used in OpenAI models: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-15bfca9cfadf10d54b4d3f66e3050044317d62c5%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=c0e648a7&sv=2) Or you can use the Llama-3 template itself (which only functions by using the instruct version of Llama-3): We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-80a2ed4de2ca323ac192c513cac65e9e8bf475db%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=467af04b&sv=2) Or in the Titanic prediction task where you had to predict if a passenger died or survived in this Colab notebook which includes CSV and Excel uploading: [https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharingarrow-up-right](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fgit-blob-20911ab305c1a10e85859c703157b80175141eb1%252Fimage.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=957f7f78&sv=2) [hashtag](https://unsloth.ai/docs/basics/chat-templates#applying-chat-templates-with-unsloth) Applying Chat Templates with Unsloth --------------------------------------------------------------------------------------------------------------------------------------- For datasets that usually follow the common chatml format, the process of preparing the dataset for training or finetuning, consists of four simple steps: * Check the chat templates that Unsloth currently supports:\\ This will print out the list of templates currently supported by Unsloth. Here is an example output:\\ \\ * Use `get_chat_template` to apply the right chat template to your tokenizer:\\ \\ * Define your formatting function. Here's an example:\\ This function loops through your dataset applying the chat template you defined to each sample.\\ * Finally, let's load the dataset and apply the required modifications to our dataset: \\ If your dataset uses the ShareGPT format with "from"/"value" keys instead of the ChatML "role"/"content" format, you can use the `standardize_sharegpt` function to convert it first. The revised code will now look as follows: \\ [hashtag](https://unsloth.ai/docs/basics/chat-templates#more-information) More Information ----------------------------------------------------------------------------------------------- Assuming your dataset is a list of list of dictionaries like the below: You can use our `get_chat_template` to format it. Select `chat_template` to be any of `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth`, and use `mapping` to map the dictionary values `from`, `value` etc. `map_eos_token` allows you to map `<|im_end|>` to EOS without any training. You can also make your own custom chat templates! For example our internal chat template we use is below. You must pass in a `tuple` of `(custom_template, eos_token)` where the `eos_token` must be used inside the template. [PreviousHugging Face Hub, XET debuggingchevron-left](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging) [NextUnsloth Environment Flagschevron-right](https://unsloth.ai/docs/basics/unsloth-environment-flags) Last updated 1 month ago Was this helpful? * [Adding new tokens](https://unsloth.ai/docs/basics/chat-templates#adding-new-tokens) * [Multi turn conversations](https://unsloth.ai/docs/basics/chat-templates#multi-turn-conversations) * [Customizable Chat Templates](https://unsloth.ai/docs/basics/chat-templates#customizable-chat-templates) * [Applying Chat Templates with Unsloth](https://unsloth.ai/docs/basics/chat-templates#applying-chat-templates-with-unsloth) * [More Information](https://unsloth.ai/docs/basics/chat-templates#more-information) Was this helpful? sun-brightdesktopmoon Copy from unsloth.chat_templates import CHAT_TEMPLATES print(list(CHAT_TEMPLATES.keys())) Copy ['unsloth', 'zephyr', 'chatml', 'mistral', 'llama', 'vicuna', 'vicuna_old', 'vicuna old', 'alpaca', 'gemma', 'gemma_chatml', 'gemma2', 'gemma2_chatml', 'llama-3', 'llama3', 'phi-3', 'phi-35', 'phi-3.5', 'llama-3.1', 'llama-31', 'llama-3.2', 'llama-3.3', 'llama-32', 'llama-33', 'qwen-2.5', 'qwen-25', 'qwen25', 'qwen2.5', 'phi-4', 'gemma-3', 'gemma3'] Copy from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "gemma-3", # change this to the right chat_template name ) Copy def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } Copy # Import and load dataset from datasets import load_dataset dataset = load_dataset("repo_name/dataset_name", split = "train") # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) Copy # Import dataset from datasets import load_dataset dataset = load_dataset("mlabonne/FineTome-100k", split = "train") # Convert your dataset to the "role"/"content" format if necessary from unsloth.chat_templates import standardize_sharegpt dataset = standardize_sharegpt(dataset) # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) Copy [\ [{'from': 'human', 'value': 'Hi there!'},\ {'from': 'gpt', 'value': 'Hi how can I help?'},\ {'from': 'human', 'value': 'What is 2+2?'}],\ [{'from': 'human', 'value': 'What's your name?'},\ {'from': 'gpt', 'value': 'I'm Daniel!'},\ {'from': 'human', 'value': 'Ok! Nice!'},\ {'from': 'gpt', 'value': 'What can I do for you?'},\ {'from': 'human', 'value': 'Oh nothing :)'},],\ ] Copy from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to instead ) def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass from datasets import load_dataset dataset = load_dataset("philschmid/guanaco-sharegpt-style", split = "train") dataset = dataset.map(formatting_prompts_func, batched = True,) Copy unsloth_template = \ "{{ bos_token }}"\ "{{ 'You are a helpful assistant to the user\n' }}"\ ""\ "
"\ "
"\ "{{ '>>> User: ' + message['content'] + '\n' }}"\ "
"\ "{{ '>>> Assistant: ' + message['content'] + eos_token + '\n' }}"\ "
"\ "
"\ "
"\ "{{ '>>> Assistant: ' }}"\ "
" unsloth_eos_token = "eos_token" tokenizer = get_chat_template( tokenizer, chat_template = (unsloth_template, unsloth_eos_token,), # You must provide a template and EOS token mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to instead ) sun-brightdesktopmoon --- # Export models with Unsloth Studio | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close Use [Unsloth Studio](https://unsloth.ai/docs/new/studio) to export, save, or convert models to GGUF, Safetensors, or LoRA for deployment, sharing, or local inference in Unsloth, llama.cpp, Ollama, vLLM, and more. Export a trained checkpoint or convert any existing model. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrrFY8YczW3dDpfYi1k9f%252FScreenshot%25202026-03-15%2520at%25209.28.19%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd2729e16-799f-48f0-8b07-0248b93fa599&width=768&dpr=3&quality=100&sign=caee257&sv=2) 1 ### [hashtag](https://unsloth.ai/docs/new/studio/export#select-training-run) Select Training Run Start by selecting the training run you want to export from. Each run represents a complete training session and may contain multiple checkpoints. After choosing a run, select the checkpoint to export. A checkpoint is a saved version of the model created during training. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzB12XFNP3UjoAT1l9vz3%252Fimage.png%3Falt%3Dmedia%26token%3D021b8864-b2c5-4a92-927e-e23350610036&width=768&dpr=3&quality=100&sign=7b9f510&sv=2) 2 ### [hashtag](https://unsloth.ai/docs/new/studio/export#select-checkpoint) Select Checkpoint Later checkpoints typically represent the final trained model, but you can export any checkpoint depending on your needs. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8VfRPUcY3w6zYfNmAIDn%252Fimage.png%3Falt%3Dmedia%26token%3D42565a7d-e62f-4cf0-bd33-90422f1b2194&width=768&dpr=3&quality=100&sign=d0d8f72f&sv=2) 3 ### [hashtag](https://unsloth.ai/docs/new/studio/export#export-methods) Export Methods Depending on your workflow, you can export a merged model, LoRA adapter weights, or a GGUF model for local inference. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh4sPts9rJhHiGqf0UxIs%252Fimage.png%3Falt%3Dmedia%26token%3D4f1d6a76-bd40-4471-ab8d-0b2fe33d0410&width=768&dpr=3&quality=100&sign=c5db7ae3&sv=2) Each export method produces a different version of the model depending on how you plan to run or share it. The table below explains what each option exports. Export Type Description Merged Model **16-bit model** with the LoRA adapter merged into the base weights. LoRA Only Exports **only the adapter weights**. Requires the original base model. GGUF / llama.cpp Converts the model to **GGUF format** for Unsloth / llama.cpp **/** Ollama / LM Studio inference. 4 ### [hashtag](https://unsloth.ai/docs/new/studio/export#export-save-locally) Export / Save Locally When exporting a model, you can choose where the resulting files should be saved. Models can be downloaded directly to your machine or pushed to the Hugging Face Hub for hosting and sharing. Save the exported model files directly to your machine. This option is useful for running the model locally, distributing files manually, or integrating with local inference tools. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfsBaE8V2o69jSyCVGIz4%252Fimage.png%3Falt%3Dmedia%26token%3D4ef3fa06-d25b-424a-91e3-42debd3b6908&width=768&dpr=3&quality=100&sign=b8ec7aac&sv=2) 5 ### [hashtag](https://unsloth.ai/docs/new/studio/export#push-to-hub) Push to Hub Upload the exported model to the Hugging Face Hub. This allows you to host, share, and deploy the model from a central repository. You will need a Hugging Face write token to publish the model. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrvVnuVUYQWv2nkrgFxpK%252Fimage.png%3Falt%3Dmedia%26token%3D5e0b91fe-5225-4bff-9fa9-ec1fb3867b1a&width=768&dpr=3&quality=100&sign=6722678d&sv=2) circle-check If you are already authenticated with the Hugging Face CLI, the write token can be left empty. [PreviousData Recipeschevron-left](https://unsloth.ai/docs/new/studio/data-recipe) [NextUnsloth Updateschevron-right](https://unsloth.ai/docs/new/changelog) Last updated 17 days ago Was this helpful? * [Select Training Run](https://unsloth.ai/docs/new/studio/export#select-training-run) * [Select Checkpoint](https://unsloth.ai/docs/new/studio/export#select-checkpoint) * [Export Methods](https://unsloth.ai/docs/new/studio/export#export-methods) * [Export / Save Locally](https://unsloth.ai/docs/new/studio/export#export-save-locally) * [Push to Hub](https://unsloth.ai/docs/new/studio/export#push-to-hub) Was this helpful? sun-brightdesktopmoon sun-brightdesktopmoon --- # How to Run Local LLMs with OpenAI Codex | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close This guide will walk you through connecting open LLMs to the Codex CLI **entirely locally**. It works with any OpenAI or API compatible local model setup including: DeepSeek, Qwen, Gemma, and more. In this tutorial, we’ll use [GLM-4.7-Flasharrow-up-right](https://open-2v.gitbook.com/url/preview/site_mXXTe/~/revisions/NYG3pIjIP3JF6zgJgfjl/models/glm-4.7-flash) (a 30B MoE, agentic + coding model) which fits nicely on a 24GB RAM/unified memory device to autonomously fine-tune an LLM using [Unslotharrow-up-right](https://github.com/unslothai/unsloth) . Prefer a different model? Swap in [any other modelarrow-up-right](https://open-2v.gitbook.com/url/preview/site_mXXTe/~/revisions/NYG3pIjIP3JF6zgJgfjl/models/tutorials) by updating the model names in the scripts. [openaiOpenAI Codex Tutorial](https://unsloth.ai/docs/basics/codex#openai-codex-cli-tutorial) For model quants, we’ll use Unsloth [Dynamic GGUFsarrow-up-right](https://open-2v.gitbook.com/url/preview/site_mXXTe/~/revisions/NYG3pIjIP3JF6zgJgfjl/basics/unsloth-dynamic-2.0-ggufs) so you can run quantized GGUF models while preserving as much quality as possible. We’ll use [`llama.cpp`arrow-up-right](https://github.com/ggml-org/llama.cpp) , an open-source runtime for running LLMs on macOS, Linux, and Windows. Its `llama-server` component lets you serve models efficiently via a single **OpenAI-compatible** HTTP endpoint. In this setup, the model is served on **port 8001**, and all agent tool calls are routed through that one endpoint. ### [hashtag](https://unsloth.ai/docs/basics/codex#id-1-setup-tutorial) 📖 #1: Setup Tutorial 1 #### [hashtag](https://unsloth.ai/docs/basics/codex#install-llama.cpp) Install llama.cpp We need to install `llama.cpp` to deploy/serve local LLMs to use in Codex etc. We follow the official build instructions for correct GPU bindings and maximum performance. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev git-all -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4DmycqgjxOz6TOQd9PLJ%252Fimage.png%3Falt%3Dmedia%26token%3Dc94db0b5-8e4a-4043-b2a3-c68bad93213e&width=768&dpr=3&quality=100&sign=bdf790c4&sv=2) 2 #### [hashtag](https://unsloth.ai/docs/basics/codex#download-and-use-models-locally) Download and use models locally Download the model via `huggingface_hub` in Python (after installing via `pip install huggingface_hub hf_transfer`). We use the **UD-Q4\_K\_XL** quant for the best size/accuracy balance. You can find all Unsloth GGUF uploads in our [Collection here](https://unsloth.ai/docs/get-started/unsloth-model-catalog) . If downloads get stuck, see [Hugging Face Hub, XET debugging](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging) circle-check We used `unsloth/GLM-4.7-Flash-GGUF` , but you can use anything like `unsloth/Qwen3-Coder-Next-GGUF` - see [Qwen3-Coder-Next](https://unsloth.ai/docs/models/qwen3-coder-next) Copy import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/GLM-4.7-Flash-GGUF", local_dir = "unsloth/GLM-4.7-Flash-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxlIrQGQ0cevb1ckkSFy5%252Fimage.png%3Falt%3Dmedia%26token%3Db1a42562-927a-4ad2-85f8-29c2993c46aa&width=768&dpr=3&quality=100&sign=f1f734a9&sv=2) 3 #### [hashtag](https://unsloth.ai/docs/basics/codex#start-the-llama-server) Start the Llama-server To deploy GLM-4.7-Flash for agentic workloads, we use `llama-server`. We apply Z.ai's recommended sampling parameters (`temp 1.0`, `top_p 0.95`) and enable `--jinja` for proper tool calling support. Run this command in a new terminal (use `tmux` or open a new terminal). The below should **fit perfectly in a 24GB GPU (RTX 4090) (uses 23GB)** `--fit on` will also auto offload, but if you see bad performance, reduce `--ctx-size` . We used `--cache-type-k q8_0 --cache-type-v q8_0` for KV cache quantization to reduce VRAM usage. Copy ./llama.cpp/llama-server \ --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf \ --alias "unsloth/GLM-4.7-Flash" \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --port 8001 \ --kv-unified \ --cache-type-k q8_0 --cache-type-v q8_0 \ --flash-attn on \ --batch-size 4096 --ubatch-size 1024 \ --ctx-size 131072 circle-check You can also disable thinking for GLM-4.7-Flash which can improve performance for agentic coding stuff. To disable thinking with llama.cpp add this to the llama-server command: `--chat-template-kwargs "{\"enable_thinking\": false}"` ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FyKf6guCV8snRaAV16Zxc%252FG_16XLgXUAEnSWH.jpg%3Falt%3Dmedia%26token%3D3b557c6d-3f6f-4515-ba9f-4cc8b50bcef1&width=300&dpr=3&quality=100&sign=d1173ec1&sv=2) [hashtag](https://unsloth.ai/docs/basics/codex#openai-codex-cli-tutorial) openai OpenAI Codex CLI Tutorial --------------------------------------------------------------------------------------------------------------- [Codex arrow-up-right](https://github.com/openai/codex) is OpenAI's official coding agent that runs locally. While designed for ChatGPT, it supports custom API endpoints, making it perfect for local LLMs. For installing on [Windowsarrow-up-right](https://developers.openai.com/codex/windows/) - it's best to use WSL. #### [hashtag](https://unsloth.ai/docs/basics/codex#install) **Install** **Mac (Homebrew):** Copy brew install --cask codex **Universal (NPM) for Linux** Copy apt update apt install nodejs npm -y npm install -g @openai/codex **Configure** First run `codex` to login and setup things, then create or edit the configuration file at `~/.codex/config.toml` (Mac/Linux) or `%USERPROFILE%\.codex\config.toml` (Windows). Use `cat > ~/.codex/config.toml` for Linux / Mac: Navigate to your project folder (`mkdir project ; cd project`) and run: Or to allow any code to execute. **(BEWARE this will make Codex do and execute code however it likes without any approvals!)** And you will see: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FiTjL7DtyNg0GREwR7p54%252Fimage.png%3Falt%3Dmedia%26token%3D9f793df2-e91b-4631-a7c8-00e659fd1a07&width=768&dpr=3&quality=100&sign=f5bba3fb&sv=2) circle-exclamation OpenAI's Codex is removing `wire_api = "chat"` support it seems - it still works as of January 29th 2026. We should switch to `wire_api = "responses"` but it keeps error-ing out with: `{"error":{"code":400,"message":"'type' of tool must be 'function'","type":"invalid_request_error"}}` Try this prompt to install and run a simple Unsloth finetune: and you will see: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fg2R9WPuNRNUUFnPRXE2F%252Fimage.png%3Falt%3Dmedia%26token%3D686f4be8-7a50-4f6b-86cb-327cec36de81&width=768&dpr=3&quality=100&sign=dc5429f4&sv=2) and if we wait a little longer, we finally get: ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHTKt5sGFpkYzS8DlM9E7%252Fimage.png%3Falt%3Dmedia%26token%3Df4fa2e27-10d7-4c4e-8af0-448170336af9&width=768&dpr=3&quality=100&sign=91a992f7&sv=2) [PreviousClaude Codechevron-left](https://unsloth.ai/docs/basics/claude-code) [NextMulti-GPU Training Unslothchevron-right](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) Last updated 1 month ago Was this helpful? * [📖 #1: Setup Tutorial](https://unsloth.ai/docs/basics/codex#id-1-setup-tutorial) * [OpenAI Codex CLI Tutorial](https://unsloth.ai/docs/basics/codex#openai-codex-cli-tutorial) Was this helpful? sun-brightdesktopmoon Copy [model_providers.llama_cpp] name = "llama_cpp API" base_url = "http://localhost:8001/v1" wire_api = "responses" stream_idle_timeout_ms = 10000000 Copy codex --model unsloth/GLM-4.7-Flash -c model_provider=llama_cpp --search Copy codex --model unsloth/GLM-4.7-Flash -c model_provider=llama_cpp --search --dangerously-bypass-approvals-and-sandbox Copy You can only work in the cwd project/. Do not search for AGENTS.md - this is it. Install Unsloth via a virtual environment via uv. See https://unsloth.ai/docs/get-started/install/pip-install on how (get it and read). Then do a simple Unsloth finetuning run described in https://github.com/unslothai/unsloth. You have access to 1 GPU. sun-brightdesktopmoon --- # 500K Context Length Fine-tuning | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close We’re introducing new algorithms in Unsloth that push the limits of long-context training for **any LLM and VLM**. Training LLMs like gpt-oss-20b can now reach **500K+ context lengths** on a single 80GB H100 GPU, compared to 80K previously with no accuracy degradation. You can reach >**750K context windows** on a B200 192GB GPU. > **Try 500K-context gpt-oss-20b fine-tuning on our** [**80GB A100 Colab notebook**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) > **.** We’ve significantly improved how Unsloth handles memory usage patterns, speed, and context lengths: * **60% lower VRAM use** with **3.2x longer context** via Unsloth’s new [fused and chunked cross-entropy](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-loss-refactoring-chunk-and-fuse) loss, with no degradation in speed or accuracy * Enhanced activation offloading in Unsloth’s [**Gradient Checkpointing**](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhanced) * Collabing with Stas Bekman from Snowflake on [Tiled MLP](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#tiled-mlp-unlocking-500k) , enabling 2× more contexts Unsloth’s algorithms allows gpt-oss-20b QLoRA (4bit) with 290K context possible on a H100 with no accuracy loss, and 500K+ with Tiled MLP enabled, altogether delivering >**6.4x longer context lengths.** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8Ha930qR5XXBOK7M7oiy%252Fline_chart_light_tiled.png%3Falt%3Dmedia%26token%3D51467f68-a77b-4037-b9d9-e668223868c5&width=768&dpr=3&quality=100&sign=74316519&sv=2) ### [hashtag](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-loss-refactoring-chunk-and-fuse) 📐 Unsloth Loss Refactoring: Chunk & Fuse Our new fused loss implementation adds **dynamic sequence chunking**: instead of computing language model head logits and cross-entropies over the entire sequence at once, we process manageable slices along the flattened sequence dimension. This cuts peak memory from GBs to a smaller chunk sizes. Each chunk still runs a fully fused forward + backward pass via `torch.func.grad_and_value` , and retains mixed precision accuracy by upcasting to float32 if necessary. **These changes do not degrade training speed or accuracy.** ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFF43WA1X8Y4vADBrCi8T%252Fline_chart_light.png%3Falt%3Dmedia%26token%3D7afc7f73-bc54-403a-9674-8a16841ec659&width=768&dpr=3&quality=100&sign=43904d88&sv=2) The key innovation is that the **chunk size is chosen automatically at runtime** based on available VRAM. * If you have more free VRAM, larger chunks are used for faster runs * If you have less VRAM, it increases the number of chunks to avoid memory blowouts. This **removes manual tuning** and keeps our algorithm robust across old and new GPUs, workloads and different sequence lengths. circle-check Due to automatic tuning, **smaller contexts will use more VRAM** (fewer chunks) to **avoid unnecessary overhead**. For the plots above, we adjust the number of loss chunks to reflect realistic VRAM tiers. With 80GB VRAM, this yields >3.2× longer contexts. ### [hashtag](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhancements) 🏁 Unsloth Gradient Checkpointing Enhancements Our [Unsloth Gradient Checkpointingarrow-up-right](https://unsloth.ai/blog/long-context) algorithm, **introduced in April 2024**, quickly became popular and the standard across the industry, having been integrated into most training packages nowadays. It offloads activations to CPU RAM which allowed 10x longer context lengths. Our new enhancements uses CUDA Streams and other tricks to add at most **0.1%** training overhead with no impact on accuracy. Previously it added 1 to 3% training overhead. By offloading activations as soon as they are produced, we minimize peak activation footprint and free GPU memory exactly when it’s needed. This sharply reduces memory pressure in long-context or large-batch training, where a single decoder layer’s activations can exceed 2 GB. > **Thus, Unsloth’s new algorithms & Gradient Checkpointing contributes to most improvements (3.2x), enabling 290k-context-length QLoRA GPT-OSS fine-tuning on a single H100.** ### [hashtag](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#tiled-mlp-unlocking-500k) 🔓 Tiled MLP: Unlocking 500K+ With help from [Stas Bekmanarrow-up-right](https://x.com/StasBekman) (Snowflake), we integrated Tiled MLP from Snowflake’s Arctic Long Sequence Training [paperarrow-up-right](https://arxiv.org/abs/2506.13996) and blog post. TiledMLP reduces activation memory and enables much longer sequence lengths by tiling hidden states along the sequence dimension before heavy MLP projections. **We also introduce a few quality-of-life improvements:** We preserve RNG state across tiled forward recomputations so dropout and other stochastic ops are consistent between forward and backward replays. This keeps nested checkpointed computations stable and numerically identical. circle-check Our implementation auto patches any module named or typed as `mlp`, so **nearly all models with MLP modules are supported out of the box for Tiled MLP.** **Tradeoffs to keep in mind** TiledMLP saves VRAM at the cost of extra forward passes. Because it lives inside a checkpointed transformer block and is itself written in a checkpoint style, it effectively becomes a nested checkpoint: one **MLP now performs ~3 forward passes and 1 backward pass per step**. In return, we can drop almost all intermediate MLP activations from VRAM while still supporting extremely long sequences. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdeOJEEqucGYtbXbb7nqB%252Fbaseline_vs_unsloth_spike.png%3Falt%3Dmedia%26token%3D3b1cdfd3-dd24-4c94-b7ec-5d1366464afb&width=768&dpr=3&quality=100&sign=29b337f3&sv=2) The plots compare active memory timelines for a single decoder layer’s forward and backward during a long-context training step, without Tiled MLP (left) and with it (right). Without Tiled MLP, peak VRAM occurs during the MLP backward; with Tiled MLP, it shifts to the fused loss calculation. We see ~40% lower VRAM usage, and because the fused loss auto chunks dynamically based on available VRAM, the peak with Tiled MLP would be even smaller on smaller GPUs. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FUCx0X7S5FvaD3hUsma5j%252Fbaseline_vs_unsloth_nospike.png%3Falt%3Dmedia%26token%3Da81b8639-21d0-43aa-a837-8209949e8742&width=768&dpr=3&quality=100&sign=78c626c&sv=2) To show cross-entropy loss is not the new bottleneck, we fix its chunk size instead of choosing it dynamically and then double the number of chunks. This significantly reduces the loss-related memory spikes. The max memory now occurs during backward in both cases, and overall timing is similar, though Tiled MLP adds a small overhead: one large GEMM becomes many sequential matmuls, plus the extra forward pass mentioned above. Overall, the trade-off is worth it: without Tiled MLP, long-context training can require roughly 2× the memory usage, while with **Tiled MLP a single GPU pays only about a 1.3× increase in step time for the same context length.** **Enabling Tiled MLP in Unsloth:** Just set `unsloth_tiled_mlp = True` in `from_pretrained` and Tiled MLP is enabled. We follow the same logic as the Arctic paper and choose `num_shards = ceil(seq_len/hidden_size)`. Each tile will operate on sequence lengths which are the same size of the hidden dimension of the model to balance throughput and memory savings. We also discussed how Tiled MLP effectively does 3 forward passes and 1 backward, compared to normal gradient checkpointing which does 2 forward passes and 1 backward with Stas Bekman and [DeepSpeedarrow-up-right](https://github.com/deepspeedai/DeepSpeed/pull/7664) provided a doc update for Tiled MLP within DeepSpeed. circle-check Next time fine-tuning runs out of memory, try turning on `unsloth_tiled_mlp = True`. This should save some VRAM as long as the context length is longer than the LLM's hidden dimension. * * * **With our latest update, it is possible to now reach 1M context length with a smaller model on a single GPU!** **Try 500K-context gpt-oss-20b fine-tuning on our** [**80GB A100 Colab notebook**arrow-up-right](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_500K_Context_Fine_tuning.ipynb) **.** If you've made it this far, we're releasing a new blog on our latest improvements in training speed this week so stay tuned by joining our [Reddit r/unslotharrow-up-right](https://www.reddit.com/r/unsloth/) or our Docs. [PreviousNew 3x Faster Trainingchevron-left](https://unsloth.ai/docs/blog/3x-faster-training-packing) [NextQuantization-Aware Trainingchevron-right](https://unsloth.ai/docs/blog/quantization-aware-training-qat) Last updated 3 months ago Was this helpful? * [📐 Unsloth Loss Refactoring: Chunk & Fuse](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-loss-refactoring-chunk-and-fuse) * [🏁 Unsloth Gradient Checkpointing Enhancements](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhancements) * [🔓 Tiled MLP: Unlocking 500K+](https://unsloth.ai/docs/blog/500k-context-length-fine-tuning#tiled-mlp-unlocking-500k) Was this helpful? sun-brightdesktopmoon Copy # Original Unsloth version released April 2024 - LGPLv3 Licensed class Unsloth_Offloaded_Gradient_Checkpointer(torch.autograd.Function): @staticmethod @torch_amp_custom_fwd def forward(ctx, forward_function, hidden_states, *args): ctx.device = hidden_states.device saved_hidden_states = hidden_states.to("cpu", non_blocking = True) with torch.no_grad(): output = forward_function(hidden_states, *args) ctx.save_for_backward(saved_hidden_states) ctx.forward_function, ctx.args = forward_function, args return output @staticmethod @torch_amp_custom_bwd def backward(ctx, dY): (hidden_states,) = ctx.saved_tensors hidden_states = hidden_states.to(ctx.device, non_blocking = True).detach() hidden_states.requires_grad_(True) with torch.enable_grad(): (output,) = ctx.forward_function(hidden_states, *ctx.args) torch.autograd.backward(output, dY) return (None, hidden_states.grad,) + (None,)*len(ctx.args) chevron-downShow all 23 lines Copy model, tokenizer = FastLanguageModel.from_pretrained( ..., unsloth_tiled_mlp = True, ) sun-brightdesktopmoon --- # MiniMax-M2.7 - How to Run Locally | Unsloth Documentation [circle-check\ \ Introducing Unsloth Studio: a new web UI for local AI\ \ 🦥chevron-right](https://unsloth.ai/docs/new/studio) close MiniMax-M2.7 is a new open model for agentic coding and chat use-cases. The model achieves SOTA performance in SWE-Pro (56.22%) and Terminal Bench 2 (57.0%). The **230B parameters** (10B active) model is the successor to [MiniMax-M25](https://unsloth.ai/docs/models/tutorials/minimax-m25) and has a **200K context** window. The unquantized bf16 requires **457GB**. Unsloth Dynamic **4-bit** GGUF reduces the size to **108GB** **(-60%)** so it can run on a **128GB RAM** device**:** [**MiniMax-M2.7 GGUF**arrow-up-right](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF) All uploads use Unsloth [Dynamic 2.0](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) for SOTA quantization performance - so important layers are upcasted to higher bits (e.g. 8 or 16-bit). Thank you MiniMax for day zero access. circle-exclamation Do NOT use CUDA 13.2 to run any model as it may cause gibberish or poor outputs. NVIDIA is working on a fix. ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#usage-guide) ⚙️ Usage Guide The 4-bit dynamic quant `UD-IQ4_XS` uses **108GB** of disk space - this fits nicely on a **128GB unified memory Mac** for ~15+ tokens/s, and also works faster with a **1x16GB GPU and 96GB of RAM** for 25+ tokens/s. **2-bit** quants or the biggest 2-bit will fit on a 96GB device. For near **full precision**, use `Q8_0` (8-bit) which utilizes 243GB and will fit on a 256GB RAM device / Mac for 15+ tokens/s. circle-check For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower. ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#recommended-settings) Recommended Settings MiniMax recommends using the following parameters for best performance: `temperature=1.0`, `top_p = 0.95`, `top_k = 40`. Default Settings (Most Tasks) `temperature = 1.0` `top_p = 0.95` `top_k = 40` * **Maximum context window:** `196,608` * Default system prompt: Copy You are a helpful assistant. Your name is MiniMax-M2.7 and is built by MiniMax. [hashtag](https://unsloth.ai/docs/models/minimax-m27#run-minimax-m2.7-tutorials) Run MiniMax-M2.7 Tutorials: ----------------------------------------------------------------------------------------------------------------- To make MiniMax-M2.7 work on a 128GB RAM device, we will be utilizing the 4-bit [`UD-IQ4_XS` quantarrow-up-right](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF?show_file_info=UD-IQ4_XS%2FMiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf) . You can now run MiniMax-M2.7 in [llama.cpp](https://unsloth.ai/docs/models/minimax-m27#run-in-llama.cpp) and [Unsloth Studio](https://unsloth.ai/docs/models/minimax-m27#run-in-unsloth-studio) . circle-exclamation Do NOT use CUDA 13.2 to run any model as it may cause gibberish or poor outputs. NVIDIA is working on a fix. ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#run-in-unsloth-studio) 🦥 Run in Unsloth Studio MiniMax-M2.7 can now runs in [Unsloth Studio](https://unsloth.ai/docs/new/studio) , our new open-source web UI for local AI. Unsloth Studio lets you run models locally on **MacOS, Windows**, Linux and: * Search, download, [run GGUFs](https://unsloth.ai/docs/new/studio#run-models-locally) and safetensor models * [**Self-healing** tool calling](https://unsloth.ai/docs/new/studio#execute-code--heal-tool-calling) + **web search** * [**Code execution**](https://unsloth.ai/docs/new/studio#run-models-locally) (Python, Bash) * [Automatic inference](https://unsloth.ai/docs/new/studio#model-arena) parameter tuning (temp, top-p, etc.) * Uses llama.cpp for Fast CPU + GPU inference and CPU offloading ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FstfdTMsoBMmsbQsgQ1Ma%252Flandscape%2520clip%2520gemma.gif%3Falt%3Dmedia%26token%3Deec5f2f7-b97a-4c1c-ad01-5a041c3e4013&width=768&dpr=3&quality=100&sign=e4b21b2d&sv=2) 1 #### [hashtag](https://unsloth.ai/docs/models/minimax-m27#install-unsloth) Install Unsloth Run in your terminal: **MacOS, Linux, WSL:** **Windows PowerShell:** 2 #### [hashtag](https://unsloth.ai/docs/models/minimax-m27#launch-unsloth) Launch Unsloth **MacOS, Linux, WSL and Windows:** **Then open** `**http://localhost:8888**` **in your browser.** 3 #### [hashtag](https://unsloth.ai/docs/models/minimax-m27#search-and-download-minimax-m2.7) Search and download MiniMax-M2.7 On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time. You can choose `UD-IQ4_XS` (dynamic 4bit quant) or other quantized versions like `UD-Q4_K_XL` . If downloads get stuck, see [Hugging Face Hub, XET debugging](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging) Then go to the [Studio Chat](https://unsloth.ai/docs/new/studio/chat) tab and search for MiniMax-M2.7 in the search bar and download your desired model and quant. It will take some time to download due to the size so please wait. To ensure fast inference, ensure you have [enough RAM/VRAM](https://unsloth.ai/docs/models/minimax-m27#usage-guide) , otherwise inference will still work, but Unsloth will offload to your CPU. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh6qv7Mh2VqtdhZaixrnO%252FScreenshot%25202026-04-11%2520at%25206.46.55%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3De2568c00-86eb-452f-a4eb-10bcc0194ddf&width=768&dpr=3&quality=100&sign=aef71268&sv=2) 4 #### [hashtag](https://unsloth.ai/docs/models/minimax-m27#run-minimax-m2.7) Run MiniMax-M2.7 Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings. For more information, you can view our [Unsloth Studio inference guide](https://unsloth.ai/docs/new/studio/chat) . ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#run-in-llama.cpp) ✨ Run in llama.cpp circle-exclamation Do NOT use CUDA 13.2 to run any model as it may cause gibberish or poor outputs. NVIDIA is working on a fix. 1 Obtain the latest `llama.cpp` on [GitHub herearrow-up-right](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default. 2 If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ4\_XS) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 200K context length. Follow this for **most default** use-cases: 3 Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-IQ4\_XS (dynamic 4-bit quant) or other quantized versions like `UD-Q6_K_XL` . We recommend using our 4bit dynamic quant UD-IQ4\_XS to balance size and accuracy. If downloads get stuck, see [Hugging Face Hub, XET debugging](https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging) 4 You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. #### [hashtag](https://unsloth.ai/docs/models/minimax-m27#llama-server-and-openais-completion-library) 🦙 Llama-server & OpenAI's completion library To deploy MiniMax-M2.7 for production, we use `llama-server` or OpenAI API. In a new terminal say via tmux, deploy the model via: Then in a new terminal, after doing `pip install openai`, do: [hashtag](https://unsloth.ai/docs/models/minimax-m27#benchmarks) 📊 Benchmarks ----------------------------------------------------------------------------------- ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#gguf-benchmarks) GGUF Benchmarks Because MiniMax-M2.7 utilizes the same architecture as MiniMax-M2.5, GGUF quantization benchmarks for M2.7 should be very similar to M2.5. So, we'll refer to previous quant benchmark conducted for M2.5. ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhfO2gsbz2lWrZXg3ojyE%252FHCGBTzgboAASv_A.png%3Falt%3Dmedia%26token%3D7d6334ca-4f3c-4946-aacd-d55527375fce&width=768&dpr=3&quality=100&sign=d36871c9&sv=2) [Benjamin Marie (third-party) benchmarkedarrow-up-right](https://x.com/bnjmn_marie/status/2027043753484021810/photo/1) **MiniMax-M2.5** using **Unsloth GGUF quantizations** on a **750-prompt mixed suite** (LiveCodeBench v6, MMLU Pro, GPQA, Math500), reporting both **overall accuracy** and **relative error increase** (how much more often the quantized model makes mistakes vs. the original). Unsloth quants, no matter their precision perform much better than their non-Unsloth counterparts for both accuracy and relative error (despite being 8GB smaller). **Key results:** * **Best quality/size tradeoff here:** `**unsloth UD-Q4_K_XL**`**.** It’s the closest to Original: only **6.0 points** down, and “only” **+22.8%** more errors than baseline. * **Other Unsloth Q4 quants perform closely together (~64.5–64.9 accuracy).** `IQ4_NL`, `MXFP4_MOE`, and `UD-IQ2_XXS` are all basically the same quality on this benchmark, with **~33–35%** more errors than Original. * Unsloth GGUFs perform much better than other non-Unsloth GGUFs, e.g. see `lmstudio-community - Q4_K_M` (despite being 8GB smaller) and `AesSedai - IQ3_S`. ### [hashtag](https://unsloth.ai/docs/models/minimax-m27#official-benchmarks) Official Benchmarks ![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fn5Xz2P6kzHRH2sQGPsHH%252Fminimaxm2.7%2520model.jpg%3Falt%3Dmedia%26token%3D04f4b3fd-9d04-4e80-9f06-09afd8ce884d&width=768&dpr=3&quality=100&sign=77299bca&sv=2) [PreviousGLM-5.1chevron-left](https://unsloth.ai/docs/models/glm-5.1) [NextNVIDIA Nemotron 3chevron-right](https://unsloth.ai/docs/models/nemotron-3) Last updated 1 day ago Was this helpful? * [⚙️ Usage Guide](https://unsloth.ai/docs/models/minimax-m27#usage-guide) * [Recommended Settings](https://unsloth.ai/docs/models/minimax-m27#recommended-settings) * [Run MiniMax-M2.7 Tutorials:](https://unsloth.ai/docs/models/minimax-m27#run-minimax-m2.7-tutorials) * [🦥 Run in Unsloth Studio](https://unsloth.ai/docs/models/minimax-m27#run-in-unsloth-studio) * [✨ Run in llama.cpp](https://unsloth.ai/docs/models/minimax-m27#run-in-llama.cpp) * [📊 Benchmarks](https://unsloth.ai/docs/models/minimax-m27#benchmarks) * [GGUF Benchmarks](https://unsloth.ai/docs/models/minimax-m27#gguf-benchmarks) * [Official Benchmarks](https://unsloth.ai/docs/models/minimax-m27#official-benchmarks) Was this helpful? sun-brightdesktopmoon Copy curl -fsSL https://unsloth.ai/install.sh | sh Copy irm https://unsloth.ai/install.ps1 | iex Copy unsloth studio -H 0.0.0.0 -p 8888 Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp Copy export LLAMA_CACHE="unsloth/MiniMax-M2.7-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/MiniMax-M2.7-GGUF:UD-IQ4_XS \ --temp 1.0 \ --top-p 0.95 \ --top-k 40 Copy hf download unsloth/MiniMax-M2.7-GGUF \ --local-dir unsloth/MiniMax-M2.7-GGUF \ --include "*UD-IQ4_XS*" # Use "*Q8_0*" for 8-bit Copy ./llama.cpp/llama-cli \ --model unsloth/MiniMax-M2.7-GGUF/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \ --temp 1.0 \ --top-p 0.95 \ --top-k 40 Copy ./llama.cpp/llama-server \ --model unsloth/MiniMax-M2.7-GGUF/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \ --alias "unsloth/MiniMax-M2.7" \ --prio 3 \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --top-k 40 \ --port 8001 Copy from openai import OpenAI import json openai_client = OpenAI( base_url = "http://127.0.0.1:8001/v1", api_key = "sk-no-key-required", ) completion = openai_client.chat.completions.create( model = "unsloth/MiniMax-M2.7", messages = [{"role": "user", "content": "Create a Snake game."},], ) print(completion.choices[0].message.content) sun-brightdesktopmoon ---