# Table of Contents - [Unsloth Docs | Unsloth Documentation](#unsloth-docs-unsloth-documentation) - [Beginner? Start here! | Unsloth Documentation](#beginner-start-here-unsloth-documentation) - [FAQ + Is Fine-tuning Right For Me? | Unsloth Documentation](#faq-is-fine-tuning-right-for-me-unsloth-documentation) - [Unsloth Requirements | Unsloth Documentation](#unsloth-requirements-unsloth-documentation) - [Installing + Updating | Unsloth Documentation](#installing-updating-unsloth-documentation) - [Updating | Unsloth Documentation](#updating-unsloth-documentation) - [Pip Install | Unsloth Documentation](#pip-install-unsloth-documentation) - [Google Colab | Unsloth Documentation](#google-colab-unsloth-documentation) - [What Model Should I Use? | Unsloth Documentation](#what-model-should-i-use-unsloth-documentation) - [Unsloth Notebooks | Unsloth Documentation](#unsloth-notebooks-unsloth-documentation) - [Conda Install | Unsloth Documentation](#conda-install-unsloth-documentation) - [Windows Installation | Unsloth Documentation](#windows-installation-unsloth-documentation) - [Fine-tuning LLMs Guide | Unsloth Documentation](#fine-tuning-llms-guide-unsloth-documentation) - [LoRA Hyperparameters Guide | Unsloth Documentation](#lora-hyperparameters-guide-unsloth-documentation) - [Saving to VLLM | Unsloth Documentation](#saving-to-vllm-unsloth-documentation) - [Running & Saving Models | Unsloth Documentation](#running-saving-models-unsloth-documentation) - [Finetuning from Last Checkpoint | Unsloth Documentation](#finetuning-from-last-checkpoint-unsloth-documentation) - [Reinforcement Learning - DPO, ORPO & KTO | Unsloth Documentation](#reinforcement-learning-dpo-orpo-kto-unsloth-documentation) - [Unsloth Environment Flags | Unsloth Documentation](#unsloth-environment-flags-unsloth-documentation) - [Phi-4 Reasoning: How to Run & Fine-tune | Unsloth Documentation](#phi-4-reasoning-how-to-run-fine-tune-unsloth-documentation) - [DeepSeek-V3.1 | Unsloth Documentation](#deepseek-v3-1-unsloth-documentation) - [Vision Fine-tuning | Unsloth Documentation](#vision-fine-tuning-unsloth-documentation) - [Inference | Unsloth Documentation](#inference-unsloth-documentation) - [Tutorial: Train your own Reasoning model with GRPO | Unsloth Documentation](#tutorial-train-your-own-reasoning-model-with-grpo-unsloth-documentation) - [Training AI Agents with RL | Unsloth Documentation](#training-ai-agents-with-rl-unsloth-documentation) - [Saving to GGUF | Unsloth Documentation](#saving-to-gguf-unsloth-documentation) - [Saving to Ollama | Unsloth Documentation](#saving-to-ollama-unsloth-documentation) - [Troubleshooting | Unsloth Documentation](#troubleshooting-unsloth-documentation) - [Unsloth Benchmarks | Unsloth Documentation](#unsloth-benchmarks-unsloth-documentation) - [Kimi K2: How to Run Locally | Unsloth Documentation](#kimi-k2-how-to-run-locally-unsloth-documentation) - [Troubleshooting & FAQs | Unsloth Documentation](#troubleshooting-faqs-unsloth-documentation) - [gpt-oss: How to Run & Fine-tune | Unsloth Documentation](#gpt-oss-how-to-run-fine-tune-unsloth-documentation) - [Gemma 3: How to Run & Fine-tune | Unsloth Documentation](#gemma-3-how-to-run-fine-tune-unsloth-documentation) - [Text-to-Speech (TTS) Fine-tuning | Unsloth Documentation](#text-to-speech-tts-fine-tuning-unsloth-documentation) - [Multi-GPU Training with Unsloth | Unsloth Documentation](#multi-gpu-training-with-unsloth-unsloth-documentation) - [Continued Pretraining | Unsloth Documentation](#continued-pretraining-unsloth-documentation) - [Gemma 3n: How to Run & Fine-tune | Unsloth Documentation](#gemma-3n-how-to-run-fine-tune-unsloth-documentation) - [Cogito v2: How to Run Locally | Unsloth Documentation](#cogito-v2-how-to-run-locally-unsloth-documentation) - [Qwen3: How to Run & Fine-tune | Unsloth Documentation](#qwen3-how-to-run-fine-tune-unsloth-documentation) - [Llama 4: How to Run & Fine-tune | Unsloth Documentation](#llama-4-how-to-run-fine-tune-unsloth-documentation) - [Chat Templates | Unsloth Documentation](#chat-templates-unsloth-documentation) - [Devstral: How to Run & Fine-tune | Unsloth Documentation](#devstral-how-to-run-fine-tune-unsloth-documentation) - [DeepSeek-R1-0528: How to Run Locally | Unsloth Documentation](#deepseek-r1-0528-how-to-run-locally-unsloth-documentation) - [Training LLMs with Blackwell, RTX 50 series & Unsloth | Unsloth Documentation](#training-llms-with-blackwell-rtx-50-series-unsloth-unsloth-documentation) - [Qwen3-2507 | Unsloth Documentation](#qwen3-2507-unsloth-documentation) - [Qwen3-Coder: How to Run Locally | Unsloth Documentation](#qwen3-coder-how-to-run-locally-unsloth-documentation) - [Tutorial: How to Fine-tune gpt-oss | Unsloth Documentation](#tutorial-how-to-fine-tune-gpt-oss-unsloth-documentation) - [DeepSeek-V3-0324: How to Run Locally | Unsloth Documentation](#deepseek-v3-0324-how-to-run-locally-unsloth-documentation) - [Tutorials: How To Fine-tune & Run LLMs | Unsloth Documentation](#tutorials-how-to-fine-tune-run-llms-unsloth-documentation) - [DeepSeek-R1: How to Run Locally | Unsloth Documentation](#deepseek-r1-how-to-run-locally-unsloth-documentation) - [QwQ-32B: How to Run effectively | Unsloth Documentation](#qwq-32b-how-to-run-effectively-unsloth-documentation) - [DeepSeek-R1 Dynamic 1.58-bit | Unsloth Documentation](#deepseek-r1-dynamic-1-58-bit-unsloth-documentation) - [Datasets Guide | Unsloth Documentation](#datasets-guide-unsloth-documentation) - [Unsloth Dynamic 2.0 GGUFs | Unsloth Documentation](#unsloth-dynamic-2-0-ggufs-unsloth-documentation) - [Reinforcement Learning (RL) Guide | Unsloth Documentation](#reinforcement-learning-rl-guide-unsloth-documentation) - [Magistral: How to Run & Fine-tune | Unsloth Documentation](#magistral-how-to-run-fine-tune-unsloth-documentation) - [Tutorial: How to Finetune Llama-3 and Use In Ollama | Unsloth Documentation](#tutorial-how-to-finetune-llama-3-and-use-in-ollama-unsloth-documentation) - [Email Protection | Cloudflare](#email-protection-cloudflare) - [AI Engineer's 2025 | Unsloth Documentation](#ai-engineer-s-2025-unsloth-documentation) --- # Unsloth Docs | Unsloth Documentation OpenAI gpt-oss & all model types now supported! At [Unsloth](https://docs.unsloth.ai/) , our mission is to make AI as accurate and accessible as possible. Train, run, evaluate and save gpt-oss, Llama, DeepSeek, TTS, Qwen, Mistral, Gemma LLMs 2x faster with 70% less VRAM. Our docs will guide you through running & training your own model locally. [Get started](https://docs.unsloth.ai/get-started/beginner-start-here) [Our GitHub](https://github.com/unslothai/unsloth) [](https://docs.unsloth.ai/basics/deepseek-v3.1) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FOFWy2bZ6L6qr12m9fbEM%252Fdeepseek%2520v3.1%2520logo.png%3Falt%3Dmedia%26token%3Ddd75f159-9266-4208-995f-b71d8e2ed4d3&width=245&dpr=4&quality=100&sign=32847d2c&sv=2) **DeepSeek-V3.1** DeepSeek's new hybrid reasoning model. [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FX0pJKFv8zDMf4TJomAts%252Fgpt-oss%2520image.png%3Falt%3Dmedia%26token%3D60c73c0d-cf83-4269-9619-f4b71e25767a&width=245&dpr=4&quality=100&sign=d3cd6098&sv=2) **gpt-oss** Run OpenAI's new open-source models with Unsloth's bug fixes. [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdUKxTDoQUFZPpOixP1Cx%252Fsloth%2520with%2520comp.png%3Falt%3Dmedia%26token%3D16fbc4a3-3d03-4e6c-bc74-75cf1121c797&width=245&dpr=4&quality=100&sign=f2f7d004&sv=2) **Finetune gpt-oss Guide** Learn how to fine-tune gpt-oss locally with our tutorial. [šŸ§¬Fine-tuning LLMs Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) [šŸ“’Unsloth Notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) [šŸ”®All Our Models](https://docs.unsloth.ai/get-started/all-our-models) [šŸš€Tutorials: How To Fine-tune & Run LLMs](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms) [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBszehKqh4ex9879rI5jv%252FGemma%25203%2520text%2520only.png%3Falt%3Dmedia%26token%3Db66212ab-409b-4603-80fa-337bea439531&width=245&dpr=4&quality=100&sign=34b6d65b&sv=2) **Gemma 3n + 270M** Fine-tune & run Google's new multimodal models [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEj2zfXu3PPd39PvAmQtx%252Fqwen3-2507.png%3Falt%3Dmedia%26token%3Dc070db7b-bfe9-4a7f-9e75-bbd0b0a01a4d&width=245&dpr=4&quality=100&sign=39aeab4b&sv=2) **Qwen3-2507** Run the new SOTA Thinking & Instruct LLMs. [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FeDz30Gy6kQ8zzdMaxr5m%252Fqwen3-coder%25201920.png%3Falt%3Dmedia%26token%3Defad8f53-6d06-48bd-98e6-96bde543702d&width=245&dpr=4&quality=100&sign=45822cb4&sv=2) **Qwen3-Coder** Run Qwen's coding and agentic models. ### [](https://docs.unsloth.ai/#why-unsloth) šŸ¦„ Why Unsloth? * Unsloth simplifies model training locally and on platforms like Google Colab and Kaggle. Our streamlined workflow handles everything from model loading and quantization to training, evaluation, saving, exporting, and integration with inference engines like Ollama, llama.cpp and vLLM. * The **key advantage** of Unsloth is our active role in _**fixing critical bugs**_ in major models. We've collaborated directly with teams behind [Qwen3](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/) , [Meta (Llama 4)](https://github.com/ggml-org/llama.cpp/pull/12889) , [Mistral (Devstral)](https://app.gitbook.com/o/HpyELzcNe0topgVLGCZY/s/xhOjnexMCB3dmuQFQ2Zq/~/changes/618/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune) , [Google (Gemma 1ā€“3)](https://news.ycombinator.com/item?id=39671146) and [Microsoft (Phi-3/4)](https://simonwillison.net/2025/Jan/11/phi-4-bug-fixes) , contributing essential fixes that significantly boost accuracy. * Unsloth is the only training framework which supports **all model types** including [vision](https://docs.unsloth.ai/basics/vision-fine-tuning) , [text-to-speech (TTS)](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) , BERT, [reinforcement learning (RL)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) , video, and all transformer-based models. Unsloth is also highly customizable, allowing modifications in chat templates, dataset formatting and we provide user-friendly notebooks for many use-cases. ### [](https://docs.unsloth.ai/#key-features) ā­ Key Features * Supports **full-finetuning**, pretraining, 4-bit, 16-bit and **8-bit** training. * [MultiGPU](https://docs.unsloth.ai/basics/multi-gpu-training-with-unsloth) is in the works and soon to come! * Supports **all transformer-style models** including [TTS, STT](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) , multimodal, diffusion, [BERT](https://docs.unsloth.ai/get-started/unsloth-notebooks#other-important-notebooks) and more! * All kernels written in [OpenAI's Triton](https://openai.com/index/triton/) language. **Manual backprop engine**. * **0% loss in accuracy** - no approximation methods - all exact. * Unsloth Supports **Linux, Windows,** Google Colab, Kaggle, **NVIDIA** and soon **AMD** & **Intel setups**. Most use Unsloth through Colab which provides a free GPU to train with. See: [šŸ› ļøUnsloth Requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) ### [](https://docs.unsloth.ai/#quickstart) Quickstart **Install locally with pip (recommended)** for Linux devices: Copy pip install unsloth For Windows install instructions, see [here](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation) . [šŸ“„Installing + Updating](https://docs.unsloth.ai/get-started/installing-+-updating) ### [](https://docs.unsloth.ai/#what-is-fine-tuning-and-rl-why) What is Fine-tuning and RL? Why? **Fine-tuning** an LLM customizes its behavior, enhances domain knowledge, and optimizes performance for specific tasks. By fine-tuning a pre-trained model (e.g. Llama-3.1-8B) on a dataset, you can: * **Update Knowledge**: Introduce new domain-specific information. * **Customize Behavior**: Adjust the modelā€™s tone, personality, or response style. * **Optimize for Tasks**: Improve accuracy and relevance for specific use cases. [**Reinforcement Learning (RL)**](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) is where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**. * **Action:** What the model generates (e.g., a sentence). * **Reward:** A signal indicating how good or bad the model's action was (e.g., did the response follow instructions? was it helpful?). * **Environment:** The scenario or task the model is working on (e.g., answering a userā€™s question). **Example usecases of fine-tuning or RL:** * Train LLM to predict if a headline impacts a company positively or negatively. * Use historical customer interactions for more accurate and custom responses. * Train LLM on legal texts for contract analysis, case law research, and compliance. You can think of a fine-tuned model as a specialized agent designed to do specific tasks more effectively and efficiently. **Fine-tuning can replicate all of RAG's capabilities**, but not vice versa. [šŸ¤”FAQ + Is Fine-tuning Right For Me?](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) [šŸ’”Reinforcement Learning (RL) Guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FLrqITvuoKyiMl8mqfu5B%252Flarge%2520sloth%2520wave.png%3Falt%3Dmedia%26token%3D3077792b-90ff-459d-aa52-57abcf219adf&width=768&dpr=4&quality=100&sign=5c6df706&sv=2) [NextBeginner? Start here!](https://docs.unsloth.ai/get-started/beginner-start-here) Last updated 9 hours ago Was this helpful? --- # Beginner? Start here! | Unsloth Documentation OpenAI gpt-oss & all model types now supported! If you're a beginner, here might be the first questions you'll ask before your first fine-tune. You can also always ask our community by joining our [Reddit page](https://www.reddit.com/r/unsloth/) . [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) šŸ§¬[Fine-tuning LLMs Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) Step-by-step on how to fine-tune! Learn the core basics of training. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use) ā“[What Model Should I Use?](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use) Instruct or Base Model? How big should my dataset be? [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms) šŸš€[Tutorials: How To Fine-tune & Run LLMs](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms) How to Run & Fine-tune DeepSeek? What settings should I set when running Gemma 3? [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) šŸ¤”[FAQ + Is Fine-tuning Right For Me?](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) What can fine-tuning do for me? RAG vs. Fine-tuning? [](https://docs.unsloth.ai/get-started/installing-+-updating) šŸ“„[Installing + Updating](https://docs.unsloth.ai/get-started/installing-+-updating) How do I install Unsloth locally? How to update Unsloth? šŸ“ˆ[Datasets Guide](https://docs.unsloth.ai/basics/datasets-guide) How do I structure/prepare my dataset? How do I collect data? [](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) šŸ› ļø[Unsloth Requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) Does Unsloth work on my GPU? How much VRAM will I need? [](https://docs.unsloth.ai/basics/running-and-saving-models) šŸ–„ļø[Running & Saving Models](https://docs.unsloth.ai/basics/running-and-saving-models) How do I save my model locally? How do I run my model via Ollama or vLLM? šŸ§ [LoRA Hyperparameters Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) What happens when I change a parameter? What parameters should I change? ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FjT759hR4zq8ygzg1oEwI%252FLarge%2520sloth%2520Question%2520mark.png%3Falt%3Dmedia%26token%3Dca8d2f56-889a-4da8-8106-da88d22e69d2&width=768&dpr=4&quality=100&sign=1635f07f&sv=2) [PreviousUnsloth Docs](https://docs.unsloth.ai/) [NextUnsloth Requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) Last updated 1 month ago Was this helpful? --- # FAQ + Is Fine-tuning Right For Me? | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#understanding-fine-tuning) Understanding Fine-Tuning ------------------------------------------------------------------------------------------------------------------------------------------------------ Fine-tuning an LLM customizes its behavior, deepens its domain expertise, and optimizes its performance for specific tasks. By refining a pre-trained model (e.g. _Llama-3.1-8B_) with specialized data, you can: * **Update Knowledge** ā€“ Introduce new, domain-specific information that the base model didnā€™t originally include. * **Customize Behavior** ā€“ Adjust the modelā€™s tone, personality, or response style to fit specific needs or a brand voice. * **Optimize for Tasks** ā€“ Improve accuracy and relevance on particular tasks or queries your use-case requires. Think of fine-tuning as creating a specialized expert out of a generalist model. Some debate whether to use Retrieval-Augmented Generation (RAG) instead of fine-tuning, but fine-tuning can incorporate knowledge and behaviors directly into the model in ways RAG cannot. In practice, combining both approaches yields the best results - leading to greater accuracy, better usability, and fewer hallucinations. ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#real-world-applications-of-fine-tuning) Real-World Applications of Fine-Tuning Fine-tuning can be applied across various domains and needs. Here are a few practical examples of how it makes a difference: * **Sentiment Analysis for Finance** ā€“ Train an LLM to determine if a news headline impacts a company positively or negatively, tailoring its understanding to financial context. * **Customer Support Chatbots** ā€“ Fine-tune on past customer interactions to provide more accurate and personalized responses in a companyā€™s style and terminology. * **Legal Document Assistance** ā€“ Fine-tune on legal texts (contracts, case law, regulations) for tasks like contract analysis, case law research, or compliance support, ensuring the model uses precise legal language. [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#the-benefits-of-fine-tuning) The Benefits of Fine-Tuning ---------------------------------------------------------------------------------------------------------------------------------------------------------- Fine-tuning offers several notable benefits beyond what a base model or a purely retrieval-based system can provide: #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#fine-tuning-vs.-rag-whats-the-difference) Fine-Tuning vs. RAG: Whatā€™s the Difference? Fine-tuning can do mostly everything RAG can - but not the other way around. During training, fine-tuning embeds external knowledge directly into the model. This allows the model to handle niche queries, summarize documents, and maintain context without relying on an outside retrieval system. Thatā€™s not to say RAG lacks advantages as it is excels at accessing up-to-date information from external databases. It is in fact possible to retrieve fresh data with fine-tuning as well, however it is better to combine RAG with fine-tuning for efficiency. #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#task-specific-mastery) Task-Specific Mastery Fine-tuning deeply integrates domain knowledge into the model. This makes it highly effective at handling structured, repetitive, or nuanced queries, scenarios where RAG-alone systems often struggle. In other words, a fine-tuned model becomes a specialist in the tasks or content it was trained on. #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#independence-from-retrieval) Independence from Retrieval A fine-tuned model has no dependency on external data sources at inference time. It remains reliable even if a connected retrieval system fails or is incomplete, because all needed information is already within the modelā€™s own parameters. This self-sufficiency means fewer points of failure in production. #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#faster-responses) Faster Responses Fine-tuned models donā€™t need to call out to an external knowledge base during generation. Skipping the retrieval step means they can produce answers much more quickly. This speed makes fine-tuned models ideal for time-sensitive applications where every second counts. #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#custom-behavior-and-tone) Custom Behavior and Tone Fine-tuning allows precise control over how the model communicates. This ensures the modelā€™s responses stay consistent with a brandā€™s voice, adhere to regulatory requirements, or match specific tone preferences. You get a model that not only knows _what_ to say, but _how_ to say it in the desired style. #### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#reliable-performance) Reliable Performance Even in a hybrid setup that uses both fine-tuning and RAG, the fine-tuned model provides a reliable fallback. If the retrieval component fails to find the right information or returns incorrect data, the modelā€™s built-in knowledge can still generate a useful answer. This guarantees more consistent and robust performance for your system. [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#common-misconceptions) Common Misconceptions ---------------------------------------------------------------------------------------------------------------------------------------------- Despite fine-tuningā€™s advantages, a few myths persist. Letā€™s address two of the most common misconceptions about fine-tuning: ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#does-fine-tuning-add-new-knowledge-to-a-model) Does Fine-Tuning Add New Knowledge to a Model? **Yes - it absolutely can.** A common myth suggests that fine-tuning doesnā€™t introduce new knowledge, but in reality it does. If your fine-tuning dataset contains new domain-specific information, the model will learn that content during training and incorporate it into its responses. In effect, fine-tuning _can and does_ teach the model new facts and patterns from scratch. ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#is-rag-always-better-than-fine-tuning) Is RAG Always Better Than Fine-Tuning? **Not necessarily.** Many assume RAG will consistently outperform a fine-tuned model, but thatā€™s not the case when fine-tuning is done properly. In fact, a well-tuned model often matches or even surpasses RAG-based systems on specialized tasks. Claims that ā€œRAG is always betterā€ usually stem from fine-tuning attempts that werenā€™t optimally configured - for example, using incorrect [LoRA parameters](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) or insufficient training. Unsloth takes care of these complexities by automatically selecting the best parameter configurations for you. All you need is a good-quality dataset, and you'll get a fine-tuned model that performs to its fullest potential. ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#is-fine-tuning-expensive) Is Fine-Tuning Expensive? **Not at all!** While full fine-tuning or pretraining can be costly, these are not necessary (pretraining is especially not necessary). In most cases, LoRA or QLoRA fine-tuning can be done for minimal cost. In fact, with Unslothā€™s [free notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) for Colab or Kaggle, you can fine-tune models without spending a dime. Better yet, you can even fine-tune locally on your own device. [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#faq) FAQ: ----------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#why-you-should-combine-rag-and-fine-tuning) Why You Should Combine RAG & Fine-Tuning Instead of choosing between RAG and fine-tuning, consider using **both** together for the best results. Combining a retrieval system with a fine-tuned model brings out the strengths of each approach. Hereā€™s why: * **Task-Specific Expertise** ā€“ Fine-tuning excels at specialized tasks or formats (making the model an expert in a specific area), while RAG keeps the model up-to-date with the latest external knowledge. * **Better Adaptability** ā€“ A fine-tuned model can still give useful answers even if the retrieval component fails or returns incomplete information. Meanwhile, RAG ensures the system stays current without requiring you to retrain the model for every new piece of data. * **Efficiency** ā€“ Fine-tuning provides a strong foundational knowledge base within the model, and RAG handles dynamic or quickly-changing details without the need for exhaustive re-training from scratch. This balance yields an efficient workflow and reduces overall compute costs. ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#lora-vs.-qlora-which-one-to-use) LoRA vs. QLoRA: Which One to Use? When it comes to implementing fine-tuning, two popular techniques can dramatically cut down the compute and memory requirements: **LoRA** and **QLoRA**. Hereā€™s a quick comparison of each: * **LoRA (Low-Rank Adaptation)** ā€“ Fine-tunes only a small set of additional ā€œadapterā€ weight matrices (in 16-bit precision), while leaving most of the original model unchanged. This significantly reduces the number of parameters that need updating during training. * **QLoRA (Quantized LoRA)** ā€“ Combines LoRA with 4-bit quantization of the model weights, enabling efficient fine-tuning of very large models on minimal hardware. By using 4-bit precision where possible, it dramatically lowers memory usage and compute overhead. We recommend starting with **QLoRA**, as itā€™s one of the most efficient and accessible methods available. Thanks to Unslothā€™s [dynamic 4-bit](https://unsloth.ai/blog/dynamic-4bit) quants, the accuracy loss compared to standard 16-bit LoRA fine-tuning is now negligible. ### [](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#experimentation-is-key) Experimentation is Key Thereā€™s no single ā€œbestā€ approach to fine-tuning - only best practices for different scenarios. Itā€™s important to experiment with different methods and configurations to find what works best for your dataset and use case. A great starting point is **QLoRA (4-bit)**, which offers a very cost-effective, resource-friendly way to fine-tune models without heavy computational requirements. [šŸ§ LoRA Hyperparameters Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) [PreviousUnsloth Requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) [NextUnsloth Notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) Last updated 5 months ago Was this helpful? --- # Unsloth Requirements | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements#system-requirements) System Requirements ----------------------------------------------------------------------------------------------------------------------------- * **Operating System**: Works on Linux and Windows. * Supports NVIDIA GPUs since 2018+ including [Blackwell RTX 50](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) series. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow. * Unsloth should work on [AMD](https://github.com/unslothai/unsloth/pull/2520) and [Intel](https://github.com/unslothai/unsloth/pull/2621) GPUs! Apple/Silicon/MLX is in the works. * If you have different versions of torch, transformers etc., `pip install unsloth` will automatically install all the latest versions of those libraries so you don't need to worry about version compatibility. * Your device must have `xformers`, `torch`, `BitsandBytes` and `triton` support. Python 3.13 is now supported! [](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements#fine-tuning-vram-requirements) Fine-tuning VRAM requirements: -------------------------------------------------------------------------------------------------------------------------------------------------- How much GPU memory do I need for LLM fine-tuning using Unsloth? A common issue when you OOM or run out of memory is because you set your batch size too high. Set it to 1, 2, or 3 to use less VRAM. **For context length benchmarks, see** [**here**](https://docs.unsloth.ai/basics/unsloth-benchmarks#context-length-benchmarks) **.** Check this table for VRAM requirements sorted by model parameters and fine-tuning method. QLoRA uses 4-bit, LoRA uses 16-bit. Keep in mind that sometimes more VRAM is required depending on the model so these numbers are the absolute minimum: Model parameters QLoRA (4-bit) VRAM LoRA (16-bit) VRAM 3B 3.5 GB 8 GB 7B 5 GB 19 GB 8B 6 GB 22 GB 9B 6.5 GB 24 GB 11B 7.5 GB 29 GB 14B 8.5 GB 33 GB 27B 22GB 64GB 32B 26 GB 76 GB 40B 30GB 96GB 70B 41 GB 164 GB 81B 48GB 192GB 90B 53GB 212GB 405B 237 GB 950 GB [PreviousBeginner? Start here!](https://docs.unsloth.ai/get-started/beginner-start-here) [NextFAQ + Is Fine-tuning Right For Me?](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) Last updated 9 days ago Was this helpful? --- # Installing + Updating | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Unsloth works on Linux, Windows directly, Kaggle, Google Colab and more. See our [system requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) . **Recommended installation method:** Copy pip install unsloth [](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install) [Pip Install](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install) [Windows Installation](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation) [](https://docs.unsloth.ai/get-started/installing-+-updating/updating) [Updating](https://docs.unsloth.ai/get-started/installing-+-updating/updating) [](https://docs.unsloth.ai/get-started/installing-+-updating/conda-install) [Conda Install](https://docs.unsloth.ai/get-started/installing-+-updating/conda-install) [](https://docs.unsloth.ai/get-started/installing-+-updating/google-colab) [Google Colab](https://docs.unsloth.ai/get-started/installing-+-updating/google-colab) [PreviousAll Our Models](https://docs.unsloth.ai/get-started/all-our-models) [NextUpdating](https://docs.unsloth.ai/get-started/installing-+-updating/updating) Last updated 5 months ago Was this helpful? --- # Updating | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/installing-+-updating/updating#standard-updating-recommended) Standard Updating (recommended): ------------------------------------------------------------------------------------------------------------------------------------------ Copy pip install --upgrade unsloth unsloth_zoo ### [](https://docs.unsloth.ai/get-started/installing-+-updating/updating#updating-without-dependency-updates) Updating without dependency updates: Copy pip install --upgrade --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git pip install --upgrade --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth-zoo.git [](https://docs.unsloth.ai/get-started/installing-+-updating/updating#to-use-an-old-version-of-unsloth) To use an old version of Unsloth: ---------------------------------------------------------------------------------------------------------------------------------------------- Copy pip install --force-reinstall --no-cache-dir --no-deps unsloth==2025.1.5 '2025.1.5' is one of the previous old versions of Unsloth. Change it to a specific release listed on our [Github here](https://github.com/unslothai/unsloth/releases) . [PreviousInstalling + Updating](https://docs.unsloth.ai/get-started/installing-+-updating) [NextPip Install](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install) Last updated 4 months ago Was this helpful? --- # Pip Install | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install#recommended-installation) **Recommended installation:** ------------------------------------------------------------------------------------------------------------------------------------- **Install with pip (recommended) for the latest pip release:** Copy pip install unsloth Python 3.13 is now supported! **To install the latest main branch of Unsloth:** Copy pip uninstall unsloth unsloth_zoo -y && pip install --no-deps git+https://github.com/unslothai/unsloth_zoo.git && pip install --no-deps git+https://github.com/unslothai/unsloth.git If you're installing Unsloth in Jupyter, Colab, or other notebooks, be sure to prefix the command with `!`. This isn't necessary when using a terminal [](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install#uninstall--reinstall) Uninstall + Reinstall ------------------------------------------------------------------------------------------------------------------------- If you're still encountering dependency issues with Unsloth, many users have resolved them by forcing uninstalling and reinstalling Unsloth: Copy pip install --upgrade --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git pip install --upgrade --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth-zoo.git * * * [](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install#advanced-pip-installation) Advanced Pip Installation ---------------------------------------------------------------------------------------------------------------------------------- Do **NOT** use this if you have [Conda](https://docs.unsloth.ai/get-started/installing-+-updating/conda-install) . Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5` and CUDA versions. For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`. For example, if you have `torch 2.4` and `CUDA 12.1`, use: Copy pip install --upgrade pip pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git" Another example, if you have `torch 2.5` and `CUDA 12.4`, use: Copy pip install --upgrade pip pip install "unsloth[cu124-torch250] @ git+https://github.com/unslothai/unsloth.git" And other examples: Copy pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu118-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu118-torch240] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121-torch250] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu124-ampere-torch250] @ git+https://github.com/unslothai/unsloth.git" Or, run the below in a terminal to get the **optimal** pip installation command: Copy wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python - Or, run the below manually in a Python REPL: Copy try: import torch except: raise ImportError('Install torch via `pip install torch`') from packaging.version import Version as V v = V(torch.__version__) cuda = str(torch.version.cuda) is_ampere = torch.cuda.get_device_capability()[0] >= 8 if cuda != "12.1" and cuda != "11.8" and cuda != "12.4": raise RuntimeError(f"CUDA = {cuda} not supported!") if v <= V('2.1.0'): raise RuntimeError(f"Torch = {v} too old!") elif v <= V('2.1.1'): x = 'cu{}{}-torch211' elif v <= V('2.1.2'): x = 'cu{}{}-torch212' elif v < V('2.3.0'): x = 'cu{}{}-torch220' elif v < V('2.4.0'): x = 'cu{}{}-torch230' elif v < V('2.5.0'): x = 'cu{}{}-torch240' elif v < V('2.6.0'): x = 'cu{}{}-torch250' else: raise RuntimeError(f"Torch = {v} too new!") x = x.format(cuda.replace(".", ""), "-ampere" if is_ampere else "") print(f'pip install --upgrade pip && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git"') [PreviousUpdating](https://docs.unsloth.ai/get-started/installing-+-updating/updating) [NextWindows Installation](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation) Last updated 9 days ago Was this helpful? --- # Google Colab | Unsloth Documentation OpenAI gpt-oss & all model types now supported! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQzuUQL60uFWHpaAvDPYD%252FColab%2520Options.png%3Falt%3Dmedia%26token%3Dfb808ec5-20c5-4f42-949e-14ed26a44987&width=768&dpr=4&quality=100&sign=be097a14&sv=2) If you have never used a Colab notebook, a quick primer on the notebook itself: 1. **Play Button at each "cell".** Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter errors, simply rerun the cell you did not run. Another option is to click CTRL + ENTER if you don't want to click the play button. 2. **Runtime Button in the top toolbar.** You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, but is a good first try. 3. **Connect / Reconnect T4 button.** T4 is the free GPU Google is providing. It's quite powerful! The first installation cell looks like below: Remember to click the PLAY button in the brackets \[ \]. We grab our open source Github package, and install some other packages. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FIz2XUXhcmjheDtxfvbLA%252Fimage.png%3Falt%3Dmedia%26token%3Db9da0e5c-075c-48f8-8abb-5db6fdf9866b&width=768&dpr=4&quality=100&sign=e33e1780&sv=2) [PreviousConda Install](https://docs.unsloth.ai/get-started/installing-+-updating/conda-install) [NextFine-tuning LLMs Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) Last updated 3 months ago Was this helpful? --- # What Model Should I Use? | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#llama-qwen-mistral-phi-or) Llama, Qwen, Mistral, Phi or? --------------------------------------------------------------------------------------------------------------------------------------------------- When preparing for fine-tuning, one of the first decisions you'll face is selecting the right model. Here's a step-by-step guide to help you choose: 1 #### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#choose-a-model-that-aligns-with-your-usecase) Choose a model that aligns with your usecase * E.g. For image-based training, select a vision model such as _Llama 3.2 Vision_. For code datasets, opt for a specialized model like _Qwen Coder 2.5_. * **Licensing and Requirements**: Different models may have specific licensing terms and [system requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements#system-requirements) . Be sure to review these carefully to avoid compatibility issues. 2 #### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#assess-your-storage-compute-capacity-and-dataset) **Assess your storage, compute capacity and dataset** * Use our [VRAM guideline](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements#approximate-vram-requirements-based-on-model-parameters) to determine the VRAM requirements for the model youā€™re considering. * Your dataset will reflect the type of model you will use and amount of time it will take to train 3 #### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#select-a-model-and-parameters) **Select a Model and Parameters** * We recommend using the latest model for the best performance and capabilities. For instance, as of January 2025, the leading 70B model is _Llama 3.3_. * You can stay up to date by exploring our [model catalog](https://docs.unsloth.ai/get-started/all-our-models) to find the newest and relevant options. 4 #### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#choose-between-base-and-instruct-models) **Choose Between Base and Instruct Models** Further details below: [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model) Instruct or Base Model? ------------------------------------------------------------------------------------------------------------------------------------------ When preparing for fine-tuning, one of the first decisions you'll face is whether to use an instruct model or a base model. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-models) Instruct Models Instruct models are pre-trained with built-in instructions, making them ready to use without any fine-tuning. These models, including GGUFs and others commonly available, are optimized for direct usage and respond effectively to prompts right out of the box. Instruct models work with conversational chat templates like ChatML or ShareGPT. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#base-models) **Base Models** Base models, on the other hand, are the original pre-trained versions without instruction fine-tuning. These are specifically designed for customization through fine-tuning, allowing you to adapt them to your unique needs. Base models are compatible with instruction-style templates like [Alpaca or Vicuna](https://docs.unsloth.ai/basics/chat-templates) , but they generally do not support conversational chat templates out of the box. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#should-i-choose-instruct-or-base) Should I Choose Instruct or Base? The decision often depends on the quantity, quality, and type of your data: * **1,000+ Rows of Data**: If you have a large dataset with over 1,000 rows, it's generally best to fine-tune the base model. * **300ā€“1,000 Rows of High-Quality Data**: With a medium-sized, high-quality dataset, fine-tuning the base or instruct model are both viable options. * **Less than 300 Rows**: For smaller datasets, the instruct model is typically the better choice. Fine-tuning the instruct model enables it to align with specific needs while preserving its built-in instructional capabilities. This ensures it can follow general instructions without additional input unless you intend to significantly alter its functionality. * For information how how big your dataset should be, [see here](https://docs.unsloth.ai/basics/datasets-guide#how-big-should-my-dataset-be) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#fine-tuning-models-with-unsloth) Fine-tuning models with Unsloth ----------------------------------------------------------------------------------------------------------------------------------------------------------- You can change the model name to whichever model you like by matching it with model's name on Hugging Face e.g. 'unsloth/llama-3.1-8b-unsloth-bnb-4bit'. We recommend starting with **Instruct models**, as they allow direct fine-tuning using conversational chat templates (ChatML, ShareGPT etc.) and require less data compared to **Base models** (which uses Alpaca, Vicuna etc). Learn more about the differences between [instruct and base models here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model) . * Model names ending in `**unsloth-bnb-4bit**` indicate they are [**Unsloth dynamic 4-bit**](https://unsloth.ai/blog/dynamic-4bit) **quants**. These models consume slightly more VRAM than standard BitsAndBytes 4-bit models but offer significantly higher accuracy. * If a model name ends with just `**bnb-4bit**`, without "unsloth", it refers to a standard BitsAndBytes 4-bit quantization. * Models with **no suffix** are in their original **16-bit or 8-bit formats**. While they are the original models from the official model creators, we sometimes include important fixes - such as chat template or tokenizer fixes. So it's recommended to use our versions when available. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#experimentation-is-key) Experimentation is Key We recommend experimenting with both models when possible. Fine-tune each one and evaluate the outputs to see which aligns better with your goals. [PreviousFine-tuning LLMs Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) [NextLoRA Hyperparameters Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) Last updated 1 month ago Was this helpful? --- # Unsloth Notebooks | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Also see our GitHub repo for our notebooks: [github.com/unslothai/notebooks](https://github.com/unslothai/notebooks/) [GRPO (RL)](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) [Text-to-speech (TTS)](https://docs.unsloth.ai/get-started/unsloth-notebooks#text-to-speech-tts-notebooks) [Vision](https://docs.unsloth.ai/get-started/unsloth-notebooks#vision-multimodal-notebooks) [Use-case](https://docs.unsloth.ai/get-started/unsloth-notebooks#other-important-notebooks) ā€¢ Google Colab [](https://docs.unsloth.ai/get-started/unsloth-notebooks#tab-google-colab) ā€¢ Kaggle [](https://docs.unsloth.ai/get-started/unsloth-notebooks#tab-kaggle) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#standard-notebooks) Standard notebooks: * [**gpt-oss (20b)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) ā€¢ [Inference](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_(20B)-Inference.ipynb) ā€¢ [Fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) **\- new** * [**Qwen3-2507-4B**](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) ā€¢ [Thinking](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Thinking.ipynb) ā€¢ [Instruct](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) - new * [Gemma 3n (E4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) ā€¢ [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) ā€¢ [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) ā€¢ [Audio](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) * [Qwen3 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) * [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) ā€¢ [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) ā€¢ [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) ā€¢ [270M](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) - new * [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb) * [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) * [Llama 3.2 (1B + 3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [Qwen 2.5 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(7B)-Alpaca.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) GRPO (Reasoning RL) notebooks: * [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) **-** Advanced GRPO LoRA * [**DeepSeek-R1-0528-Qwen3 (8B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) (for multilingual usecase) * [Gemma 3 (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) * [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_(3B)_GRPO_LoRA.ipynb) - Advanced GRPO LoRA * [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb) * [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) * [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-GRPO.ipynb) * [Qwen2.5 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#text-to-speech-tts-notebooks) Text-to-Speech (TTS) notebooks: * [Sesame-CSM (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb) - new * [Orpheus-TTS (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) - Speech-to-Text (STT) * [Llasa-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_(1B).ipynb) * [Spark-TTS (0.5B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb) * [Oute-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#vision-multimodal-notebooks) Vision (Multimodal) notebooks: * [Gemma 3 (4B) vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) - new * [Llama 3.2 Vision (11B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb) * [Qwen2.5-VL (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral (12B) 2409](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Pixtral_(12B)-Vision.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#other-important-notebooks) Other important notebooks: * [**ModernBERT-large**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) **- new as of Aug 19** * [**Synthetic Data Generation Llama 3.2 (3B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) - new * [**Tool Calling**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) **- new** * [Mistral v0.3 Instruct (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Conversational.ipynb) * [Ollama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [ORPO](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-ORPO.ipynb) * [Continued Pretraining](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyr](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [_**Inference only**_](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Inference.ipynb) * [Llama 3 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Alpaca.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#specific-use-case-notebooks) Specific use-case notebooks: * [DPO Zephyr](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [**BERT - Text Classification**](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) **- new as of Aug 19** * [Ollama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [**Tool Calling**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb) **- new** * [Continued Pretraining (CPT)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) * [Multiple Datasets](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) by Flail * [KTO](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) by Jeffrey * [Inference chat UI](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Unsloth_Studio.ipynb) * [Conversational](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [ChatML](https://colab.research.google.com/drive/15F1xyn8497_dUbxZP4zWmPZ3PJx1Oymv?usp=sharing) * [Text Completion](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#rest-of-notebooks) Rest of notebooks: * [IBM Granite-3.3 (2B)](https://colab.research.google.com/github/ibm-granite-community/granite-snack-cookbook/blob/main/recipes/Fine_Tuning/FineTuning_with_Unsloth.ipynb) * [Gemma 2 (9B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(9B)-Alpaca.ipynb) * [Mistral NeMo (12B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Nemo_(12B)-Alpaca.ipynb) * [Phi-3.5 (mini)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3.5_Mini-Conversational.ipynb) * [Phi-3 (medium)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3_Medium-Conversational.ipynb) * [Gemma 2 (2B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(2B)-Alpaca.ipynb) * [Qwen 2.5 Coder (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(14B)-Conversational.ipynb) * [Mistral Small (22B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/TinyLlama_(1.1B)-Alpaca.ipynb) * [CodeGemma (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/CodeGemma_(7B)-Conversational.ipynb) * [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Alpaca.ipynb) * [Qwen2 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_(7B)-Alpaca.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#standard-notebooks-1) Standard notebooks: * [**gpt-oss (20B)**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-gpt-oss-(20B)-Fine-tuning.ipynb&accelerator=nvidiaTeslaT4) **- new** * [Gemma 3n (E4B)](https://www.kaggle.com/code/danielhanchen/gemma-3n-4b-multimodal-finetuning-inference) * [Qwen3 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(14B).ipynb) * [Gemma 3 (4B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(4B).ipynb) * [Phi-4 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4-Conversational.ipynb) * [Llama 3.1 (8B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Alpaca.ipynb) * [Llama 3.2 (1B + 3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(1B_and_3B)-Conversational.ipynb) * [Qwen 2.5 (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(7B)-Alpaca.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-notebooks) GRPO (Reasoning) notebooks: * [Qwen3 (4B)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3_(4B)-GRPO.ipynb&accelerator=nvidiaTeslaT4) * [Gemma 3 (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_(1B)-GRPO.ipynb) * [Llama 3.1 (8B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-GRPO.ipynb) * [Phi-4 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4_(14B)-GRPO.ipynb) * [Qwen 2.5 (3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_(3B)-GRPO.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#text-to-speech-tts-notebooks-1) Text-to-Speech (TTS) notebooks: * [Sesame-CSM (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Sesame_CSM_(1B)-TTS.ipynb) * [Orpheus-TTS (3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Orpheus_(3B)-TTS.ipynb) * [Whisper Large V3](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Whisper.ipynb) ā€“ Speech-to-Text * [Llasa-TTS (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llasa_TTS_(1B).ipynb) * [Spark-TTS (0.5B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Spark_TTS_(0_5B).ipynb) * [Oute-TTS (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Oute_TTS_(1B).ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#vision-multimodal-notebooks-1) Vision (Multimodal) notebooks: * [Llama 3.2 Vision (11B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_(11B)-Vision.ipynb) * [Qwen 2.5-VL (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_VL_(7B)-Vision.ipynb) * [Pixtral (12B) 2409](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Pixtral_(12B)-Vision.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#specific-use-case-notebooks-1) Specific use-case notebooks: * [Tool Calling](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_Coder_(14B)-Tool_Calling.ipynb) * [ORPO](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-ORPO.ipynb) * [Continued Pretraining](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_v0.3_(7B)-CPT.ipynb) * [DPO Zephyr](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Zephyr_(7B)-DPO.ipynb) * [Inference only](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_(8B)-Inference.ipynb) * [Ollama](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_(8B)-Ollama.ipynb) * [Text Completion](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_(7B)-Text_Completion.ipynb) * [CodeForces-cot (Reasoning)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeForces-cot-Finetune_for_Reasoning_on_CodeForces.ipynb) * [Unsloth Studio (chat UI)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Unsloth_Studio.ipynb) #### [](https://docs.unsloth.ai/get-started/unsloth-notebooks#rest-of-notebooks-1) Rest of notebooks: * [Gemma 2 (9B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(9B)-Alpaca.ipynb) * [Gemma 2 (2B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_(2B)-Alpaca.ipynb) * [CodeGemma (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeGemma_(7B)-Conversational.ipynb) * [Mistral NeMo (12B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Nemo_(12B)-Alpaca.ipynb) * [Mistral Small (22B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Small_(22B)-Alpaca.ipynb) * [TinyLlama (1.1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-TinyLlama_(1.1B)-Alpaca.ipynb) To view a complete list of all our Kaggle notebooks, [click here](https://github.com/unslothai/notebooks#-kaggle-notebooks) . Feel free to contribute to the notebooks by visiting our [repo](https://github.com/unslothai/notebooks) ! [PreviousFAQ + Is Fine-tuning Right For Me?](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) [NextAll Our Models](https://docs.unsloth.ai/get-started/all-our-models) Last updated 7 days ago Was this helpful? --- # Conda Install | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Only use Conda if you have it. If not, use [Pip](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install) . Select either `pytorch-cuda=11.8,12.1` for CUDA 11.8 or CUDA 12.1. We support `python=3.10,3.11,3.12`. Copy conda create --name unsloth_env \ python=3.11 \ pytorch-cuda=12.1 \ pytorch cudatoolkit xformers -c pytorch -c nvidia -c xformers \ -y conda activate unsloth_env pip install unsloth If you're looking to install Conda in a Linux environment, [read here](https://docs.anaconda.com/miniconda/) , or run the below: Copy mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh ~/miniconda3/bin/conda init bash ~/miniconda3/bin/conda init zsh [PreviousWindows Installation](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation) [NextGoogle Colab](https://docs.unsloth.ai/get-started/installing-+-updating/google-colab) Last updated 5 months ago Was this helpful? --- # Windows Installation | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#method-1-windows-directly) Method #1 - Windows directly: ----------------------------------------------------------------------------------------------------------------------------------------------- Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10. Need help or experiencing an error? Ask on our [GitHub Discussions](https://github.com/unslothai/unsloth/discussions/1849) thread for Windows support! 1 **Install NVIDIA Video Driver** You should install the latest version of your GPUs driver. Download drivers here: [NVIDIA GPU Drive](https://www.nvidia.com/Download/index.aspx) 2 **Install Visual Studio C++** You will need Visual Studio, with C++ installed. By default, C++ is not installed with Visual Studio, so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK. * Launch the Installer here: [Visual Studio Community Edition](https://visualstudio.microsoft.com/vs/community/) * In the installer, navigate to individual components and select all the options listed here: * **.NET Framework 4.8 SDK** * **.NET Framework 4.7.2 targeting pack** * **C# and Visual Basic Roslyn compilers** * **MSBuild** * **MSVC v143 - VS 2022 C++ x64/x86 build tools** * **C++ 2022 Redistributable Update** * **C++ CMake tools for Windows** * **C++/CLI support for v143 build tools (Latest)** * **MSBuild support for LLVM (clang-cl) toolset** * **C++ Clang Compiler for Windows (19.1.1)** * **Windows 11 SDK (10.0.22621.0)** * **Windows Universal CRT SDK** * **C++ 2022 Redistributable MSMs** **Easier method:** Or you can open an elevated Command Prompt or PowerShell: * Search for "cmd" or "PowerShell", right-click it, and choose "Run as administrator." * Paste and run this command (update the Visual Studio path if necessary): Copy "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" modify ^ --installPath "C:\Program Files\Microsoft Visual Studio\2022\Community" ^ --add Microsoft.Net.Component.4.8.SDK ^ --add Microsoft.Net.Component.4.7.2.TargetingPack ^ --add Microsoft.VisualStudio.Component.Roslyn.Compiler ^ --add Microsoft.Component.MSBuild ^ --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ^ --add Microsoft.VisualStudio.Component.VC.Redist.14.Latest ^ --add Microsoft.VisualStudio.Component.VC.CMake.Project ^ --add Microsoft.VisualStudio.Component.VC.CLI.Support ^ --add Microsoft.VisualStudio.Component.VC.Llvm.Clang ^ --add Microsoft.VisualStudio.ComponentGroup.ClangCL ^ --add Microsoft.VisualStudio.Component.Windows11SDK.22621 ^ --add Microsoft.VisualStudio.Component.Windows10SDK.19041 ^ --add Microsoft.VisualStudio.Component.UniversalCRT.SDK ^ --add Microsoft.VisualStudio.Component.VC.Redist.MSM 3 **Install Python and CUDA Toolkit** Follow the instructions to install [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive) . Then install Miniconda (which has Python) here: [https://www.anaconda.com/docs/getting-started/miniconda/install](https://www.anaconda.com/docs/getting-started/miniconda/install#quickstart-install-instructions) 4 **Install PyTorch** You will need the correct version of PyTorch that is compatible with your CUDA drivers, so make sure to select them carefully. [Install PyTorch](https://pytorch.org/get-started/locally/) 5 **Install Unsloth** Open Conda command prompt or your terminal with Python and run the command: Copy pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git" If you're using GRPO or plan to use vLLM, currently vLLM does not support Windows directly but only via WSL or Linux. ### [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#notes) **Notes** To run Unsloth directly on Windows: * Install Triton from this Windows fork and follow the instructions [here](https://github.com/woct0rdho/triton-windows) (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12) * In the SFTTrainer, set `dataset_num_proc=1` to avoid a crashing issue: Copy trainer = SFTTrainer( dataset_num_proc=1, ... ) ### [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#advanced-troubleshooting) **Advanced/Troubleshooting** For **advanced installation instructions** or if you see weird errors during installations: 1. Install `torch` and `triton`. Go to https://pytorch.org to install it. For example `pip install torch torchvision torchaudio triton` 2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers. 3. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to https://github.com/facebookresearch/xformers. Another option is to install `flash-attn` for Ampere GPUs. 4. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful. 5. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes` * * * [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#method-2-windows-using-powershell) Method #2 - Windows using PowerShell: --------------------------------------------------------------------------------------------------------------------------------------------------------------- #### [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#step-1-install-prerequisites) **Step 1: Install Prerequisites** 1. **Install NVIDIA CUDA Toolkit**: * Download and install the appropriate version of the **NVIDIA CUDA Toolkit** from [CUDA Downloads](https://developer.nvidia.com/cuda-downloads) . * Reboot your system after installation if prompted. * **Note**: No additional setup is required after installation for Unsloth. 2. **Install Microsoft C++ Build Tools**: * Download and install **Microsoft Build Tools for Visual Studio** from the [official website](https://visualstudio.microsoft.com/visual-cpp-build-tools/) . * During installation, select the **C++ build tools** workload. Ensure the **MSVC compiler toolset** is included. 3. **Set Environment Variables for the C++ Compiler**: * Open the **System Properties** window (search for "Environment Variables" in the Start menu). * Click **"Environment Variablesā€¦"**. * Add or update the following under **System variables**: * **CC**: Path to the `cl.exe` C++ compiler. Example (adjust if your version differs): Copy C:\Program Files\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\Hostx64\x64\cl.exe * **CXX**: Same path as `CC`. * Click **OK** to save changes. * Verify: Open a new terminal and type `cl`. It should show version info. 4. **Install Conda** 1. Download and install **Miniconda** from the [official website](https://docs.anaconda.com/miniconda/install/#quick-command-line-install) 2. Follow installation instruction from the website 3. To check whether `conda` is already installed, you can test it with `conda` in your PowerShell #### [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#step-2-run-the-unsloth-installation-script) **Step 2: Run the Unsloth Installation Script** 1. **Download the** [**unsloth\_windows.ps1**](https://github.com/unslothai/notebooks/blob/main/unsloth_windows.ps1) **PowerShell script by going through this link**. 2. **Open PowerShell as Administrator**: * Right-click Start and select **"Windows PowerShell (Admin)"**. 3. **Navigate to the scriptā€™s location** using `cd`: Copy cd path\to\script\folder 4. **Run the script**: Copy powershell.exe -ExecutionPolicy Bypass -File .\unsloth_windows.ps1 #### [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#step-3-using-unsloth) **Step 3: Using Unsloth** Activate the environment after the installation completes: Copy conda activate unsloth_env **Unsloth and its dependencies are now ready!** * * * [](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation#method-3-windows-via-wsl) Method #3 - Windows via WSL: --------------------------------------------------------------------------------------------------------------------------------------------- WSL is Window's subsystem for Linux. 1. Install python though [Python's official site](https://www.python.org/downloads/windows/) . 2. Start WSL (Should already be preinstalled). Open command prompt as admin then run: Copy wsl -d ubuntu Optional: If WSL is not preinstalled, go to the Microsoft store and search "Ubuntu" and the app that says Ubuntu will be WSL. Install it and run it and continue from there. 1. Update WSL: Copy sudo apt update && sudo apt upgrade -y 1. Install pip: Copy sudo apt install python3-pip 1. Install unsloth: Copy pip install unsloth 1. Optional: Install Jupyter Notebook to run in a Colab like environment: Copy pip3 install notebook 1. Launch Jupyter Notebook: Copy jupyter notebook 1. Download any Colab notebook from Unsloth, import it into your Jupyter Notebook, adjust the parameters as needed, and execute the script. [PreviousPip Install](https://docs.unsloth.ai/get-started/installing-+-updating/pip-install) [NextConda Install](https://docs.unsloth.ai/get-started/installing-+-updating/conda-install) Last updated 5 months ago Was this helpful? --- # Fine-tuning LLMs Guide | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-1.-understand-fine-tuning) 1\. Understand Fine-tuning --------------------------------------------------------------------------------------------------------------------------- Fine-tuning an LLM customizes its behavior, enhances + injects knowledge, and optimizes performance for domains/specific tasks. For example: * **GPT-4** serves as a base model; however, OpenAI fine-tuned it to better comprehend instructions and prompts, leading to the creation of ChatGPT-4 which everyone uses today. * ā€‹**DeepSeek-R1-Distill-Llama-8B** is a fine-tuned version of Llama-3.1-8B. DeepSeek utilized data generated by DeepSeek-R1, to fine-tune Llama-3.1-8B. This process, known as distillation (a subcategory of fine-tuning), injects the data into the Llama model to learn reasoning capabilities. With [Unsloth](https://github.com/unslothai/unsloth) , you can fine-tune for free on Colab, Kaggle, or locally with just 3GB VRAM by using our [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) . By fine-tuning a pre-trained model (e.g. Llama-3.1-8B) on a specialized dataset, you can: * **Update + Learn New Knowledge**: Inject and learn new domain-specific information. * **Customize Behavior**: Adjust the modelā€™s tone, personality, or response style. * **Optimize for Tasks**: Improve accuracy and relevance for specific use cases. **Example usecases**: * Train LLM to predict if a headline impacts a company positively or negatively. * Use historical customer interactions for more accurate and custom responses. * Fine-tune LLM on legal texts for contract analysis, case law research, and compliance. You can think of a fine-tuned model as a specialized agent designed to do specific tasks more effectively and efficiently. **Fine-tuning can replicate all of RAG's capabilities**, but not vice versa. #### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#fine-tuning-misconceptions) Fine-tuning misconceptions: You may have heard that fine-tuning does not make a model learn new knowledge or RAG performs better than fine-tuning. That is **false**. Read more FAQ + misconceptions [here](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#fine-tuning-vs.-rag-whats-the-difference) : [šŸ¤”FAQ + Is Fine-tuning Right For Me?](https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-2.-choose-the-right-model--method) 2\. Choose the Right Model + Method -------------------------------------------------------------------------------------------------------------------------------------------- If you're a beginner, it is best to start with a small instruct model like Llama 3.1 (8B) and experiment from there. You'll also need to decide between QLoRA and LoRA training: * **LoRA:** Fine-tunes small, trainable matrices in 16-bit without updating all model weights. * **QLoRA:** Combines LoRA with 4-bit quantization to handle very large models with minimal resources. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDpWv59wCNJUR38sVMjT6%252Fmodel%2520name%2520change.png%3Falt%3Dmedia%26token%3D1283a92d-9df7-4de0-b1a1-9fc7cc483381&width=768&dpr=4&quality=100&sign=f8be6cd7&sv=2) You can change the model name to whichever model you like by matching it with model's name on Hugging Face e.g. 'unsloth/llama-3.1-8b-unsloth-bnb-4bit'. We recommend starting with **Instruct models**, as they allow direct fine-tuning using conversational chat templates (ChatML, ShareGPT etc.) and require less data compared to **Base models** (which uses Alpaca, Vicuna etc). Learn more about the differences between [instruct and base models here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model) . * Model names ending in `**unsloth-bnb-4bit**` indicate they are [**Unsloth dynamic 4-bit**](https://unsloth.ai/blog/dynamic-4bit) **quants**. These models consume slightly more VRAM than standard BitsAndBytes 4-bit models but offer significantly higher accuracy. * If a model name ends with just `**bnb-4bit**`, without "unsloth", it refers to a standard BitsAndBytes 4-bit quantization. * Models with **no suffix** are in their original **16-bit or 8-bit formats**. While they are the original models from the official model creators, we sometimes include important fixes - such as chat template or tokenizer fixes. So it's recommended to use our versions when available. There are other settings which you can toggle: * `**max_seq_length = 2048**` ā€“ Controls context length. While Llama-3 supports 8192, we recommend 2048 for testing. Unsloth enables 4Ɨ longer context fine-tuning. * `**dtype = None**` ā€“ Defaults to None; use `torch.float16` or `torch.bfloat16` for newer GPUs. * `**load_in_4bit = True**` ā€“ Enables 4-bit quantization, reducing memory use 4Ɨ for fine-tuning. Disabling it allows for LoRA 16-bit fine-tuning to be enabled. * To enable full fine-tuning (FFT), set `full_finetuning = True`. For 8-bit fine-tuning, set `load_in_8bit = True`. **Note:** Only one training method can be set to `True` at a time. We recommend starting with QLoRA, as it is one of the most accessible and effective methods for training models. Our [dynamic 4-bit](https://unsloth.ai/blog/dynamic-4bit) quants, the accuracy loss for QLoRA compared to LoRA is now largely recovered. You can also do [Text-to-speech (TTS)](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) , [reasoning (GRPO)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) , [vision](https://docs.unsloth.ai/basics/vision-fine-tuning) , [reinforcement learning](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/reinforcement-learning-dpo-orpo-and-kto) (DPO, ORPO, KTO), [continued pretraining](https://docs.unsloth.ai/basics/continued-pretraining) , text completion and other training methodologies with Unsloth. Read our detailed guide on choosing the right model: [ā“What Model Should I Use?](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-3.-your-dataset) 3\. Your Dataset ------------------------------------------------------------------------------------------------------- For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized. * You will need to create a dataset usually with 2 columns - question and answer. The quality and amount will largely reflect the end result of your fine-tune so it's imperative to get this part right. * You can [synthetically generate data](https://docs.unsloth.ai/basics/datasets-guide#synthetic-data-generation) and structure your dataset (into QA pairs) using ChatGPT or local LLMs. * You can also use our new Synthetic Dataset notebook which automatically parses documents (PDFs, videos etc.), generates QA pairs and auto cleans data using local models like Llama 3.2. [Access the notebook here.](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) * Fine-tuning can learn from an existing repository of documents and continuously expand its knowledge base, but just dumping data alone wonā€™t work as well. For optimal results, curate a well-structured dataset, ideally as question-answer pairs. This enhances learning, understanding, and response accuracy. * But, that's not always the case, e.g. if you are fine-tuning a LLM for code, just dumping all your code data can actually enable your model to yield significant performance improvements, even without structured formatting. So it really depends on your use case. _**Read more about creating your dataset:**_ [šŸ“ˆDatasets Guide](https://docs.unsloth.ai/basics/datasets-guide) For most of our notebook examples, we utilize the [Alpaca dataset](https://docs.unsloth.ai/basics/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset) however other notebooks like Vision will use different datasets which may need images in the answer ouput as well. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-4.-understand-training-hyperparameters) 4\. Understand Training Hyperparameters ----------------------------------------------------------------------------------------------------------------------------------------------------- Learn how to choose the right hyperparameters using best practices from research and real-world experiments ā€” and understand how each one affects your model's performance. **For a complete guide on how hyperparameters affect training, see:** [šŸ§ LoRA Hyperparameters Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-5.-installing--requirements) 5\. Installing + Requirements -------------------------------------------------------------------------------------------------------------------------------- We would recommend beginners to utilise our pre-made [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) first as it's the easiest way to get started with guided steps. However, if installing locally is a must, you can install and use Unsloth - just make sure you have all the right requirements necessary. Also depending on the model and quantization you're using, you'll need enough VRAM and resources. See all the details here: [šŸ› ļøUnsloth Requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) Next, you'll need to install Unsloth. Unsloth currently only supports Windows and Linux devices. Once you install Unsloth, you can copy and paste our notebooks and use them in your own local environment. We have many installation methods: [šŸ“„Installing + Updating](https://docs.unsloth.ai/get-started/installing-+-updating) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-6.-training--evaluation) 6\. Training + Evaluation ------------------------------------------------------------------------------------------------------------------------ Once you have everything set, it's time to train! If something's not working, remember you can always change hyperparameters, your dataset etc. You will see a log of some numbers whilst training! This is the training loss, and your job is to set parameters to make this go to as close to 0.5 as possible! If your finetune is not reaching 1, 0.8 or 0.5, you might have to adjust some numbers. If your loss goes to 0, that's probably not a good sign as well! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxwOA09mtcimcQOCjP4PG%252Fimage.png%3Falt%3Dmedia%26token%3D39a0f525-6d4e-4c3b-af0d-82d8960d87be&width=768&dpr=4&quality=100&sign=853c0062&sv=2) The training loss will appear as numbers We generally recommend keeping the default settings unless you need longer training or larger batch sizes. * `**per_device_train_batch_size = 2**` ā€“ Increase for better GPU utilization but beware of slower training due to padding. Instead, increase `gradient_accumulation_steps` for smoother training. * `**gradient_accumulation_steps = 4**` ā€“ Simulates a larger batch size without increasing memory usage. * `**max_steps = 60**` ā€“ Speeds up training. For full runs, replace with `num_train_epochs = 1` (1ā€“3 epochs recommended to avoid overfitting). * `**learning_rate = 2e-4**` ā€“ Lower for slower but more precise fine-tuning. Try values like `1e-4`, `5e-5`, or `2e-5`. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#evaluation) Evaluation In order to evaluate, you could do manually evaluation by just chatting with the model and see if it's to your liking. You can also enable evaluation for Unsloth, but keep in mind it can be time-consuming depending on the dataset size. To speed up evaluation you can: reduce the evaluation dataset size or set `evaluation_steps = 100`. For testing, you can also take 20% of your training data and use that for testing. If you already used all of the training data, then you have to manually evaluate it. You can also use automatic eval tools like EleutherAIā€™s [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) . Keep in mind that automated tools may not perfectly align with your evaluation criteria. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-7.-running--saving-the-model) 7\. Running + Saving the model ---------------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRX9Byv1hlSpvmonT1PLw%252Fimage.png%3Falt%3Dmedia%26token%3D6043cd8c-c6a3-4cc5-a019-48baeed3b5a2&width=768&dpr=4&quality=100&sign=7c7ce43f&sv=2) Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6DXSlsHkN8cZiiAxAV0Z%252Fimage.png%3Falt%3Dmedia%26token%3D846307de-7386-4bbe-894e-7d9e572244fe&width=768&dpr=4&quality=100&sign=6482b95b&sv=2) Reminder Unsloth itself provides **2x faster inference** natively as well, so always do not forget to call `FastLanguageModel.for_inference(model)`. If you want the model to output longer responses, set `max_new_tokens = 128` to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well! ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#saving-the-model) Saving the model For saving and using your model in desired inference engines like Ollama, vLLM, Open WebUI, we can have more information here: [šŸ–„ļøRunning & Saving Models](https://docs.unsloth.ai/basics/running-and-saving-models) We can now save the finetuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face token via: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and add your token! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBz0YDi6Sc2oEP5QWXgSz%252Fimage.png%3Falt%3Dmedia%26token%3D33d9e4fd-e7dc-4714-92c5-bfa3b00f86c4&width=768&dpr=4&quality=100&sign=d6933a01&sv=2) After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzymBQrqwt4GUmCIN0Iec%252Fimage.png%3Falt%3Dmedia%26token%3D41a110e4-8263-426f-8fa7-cdc295cc8210&width=768&dpr=4&quality=100&sign=b2a207c3&sv=2) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-8.-were-done) 8\. We're done! --------------------------------------------------------------------------------------------------- You've successfully finetuned a language model and exported it to your desired inference engine with Unsloth! To learn more about finetuning tips and tricks, head over to our blogs which provide tremendous and educational value: [https://unsloth.ai/blog/](https://unsloth.ai/blog/) If you need any help on finetuning, you can also join our Discord server [here](https://discord.gg/unsloth) . Thanks for reading and hopefully this was helpful! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FPEvp4xsbVObJZ1lawDj8%252Fsloth%2520sparkling%2520square.png%3Falt%3Dmedia%26token%3D876bf67d-7470-4977-a6cc-3ee02cc9440b&width=768&dpr=4&quality=100&sign=d5ba19e6&sv=2) [PreviousGoogle Colab](https://docs.unsloth.ai/get-started/installing-+-updating/google-colab) [NextWhat Model Should I Use?](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use) Last updated 2 months ago Was this helpful? --- # LoRA Hyperparameters Guide | Unsloth Documentation OpenAI gpt-oss & all model types now supported! LoRA hyperparameters are adjustable parameters that control how Low-Rank Adaptation (LoRA) fine-tunes LLMs. With many options (such as learning rate and epochs) and millions of possible combinations, selecting the right values is crucial for achieving accuracy, stability, quality, and fewer hallucinations during fine-tuning. You'll learn the best practices for these parameters, based on insights from hundreds of research papers and experiments, and see how they impact the model. **While we recommend using Unsloth's defaults**, understanding these concepts will give you full control. The goal is to change hyperparameter numbers to increase accuracy while counteracting [**overfitting or underfitting**](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#overfitting-poor-generalization-too-specialized) . Overfitting occurs when the model memorizes the training data, harming its ability to generalize to new, unseen inputs. The objective is a model that generalizes well, not one that simply memorizes. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#but-what-is-lora) ā“But what is LoRA? In LLMs, we have model weights. Llama 70B has 70 billion numbers. Instead of changing all 70b numbers, we instead add thin matrices A and B to each weight, and optimize those. This means we only optimize 1% of weights. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fx6UtLPuzEudHY7SjLDAm%252Fimage.png%3Falt%3Dmedia%26token%3Dca891bda-e67e-4219-b74e-4a3a9c137700&width=768&dpr=4&quality=100&sign=6fd5d99b&sv=2) Instead of optimizing Model Weights (yellow), we optimize 2 thin matrices A and B. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#key-fine-tuning-hyperparameters) šŸ”¢ Key Fine-tuning Hyperparameters ----------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#learning-rate) **Learning Rate** Defines how much the modelā€™s weights are adjusted during each training step. * **Higher Learning Rates**: Lead to faster initial convergence but can cause training to become unstable or fail to find an optimal minimum if set too high. * **Lower Learning Rates**: Result in more stable and precise training but may require more epochs to converge, increasing overall training time. While low learning rates are often thought to cause underfitting, they actually can lead to **overfitting** or even prevent the model from learning. * **Typical Range**: `2e-4` (0.0002) to `5e-6` (0.000005). šŸŸ© _**For normal LoRA/QLoRA Fine-tuning**_, _we recommend_ `**2e-4**` _as a starting point._ šŸŸ¦ _**For Reinforcement Learning**_ _(DPO, GRPO etc.), we recommend_ `**5e-6**` **.** ā¬œ _**For Full Fine-tuning,**_ _lower learning rates are generally more appropriate._ ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#epochs) **Epochs** The number of times the model sees the full training dataset. * **More Epochs:** Can help the model learn better, but a high number can cause it to **memorize the training data**, hurting its performance on new tasks. * **Fewer Epochs:** Reduces training time and can prevent overfitting, but may result in an undertrained model if the number is insufficient for the model to learn the dataset's underlying patterns. * **Recommended:** 1-3 epochs. For most instruction-based datasets, training for more than 3 epochs offers diminishing returns and increases the risk of overfitting. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-or-qlora) **LoRA or QLoRA** LoRA uses 16-bit precision, while QLoRA is a 4-bit fine-tuning method. * **LoRA:** 16-bit fine-tuning. It's slightly faster and slightly more accurate, but consumes significantly more VRAM (4Ɨ more than QLoRA). Recommended for 16-bit environments and scenarios where maximum accuracy is required. * **QLoRA:** 4-bit fine-tuning. Slightly slower and marginally less accurate, but uses much less VRAM (4Ɨ less). šŸ¦„ _70B LLaMA fits in <48GB VRAM with QLoRA in Unsloth -_ [_more details here_](https://unsloth.ai/blog/llama3-3) _._ ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#hyperparameters-and-recommendations) Hyperparameters & Recommendations: Hyperparameter Function Recommended Settings **LoRA Rank** (`r`) Controls the number of trainable parameters in the LoRA adapter matrices. A higher rank increases model capacity but also memory usage. 8, 16, 32, 64, 128 Choose 16 or 32 **LoRA Alpha** (`lora_alpha`) Scales the strength of the fine-tuned adjustments in relation to the rank (`r`). `r` (standard) or `r * 2` (common heuristic). [More details here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-alpha-and-rank-relationship) . **LoRA Dropout** A regularization technique that randomly sets a fraction of LoRA activations to zero during training to prevent overfitting. **Not that useful**, so we default set it to 0. 0 (default) to 0.1 **Weight Decay** A regularization term that penalizes large weights to prevent overfitting and improve generalization. Don't use too large numbers! 0.01 (recommended) - 0.1 **Warmup Steps** Gradually increases the learning rate at the start of training. 5-10% of total steps **Scheduler Type** Adjusts the learning rate dynamically during training. `linear` or `cosine` **Seed (**`**random_state**`**)** A fixed number to ensure reproducibility of results. Any integer (e.g., `42`, `3407`) **Target Modules** Specify which parts of the model you want to apply LoRA adapters to ā€” either the attention, the MLP, or both. Attention: `q_proj, k_proj, v_proj, o_proj` MLP: `gate_proj, up_proj, down_proj` Recommended to target all major linear layers: `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#gradient-accumulation-and-batch-size-equivalency) šŸŒ³ Gradient Accumulation and Batch Size equivalency --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#effective-batch-size) Effective Batch Size Correctly configuring your batch size is critical for balancing training stability with your GPU's VRAM limitations. This is managed by two parameters whose product is the **Effective Batch Size**. **Effective Batch Size** = `batch_size * gradient_accumulation_steps` * A **larger Effective Batch Size** generally leads to smoother, more stable training. * A **smaller Effective Batch Size** may introduce more variance. While every task is different, the following configuration provides a great starting point for achieving a stable **Effective Batch Size** of 16, which works well for most fine-tuning tasks on modern GPUs. Parameter Description Recommended Setting **Batch Size** (`batch_size`) The number of samples processed in a single forward/backward pass on one GPU. **Primary Driver of VRAM Usage**. Higher values can improve hardware utilization and speed up training, but only if they fit in memory. 2 **Gradient Accumulation** (`gradient_accumulation_steps`) The number of micro-batches to process before performing a single model weight update. **Primary Driver of Training Time.** Allows simulation of a larger `batch_size` to conserve VRAM. Higher values increase training time per epoch. 8 **Effective Batch Size** (Calculated) The true batch size used for each gradient update. It directly influences training stability, quality, and final model performance. 4 to 16 Recommended: 16 (from 2 \* 8) ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#the-vram-and-performance-trade-off) The VRAM & Performance Trade-off Assume you want 32 samples of data per training step. Then you can use any of the following configurations: * `batch_size = 32, gradient_accumulation_steps = 1` * `batch_size = 16, gradient_accumulation_steps = 2` * `batch_size = 8, gradient_accumulation_steps = 4` * `batch_size = 4, gradient_accumulation_steps = 8` * `batch_size = 2, gradient_accumulation_steps = 16` * `batch_size = 1, gradient_accumulation_steps = 32` While all of these are equivalent for the model's weight updates, they have vastly different hardware requirements. The first configuration (`batch_size = 32`) uses the **most VRAM** and will likely fail on most GPUs. The last configuration (`batch_size = 1`) uses the **least VRAM,** but at the cost of slightly slower training**.** To avoid OOM (out of memory) errors, always prefer to set a smaller `batch_size` and increase `gradient_accumulation_steps` to reach your target **Effective Batch Size**. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#unsloth-gradient-accumulation-fix) šŸ¦„ Unsloth Gradient Accumulation Fix Gradient accumulation and batch sizes **are now fully equivalent in Unsloth** due to our bug fixes for gradient accumulation. We have implemented specific bug fixes for gradient accumulation that resolve a common issue where the two methods did not produce the same results. This was a known challenge in the wider community, but for Unsloth users, the two methods are now interchangeable. [Read our blog post](https://unsloth.ai/blog/gradient) for more details. Prior to our fixes, combinations of `batch_size` and `gradient_accumulation_steps` that yielded the same **Effective Batch Size** (i.e., `batch_size Ɨ gradient_accumulation_steps = 16`) did not result in equivalent training behavior. For example, configurations like `b1/g16`, `b2/g8`, `b4/g4`, `b8/g2`, and `b16/g1` all have an **Effective Batch Size** of 16, but as shown in the graph, the loss curves did not align when using standard gradient accumulation: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfbTkE4kv2tVwCIdyxWKe%252FBefore_-_Standard_gradient_accumulation_UQOFkUggudXuV9dzrh8MA.svg%3Falt%3Dmedia%26token%3Dc3297fd4-a96b-45d0-9925-0010165d85c6&width=768&dpr=4&quality=100&sign=7934c98b&sv=2) (Before - Standard Gradient Accumulation) After applying our fixes, the loss curves now align correctly, regardless of how the **Effective Batch Size** of 16 is achieved: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBtwCpRAye5yq1Yvhlwn2%252FAfter_-_Unsloth_gradient_accumulation_6Y4pJdJF0vruzradUpymY.svg%3Falt%3Dmedia%26token%3D3b53d4ca-44f2-45b2-af41-cbf6b24fc80b&width=768&dpr=4&quality=100&sign=4fee1fa1&sv=2) (After - šŸ¦„ Unsloth Gradient Accumulation) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-hyperparameters-in-unsloth) šŸ¦„ **LoRA Hyperparameters in Unsloth** --------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following demonstrates a standard configuration. **While Unsloth provides optimized defaults**, understanding these parameters is key to manual tuning. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FmxdGwpEiv0XReahK4zDf%252Fnotebook_parameter_screenshott.png%3Falt%3Dmedia%26token%3D2e11c53c-9a23-4132-8c6e-cb81f3d78172&width=768&dpr=4&quality=100&sign=cb897330&sv=2) 1. Copy r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 The rank (`r`) of the fine-tuning process. A larger rank uses more memory and will be slower, but can increase accuracy on complex tasks. We suggest ranks like 8 or 16 (for fast fine-tunes) and up to 128. Using a rank that is too large can cause overfitting and harm your model's quality. 2. Copy target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], For optimal performance, **LoRA should be applied to all major linear layers**. [Research has shown](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-target-modules-and-qlora-vs-lora) that targeting all major layers is crucial for matching the performance of full fine-tuning. While it's possible to remove modules to reduce memory usage, we strongly advise against it to preserve maximum quality as the savings are minimal. 3. Copy lora_alpha = 16, A scaling factor that controls the strength of the fine-tuned adjustments. Setting it equal to the rank (`r`) is a reliable baseline. A popular and effective heuristic is to set it to double the rank (`r * 2`), which makes the model learn more aggressively by giving more weight to the LoRA updates. [More details here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-alpha-and-rank-relationship) . 4. Copy lora_dropout = 0, # Supports any, but = 0 is optimized A regularization technique that helps [prevent overfitting](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#overfitting-poor-generalization-too-specialized) by randomly setting a fraction of the LoRA activations to zero during each training step. [Recent research suggests](https://arxiv.org/abs/2410.09692) that for **the short training runs** common in fine-tuning, `lora_dropout` may be an unreliable regularizer. šŸ¦„ _Unsloth's internal code can optimize training when_ `lora_dropout = 0`_, making it slightly faster, but we recommend a non-zero value if you suspect overfitting._ 5. Copy bias = "none", # Supports any, but = "none" is optimized Leave this as `"none"` for faster training and reduced memory usage. This setting avoids training the bias terms in the linear layers, which adds trainable parameters for little to no practical gain. 6. Copy use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context Options are `True`, `False`, and `"unsloth"`. šŸ¦„ _We recommend_ `"unsloth"` _as it reduces memory usage by an extra 30% and supports extremely long context fine-tunes. You can read more on_ [_our blog post about long context training_](https://unsloth.ai/blog/long-context) _._ 7. Copy random_state = 3407, The seed to ensure deterministic, reproducible runs. Training involves random numbers, so setting a fixed seed is essential for consistent experiments. 8. Copy use_rslora = False, # We support rank stabilized LoRA An advanced feature that implements [**Rank-Stabilized LoRA**](https://arxiv.org/abs/2312.03732) . If set to `True`, the effective scaling becomes `lora_alpha / sqrt(r)` instead of the standard `lora_alpha / r`. This can sometimes improve stability, particularly for higher ranks. [More details here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-alpha-and-rank-relationship) . 9. Copy loftq_config = None, # And LoftQ An advanced technique, as proposed in [**LoftQ**](https://arxiv.org/abs/2310.08659) , initializes LoRA matrices with the top 'r' singular vectors from the pretrained weights. This can improve accuracy but may cause a significant memory spike at the start of training. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#verifying-lora-weight-updates) **Verifying LoRA Weight Updates:** When validating that **LoRA** adapter weights have been updated after fine-tuning, avoid using **np.allclose()** for comparison. This method can miss subtle but meaningful changes, particularly in **LoRA A**, which is initialized with small Gaussian values. These changes may not register as significant under loose numerical tolerances. Thanks to [contributors](https://github.com/unslothai/unsloth/issues/3035) for this section. To reliably confirm weight updates, we recommend: * Using **checksum or hash comparisons** (e.g., MD5) * Computing the **sum of absolute differences** between tensors * Inspecting t**ensor statistics** (e.g., mean, variance) manually * Or using **np.array\_equal()** if exact equality is expected [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-alpha-and-rank-relationship) šŸ“LoRA Alpha and Rank relationship ------------------------------------------------------------------------------------------------------------------------------------------------------------------ It's best to set `lora_alpha = 2 * lora_rank` or `lora_alpha = lora_rank` W^\=W+Ī±rankƗAB\\hat{W} = W + \\frac{\\alpha}{\\text{rank}} \\times ABW^\=W+rankĪ±ā€‹Ć—AB ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfrlYmBPuCMy1GaXVYpIp%252Fimage.png%3Falt%3Dmedia%26token%3Db4cdfb81-8117-4852-a552-4869d27ea141&width=768&dpr=4&quality=100&sign=5a70cdfb&sv=2) rsLoRA other scaling options. sqrt(r) is the best. W^rslora\=W+Ī±rankƗAB\\hat{W}\_{\\text{rslora}} = W + \\frac{\\alpha}{\\sqrt{\\text{rank}}} \\times ABW^rsloraā€‹\=W+rankā€‹Ī±ā€‹Ć—AB The formula for LoRA is on the left. We need to scale the thin matrices A and B by alpha divided by the rank. **This means we should keep alpha/rank at least = 1**. According to the [rsLoRA (rank stabilized lora) paper](https://arxiv.org/abs/2312.03732) , we should instead scale alpha by the sqrt of the rank. Other options exist, but theoretically this is the optimum. The left plot shows other ranks and their perplexities (lower is better). To enable this, set `use_rslora = True` in Unsloth. Our recommendation is to set the **alpha to equal to the rank, or at least 2 times the rank.** This means alpha/rank = 1 or 2. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#lora-target-modules-and-qlora-vs-lora) šŸŽÆ LoRA Target Modules and QLoRA vs LoRA ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use: `target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",]` to target both **MLP** and **attention** layers to increase accuracy. **QLoRA uses 4-bit precision**, reducing VRAM usage by over 75%. **LoRA (16-bit)** is slightly more accurate and faster. According to empirical experiments and research papers like the original [QLoRA paper](https://arxiv.org/pdf/2305.14314) , it's best to apply LoRA to both attention and MLP layers. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FeTeDWK5yQhRv1YxmKyQ5%252Fimage.png%3Falt%3Dmedia%26token%3Da4d21361-9128-46e0-bc17-a31d212d16a1&width=768&dpr=4&quality=100&sign=2a279d50&sv=2) The chart shows RougeL scores (higher is better) for different target module configurations, comparing LoRA vs QLoRA. The first 3 dots show: 1. **QLoRA-All:** LoRA applied to all FFN/MLP and Attention layers. šŸ”„ _This performs best overall._ 2. **QLoRA-FFN**: LoRA only on FFN. Equivalent to: `gate_proj`, `up_proj`, `down_proj.` 3. **QLoRA-Attention**: LoRA applied only to Attention layers. Equivalent to: `q_proj`, `k_proj`, `v_proj`, `o_proj`. [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#training-on-completions-only-masking-out-inputs) šŸ˜Ž Training on completions only, masking out inputs -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [QLoRA paper](https://arxiv.org/pdf/2305.14314) shows that masking out inputs and **training only on completions** (outputs or assistant messages) can further **increase accuracy** by a few percentage points (_1%_). Below demonstrates how this is done in Unsloth: **NOT** training on completions only: **USER:** Hello what is 2+2? **ASSISTANT:** The answer is 4. **USER:** Hello what is 3+3? **ASSISTANT:** The answer is 6. **Training** on completions only: **USER:** ~Hello what is 2+2?~ **ASSISTANT:** The answer is 4. **USER:** ~Hello what is 3+3?~ **ASSISTANT:** The answer is 6**.** The QLoRA paper states that **training on completions only** increases accuracy by quite a bit, especially for multi-turn conversational finetunes! We do this in our [conversational notebooks here](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) . ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fe8oeF4J6Pe2kpDE4hosL%252Fimage.png%3Falt%3Dmedia%26token%3D7e59cb98-10d4-4563-9e25-26d3f3fb35cb&width=768&dpr=4&quality=100&sign=58852d62&sv=2) To enable **training on completions** in Unsloth, you will need to define the instruction and assistant parts. šŸ¦„ _We plan to further automate this for you in the future!_ For Llama 3, 3.1, 3.2, 3.3 and 4 models, you define the parts as follows: Copy from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", ) For Gemma 2, 3, 3n models, you define the parts as follows: Copy from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part = "user\n", response_part = "model\n", ) [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting) šŸ”‘ **Avoiding Overfitting & Underfitting** ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#overfitting-poor-generalization-too-specialized) **Overfitting** (Poor Generalization/Too Specialized) The model memorizes the training data, including its statistical noise, and consequently fails to generalize to unseen data. If your training loss drops below 0.2, your model is likely **overfitting** ā€” meaning it may perform poorly on unseen tasks. One simple trick is LoRA alpha scaling ā€” just multiply the alpha value of each LoRA matrix by 0.5. This effectively scales down the impact of fine-tuning. **This is closely related to merging / averaging weights.** You can take the original base (or instruct) model, add the LoRA weights, then divide the result by 2. This gives you an averaged model ā€” which is functionally equivalent to reducing the `alpha` by half. **Solution:** * **Adjust the learning rate:** A high learning rate often leads to overfitting, especially during short training runs. For longer training, a higher learning rate may work better. Itā€™s best to experiment with both to see which performs best. * **Reduce the number of training epochs**. Stop training after 1, 2, or 3 epochs. * **Increase** `weight_decay`. A value of `0.01` or `0.1` is a good starting point. * **Increase** `lora_dropout`. Use a value like `0.1` to add regularization. * **Increase batch size or gradient accumulation steps**. * **Dataset expansion** \- make your dataset larger by combining or concatenating open source datasets with your dataset. Choose higher quality ones. * **Evaluation early stopping** - enable evaluation and stop when the evaluation loss increases for a few steps. * **LoRA Alpha Scaling** - scale the alpha down after training and during inference - this will make the finetune less pronounced. * **Weight averaging** - literally add the original instruct model and the finetune and divide the weights by 2. ### [](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#underfitting-too-generic) **Underfitting** (Too Generic) The model fails to capture the underlying patterns in the training data, often due to insufficient complexity or training duration. **Solution:** * **Adjust the Learning Rate:** If the current rate is too low, increasing it may speed up convergence, especially for short training runs. For longer runs, try lowering the learning rate instead. Test both approaches to see which works best. * **Increase Training Epochs:** Train for more epochs, but monitor validation loss to avoid overfitting. * **Increase LoRA Rank** (`r`) and alpha: Rank should at least equal to the alpha number, and rank should be bigger for smaller models/more complex datasets; it usually is between 4 and 64. * **Use a More Domain-Relevant Dataset**: Ensure the training data is high-quality and directly relevant to the target task. * **Decrease batch size to 1**. This will cause the model to update more vigorously. Fine-tuning has no single "best" approach, only best practices. Experimentation is key to finding what works for your specific needs. Our notebooks automatically set optimal parameters based on many papers research and our experiments, giving you a great starting point. Happy fine-tuning! _**Acknowledgements:**_ _A huge thank you to_ [_Eyera_](https://huggingface.co/Orenguteng) _for contributing to this guide!_ [PreviousWhat Model Should I Use?](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/what-model-should-i-use) [NextDeepSeek-V3.1](https://docs.unsloth.ai/basics/deepseek-v3.1) Last updated 28 days ago Was this helpful? --- # Saving to VLLM | Unsloth Documentation OpenAI gpt-oss & all model types now supported! To save to 16bit for VLLM, use: Copy model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "") To merge to 4bit to load on HuggingFace, first call `merged_4bit`. Then use `merged_4bit_forced` if you are certain you want to merge to 4bit. I highly discourage you, unless you know what you are going to do with the 4bit model (ie for DPO training for eg or for HuggingFace's online inference engine) Copy model.save_pretrained_merged("model", tokenizer, save_method = "merged_4bit",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "") To save just the LoRA adapters, either use: Copy model.save_pretrained(...) AND tokenizer.save_pretrained(...) Or just use our builtin function to do that: Copy model.save_pretrained_merged("model", tokenizer, save_method = "lora",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "") [PreviousSaving to Ollama](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama) [NextTroubleshooting](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting) Last updated 1 year ago Was this helpful? --- # Running & Saving Models | Unsloth Documentation OpenAI gpt-oss & all model types now supported! You can also run your fine-tuned models by using [Unsloth's 2x faster inference](https://docs.unsloth.ai/basics/running-and-saving-models/inference) . [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) [Saving to GGUF](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama) [Saving to Ollama](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-vllm) [Saving to VLLM](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-vllm) [](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting) [Troubleshooting](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting) [Inference](https://docs.unsloth.ai/basics/running-and-saving-models/inference) [PreviousCogito v2: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally) [NextSaving to GGUF](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) Last updated 6 months ago Was this helpful? --- # Finetuning from Last Checkpoint | Unsloth Documentation OpenAI gpt-oss & all model types now supported! You must edit the `Trainer` first to add `save_strategy` and `save_steps`. Below saves a checkpoint every 50 steps to the folder `outputs`. Copy trainer = SFTTrainer( .... args = TrainingArguments( .... output_dir = "outputs", save_strategy = "steps", save_steps = 50, ), ) Then in the trainer do: Copy trainer_stats = trainer.train(resume_from_checkpoint = True) Which will start from the latest checkpoint and continue training. ### [](https://docs.unsloth.ai/basics/finetuning-from-last-checkpoint#wandb-integration) Wandb Integration Copy # Install library !pip install wandb --upgrade # Setting up Wandb !wandb login import os os.environ["WANDB_PROJECT"] = "" os.environ["WANDB_LOG_MODEL"] = "checkpoint" Then in `TrainingArguments()` set Copy report_to = "wandb", logging_steps = 1, # Change if needed save_steps = 100 # Change if needed run_name = "" # (Optional) To train the model, do `trainer.train()`; to resume training, do Copy import wandb run = wandb.init() artifact = run.use_artifact('//', type='model') artifact_dir = artifact.download() trainer.train(resume_from_checkpoint=artifact_dir) [](https://docs.unsloth.ai/basics/finetuning-from-last-checkpoint#how-do-i-do-early-stopping) ā“How do I do Early Stopping? ------------------------------------------------------------------------------------------------------------------------------- If you want to stop or pause the finetuning / training run since the evaluation loss is not decreasing, then you can use early stopping which stops the training process. Use `EarlyStoppingCallback`. As usual, set up your trainer and your evaluation dataset. The below is used to stop the training run if the `eval_loss` (the evaluation loss) is not decreasing after 3 steps or so. Copy from trl import SFTConfig, SFTTrainer trainer = SFTTrainer( args = SFTConfig( fp16_full_eval = True, per_device_eval_batch_size = 2, eval_accumulation_steps = 4, output_dir = "training_checkpoints", # location of saved checkpoints for early stopping save_strategy = "steps", # save model every N steps save_steps = 10, # how many steps until we save the model save_total_limit = 3, # keep ony 3 saved checkpoints to save disk space eval_strategy = "steps", # evaluate every N steps eval_steps = 10, # how many steps until we do evaluation load_best_model_at_end = True, # MUST USE for early stopping metric_for_best_model = "eval_loss", # metric we want to early stop on greater_is_better = False, # the lower the eval loss, the better ), model = model, tokenizer = tokenizer, train_dataset = new_dataset["train"], eval_dataset = new_dataset["test"], ) We then add the callback which can also be customized: Copy from transformers import EarlyStoppingCallback early_stopping_callback = EarlyStoppingCallback( early_stopping_patience = 3, # How many steps we will wait if the eval loss doesn't decrease # For example the loss might increase, but decrease after 3 steps early_stopping_threshold = 0.0, # Can set higher - sets how much loss should decrease by until # we consider early stopping. For eg 0.01 means if loss was # 0.02 then 0.01, we consider to early stop the run. ) trainer.add_callback(early_stopping_callback) Then train the model as usual via `trainer.train() .` [PreviousVision Fine-tuning](https://docs.unsloth.ai/basics/vision-fine-tuning) [NextTroubleshooting & FAQs](https://docs.unsloth.ai/basics/troubleshooting-and-faqs) Last updated 1 month ago Was this helpful? --- # Reinforcement Learning - DPO, ORPO & KTO | Unsloth Documentation OpenAI gpt-oss & all model types now supported! DPO (Direct Preference Optimization), ORPO (Odds Ratio Preference Optimization), PPO, KTO Reward Modelling all work with Unsloth. We have Google Colab notebooks for reproducing GRPO, ORPO, DPO Zephyr, KTO and SimPO: * [GRPO notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) * [ORPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-ORPO.ipynb) * [DPO Zephyr notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_(7B)-DPO.ipynb) * [KTO notebook](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) * [SimPO notebook](https://colab.research.google.com/drive/1Hs5oQDovOay4mFA6Y9lQhVJ8TnbFLFh2?usp=sharing) We're also in šŸ¤—Hugging Face's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth) . [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/reinforcement-learning-dpo-orpo-and-kto#dpo-code) DPO Code --------------------------------------------------------------------------------------------------------------------------------- Copy python import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Optional set GPU device ID from unsloth import FastLanguageModel, PatchDPOTrainer from unsloth import is_bfloat16_supported PatchDPOTrainer() import torch from transformers import TrainingArguments from trl import DPOTrainer model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/zephyr-sft-bnb-4bit", max_seq_length = max_seq_length, dtype = None, load_in_4bit = True, ) # Do model patching and add fast LoRA weights model = FastLanguageModel.get_peft_model( model, r = 64, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 64, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, max_seq_length = max_seq_length, ) dpo_trainer = DPOTrainer( model = model, ref_model = None, args = TrainingArguments( per_device_train_batch_size = 4, gradient_accumulation_steps = 8, warmup_ratio = 0.1, num_train_epochs = 3, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", seed = 42, output_dir = "outputs", ), beta = 0.1, train_dataset = YOUR_DATASET_HERE, # eval_dataset = YOUR_DATASET_HERE, tokenizer = tokenizer, max_length = 1024, max_prompt_length = 512, ) dpo_trainer.train() [PreviousTutorial: Train your own Reasoning model with GRPO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) [NextTraining AI Agents with RL](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl) Last updated 1 month ago Was this helpful? --- # Unsloth Environment Flags | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Environment variable Purpose `os.environ["UNSLOTH_RETURN_LOGITS"] = "1"` Forcibly returns logits - useful for evaluation if logits are needed. `os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"` Disables auto compiler. Could be useful to debug incorrect finetune results. `os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"` Disables fast generation for generic models. `os.environ["UNSLOTH_ENABLE_LOGGING"] = "1"` Enables auto compiler logging - useful to see which functions are compiled or not. `os.environ["UNSLOTH_FORCE_FLOAT32"] = "1"` On float16 machines, use float32 and not float16 mixed precision. Useful for Gemma 3. `os.environ["UNSLOTH_STUDIO_DISABLED"] = "1"` Disables extra features. `os.environ["UNSLOTH_COMPILE_DEBUG"] = "1"` Turns on extremely verbose `torch.compile`logs. `os.environ["UNSLOTH_COMPILE_MAXIMUM"] = "0"` Enables maximum `torch.compile`optimizations - not recommended. `os.environ["UNSLOTH_COMPILE_IGNORE_ERRORS"] = "1"` Can turn this off to enable fullgraph parsing. `os.environ["UNSLOTH_FULLGRAPH"] = "0"` Enable `torch.compile` fullgraph mode `os.environ["UNSLOTH_DISABLE_AUTO_UPDATES"] = "1"` Forces no updates to `unsloth-zoo` Another possiblity is maybe the model uploads we uploaded are corrupted, but unlikely. Try the following: Copy model, tokenizer = FastVisionModel.from_pretrained( "Qwen/Qwen2-VL-7B-Instruct", use_exact_model_name = True, ) [PreviousTroubleshooting & FAQs](https://docs.unsloth.ai/basics/troubleshooting-and-faqs) [NextTraining LLMs with Blackwell, RTX 50 series & Unsloth](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) Last updated 5 months ago Was this helpful? --- # Phi-4 Reasoning: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Microsoft's new Phi-4 reasoning models are now supported in Unsloth. The 'plus' variant performs on par with OpenAI's o1-mini, o3-mini and Sonnet 3.7. The 'plus' and standard reasoning models are 14B parameters while the 'mini' has 4B parameters. All Phi-4 reasoning uploads use our [Unsloth Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) methodology. #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#phi-4-reasoning-unsloth-dynamic-2.0-uploads) **Phi-4 reasoning - Unsloth Dynamic 2.0 uploads:** Dynamic 2.0 GGUF (to run) Dynamic 4-bit Safetensor (to finetune/deploy) * [Reasoning-plus](https://huggingface.co/unsloth/Phi-4-reasoning-plus-GGUF/) (14B) * [Reasoning](https://huggingface.co/unsloth/Phi-4-reasoning-GGUF) (14B) * [Mini-reasoning](https://huggingface.co/unsloth/Phi-4-mini-reasoning-GGUF/) (4B) * [Reasoning-plus](https://huggingface.co/unsloth/Phi-4-reasoning-plus-unsloth-bnb-4bit) * [Reasoning](https://huggingface.co/unsloth/phi-4-reasoning-unsloth-bnb-4bit) * [Mini-reasoning](https://huggingface.co/unsloth/Phi-4-mini-reasoning-unsloth-bnb-4bit) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#running-phi-4-reasoning) šŸ–„ļø **Running Phi-4 reasoning** -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings According to Microsoft, these are the recommended settings for inference: * **Temperature = 0.8** * Top\_P = 0.95 ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#phi-4-reasoning-chat-templates) **Phi-4 reasoning Chat templates** Please ensure you use the correct chat template as the 'mini' variant has a different one. #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#phi-4-mini) **Phi-4-mini:** Copy <|system|>Your name is Phi, an AI math expert developed by Microsoft.<|end|><|user|>How to solve 3*x^2+4*x+5=1?<|end|><|assistant|> #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#phi-4-reasoning-and-phi-4-reasoning-plus) **Phi-4-reasoning and Phi-4-reasoning-plus:** This format is used for general conversation and instructions: Copy <|im_start|>system<|im_sep|>You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: {Thought section} {Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines:<|im_end|><|im_start|>user<|im_sep|>What is 1+1?<|im_end|><|im_start|>assistant<|im_sep|> Yes, the chat template/prompt format is this long! ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#ollama-run-phi-4-reasoning-tutorial) šŸ¦™ Ollama: Run Phi-4 reasoning Tutorial 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails. We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload. Copy ollama run hf.co/unsloth/Phi-4-mini-reasoning-GGUF:Q4_K_XL ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#llama.cpp-run-phi-4-reasoning-tutorial) šŸ“– Llama.cpp: Run Phi-4 reasoning Tutorial You must use `--jinja` in llama.cpp to enable reasoning for the models, expect for the 'mini' variant. Otherwise no token will be provided. 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Phi-4-mini-reasoning-GGUF", local_dir = "unsloth/Phi-4-mini-reasoning-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) 1. Run the model in conversational mode in llama.cpp. You must use `--jinja` in llama.cpp to enable reasoning for the models. This is however not needed if you're using the 'mini' variant. Copy ./llama.cpp/llama-cli \ --model unsloth/Phi-4-mini-reasoning-GGUF/Phi-4-mini-reasoning-UD-Q4_K_XL.gguf \ --threads -1 \ --n-gpu-layers 99 \ --prio 3 \ --temp 0.8 \ --top-p 0.95 \ --jinja \ --min_p 0.00 \ --ctx-size 32768 \ --seed 3407 [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune#fine-tuning-phi-4-with-unsloth) šŸ¦„ Fine-tuning Phi-4 with Unsloth ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Phi-4 fine-tuning](https://unsloth.ai/blog/phi4) for the models are also now supported in Unsloth. To fine-tune for free on Google Colab, just change the `model_name` of 'unsloth/Phi-4' to 'unsloth/Phi-4-mini-reasoning' etc. * [Phi-4 (14B) fine-tuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb) [PreviousQwQ-32B: How to Run effectively](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively) [NextCogito v2: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally) Last updated 3 months ago Was this helpful? --- # DeepSeek-V3.1 | Unsloth Documentation OpenAI gpt-oss & all model types now supported! DeepSeekā€™s V3.1 update introduces hybrid reasoning inference, combining 'think' and 'non-think' into one model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic 2-bit version uses 245GB (-75% reduction in size). GGUF: [**DeepSeek-V3.1-GGUF**](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF) Our DeepSeek-V3.1 GGUFs include Unsloth [chat template fixes](https://docs.unsloth.ai/basics/deepseek-v3.1#chat-template-bug-fixes) for llama.cpp supported backends. All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss. **Tutorials navigation:** [Run in llama.cpp](https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-llama.cpp) [Run in Ollama/Open WebUI](https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-ollama-open-webui) [](https://docs.unsloth.ai/basics/deepseek-v3.1#recommended-settings) āš™ļø Recommended Settings -------------------------------------------------------------------------------------------------- The 1-bit dynamic quant TQ1\_0 (1bit for unimportant MoE layers, 2-4bit for important MoE, and 6-8bit for rest) uses 170GB of disk space - this works well in a **1x24GB card and 128GB of RAM** with MoE offloading - it also **works natively in Ollama**! You must use `--jinja` for llama.cpp quants - this uses our [fixed chat templates](https://docs.unsloth.ai/basics/deepseek-v3.1#chat-template-bug-fixes) and enables the correct template! You might get incorrect results if you do not use `--jinja` The 2-bit quants will fit in a 1x 24GB GPU (with MoE layers offloaded to RAM). Expect around 7 tokens/s with this setup if you have bonus 128GB RAM as well. It is recommended to have at least 246GB RAM to run this quant. For optimal performance you will need at least 246GB unified memory or 246GB combined RAM+VRAM for 5+ tokens/s. To learn how to increase generation speed and fit longer contexts, [read here](https://docs.unsloth.ai/basics/deepseek-v3.1#improving-generation-speed) . Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower. [](https://docs.unsloth.ai/basics/deepseek-v3.1#chat-template-bug-fixes) šŸ¦‹Chat template bug fixes ------------------------------------------------------------------------------------------------------- We fixed a few issues with DeepSeek V3.1's chat template since they did not function correctly in llama.cpp and other engines: 1. DeepSeek V3.1 is a hybrid reasoning model, meaning you can change the chat template to enable reasoning. The chat template introduced `thinking = True` , but other models use `enable_thinking = True` . We added the option to use `enable_thinking` as a keyword instead. 2. llama.cpp's jinja renderer via [minja](https://github.com/google/minja) does not allow the use of extra arguments in the `.split()` command, so using `.split(text, 1)` works in Python, but not in minja. We had to change this to make llama.cpp function correctly without erroring out. You will get the following error when using other quants: `terminate called after throwing an instance of 'std::runtime_error' what(): split method must have between 1 and 1 positional arguments and between 0 and 0 keyword arguments at row 3, column 1908` We fixed it in all our quants! ### [](https://docs.unsloth.ai/basics/deepseek-v3.1#official-recommended-settings) šŸ³Official Recommended Settings According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) , these are the recommended settings for V3.1 inference: * Set the **temperature 0.6** to reduce repetition and incoherence. * Set **top\_p to 0.95** (recommended) * **128K context length** or less * Use `--jinja` for llama.cpp variants - we **fixed some chat template issues as well!** * **Use** `enable_thinking = True` to use reasoning/ thinking mode. By default it's set to non reasoning. #### [](https://docs.unsloth.ai/basics/deepseek-v3.1#chat-template-prompt-format) šŸ”¢ Chat template/prompt format You do not need to force `\n` , but you can still add it in! With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token ``. Copy <ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ> A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call `tokenizer.encode(..., add_special_tokens = False)` since the chat template auto adds a BOS token as well. For llama.cpp / GGUF inference, you should skip the BOS since itā€™ll auto add it. #### [](https://docs.unsloth.ai/basics/deepseek-v3.1#non-thinking-mode-use-thinking-falseor-enable_thinking-false-and-is-by-default) šŸ“” Non-Thinking Mode (use `thinking = False`or `enable_thinking = False` and is by default) **First-Turn** Prefix: `<ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>` With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token ``. **Multi-Turn** Context: `<ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>{response}<ļ½œendā–ofā–sentenceļ½œ>...<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>{response}<ļ½œendā–ofā–sentenceļ½œ>` Prefix: `<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>` By concatenating the context and the prefix, we obtain the correct prompt for the query. #### [](https://docs.unsloth.ai/basics/deepseek-v3.1#thinking-mode-use-thinking-trueor-enable_thinking-true-and-is-by-default) šŸ“š Thinking Mode (use `thinking = True`or `enable_thinking = True` and is by default) **First-Turn** Prefix: `<ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>` The prefix of thinking mode is similar to DeepSeek-R1. **Multi-Turn** Context: `<ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>{response}<ļ½œendā–ofā–sentenceļ½œ>...<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>{response}<ļ½œendā–ofā–sentenceļ½œ>` Prefix: `<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>` The multi-turn template is the same with non-thinking multi-turn chat template. It means the thinking token in the last turn will be dropped but the `` is retained in every turn of context. #### [](https://docs.unsloth.ai/basics/deepseek-v3.1#tool-calling) šŸ¹ Tool Calling Tool calling is supported in non-thinking mode. The format is: `<ļ½œbeginā–ofā–sentenceļ½œ>{system prompt}{tool_description}<ļ½œUserļ½œ>{query}<ļ½œAssistantļ½œ>` where we populate the tool\_description is area after the system prompt. [](https://docs.unsloth.ai/basics/deepseek-v3.1#run-deepseek-v3.1-tutorials) ā–¶ļøRun DeepSeek-V3.1 Tutorials: ---------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-ollama-open-webui) šŸ¦™ Run in Ollama/Open WebUI 1 Install `ollama` if you haven't already! To run more variants of the model, [see here](https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-llama.cpp) . Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 2 Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! **(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1\_0 (170GB quant):** Copy OLLAMA_MODELS=unsloth ollama serve & OLLAMA_MODELS=unsloth ollama run hf.co/unsloth/DeepSeek-V3.1-GGUF:TQ1_0 3 To run other quants, you need to first merge the GGUF split files into 1 like the code below. Then you will need to run the model locally. Copy ./llama.cpp/llama-gguf-split --merge \ DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ merged_file.gguf Copy OLLAMA_MODELS=unsloth ollama serve & OLLAMA_MODELS=unsloth ollama run merged_file.gguf 4 Open WebUI also made a [step-by-step tutorial](https://docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/) on how to run R1 and for V3.1, you will just need to replace R1 with the new V3.1 quant. ### [](https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-llama.cpp) āœØ Run in llama.cpp 1 Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server cp llama.cpp/build/bin/llama-* llama.cpp 2 If you want to use `llama.cpp` directly to load models, you can do the below: (:Q2\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length. Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. Copy export LLAMA_CACHE="unsloth/DeepSeek-V3.1-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/DeepSeek-V3.1-GGUF:Q2_K_XL \ --cache-type-k q4_0 \ --jinja \ --n-gpu-layers 99 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" 3 Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-`Q2\_K\_XL (dynamic 2bit quant) or other quantized versions like `Q4_K_M` . We **recommend using our 2.7bit dynamic quant** `**UD-Q2_K_XL**` **to balance size and accuracy**. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/DeepSeek-V3.1-GGUF", local_dir = "unsloth/DeepSeek-V3.1-GGUF", allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2bit Use "*UD-TQ1_0*" for Dynamic 1bit ) 4 You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Copy ./llama.cpp/llama-cli \ --model unsloth/DeepSeek-V3.1-GGUF/UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ --cache-type-k q4_0 \ --jinja \ --threads -1 \ --n-gpu-layers 99 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" 5 Get the 1bit version (170GB) if you don't have enough combined RAM and VRAM: Copy from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/DeepSeek-V3.1-GGUF", local_dir = "unsloth/DeepSeek-V3.1-GGUF", allow_patterns = ["*UD-TQ1_0*"], # Use "*UD-Q2_K_XL*" for Dynamic 2bit ) ### [](https://docs.unsloth.ai/basics/deepseek-v3.1#deploy-with-llama-server-and-openais-completion-library) āœØ Deploy with llama-server and OpenAI's completion library To use llama-server for deployment, use the following command: Copy ./llama.cpp/llama-server \ --model unsloth/DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-TQ1_0.gguf \ --alias "unsloth/DeepSeek-V3.1" \ --threads -1 \ --n-gpu-layers 999 \ -ot ".ffn_.*_exps.=CPU" \ --prio 3 \ --min_p 0.01 \ --ctx-size 16384 \ --port 8001 \ --jinja Then use OpenAI's Python library after `pip install openai` : Copy from openai import OpenAI import json openai_client = OpenAI( base_url = "http://127.0.0.1:8001/v1", api_key = "sk-no-key-required", ) completion = openai_client.chat.completions.create( model = "unsloth/DeepSeek-V3.1", messages = [{"role": "user", "content": "What is 2+2?"},], ) print(completion.choices[0].message.content) [](https://docs.unsloth.ai/basics/deepseek-v3.1#model-uploads) šŸ’½Model uploads ----------------------------------------------------------------------------------- **ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks. * Full DeepSeek-V3.1 model uploads below: We also uploaded [IQ4\_NL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/IQ4_NL) and [Q4\_1](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/Q4_1) quants which run specifically faster for ARM and Apple devices respectively. MoE Bits Type + Link Disk Size Details 1.66bit [TQ1\_0](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF?show_file_info=DeepSeek-V3.1-UD-TQ1_0.gguf) **170GB** 1.92/1.56bit 1.78bit [IQ1\_S](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ1_S) **185GB** 2.06/1.56bit 1.93bit [IQ1\_M](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ1_M) **200GB** 2.5/2.06/1.56 2.42bit [IQ2\_XXS](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ2_XXS) **216GB** 2.5/2.06bit 2.71bit [Q2\_K\_XL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q2_K_XL) **251GB** 3.5/2.5bit 3.12bit [IQ3\_XXS](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ3_XXS) **273GB** 3.5/2.06bit 3.5bit [Q3\_K\_XL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q3_K_XL) **296GB** 4.5/3.5bit 4.5bit [Q4\_K\_XL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q4_K_XL) **384GB** 5.5/4.5bit 5.5bit [Q5\_K\_XL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q5_K_XL) **481GB** 6.5/5.5bit We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/DeepSeek-V3.1-BF16) , and original [FP8 (float8) format](https://huggingface.co/unsloth/DeepSeek-V3.1) . [](https://docs.unsloth.ai/basics/deepseek-v3.1#improving-generation-speed) šŸ‚ Improving generation speed -------------------------------------------------------------------------------------------------------------- If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves. Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel) . You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. [](https://docs.unsloth.ai/basics/deepseek-v3.1#how-to-fit-long-context-full-128k) šŸ“How to fit long context (full 128K) ----------------------------------------------------------------------------------------------------------------------------- To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below. `--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1` You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1` You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. Then you can use together with `--cache-type-k` : `--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1` [PreviousLoRA Hyperparameters Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) [Nextgpt-oss: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune) Last updated 3 days ago Was this helpful? --- # Vision Fine-tuning | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Fine-tuning vision models has numerous use cases across various industries, enabling models to adapt to specific tasks and datasets. We provided 3 example notebooks for vision finetuning. 1. **NEW: Gemma 3 (4B) Vision fine-tuning:** [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) 2. **Llama 3.2 Vision** fine-tuning for radiography: [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb) How can we assist medical professionals in analyzing Xrays, CT Scans & ultrasounds faster. 3. **Qwen2.5 VL** fine-tuning for converting handwriting to LaTeX: [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb) This allows complex math formulas to be easily transcribed as LaTeX without manually writing it. 4. **Pixtral 12B 2409** vision fine-tuning for general Q&A: [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Pixtral_(12B)-Vision.ipynb) One can concatenate general Q&A datasets with more niche datasets to make the finetune not forget base model skills. It is best to ensure your dataset has images of all the same size/dimensions. Use dimensions of 300-1000px to ensure your training does not take too long or use too many resources. To finetune vision models, we now allow you to select which parts of the mode to finetune. You can select to only finetune the vision layers, or the language layers, or the attention / MLP layers! We set them all on by default! Copy model = FastVisionModel.get_peft_model( model, finetune_vision_layers = True, # False if not finetuning vision layers finetune_language_layers = True, # False if not finetuning language layers finetune_attention_modules = True, # False if not finetuning attention layers finetune_mlp_modules = True, # False if not finetuning MLP layers r = 16, # The larger, the higher the accuracy, but might overfit lora_alpha = 16, # Recommended alpha == r at least lora_dropout = 0, bias = "none", random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ target_modules = "all-linear", # Optional now! Can specify a list if needed modules_to_save=[\ "lm_head",\ "embed_tokens",\ ], ) [](https://docs.unsloth.ai/basics/vision-fine-tuning#vision-fine-tuning-dataset) Vision Fine-tuning Dataset ---------------------------------------------------------------------------------------------------------------- The dataset for fine-tuning a vision or multimodal model is similar to standard question & answer pair [datasets](https://docs.unsloth.ai/basics/datasets-guide) , but this time, they also includes image inputs. For example, the [Llama 3.2 Vision Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb#scrollTo=vITh0KVJ10qX) uses a radiography case to show how AI can help medical professionals analyze X-rays, CT scans, and ultrasounds more efficiently. We'll be using a sampled version of the ROCO radiography dataset. You can access the dataset [here](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Funsloth%2FRadiology_mini) . The dataset includes X-rays, CT scans and ultrasounds showcasing medical conditions and diseases. Each image has a caption written by experts describing it. The goal is to finetune a VLM to make it a useful analysis tool for medical professionals. Let's take a look at the dataset, and check what the 1st example shows: Copy Dataset({ features: ['image', 'image_id', 'caption', 'cui'], num_rows: 1978 }) Image Caption ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrjdETiyi6jqzAao7vg8I%252Fxray.png%3Falt%3Dmedia%26token%3Df66fdd7f-5e10-4eff-a280-5b3d63ed7849&width=768&dpr=4&quality=100&sign=4d4d6839&sv=2) Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows). To format the dataset, all vision finetuning tasks should be formatted as follows: Copy [\ { "role": "user",\ "content": [{"type": "text", "text": instruction}, {"type": "image", "image": image} ]\ },\ { "role": "assistant",\ "content": [{"type": "text", "text": answer} ]\ },\ ] We will craft an custom instruction asking the VLM to be an expert radiographer. Notice also instead of just 1 instruction, you can add multiple turns to make it a dynamic conversation. Copy instruction = "You are an expert radiographer. Describe accurately what you see in this image." def convert_to_conversation(sample): conversation = [\ { "role": "user",\ "content" : [\ {"type" : "text", "text" : instruction},\ {"type" : "image", "image" : sample["image"]} ]\ },\ { "role" : "assistant",\ "content" : [\ {"type" : "text", "text" : sample["caption"]} ]\ },\ ] return { "messages" : conversation } pass Let's convert the dataset into the "correct" format for finetuning: Copy converted_dataset = [convert_to_conversation(sample) for sample in dataset] The first example is now structured like below: Copy converted_dataset[0] Copy {'messages': [{'role': 'user',\ 'content': [{'type': 'text',\ 'text': 'You are an expert radiographer. Describe accurately what you see in this image.'},\ {'type': 'image',\ 'image': }]},\ {'role': 'assistant',\ 'content': [{'type': 'text',\ 'text': 'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'}]}]} Before we do any finetuning, maybe the vision model already knows how to analyse the images? Let's check if this is the case! Copy FastVisionModel.for_inference(model) # Enable for inference! image = dataset[0]["image"] instruction = "You are an expert radiographer. Describe accurately what you see in this image." messages = [\ {"role": "user", "content": [\ {"type": "image"},\ {"type": "text", "text": instruction}\ ]}\ ] input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True) inputs = tokenizer( image, input_text, add_special_tokens = False, return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer, skip_prompt = True) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True, temperature = 1.5, min_p = 0.1) And the result: Copy This radiograph appears to be a panoramic view of the upper and lower dentition, specifically an Orthopantomogram (OPG). * The panoramic radiograph demonstrates normal dental structures. * There is an abnormal area on the upper right, represented by an area of radiolucent bone, corresponding to the antrum. **Key Observations** * The bone between the left upper teeth is relatively radiopaque. * There are two large arrows above the image, suggesting the need for a closer examination of this area. One of the arrows is in a left-sided position, and the other is in the right-sided position. However, only For more details, view our dataset section in the [notebook here](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb#scrollTo=vITh0KVJ10qX) . [PreviousChat Templates](https://docs.unsloth.ai/basics/chat-templates) [NextFinetuning from Last Checkpoint](https://docs.unsloth.ai/basics/finetuning-from-last-checkpoint) Last updated 1 month ago Was this helpful? --- # Inference | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Unsloth supports natively 2x faster inference. For our inference only notebook, click [here](https://colab.research.google.com/drive/1aqlNQi7MMJbynFDyOQteD2t0yVfjb9Zh?usp=sharing) . All QLoRA, LoRA and non LoRA inference paths are 2x faster. This requires no change of code or any new dependencies. Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) FastLanguageModel.for_inference(model) # Enable native 2x faster inference text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 64) #### [](https://docs.unsloth.ai/basics/running-and-saving-models/inference#notimplementederror-a-utf-8-locale-is-required.-got-ansi) NotImplementedError: A UTF-8 locale is required. Got ANSI Sometimes when you execute a cell [this error](https://github.com/googlecolab/colabtools/issues/3409) can appear. To solve this, in a new cell, run the below: Copy import locale locale.getpreferredencoding = lambda: "UTF-8" [PreviousTroubleshooting](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting) [NextContinued Pretraining](https://docs.unsloth.ai/basics/continued-pretraining) Last updated 7 months ago Was this helpful? --- # Tutorial: Train your own Reasoning model with GRPO | Unsloth Documentation OpenAI gpt-oss & all model types now supported! DeepSeek developed [GRPO](https://unsloth.ai/blog/grpo) (Group Relative Policy Optimization) to train their R1 reasoning models. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#quickstart) Quickstart ----------------------------------------------------------------------------------------------------------------------------------------------- These instructions are for our pre-made Google Colab [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) . If you are installing Unsloth locally, you can also copy our notebooks inside your favorite code editor. #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#the-grpo-notebooks-we-are-using-gemma-3-1b-llama-3.1-8b-phi-4-14b-and-qwen2.5-3b) The GRPO notebooks we are using: [Gemma 3 (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) , [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/HuggingFace%20Course-Gemma3_(1B)-GRPO.ipynb) , [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) and [Qwen2.5 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) 1 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#install-unsloth) Install Unsloth If you're using our Colab notebook, click **Runtime > Run all**. We'd highly recommend you checking out our [Fine-tuning Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) before getting started. If installing locally, ensure you have the correct [requirements](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) and use `pip install unsloth` on Linux or follow our [Windows install](https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation) instructions. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FCovHTH7dI2GcwNZm5TxF%252Fimage.png%3Falt%3Dmedia%26token%3Da157e33b-ad01-4174-a01c-67f742e4e732&width=768&dpr=4&quality=100&sign=e2f6a15e&sv=2) 2 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#learn-about-grpo-and-reward-functions) Learn about GRPO & Reward Functions Before we get started, it is recommended to learn more about GRPO, reward functions and how they work. Read more about them including [tips & tricks](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#basics-tips) [here](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#basics-tips) . You will also need enough VRAM. In general, model parameters = amount of VRAM you will need. In Colab, we are using their free 16GB VRAM GPUs which can train any model up to 16B in parameters. 3 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#configure-desired-settings) Configure desired settings We have pre-selected optimal settings for the best results for you already and you can change the model to whichever you want listed in our [supported models](https://docs.unsloth.ai/get-started/all-our-models) . Would not recommend changing other settings if you're a beginner. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fyd3RkyPKInZBbvX1Memf%252Fimage.png%3Falt%3Dmedia%26token%3Da9ca4ce4-2e9f-4b5a-a65c-646d267411c8&width=768&dpr=4&quality=100&sign=89b12b4e&sv=2) 4 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#data-preparation) Data preparation We have pre-selected OpenAI's [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset which contains grade school math problems but you could change it to your own or any public one on Hugging Face. You can read more about [datasets here](https://docs.unsloth.ai/basics/datasets-guide) . Your dataset should still have at least 2 columns for question and answer pairs. However the answer must not reveal the reasoning behind how it derived the answer from the question. See below for an example: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqdTVcMEeJ3kzPToSY1X8%252Fimage.png%3Falt%3Dmedia%26token%3D3dd8d9d7-1847-42b6-a73a-f9c995b798b1&width=768&dpr=4&quality=100&sign=7dd5cdc9&sv=2) We'll structure the data to prompt the model to articulate its reasoning before delivering an answer. To start, we'll establish a clear format for both prompts and responses. Copy # Define the system prompt that instructs the model to use a specific format SYSTEM_PROMPT = """ Respond in the following format: ... ... """ XML_COT_FORMAT = """\ {reasoning} {answer} """ Now, to prepare the dataset: Copy import re from datasets import load_dataset, Dataset # Helper functions to extract answers from different formats def extract_xml_answer(text: str) -> str: answer = text.split("")[-1] answer = answer.split("")[0] return answer.strip() def extract_hash_answer(text: str) -> str | None: if "####" not in text: return None return text.split("####")[1].strip() # Function to prepare the GSM8K dataset def get_gsm8k_questions(split="train") -> Dataset: data = load_dataset("openai/gsm8k", "main")[split] data = data.map( lambda x: { "prompt": [\ {"role": "system", "content": SYSTEM_PROMPT},\ {"role": "user", "content": x["question"]},\ ], "answer": extract_hash_answer(x["answer"]), } ) return data dataset = get_gsm8k_questions() The dataset is prepared by extracting the answers and formatting them as structured strings. 5 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#reward-functions-verifier) Reward Functions/Verifier [Reward Functions/Verifiers](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-functions-verifier) lets us know if the model is doing well or not according to the dataset you have provided. Each generation run will be assessed on how it performs to the score of the average of the rest of generations. You can create your own reward functions however we have already pre-selected them for you with [Will's GSM8K](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#gsm8k-reward-functions) reward functions. With this, we have 5 different ways which we can reward each generation. You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria. **See examples** of what they can look like [here](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-function-examples) . **Example Reward Function for an Email Automation Task:** * **Question:** Inbound email * **Answer:** Outbound email * **Reward Functions:** * If the answer contains a required keyword ā†’ **+1** * If the answer exactly matches the ideal response ā†’ **+1** * If the response is too long ā†’ **\-1** * If the recipient's name is included ā†’ **+1** * If a signature block (phone, email, address) is present ā†’ **+1** ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6GRcqgUKmKn2dWCk4nWK%252Fimage.png%3Falt%3Dmedia%26token%3Dac153141-03f8-4795-9074-ad592289bd70&width=768&dpr=4&quality=100&sign=3f226098&sv=2) 6 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#train-your-model) Train your model We have pre-selected hyperparameters for the most optimal results however you could change them. Read all about [parameters here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) . ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1MpLSyaOH3j8MhQvquqX%252Fimage.png%3Falt%3Dmedia%26token%3D818034b1-f2db-464d-a108-3b2c6897edb7&width=768&dpr=4&quality=100&sign=4da81ed4&sv=2) The **GRPOConfig** defines key hyperparameters for training: * `use_vllm`: Activates fast inference using vLLM. * `learning_rate`: Determines the model's learning speed. * `num_generations`: Specifies the number of completions generated per prompt. * `max_steps`: Sets the total number of training steps. **NEW!** We now support DAPO, Dr. GRPO and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable: Copy epsilon=0.2, epsilon_high=0.28, # one sided delta=1.5 # two sided loss_type='bnpo', # or: loss_type='grpo', # or: loss_type='dr_grpo', # or: loss_type='dapo', mask_truncated_completions=True, You should see the reward increase overtime. We would recommend you train for at least 300 steps which may take 30 mins however, for optimal results, you should train for longer. If you're having issues with your GRPO model not learning, we'd highly recommend to use our [Advanced GRPO notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-notebooks) as it has a much better reward function and you should see results much faster and frequently. You will also see sample answers which allows you to see how the model is learning. Some may have steps, XML tags, attempts etc. and the idea is as trains it's going to get better and better because it's going to get scored higher and higher until we get the outputs we desire with long reasoning chains of answers. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FyRmUGe8laUKIl0RKwlE6%252Fimage.png%3Falt%3Dmedia%26token%3D3ff931cc-0d2b-4a9c-bbe1-b6289b22d157&width=768&dpr=4&quality=100&sign=40488764&sv=2) 7 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#run-and-evaluate-your-model) Run & Evaluate your model Run your model by clicking the play button. In the first example, there is usually no reasoning in the answer and in order to see the reasoning, we need to first save the LoRA weights we just trained with GRPO first using: Copy model.save_lora("grpo_saved_lora") ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkLHdlRVKN58tM7SGKp3O%252Fimage.png%3Falt%3Dmedia%26token%3Db43a8164-7eae-4ec4-bf59-976078f9be31&width=768&dpr=4&quality=100&sign=e2241c49&sv=2) The first inference example run has no reasoning. You must load the LoRA and test it to reveal the reasoning. Then we load the LoRA and test it. Our reasoning model is much better - it's not always correct, since we only trained it for an hour or so - it'll be better if we extend the sequence length and train for longer! You can then save your model to GGUF, Ollama etc. by following our [guide here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide#id-7.-running--saving-the-model) . ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYdz5ch20Ig8JlumBesle%252Fimage.png%3Falt%3Dmedia%26token%3D8aea2867-b8a8-470a-aa4b-a7b9cdd64c3c&width=768&dpr=4&quality=100&sign=a9ddceab&sv=2) If you are still not getting any reasoning, you may have either trained for too less steps or your reward function/verifier was not optimal. 8 ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#save-your-model) Save your model We have multiple options for saving your fine-tuned model, but weā€™ll focus on the easiest and most popular approaches which you can read more about [here](https://docs.unsloth.ai/basics/running-and-saving-models) **Saving in 16-bit Precision** You can save the model with 16-bit precision using the following command: Copy # Save to 16-bit precision model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit") #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#pushing-to-hugging-face-hub) **Pushing to Hugging Face Hub** To share your model, weā€™ll push it to the Hugging Face Hub using the `push_to_hub_merged` method. This allows saving the model in multiple quantization formats. Copy # Push to Hugging Face Hub (requires a token) model.push_to_hub_merged( "your-username/model-name", tokenizer, save_method="merged_16bit", token="your-token" ) #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#saving-in-gguf-format-for-llama.cpp) **Saving in GGUF Format for llama.cpp** Unsloth also supports saving in **GGUF format**, making it compatible with **llama.cpp** and **Ollama**. Copy model.push_to_hub_gguf( "your-username/model-name", tokenizer, quantization_method=["q4_k_m", "q8_0", "q5_k_m"], token="your-token", ) Once saved in GGUF format, the model can be easily deployed in lightweight environments using **llama.cpp** or used in other inference engines. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo#video-tutorials) Video Tutorials --------------------------------------------------------------------------------------------------------------------------------------------------------- Here are some video tutorials created by amazing YouTubers who we think are fantastic! Local GRPO on your own device Great to learn about how to prep your dataset and explanations behind Reinforcement Learning + GRPO basics [PreviousReinforcement Learning (RL) Guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) [NextReinforcement Learning - DPO, ORPO & KTO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/reinforcement-learning-dpo-orpo-and-kto) Last updated 2 months ago Was this helpful? --- # Training AI Agents with RL | Unsloth Documentation OpenAI gpt-oss & all model types now supported! ā€œAgenticā€ AI is becoming more popular over time. In this context, an ā€œagentā€ is an LLM that is given a high-level goal and a set of tools to achieve it. Agents are also typically ā€œmulti-turnā€ ā€” they can perform an action, see what effect it had on the environment, and then perform another action repeatedly, until they achieve their goal or fail trying. Unfortunately, even very capable LLMs can have a hard time performing complex multi-turn agentic tasks reliably. Interestingly, weā€™ve found that training agents using an RL algorithm called [GRPO (Group Relative Policy Optimization)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) can make them far more reliable! In this guide, you will learn how to to build reliable AI agents using open-source tools. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#training-rl-agents-with-art) šŸŽØ Training RL Agents with ART ------------------------------------------------------------------------------------------------------------------------------------------------------------- [ART (Agent Reinforcement Trainer)](https://github.com/openpipe/art) built on top of [Unsloth](https://github.com/unslothai/unsloth) ā€™s GRPOTrainer, is a tool that makes training multi-turn agents possible and easy. If youā€™re already using Unsloth for GRPO and need to train agents that can handle complex, multi-turn interactions, ART simplifies the process. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlIJVwkW3jsINbrOndI8w%252FScreenshot_2025-07-19_at_1.23.18_PM.webp%3Falt%3Dmedia%26token%3D5fce0b1b-870a-4c5b-a647-ba9688977eb3&width=768&dpr=4&quality=100&sign=83773ced&sv=2) Agent models trained with Unsloth+ART are often able to outperform prompted models on agentic workflows. ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#art--unsloth) ART + Unsloth ART builds on top of Unslothā€™s memory- and compute-efficient GRPO implementation. In addition, it adds the following capabilities: #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#id-1.-multi-turn-agent-training) 1\. Multi-Turn Agent Training ART introduces the concept of a ā€œtrajectoryā€, which is built up as your agent executes. These trajectories can then be scored and used for GRPO. Trajectories can be complex, and even include non-linear histories, sub-agent calls, etc. They also support tool calls and responses. #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#id-2.-flexible-integration-into-existing-codebases) 2\. Flexible Integration into Existing Codebases If you already have an agent working with a prompted model, ART tries to minimize the number of changes you need to make to wrap your existing agent loop and use it for training. Architecturally, ART is split into a ā€œfrontendā€ client that lives in your codebase and communicates via API with a ā€œbackendā€ where the actual training happens (these can also be colocated on a single machine if you prefer using ARTā€™s `LocalBackend`). This gives some key benefits: * **Minimal setup required**: The ART frontend is has minimal dependencies and can be easily added to existing Python codebases. * **Train from anywhere**: You can run the ART client on your laptop and let the ART server kick off an ephemeral GPU-enabled environment, or run on a local GPU * **OpenAI-compatible API**: The ART backend serves your model undergoing training via an OpenAI-compatible API, which is compatible with most existing codebases. #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#id-3.-ruler-zero-shot-agent-rewards) 3\. RULER: Zero-Shot Agent Rewards ART also provides a built-in general-purpose reward function called [RULER](https://art.openpipe.ai/fundamentals/ruler) (Relative Universal LLM-Elicited Rewards), which can eliminate the need for hand-crafted reward functions. Surprisingly, agents RL-trained with the RULER automatic reward function often match or surpass the performance of agents trained using hand-written reward functions. This makes getting started with RL easier. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYAeJzbOFGTNZM9r5Qkrq%252FScreenshot_2025-07-19_at_1.21.08_PM.webp%3Falt%3Dmedia%26token%3Da7b67644-f6ac-4d4f-a690-7b160948f9d2&width=768&dpr=4&quality=100&sign=3c2d375b&sv=2) Copy # Before: Hours of reward engineering def complex_reward_function(trajectory): # 50+ lines of careful scoring logic... pass # After: One line with RULER judged_group = await ruler_score_group(group, "openai/o3") ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#when-to-choose-art) When to Choose ART ART might be a good fit for projects that need: 1. **Multi-step agent capabilities**: When your use case involves agents that need to take multiple actions, use tools, or have extended conversations 2. **Rapid prototyping without reward engineering**: RULERā€™s automatic reward scoring can cut your projectā€™s development time by 2-3x 3. **Integration with existing systems**: When you need to add RL capabilities to an existing agentic codebase with minimal changes ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#code-example-art-in-action) Code Example: ART in Action Copy import art from art.rewards import ruler_score_group # Initialize model with Unsloth-supported basemodel model = art.TrainableModel( name="agent-001", project="my-agentic-task", base_model="Qwen/Qwen2.5-14B-Instruct", # Any Unsloth-supported model ) # Define your rollout function async def rollout(model: art.Model, scenario: Scenario) -> art.Trajectory: openai_client = model.openai_client() trajectory = art.Trajectory( messages_and_choices=[\ {"role": "system", "content": "..."},\ {"role": "user", "content": "..."}\ ] ) # Your agent logic here... return trajectory # Train with RULER for automatic rewards groups = await art.gather_trajectory_groups( ( art.TrajectoryGroup(rollout(model, scenario) for _ in range(8)) for scenario in scenarios ), after_each=lambda group: ruler_score_group( group, "openai/o3", swallow_exceptions=True ) ) await model.train(groups) ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl#getting-started) Getting Started To add ART to your Unsloth-based project: Copy pip install openpipe-art # or `uv add openpipe-art` Then check out the [example notebooks](https://art.openpipe.ai/getting-started/notebooks) to see ART in action with tasks like: * Email retrieval agents that beat o3 * Game-playing agents (2048, Tic Tac Toe, Codenames) * Complex reasoning tasks (Temporal Clue) [PreviousReinforcement Learning - DPO, ORPO & KTO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/reinforcement-learning-dpo-orpo-and-kto) [NextText-to-Speech (TTS) Fine-tuning](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) Last updated 26 days ago Was this helpful? --- # Saving to GGUF | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Locally [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf#tab-locally) Manual Saving [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf#tab-manual-saving) To save to GGUF, use the below to save locally: Copy model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q4_k_m") model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q8_0") model.save_pretrained_gguf("directory", tokenizer, quantization_method = "f16") To push to Hugging Face hub: Copy model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q4_k_m") model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q8_0") All supported quantization options for `quantization_method` are listed below: Copy # https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/quantize.cpp#L19 # From https://mlabonne.github.io/blog/posts/Quantize_Llama_2_models_using_ggml.html ALLOWED_QUANTS = \ { "not_quantized" : "Recommended. Fast conversion. Slow inference, big files.", "fast_quantized" : "Recommended. Fast conversion. OK inference, OK file size.", "quantized" : "Recommended. Slow conversion. Fast inference, small files.", "f32" : "Not recommended. Retains 100% accuracy, but super slow and memory hungry.", "f16" : "Fastest conversion + retains 100% accuracy. Slow and memory hungry.", "q8_0" : "Fast conversion. High resource use, but generally acceptable.", "q4_k_m" : "Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K", "q5_k_m" : "Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K", "q2_k" : "Uses Q4_K for the attention.vw and feed_forward.w2 tensors, Q2_K for the other tensors.", "q3_k_l" : "Uses Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else Q3_K", "q3_k_m" : "Uses Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else Q3_K", "q3_k_s" : "Uses Q3_K for all tensors", "q4_0" : "Original quant method, 4-bit.", "q4_1" : "Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.", "q4_k_s" : "Uses Q4_K for all tensors", "q4_k" : "alias for q4_k_m", "q5_k" : "alias for q5_k_m", "q5_0" : "Higher accuracy, higher resource usage and slower inference.", "q5_1" : "Even higher accuracy, resource usage and slower inference.", "q5_k_s" : "Uses Q5_K for all tensors", "q6_k" : "Uses Q8_K for all tensors", "iq2_xxs" : "2.06 bpw quantization", "iq2_xs" : "2.31 bpw quantization", "iq3_xxs" : "3.06 bpw quantization", "q3_k_xs" : "3-bit extra small quantization", } First save your model to 16bit: Copy model.save_pretrained_merged("merged_model", tokenizer, save_method = "merged_16bit",) Then use the terminal and do: Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp python llama.cpp/convert-hf-to-gguf.py FOLDER --outfile OUTPUT --outtype f16 Or follow the steps at https://rentry.org/llama-cpp-conversions#merging-loras-into-a-model using the model name "merged\_model" to merge to GGUF. [PreviousRunning & Saving Models](https://docs.unsloth.ai/basics/running-and-saving-models) [NextSaving to Ollama](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama) Last updated 9 hours ago Was this helpful? --- # Saving to Ollama | Unsloth Documentation OpenAI gpt-oss & all model types now supported! See our guide below for the complete process on how to save to [Ollama](https://github.com/ollama/ollama) : [šŸ¦™Tutorial: How to Finetune Llama-3 and Use In Ollama](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama#saving-on-google-colab) Saving on Google Colab -------------------------------------------------------------------------------------------------------------------------------- You can save the finetuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face token via: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and add your token! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBz0YDi6Sc2oEP5QWXgSz%252Fimage.png%3Falt%3Dmedia%26token%3D33d9e4fd-e7dc-4714-92c5-bfa3b00f86c4&width=768&dpr=4&quality=100&sign=d6933a01&sv=2) After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzymBQrqwt4GUmCIN0Iec%252Fimage.png%3Falt%3Dmedia%26token%3D41a110e4-8263-426f-8fa7-cdc295cc8210&width=768&dpr=4&quality=100&sign=b2a207c3&sv=2) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama#exporting-to-ollama) Exporting to Ollama -------------------------------------------------------------------------------------------------------------------------- Finally we can export our finetuned model to Ollama itself! First we have to install Ollama in the Colab notebook: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqNvGTAGwZKXxkMQqzloS%252Fimage.png%3Falt%3Dmedia%26token%3Ddb503499-0c74-4281-b3bf-400fa20c9ce2&width=768&dpr=4&quality=100&sign=6d57e83a&sv=2) Then we export the finetuned model we have to llama.cpp's GGUF formats like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZduLjedyfUbTmYqF85pa%252Fimage.png%3Falt%3Dmedia%26token%3Df5bac541-b99f-4d9b-82f7-033f8de780f2&width=768&dpr=4&quality=100&sign=1fdb7647&sv=2) Reminder to convert `False` to `True` for 1 row, and not change every row to `True`, or else you'll be waiting for a very time! We normally suggest the first row getting set to `True`, so we can export the finetuned model quickly to `Q8_0` format (8 bit quantization). We also allow you to export to a whole list of quantization methods as well, with a popular one being `q4_k_m`. Head over to [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) to learn more about GGUF. We also have some manual instructions of how to export to GGUF if you want here: [https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf](https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf) You will see a long list of text like below - please wait 5 to 10 minutes!! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FcuUAx0RNtrQACvU7uWCL%252Fimage.png%3Falt%3Dmedia%26token%3Ddc67801a-a363-48e2-8572-4c6d0d8d0d93&width=768&dpr=4&quality=100&sign=cc7f7372&sv=2) And finally at the very end, it'll look like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxRh07PEQjAmmz3s2HJUP%252Fimage.png%3Falt%3Dmedia%26token%3D3552a3c9-4d4f-49ee-a31e-0a64327419f0&width=768&dpr=4&quality=100&sign=1e9c9f0d&sv=2) Then, we have to run Ollama itself in the background. We use `subprocess` because Colab doesn't like asynchronous calls, but normally one just runs `ollama serve` in the terminal / command prompt. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FszDuikrg4HY8lGefwpRQ%252Fimage.png%3Falt%3Dmedia%26token%3Dec1c8762-661d-4b13-ab4f-ed1a7b9fda00&width=768&dpr=4&quality=100&sign=fc72e538&sv=2) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama#automatic-modelfile-creation) Automatic `Modelfile` creation ---------------------------------------------------------------------------------------------------------------------------------------------- The trick Unsloth provides is we automatically create a `Modelfile` which Ollama requires! This is a just a list of settings and includes the chat template which we used for the finetune process! You can also print the `Modelfile` generated like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh6inH6k5ggxUP80Gltgj%252Fimage.png%3Falt%3Dmedia%26token%3D805bafb1-2795-4743-9bd2-323ab4f0881e&width=768&dpr=4&quality=100&sign=456e8653&sv=2) We then ask Ollama to create a model which is Ollama compatible, by using the `Modelfile` ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1123bSSwmjWXliaRUL5U%252Fimage.png%3Falt%3Dmedia%26token%3D2e72f1a0-1ff8-4189-8d9c-d31e39385555&width=768&dpr=4&quality=100&sign=52a4fd99&sv=2) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama#ollama-inference) Ollama Inference -------------------------------------------------------------------------------------------------------------------- And we can now call the model for inference if you want to do call the Ollama server itself which is running on your own local machine / in the free Colab notebook in the background. Remember you can edit the yellow underlined part. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fk5mdsJ57hQ1Ar3KY6VXY%252FInference.png%3Falt%3Dmedia%26token%3D8cf0cbf9-0534-4bae-a887-89f45a3de771&width=768&dpr=4&quality=100&sign=8489fe55&sv=2) [](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-ollama#undefined) ------------------------------------------------------------------------------------------- [PreviousSaving to GGUF](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) [NextSaving to VLLM](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-vllm) Last updated 1 year ago Was this helpful? --- # Troubleshooting | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting#running-in-unsloth-works-well-but-after-exporting-and-running-on-other-platforms-the-results-are-poo) Running in Unsloth works well, but after exporting & running on other platforms, the results are poor -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might sometimes encounter an issue where your model runs and produces good results on Unsloth, but when you use it on another platform like Ollama or vLLM, the results are poor or you might get gibberish, endless/infinite generations _or_ repeated outputs**.** * The most common cause of this error is using an incorrect chat template. Itā€™s essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. When inferencing from a saved model, it's crucial to apply the correct template. * It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses! [](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting#saving-to-safetensors-not-bin-format-in-colab) Saving to `safetensors`, not `bin` format in Colab ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We save to `.bin` in Colab so it's like 4x faster, but set `safe_serialization = None` to force saving to `.safetensors`. So `model.save_pretrained(..., safe_serialization = None)` or `model.push_to_hub(..., safe_serialization = None)` [](https://docs.unsloth.ai/basics/running-and-saving-models/troubleshooting#if-saving-to-gguf-or-vllm-16bit-crashes) If saving to GGUF or vLLM 16bit crashes ----------------------------------------------------------------------------------------------------------------------------------------------------------------- You can try reducing the maximum GPU usage during saving by changing `maximum_memory_usage`. The default is `model.save_pretrained(..., maximum_memory_usage = 0.75)`. Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving. [PreviousSaving to VLLM](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-vllm) [NextInference](https://docs.unsloth.ai/basics/running-and-saving-models/inference) Last updated 5 months ago Was this helpful? --- # Unsloth Benchmarks | Unsloth Documentation OpenAI gpt-oss & all model types now supported! * For our most detailed benchmarks, read our [Llama 3.3 Blog](https://unsloth.ai/blog/llama3-3) . * Benchmarking of Unsloth was also conducted by [šŸ¤—Hugging Face](https://huggingface.co/blog/unsloth-trl) . We tested using the Alpaca Dataset, a batch size of 2, gradient accumulation steps of 4, rank = 32, and applied QLoRA on all linear layers (q, k, v, o, gate, up, down): Model VRAM šŸ¦„Unsloth speed šŸ¦„VRAM reduction šŸ¦„Longer context šŸ˜ŠHugging Face + FA2 Llama 3.3 (70B) 80GB 2x \>75% 13x longer 1x Llama 3.1 (8B) 80GB 2x \>70% 12x longer 1x [](https://docs.unsloth.ai/basics/unsloth-benchmarks#context-length-benchmarks) Context length benchmarks -------------------------------------------------------------------------------------------------------------- The more data you have, the less VRAM Unsloth uses due to our [gradient checkpointing](https://unsloth.ai/blog/long-context) algorithm + Apple's CCE algorithm! ### [](https://docs.unsloth.ai/basics/unsloth-benchmarks#llama-3.1-8b-max.-context-length) **Llama 3.1 (8B) max. context length** We tested Llama 3.1 (8B) Instruct and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads. GPU VRAM šŸ¦„Unsloth context length Hugging Face + FA2 8 GB 2,972 OOM 12 GB 21,848 932 16 GB 40,724 2,551 24 GB 78,475 5,789 40 GB 153,977 12,264 48 GB 191,728 15,502 80 GB 342,733 28,454 ### [](https://docs.unsloth.ai/basics/unsloth-benchmarks#llama-3.3-70b-max.-context-length) **Llama 3.3 (70B) max. context length** We tested Llama 3.3 (70B) Instruct on a 80GB A100 and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads. GPU VRAM šŸ¦„Unsloth context length Hugging Face + FA2 48 GB 12,106 OOM 80 GB 89,389 6,916 [PreviousTraining LLMs with Blackwell, RTX 50 series & Unsloth](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) [NextMulti-GPU Training with Unsloth](https://docs.unsloth.ai/basics/multi-gpu-training-with-unsloth) Last updated 7 months ago Was this helpful? --- # Kimi K2: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Kimi K2 is the worldā€™s most powerful open-source model, setting new SOTA performance in knowledge, reasoning, coding, and agentic tasks. The full 1T parameter model from Moonshot AI requires 1.09TB of disk space, while the quantized **Unsloth Dynamic 1.8-bit** version reduces this to just 245GB (-80% size)**:** [**Kimi-K2-GGUF**](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF) All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run quantized LLMs with minimal accuracy loss. You can now use the latest update of [llama.cpp](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#run-in-llama.cpp) to run the model! **Tool calling also got updated as at 16th July 2025** \- you can use the old GGUF files you downloaded, and re-download the first GGUF file (50GB worth) OR use `--chat-template-file NEW_FILE.jinja` . [More details here](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#tokenizer-quirks-and-bug-fixes) . [Run in llama.cpp](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#run-kimi-k2-tutorials) [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#recommended-settings) āš™ļø Recommended Settings --------------------------------------------------------------------------------------------------------------- You need **250GB of disk space** at least to run the 1bit quant! The only requirement is `**disk space + RAM + VRAM ā‰„ 250GB**`. That means you do not need to have that much RAM or VRAM (GPU) to run the model, but it will just be slower. The 1.8-bit (UD-TQ1\_0) quant will fit in a 1x 24GB GPU (with all MoE layers offloaded to system RAM or a fast disk). Expect around 5 tokens/s with this setup if you have bonus 256GB RAM as well. The full Kimi K2 Q8 quant is 1.09TB in size and will need at least 8 x H200 GPUs. For optimal performance you will need at least **250GB unified memory or 250GB combined RAM+VRAM** for 5+ tokens/s. If you have less than 250GB combined RAM+VRAM, then the speed of the model will definitely take a hit. **If you do not have 250GB of RAM+VRAM, no worries!** llama.cpp inherently has **disk offloading**, so through mmaping, it'll still work, just be slower - for example before you might get 5 to 10 tokens / second, now it's under 1 token. We suggest using our **UD-Q2\_K\_XL (381GB)** quant to balance size and accuracy! For the best performance, have your VRAM + RAM combined = the size of the quant you're downloading. If not, it'll still work via disk offloading, just it'll be slower! ### [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#official-recommended-settings) šŸŒ™ Official Recommended Settings: According to [Moonshot AI](https://huggingface.co/moonshotai/Kimi-K2-Instruct) , these are the recommended settings for Kimi K2 inference: * Set the **temperature 0.6** to reduce repetition and incoherence. * Original default system prompt is: Copy You are a helpful assistant * (Optional) Moonshot also suggests the below for the system prompt: Copy You are Kimi, an AI assistant created by Moonshot AI. We recommend setting **min\_p to 0.01** to suppress the occurrence of unlikely tokens with low probabilities. [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#chat-template-and-prompt-format) šŸ”¢ Chat template and prompt format ------------------------------------------------------------------------------------------------------------------------------------- Kimi Chat does use a BOS (beginning of sentence token). The system, user and assistant roles are all enclosed with `<|im_middle|>` which is interesting, and each get their own respective token `<|im_system|>, <|im_user|>, <|im_assistant|>`. Copy <|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|><|im_user|>user<|im_middle|>What is 1+1?<|im_end|><|im_assistant|>assistant<|im_middle|>2<|im_end|> To separate the conversational boundaries (you must remove each new line), we get: Copy <|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|> <|im_user|>user<|im_middle|>What is 1+1?<|im_end|> <|im_assistant|>assistant<|im_middle|>2<|im_end|> [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#model-uploads) šŸ’¾ Model uploads ------------------------------------------------------------------------------------------------- **ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and reasoning tasks. MoE Bits Type + Link Disk Size Details 1.66bit [UD-TQ1\_0](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-TQ1_0) **245GB** 1.92/1.56bit 1.78bit [UD-IQ1\_S](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ1_S) **281GB** 2.06/1.56bit 1.93bit [UD-IQ1\_M](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ1_M) **304GB** 2.5/2.06/1.56 2.42bit [UD-IQ2\_XXS](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ2_XXS) **343GB** 2.5/2.06bit 2.71bit [UD-Q2\_K\_XL](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q2_K_XL) **381GB** 3.5/2.5bit 3.12bit [UD-IQ3\_XXS](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ3_XXS) **417GB** 3.5/2.06bit 3.5bit [UD-Q3\_K\_XL](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q3_K_XL) **452GB** 4.5/3.5bit 4.5bit [UD-Q4\_K\_XL](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q4_K_XL) **588GB** 5.5/4.5bit 5.5bit [UD-Q5\_K\_XL](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q5_K_XL) **732GB** 6.5/5.5bit We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/Kimi-K2-Instruct-BF16) . [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#run-kimi-k2-tutorials) šŸ¢Run Kimi K2 Tutorials ---------------------------------------------------------------------------------------------------------------- You can now use the latest update of [llama.cpp](https://github.com/ggml-org/llama.cpp) to run the model: ### [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#run-in-llama.cpp) āœØ Run in llama.cpp 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:UD-IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. Copy export LLAMA_CACHE="unsloth/Kimi-K2-Instruct-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/Kimi-K2-Instruct-GGUF:TQ1_0 \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --temp 0.6 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-TQ1_0`(dynamic 1.8bit quant) or other quantized versions like `Q2_K_XL` . We **recommend using our 2bit dynamic quant** `**UD-Q2_K_XL**` **to balance size and accuracy**. More versions at: [huggingface.co/unsloth/Kimi-K2-Instruct-GGUF](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Kimi-K2-Instruct-GGUF", local_dir = "unsloth/Kimi-K2-Instruct-GGUF", allow_patterns = ["*UD-TQ1_0*"], # Dynamic 1bit (281GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (381GB) ) If you find that downloads get stuck at 90 to 95% or so, please see [https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95) 1. Run any prompt. 2. Edit `--threads -1` for the number of CPU threads (be default it's set to the maximum CPU threads), `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Set it to 99 combined with MoE CPU offloading to get the best performance. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Copy ./llama.cpp/llama-cli \ --model unsloth/Kimi-K2-Instruct-GGUF/UD-TQ1_0/Kimi-K2-Instruct-UD-TQ1_0-00001-of-00005.gguf \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --temp 0.6 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" \ -no-cnv \ --prompt "<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|><|im_user|>user<|im_middle|>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|><|im_assistant|>assistant<|im_middle|>" [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#tokenizer-quirks-and-bug-fixes) šŸ”Tokenizer quirks and bug fixes ---------------------------------------------------------------------------------------------------------------------------------- **16th July 2025: Kimi K2 updated their tokenizer to enable multiple tool calls** as per [https://x.com/Kimi\_Moonshot/status/1945050874067476962](https://x.com/Kimi_Moonshot/status/1945050874067476962) If you have the old checkpoints downloaded - now worries - simply download the first GGUF split which was changed. OR if you do not want to download any new files do: Copy wget https://huggingface.co/unsloth/Kimi-K2-Instruct/raw/main/chat_template.jinja ./llama.cpp ... --chat-template-file /dir/to/chat_template.jinja The Kimi K2 tokenizer was interesting to play around with - **it's mostly similar in action to GPT-4o's tokenizer**! We first see in the [tokenization\_kimi.py](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenization_kimi.py) file the following regular expression (regex) that Kimi K2 uses: Copy pat_str = "|".join( [\ r"""[\p{Han}]+""",\ r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",\ r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",\ r"""\p{N}{1,3}""",\ r""" ?[^\s\p{L}\p{N}]+[\r\n]*""",\ r"""\s*[\r\n]+""",\ r"""\s+(?!\S)""",\ r"""\s+""",\ ] ) After careful inspection, we find Kimi K2 is nearly identical to GPT-4o's tokenizer regex which can be found in [llama.cpp's source code](https://github.com/ggml-org/llama.cpp/blob/55c509daf51d25bfaee9c8b8ce6abff103d4473b/src/llama-vocab.cpp#L400) . Copy [^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+ Both tokenize numbers into groups of 1 to 3 numbers (9, 99, 999), and use similar patterns. The only difference looks to be the handling of "Han" or Chinese characters, which Kimi's tokenizer deals with more. [The PR](https://github.com/ggml-org/llama.cpp/pull/14654) by [https://github.com/gabriellarson](https://github.com/gabriellarson) handles these differences well after some [discussions here](https://github.com/ggml-org/llama.cpp/issues/14642#issuecomment-3067324745) . **We also find the correct EOS token should not be \[EOS\], but rather <|im\_end|>, which we have also fixed in our model conversions.** [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#heptagon-test) šŸ¦ Flappy Bird + other tests ------------------------------------------------------------------------------------------------------------- We introduced the Flappy Bird test when our 1.58bit quants for DeepSeek R1 were provided. We found Kimi K2 one of the only models to one-shot all our tasks including this one, [Heptagon](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#heptagon-test) and others tests even at 2-bit. The goal is to ask the LLM to create a Flappy Bird game but following some specific instructions: Copy Create a Flappy Bird game in Python. You must include these things: 1. You must use pygame. 2. The background color should be randomly chosen and is a light shade. Start with a light blue color. 3. Pressing SPACE multiple times will accelerate the bird. 4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color. 5. Place on the bottom some land colored as dark brown or yellow chosen randomly. 6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them. 7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade. 8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again. The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section. You can also test the dynamic quants via the Heptagon Test as per [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2O72oTw5yPUbcxXjDNKS%252Fsnapshot.jpg%3Falt%3Dmedia%26token%3Dce852f9f-20ee-4b93-9d7b-1a5f211b9e04&width=768&dpr=4&quality=100&sign=55d1134d&sv=2) The goal is to make the heptagon spin, and the balls in the heptagon should move. The prompt is below: Copy Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file. [PreviousQwen3-2507](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) [NextGemma 3n: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) Last updated 1 month ago Was this helpful? --- # Troubleshooting & FAQs | Unsloth Documentation OpenAI gpt-oss & all model types now supported! **Try always to update Unsloth if you find any issues.** `pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo` [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#running-in-unsloth-works-well-but-after-exporting-and-running-on-other-platforms-the-results-are-poo) ā“Running in Unsloth works well, but after exporting & running on other platforms, the results are poor ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might sometimes encounter an issue where your model runs and produces good results on Unsloth, but when you use it on another platform like Ollama or vLLM, the results are poor or you might get gibberish, endless/infinite generations _or_ repeated outputs**.** * The most common cause of this error is using an **incorrect chat template****.** Itā€™s essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. When inferencing from a saved model, it's crucial to apply the correct template. * It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses! * **Use our conversational notebooks to force the chat template - this will fix most issues.** * Qwen-3 14B Conversational notebook [**Open in Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_%2814B%29-Reasoning-Conversational.ipynb) * Gemma-3 4B Conversational notebook [**Open in Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_%284B%29.ipynb) * Llama-3.2 3B Conversational notebook [**Open in Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_%281B_and_3B%29-Conversational.ipynb) * Phi-4 14B Conversational notebook [**Open in Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb) * Mistral v0.3 7B Conversational notebook [**Open in Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_%287B%29-Conversational.ipynb) * **More notebooks in our** [**notebooks repo**](https://github.com/unslothai/notebooks) **.** [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#saving-to-gguf-vllm-16bit-crashes) ā“Saving to GGUF / vLLM 16bit crashes --------------------------------------------------------------------------------------------------------------------------------------- You can try reducing the maximum GPU usage during saving by changing `maximum_memory_usage`. The default is `model.save_pretrained(..., maximum_memory_usage = 0.75)`. Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving. [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-do-i-manually-save-to-gguf) ā“How do I manually save to GGUF? -------------------------------------------------------------------------------------------------------------------------------- First save your model to 16bit via: Copy model.save_pretrained_merged("merged_model", tokenizer, save_method = "merged_16bit",) Compile llama.cpp from source like below: Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp Then, save the model to F16: Copy python llama.cpp/convert_hf_to_gguf.py merged_model \ --outfile model-F16.gguf --outtype f16 \ --split-max-size 50G Copy # For BF16: python llama.cpp/convert_hf_to_gguf.py merged_model \ --outfile model-BF16.gguf --outtype bf16 \ --split-max-size 50G # For Q8_0: python llama.cpp/convert_hf_to_gguf.py merged_model \ --outfile model-Q8_0.gguf --outtype q8_0 \ --split-max-size 50G [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#why-is-q8_k_xl-slower-than-q8_0-gguf) ā“Why is Q8\_K\_XL slower than Q8\_0 GGUF? ----------------------------------------------------------------------------------------------------------------------------------------------- On Mac devices, it seems like that BF16 might be slower than F16. Q8\_K\_XL upcasts some layers to BF16, so hence the slowdown, We are actively changing our conversion process to make F16 the default choice for Q8\_K\_XL to reduce performance hits. [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-to-do-evaluation) ā“How to do Evaluation ----------------------------------------------------------------------------------------------------------- To set up evaluation in your training run, you first have to split your dataset into a training and test split. You should **always shuffle the selection of the dataset**, otherwise your evaluation is wrong! Copy new_dataset = dataset.train_test_split( test_size = 0.01, # 1% for test size can also be an integer for # of rows shuffle = True, # Should always set to True! seed = 3407, ) train_dataset = new_dataset["train"] # Dataset for training eval_dataset = new_dataset["test"] # Dataset for evaluation Then, we can set the training arguments to enable evaluation. Reminder evaluation can be very very slow especially if you set `eval_steps = 1` which means you are evaluating every single step. If you are, try reducing the eval\_dataset size to say 100 rows or something. Copy from trl import SFTTrainer, SFTConfig trainer = SFTTrainer( args = SFTConfig( fp16_full_eval = True, # Set this to reduce memory usage per_device_eval_batch_size = 2,# Increasing this will use more memory eval_accumulation_steps = 4, # You can increase this include of batch_size eval_strategy = "steps", # Runs eval every few steps or epochs. eval_steps = 1, # How many evaluations done per # of training steps ), train_dataset = new_dataset["train"], eval_dataset = new_dataset["test"], ... ) trainer.train() [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#evaluation-loop-out-of-memory-or-crashing) ā“Evaluation Loop - Out of Memory or crashing. -------------------------------------------------------------------------------------------------------------------------------------------------------- A common issue when you OOM is because you set your batch size too high. Set it lower than 2 to use less VRAM. Also use `fp16_full_eval=True` to use float16 for evaluation which cuts memory by 1/2. First split your training dataset into a train and test split. Set the trainer settings for evaluation to: Copy new_dataset = dataset.train_test_split(test_size = 0.01) from trl import SFTTrainer, SFTConfig trainer = SFTTrainer( args = SFTConfig( fp16_full_eval = True, per_device_eval_batch_size = 2, eval_accumulation_steps = 4, eval_strategy = "steps", eval_steps = 1, ), train_dataset = new_dataset["train"], eval_dataset = new_dataset["test"], ... ) This will cause no OOMs and make it somewhat faster. You can also use `bf16_full_eval=True` for bf16 machines. By default Unsloth should have set these flags on by default as of June 2025. [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-do-i-do-early-stopping) ā“How do I do Early Stopping? ------------------------------------------------------------------------------------------------------------------------ If you want to stop the finetuning / training run since the evaluation loss is not decreasing, then you can use early stopping which stops the training process. Use `EarlyStoppingCallback`. As usual, set up your trainer and your evaluation dataset. The below is used to stop the training run if the `eval_loss` (the evaluation loss) is not decreasing after 3 steps or so. Copy from trl import SFTConfig, SFTTrainer trainer = SFTTrainer( args = SFTConfig( fp16_full_eval = True, per_device_eval_batch_size = 2, eval_accumulation_steps = 4, output_dir = "training_checkpoints", # location of saved checkpoints for early stopping save_strategy = "steps", # save model every N steps save_steps = 10, # how many steps until we save the model save_total_limit = 3, # keep ony 3 saved checkpoints to save disk space eval_strategy = "steps", # evaluate every N steps eval_steps = 10, # how many steps until we do evaluation load_best_model_at_end = True, # MUST USE for early stopping metric_for_best_model = "eval_loss", # metric we want to early stop on greater_is_better = False, # the lower the eval loss, the better ), model = model, tokenizer = tokenizer, train_dataset = new_dataset["train"], eval_dataset = new_dataset["test"], ) We then add the callback which can also be customized: Copy from transformers import EarlyStoppingCallback early_stopping_callback = EarlyStoppingCallback( early_stopping_patience = 3, # How many steps we will wait if the eval loss doesn't decrease # For example the loss might increase, but decrease after 3 steps early_stopping_threshold = 0.0, # Can set higher - sets how much loss should decrease by until # we consider early stopping. For eg 0.01 means if loss was # 0.02 then 0.01, we consider to early stop the run. ) trainer.add_callback(early_stopping_callback) Then train the model as usual via `trainer.train() .` [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95) ā“Downloading gets stuck at 90 to 95% ---------------------------------------------------------------------------------------------------------------------------------------- If your model gets stuck at 90, 95% for a long time before you can disable some fast downloading processes to force downloads to be synchronous and to print out more error messages. Simply use `UNSLOTH_STABLE_DOWNLOADS=1` before any Unsloth import. Copy import os os.environ["UNSLOTH_STABLE_DOWNLOADS"] = "1" from unsloth import FastLanguageModel [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#runtimeerror-cuda-error-device-side-assert-triggered) ā“RuntimeError: CUDA error: device-side assert triggered ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Restart and run all, but place this at the start before any Unsloth import. Also please file a bug report asap thank you! Copy import os os.environ["UNSLOTH_COMPILE_DISABLE"] = "1" os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1" [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#all-labels-in-your-dataset-are-100.-training-losses-will-be-all-0) ā“All labels in your dataset are -100. Training losses will be all 0. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This means that your usage of `train_on_responses_only` is incorrect for that particular model. train\_on\_responses\_only allows you to mask the user question, and train your model to output the assistant response with higher weighting. This is known to increase accuracy by 1% or more. See our [**LoRA Hyperparameters Guide**](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) for more details. For Llama 3.1, 3.2, 3.3 type models, please use the below: Copy from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", ) For Gemma 2, 3. 3n models, use the below: Copy from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part = "user\n", response_part = "model\n", ) [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#some-weights-of-gemma3nforconditionalgeneration-were-not-initialized-from-the-model-checkpoint) ā“Some weights of Gemma3nForConditionalGeneration were not initialized from the model checkpoint --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is a critical error, since this means some weights are not parsed correctly, which will cause incorrect outputs. This can normally be fixed by upgrading Unsloth `pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo` Then upgrade transformers and timm: `pip install --upgrade --force-reinstall --no-cache-dir --no-deps transformers timm` However if the issue still persists, please file a bug report asap! [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#notimplementederror-a-utf-8-locale-is-required.-got-ansi) ā“NotImplementedError: A UTF-8 locale is required. Got ANSI ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ See https://github.com/googlecolab/colabtools/issues/3409 In a new cell, run the below: Copy import locale locale.getpreferredencoding = lambda: "UTF-8" [](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#citing-unsloth) šŸ“—Citing Unsloth ------------------------------------------------------------------------------------------------ If you are citing the usage of our model uploads, use the below Bibtex. This is for Qwen3-30B-A3B-GGUF Q8\_K\_XL: Copy @misc{unsloth_2025_qwen3_30b_a3b, author = {Unsloth AI and Han-Chen, Daniel and Han-Chen, Michael}, title = {Qwen3-30B-A3B-GGUF:Q8\_K\_XL}, year = {2025}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF}} } To cite the usage of our Github package or our work in general: Copy @misc{unsloth, author = {Unsloth AI and Han-Chen, Daniel and Han-Chen, Michael}, title = {Unsloth}, year = {2025}, publisher = {Github}, howpublished = {\url{https://github.com/unslothai/unsloth}} } [PreviousFinetuning from Last Checkpoint](https://docs.unsloth.ai/basics/finetuning-from-last-checkpoint) [NextUnsloth Environment Flags](https://docs.unsloth.ai/basics/troubleshooting-and-faqs/unsloth-environment-flags) Last updated 1 month ago Was this helpful? --- # gpt-oss: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! OpenAI releases '**gpt-oss-120b'** and '**gpt-oss-20b'**, two SOTA open language models under the Apache 2.0 license. Both 128k context models outperform similarly sized open models in reasoning, tool use, and agentic tasks. You can now run & fine-tune them locally with Unsloth! Aug 13: We've fixed our fine-tuning & inference notebooks making them much more stable! Update Unsloth using the new installation cells. It's best to train & use Unsloth quants due to our [fixes](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss) for the model. > [**Fine-tune**](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#fine-tuning-gpt-oss-with-unsloth) > **gpt-oss-20b for free with our** [**Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) Trained with [RL](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) , **gpt-oss-120b** rivals o4-mini and **gpt-oss-20b** rivals o3-mini. Both excel at function calling and CoT reasoning, surpassing o1 and GPT-4o. **Includes Unsloth's** [**chat template fixes**](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss) **. For best results, use our uploads & train with Unsloth!** [Run gpt-oss-20b](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-20b) [Run gpt-oss-120b](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-120b) [Fine-tune gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#fine-tuning-gpt-oss-with-unsloth) #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#gpt-oss-unsloth-ggufs) **gpt-oss - Unsloth GGUFs:** * 20B: [gpt-oss-**20B**](https://huggingface.co/unsloth/gpt-oss-20b-GGUF) * 120B: [gpt-oss-**120B**](https://huggingface.co/unsloth/gpt-oss-120b-GGUF) [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss) šŸ“œUnsloth fixes for gpt-oss ------------------------------------------------------------------------------------------------------------------------------ OpenAI released a standalone parsing and tokenization library called [Harmony](https://github.com/openai/harmony) which allows one to tokenize conversations to OpenAI's preferred format for gpt-oss. The official OpenAI [cookbook article](https://docs.unsloth.ai/) provides many more details on how to use the Harmony library. Inference engines generally use the jinja chat template instead and not the Harmony package, and we found some issues with them after comparing with Harmony directly. If you see below, the top is the correct rendered form as from Harmony. The below is the one rendered by the current jinja chat template. There are quite a few differences! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFqIrmxJhFtJutzMn5wLx%252FScreenshot%25202025-08-08%2520at%252008-19-49%2520Untitled151.ipynb%2520-%2520Colab.png%3Falt%3Dmedia%26token%3De740b75f-1634-45ad-9be7-55370d13cd7e&width=768&dpr=4&quality=100&sign=e3f93b3a&sv=2) We also made some functions to directly allow you to use OpenAI's Harmony library directly without a jinja chat template if you desire - you can simply parse in normal conversations like below: Copy messages = [\ {"role" : "user", "content" : "What is 1+1?"},\ {"role" : "assistant", "content" : "2"},\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Today's date is 2024-09-30."},\ {"role": "assistant", "content": "User asks: 'What is the weather in San Francisco?' We need to use get_current_temperature tool.", "thinking" : ""},\ {"role": "assistant", "content": "", "tool_calls": [{"name": "get_current_temperature", "arguments": '{"location": "San Francisco, California, United States", "unit": "celsius"}'}]},\ {"role": "tool", "name": "get_current_temperature", "content": '{"temperature": 19.9, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ ] Then use the `encode_conversations_with_harmony` function from Unsloth: Copy from unsloth_zoo import encode_conversations_with_harmony def encode_conversations_with_harmony( messages, reasoning_effort = "medium", add_generation_prompt = True, tool_calls = None, developer_instructions = None, model_identity = "You are ChatGPT, a large language model trained by OpenAI.", ) The harmony format includes multiple interesting things: 1. `reasoning_effort = "medium"` You can select low, medium or high, and this changes gpt-oss's reasoning budget - generally the higher the better the accuracy of the model. 2. `developer_instructions` is like a system prompt which you can add. 3. `model_identity` is best left alone - you can edit it, but we're unsure if custom ones will function. We find multiple issues with current jinja chat templates (there exists multiple implementations across the ecosystem): 1. Function and tool calls are rendered with `tojson`, which is fine it's a dict, but if it's a string, speech marks and other **symbols become backslashed**. 2. There are some **extra new lines** in the jinja template on some boundaries. 3. Tool calling thoughts from the model should have the `**analysis**` **tag and not** `**final**` **tag**. 4. Other chat templates seem to not utilize `<|channel|>final` at all - one should use this for the final assistant message. You should not use this for thinking traces or tool calls. Our chat templates for the GGUF, our BnB and BF16 uploads and all versions are fixed! For example when comparing both ours and Harmony's format, we get no different characters: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fq3pLyJyjBA7MTENhEX8S%252FScreenshot%25202025-08-08%2520at%252008-20-00%2520Untitled151.ipynb%2520-%2520Colab.png%3Falt%3Dmedia%26token%3Da02d2626-c535-4aa3-bd72-09bf5829ac8e&width=768&dpr=4&quality=100&sign=d0c7aec9&sv=2) ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#precision-issues) šŸ”¢ Precision issues We found multiple precision issues in Tesla T4 and float16 machines primarily since the model was trained using BF16, and so outliers and overflows existed. MXFP4 is not actually supported on Ampere and older GPUs, so Triton provides `tl.dot_scaled` for MXFP4 matrix multiplication. It upcasts the matrices to BF16 internaly on the fly. We made a [MXFP4 inference notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_(20B)-Inference.ipynb) as well in Tesla T4 Colab! [Software emulation](https://triton-lang.org/main/python-api/generated/triton.language.dot_scaled.html) enables targeting hardware architectures without native microscaling operation support. Right now for such case, microscaled lhs/rhs are upcasted to `bf16` element type beforehand for dot computation, We found if you use float16 as the mixed precision autocast data-type, you will get infinities after some time. To counteract this, we found doing the MoE in bfloat16, then leaving it in either bfloat16 or float32 precision. If older GPUs don't even have bfloat16 support (like T4), then float32 is used. We also change all precisions of operations (like the router) to float32 for float16 machines. [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#running-gpt-oss) šŸ–„ļø **Running gpt-oss** ---------------------------------------------------------------------------------------------------------------- Below are guides for the [20B](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-20b) and [120B](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-120b) variants of the model. Any quant smaller than F16, including 2-bit has minimal accuracy loss, since only some parts (e.g., attention layers) are lower bit while most remain full-precision. Thatā€™s why sizes are close to the F16 model; for example, the 2-bit (11.5 GB) version performs nearly the same as the full 16-bit (14 GB) one. Once llama.cpp supports better quantization for these models, we'll upload them ASAP. The `gpt-oss` models from OpenAI include a feature that allows users to adjust the model's "reasoning effort." This gives you control over the trade-off between the model's performance and its response speed (latency) which by the amount of token the model will use to think. The `gpt-oss` models offer three distinct levels of reasoning effort you can choose from: * **Low**: Optimized for tasks that need very fast responses and don't require complex, multi-step reasoning. * **Medium**: A balance between performance and speed. * **High**: Provides the strongest reasoning performance for tasks that require it, though this results in higher latency. ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#recommended-settings) āš™ļø Recommended Settings OpenAI recommends these inference settings for both models: `temperature=1.0`, `top_p=1.0`, `top_k=0` * **Temperature of 1.0** * Top\_K = 0 (or experiment with 100 for possible better results) * Top\_P = 1.0 * Recommended minimum context: 16,384 * Maximum context length window: 131,072 **Chat template:** Copy <|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-08-05\n\nReasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>user<|message|>Hello<|end|><|start|>assistant<|channel|>final<|message|>Hi there!<|end|><|start|>user<|message|>What is 1+1?<|end|><|start|>assistant The end of sentence/generation token: EOS is `<|return|>` ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-20b) Run gpt-oss-20B ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F5uMxZIFbSS7976wghYcR%252Fgpt-oss-20b.svg%3Falt%3Dmedia%26token%3D43e2694c-317b-49ec-9723-2c08e1cc9dd3&width=768&dpr=4&quality=100&sign=7e8691b8&sv=2) To achieve inference speeds of 6+ tokens per second for our Dynamic 4-bit quant, have at least **14GB of unified memory** (combined VRAM and RAM) or **14GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model youā€™re using. GGUF Link: [unsloth/gpt-oss-20b-GGUF](https://huggingface.co/unsloth/gpt-oss-20b-GGUF) **NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds. Follow the [**best practices above**](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#recommended-settings) . They're the same as the 120B model. You can run the model on Google Colab, Docker, LM Studio or llama.cpp for now. See below: > **You can run gpt-oss-20b for free with our** [**Google Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_(20B)-Inference.ipynb) #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#docker-run-gpt-oss-20b-tutorial) šŸ‹ Docker: Run gpt-oss-20b Tutorial If you already have Docker desktop, all your need to do is run the command below and you're done: Copy docker model pull hf.co/unsloth/gpt-oss-20b-GGUF:F16 #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#llama.cpp-run-gpt-oss-20b-tutorial) āœØ Llama.cpp: Run gpt-oss-20b Tutorial 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. You can directly pull from Hugging Face via: Copy ./llama.cpp/llama-cli \ -hf unsloth/gpt-oss-20b-GGUF:F16 \ --jinja -ngl 99 --threads -1 --ctx-size 16384 \ --temp 1.0 --top-p 1.0 --top-k 0 2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/gpt-oss-20b-GGUF", local_dir = "unsloth/gpt-oss-20b-GGUF", allow_patterns = ["*F16*"], ) ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#run-gpt-oss-120b) Run gpt-oss-120b: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FuelT8du9Slmb40yhLN9g%252Fgpt-oss-120b.svg%3Falt%3Dmedia%26token%3D3447826e-78fc-4732-b321-70dfd513804c&width=768&dpr=4&quality=100&sign=a2691f95&sv=2) To achieve inference speeds of 6+ tokens per second for our 1-bit quant, we recommend at least **66GB of unified memory** (combined VRAM and RAM) or **66GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model youā€™re using. GGUF Link: [unsloth/gpt-oss-120b-GGUF](https://huggingface.co/unsloth/gpt-oss-120b-GGUF) **NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds. Follow the [**best practices above**](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#recommended-settings) . They're the same as the 20B model. #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#llama.cpp-run-gpt-oss-120b-tutorial) šŸ“– Llama.cpp: Run gpt-oss-120b Tutorial For gpt-oss-120b, we will specifically use Llama.cpp for optimized inference. If you want a **full precision unquantized version**, use our `F16` versions! 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do: Copy ./llama.cpp/llama-cli \ -hf unsloth/gpt-oss-120b-GGUF:F16 \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 1.0 \ --min-p 0.0 \ --top-p 1.0 \ --top-k 0.0 \ 3. Or, download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions.. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/gpt-oss-120b-GGUF", local_dir = "unsloth/gpt-oss-120b-GGUF", allow_patterns = ["*F16*"], ) 4. Run the model in conversation mode and try any prompt. 5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. More options discussed [here](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#improving-generation-speed) . Copy ./llama.cpp/llama-cli \ --model unsloth/gpt-oss-120b-GGUF/gpt-oss-120b-F16.gguf \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 1.0 \ --min-p 0.0 \ --top-p 1.0 \ --top-k 0.0 \ ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#improving-generation-speed) šŸ› ļø Improving generation speed If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves. Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel) . You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#fine-tuning-gpt-oss-with-unsloth) šŸ¦„ Fine-tuning gpt-oss with Unsloth --------------------------------------------------------------------------------------------------------------------------------------------- Unsloth gpt-oss fine-tuning is 1.5x faster, uses 70% less VRAM, and supports 10x longer context lengths. gpt-oss-20b QLoRA training fits on a 14GB VRAM, and gpt-oss-120b works on 65GB VRAM. * **QLoRA requirements:** gpt-oss-20b = 14GB VRAM ā€¢ gpt-oss-120b = 65GB VRAM. * **BF16 LoRA requirements:** gpt-oss-20b = 44GB VRAM ā€¢ gpt-oss-120b = 210GB VRAM. Read our step-by-step tutorial for fine-tuning gpt-oss: [āš”Tutorial: How to Fine-tune gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) Aug 13: We've fixed our fine-tuning & inference notebooks making them much more stable! Update Unsloth using the new installation cells we put in the notebook. Currently you cannot load QLoRA fine-tuned gpt-oss models in frameworks other than Unsloth, however you can if you do LoRA fine-tuning and utilize our [bf16 weights](https://huggingface.co/unsloth/gpt-oss-20b-BF16) for fine-tuning. This means you **must** set `model_name = "unsloth/gpt-oss-20b-BF16".` Keep in mind VRAM usage will be 4x more so gpt-oss-20b will require about 45GB VRAM. We are working on better exporting for the model so stay tuned. Free Unsloth notebooks to fine-tune gpt-oss: * gpt-oss-20b [Reasoning + Conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) (recommended) * GRPO notebooks coming soon! Stay tuned! To fine-tune gpt-oss and leverage our latest updates, you must install the latest version of Unsloth: Copy pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo To enable export/usage of the model for use outside of Unsloth but with Hugging Face, llama.cpp, or vLLM, fine-tuning must be done with LoRA while leveraging our [bf16 weights](https://huggingface.co/unsloth/gpt-oss-20b-BF16) . Keep in mind VRAM usage will be 4x more so gpt-oss-20b will require 60GB VRAM. We are working on better exporting for the model so stay tuned. ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#making-efficient-gpt-oss-fine-tuning-work) šŸ’”Making efficient gpt-oss fine-tuning work We found that while MXFP4 is highly efficient, it does not natively support training with gpt-oss. To overcome this limitation, we implemented custom training functions specifically for MXFP4 layers through mimicking it via `Bitsandbytes` NF4 quantization. We utilized OpenAI's Triton Kernels library directly to allow MXFP4 inference. For finetuning / training however, the MXFP4 kernels do not yet support training, since the backwards pass is not yet implemented. We're actively working on implementing it in Triton! There is a flag called `W_TRANSPOSE` as mentioned [here](https://github.com/triton-lang/triton/blob/main/python/triton_kernels/triton_kernels/matmul_ogs_details/_matmul_ogs.py#L39) , which should be implemented. The derivative can be calculated by the transpose of the weight matrices, and so we have to implement the transpose operation. If you want to train gpt-oss with any library other than Unsloth, youā€™ll need to upcast the weights to bf16 before training. This approach, however, **significantly increases** both VRAM usage and training time by as much as **300% more memory usage**! **ALL other training methods will require a minimum of 65GB VRAM to train the 20b model while Unsloth only requires 14GB VRAM (-80%).** As both models use MoE architecture, the 20B model selects 4 experts out of 32, while the 120B model selects 4 out of 128 per token. During training and release, weights are stored in MXFP4 format as `nn.Parameter` objects, not as `nn.Linear` layers, which complicates quantization, especially since MoE/MLP experts make up about 19B of the 20B parameters. To enable `BitsandBytes` quantization and memory-efficient fine-tuning, we converted these parameters into `nn.Linear` layers. Although this slightly slows down operations, it allows fine-tuning on GPUs with limited memory, a worthwhile trade-off. ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#datasets-fine-tuning-guide) Datasets fine-tuning guide Though gpt-oss supports only reasoning, you can still fine-tune it with a non-reasoning [dataset](https://docs.unsloth.ai/basics/datasets-guide) , but this may affect its reasoning ability. If you want to maintain its reasoning capabilities (optional), you can use a mix of direct answers and chain-of-thought examples. Use at least 75% reasoning and 25% non-reasoning in your dataset to make the model retain its reasoning capabilities. Our gpt-oss-20b Conversational notebook uses OpenAI's example which is Hugging Face's Multilingual-Thinking dataset. The purpose of using this dataset is to enable the model to learn and develop reasoning capabilities in these four distinct languages. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQhnJE7SelxoTaAv6l8Ff%252Fwider%2520gptoss%2520image.png%3Falt%3Dmedia%26token%3Dfd8d11f2-0159-44aa-a773-4cd2668f0a78&width=768&dpr=4&quality=100&sign=2fda0b36&sv=2) [PreviousDeepSeek-V3.1](https://docs.unsloth.ai/basics/deepseek-v3.1) [NextTutorial: How to Fine-tune gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) Last updated 8 days ago Was this helpful? --- # Gemma 3: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Google releases Gemma 3 with a new 270M model and the previous 1B, 4B, 12B, and 27B sizes. The 270M and 1B are text-only, while larger models handle both text and vision. We provide GGUFs, and a guide of how to run it effectively, and how to finetune & do [RL](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) with Gemma 3! **NEW Aug 14, 2025 Update:** Try our fine-tuning [Gemma 3 (270M) notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) and [GGUFs to run](https://huggingface.co/collections/unsloth/gemma-3-67d12b7e8816ec6efa7e4e5b) . Also see our [Gemma 3n Guide](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) . [Running Tutorial](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#gmail-running-gemma-3-on-your-phone) [Fine-tuning Tutorial](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#fine-tuning-gemma-3-in-unsloth) **Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training.** This means Colab Notebooks with free Tesla T4 GPUs also work! * Fine-tune Gemma 3 (4B) with vision support using our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) According to the Gemma team, the optimal config for inference is `temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0` **Unsloth Gemma 3 uploads with optimal configs:** GGUF Unsloth Dynamic 4-bit Instruct 16-bit Instruct * [270M](https://huggingface.co/unsloth/gemma-3-270m-it-GGUF) - new * [1B](https://huggingface.co/unsloth/gemma-3-1b-it-GGUF) * [4B](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) * [12B](https://huggingface.co/unsloth/gemma-3-12b-it-GGUF) * [27B](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF) * [270M](https://huggingface.co/unsloth/gemma-3-270m-it-unsloth-bnb-4bit) * [1B](https://huggingface.co/unsloth/gemma-3-1b-it-bnb-4bit) * [4B](https://huggingface.co/unsloth/gemma-3-4b-it-bnb-4bit) * [12B](https://huggingface.co/unsloth/gemma-3-27b-it-unsloth-bnb-4bit) * [27B](https://huggingface.co/unsloth/gemma-3-27b-it-bnb-4bit) * [270M](https://huggingface.co/unsloth/gemma-3-270m-it) * [1B](https://huggingface.co/unsloth/gemma-3-1b) * [4B](https://huggingface.co/unsloth/gemma-3-4b) * [12B](https://huggingface.co/unsloth/gemma-3-12b) * [27B](https://huggingface.co/unsloth/gemma-3-27b) [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#official-recommended-inference-settings) āš™ļø Official Recommended Inference Settings ----------------------------------------------------------------------------------------------------------------------------------------------------------- According to the Gemma team, the official recommended settings for inference is: * Temperature of 1.0 * Top\_K of 64 * Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1) * Top\_P of 0.95 * Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers) * Chat template: Copy user\nHello!\nmodel\nHey there!\nuser\nWhat is 1+1?\nmodel\n * Chat template with `\n`newlines rendered (except for the last) Copy user Hello! model Hey there! user What is 1+1? model\n llama.cpp an other inference engines auto add a - DO NOT add TWO tokens! You should ignore the when prompting the model! ### [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#gmail-running-gemma-3-on-your-phone) āœØRunning Gemma 3 on your phone To run the models on your phone, we recommend using any mobile app that can run GGUFs locally on edge devices like phones. After fine-tuning you can export it to GGUF then run it locally on your phone. Ensure your phone has enough RAM/power to process the models as it can overheat so we recommend using Gemma 3 270M or the Gemma 3n models for this use-case. You can try the [open-source project AnythingLLM's](https://github.com/Mintplex-Labs/anything-llm) mobile app which you can download on [Android here](https://play.google.com/store/apps/details?id=com.anythingllm) or [ChatterUI](https://github.com/Vali-98/ChatterUI) , which are great apps for running GGUFs on your phone. Remember, you can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8\_K\_XL' for all the tutorials. [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#tutorial-how-to-run-gemma-3-in-ollama) šŸ¦™ Tutorial: How to Run Gemma 3 in Ollama -------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! You can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8\_K\_XL'. Copy ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_XL [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#tutorial-how-to-run-gemma-3-27b-in-llama.cpp) šŸ“– Tutorial: How to Run Gemma 3 27B in llama.cpp ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` Copy ./llama.cpp/llama-mtmd-cli \ -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL 1. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: [https://huggingface.co/unsloth/gemma-3-27b-it-GGUF](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/gemma-3-27b-it-GGUF", local_dir = "unsloth/gemma-3-27b-it-GGUF", allow_patterns = ["*Q4_K_XL*", "mmproj-BF16.gguf"], # For Q4_K_M ) 1. Run Unsloth's Flappy Bird test 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length (Gemma 3 supports 128K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. 3. For conversation mode: Copy ./llama.cpp/llama-mtmd-cli \ --model unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_XL.gguf \ --mmproj unsloth/gemma-3-27b-it-GGUF/mmproj-BF16.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 1.0 \ --repeat-penalty 1.0 \ --min-p 0.01 \ --top-k 64 \ --top-p 0.95 1. For non conversation mode to test Flappy Bird: Copy ./llama.cpp/llama-cli \ --model unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_XL.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 1.0 \ --repeat-penalty 1.0 \ --min-p 0.01 \ --top-k 64 \ --top-p 0.95 \ -no-cnv \ --prompt "user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.\nmodel\n" The full input from our [https://unsloth.ai/blog/deepseekr1-dynamic](https://unsloth.ai/blog/deepseekr1-dynamic) 1.58bit blog is: Remember to remove since Gemma 3 auto adds a ! Copy user Create a Flappy Bird game in Python. You must include these things: 1. You must use pygame. 2. The background color should be randomly chosen and is a light shade. Start with a light blue color. 3. Pressing SPACE multiple times will accelerate the bird. 4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color. 5. Place on the bottom some land colored as dark brown or yellow chosen randomly. 6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them. 7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade. 8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again. The final game should be inside a markdown section in Python. Check your code for error [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#fine-tuning-gemma-3-in-unsloth) šŸ¦„ Fine-tuning Gemma 3 in Unsloth ----------------------------------------------------------------------------------------------------------------------------------------- **Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training.** This means Colab Notebooks with free Tesla T4 GPUs also work! * Try our new [Gemma 3 (270M) notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) which makes the 270M parameter model very smart at playing chess and can predict the next chess move. * Fine-tune Gemma 3 (4B) using our notebooks for: [**Text**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) or [**Vision**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) * Or fine-tune [Gemma 3n (E4B)](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) with [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) ā€¢ [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) ā€¢ [Audio](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) ### [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#unsloth-fine-tuning-fixes) Unsloth Fine-tuning Fixes Our solution in Unsloth is 3 fold: 1. Keep all intermediate activations in bfloat16 format - can be float32, but this uses 2x more VRAM or RAM (via Unsloth's async gradient checkpointing) 2. Do all matrix multiplies in float16 with tensor cores, but manually upcasting / downcasting without the help of Pytorch's mixed precision autocast. 3. Upcast all other options that don't need matrix multiplies (layernorms) to float32. [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#gemma-3-fixes-analysis) šŸ¤” Gemma 3 Fixes Analysis ------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FpQGE6CEsuvGcQaOKrQFQ%252Foutput%281%29.png%3Falt%3Dmedia%26token%3D5f741769-3591-4a79-bb83-d6d58a4e9818&width=768&dpr=4&quality=100&sign=65c761c0&sv=2) Gemma 3 1B to 27B exceed float16's maximum of 65504 First, before we finetune or run Gemma 3, we found that when using float16 mixed precision, gradients and **activations become infinity** unfortunately. This happens in T4 GPUs, RTX 20x series and V100 GPUs where they only have float16 tensor cores. For newer GPUs like RTX 30x or higher, A100s, H100s etc, these GPUs have bfloat16 tensor cores, so this problem does not happen! **But why?** ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FXmN6s9dA64N3nvmi4Y4x%252Ffloat16%2520bfloat16.png%3Falt%3Dmedia%26token%3D3e1cb682-49d0-4083-b791-589cf01a05a8&width=768&dpr=4&quality=100&sign=b86bca81&sv=2) Wikipedia [https://en.wikipedia.org/wiki/Bfloat16\_floating-point\_format](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format) Float16 can only represent numbers up to **65504**, whilst bfloat16 can represent huge numbers up to **10^38**! But notice both number formats use only 16bits! This is because float16 allocates more bits so it can represent smaller decimals better, whilst bfloat16 cannot represent fractions well. But why float16? Let's just use float32! But unfortunately float32 in GPUs is very slow for matrix multiplications - sometimes 4 to 10x slower! So we cannot do this. [PreviousTutorial: How to Fine-tune gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) [NextQwen3-Coder: How to Run Locally](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally) Last updated 11 days ago Was this helpful? --- # Text-to-Speech (TTS) Fine-tuning | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Fine-tuning TTS models allows them to adapt to your specific dataset, use case, or desired style and tone. The goal is to customize these models to clone voices, adapt speaking styles and tones, support new languages, handle specific tasks and more. We also support **Speech-to-Text (STT)** models like OpenAI's Whisper. With [Unsloth](https://github.com/unslothai/unsloth) , you can fine-tune TTS models 1.5x faster with 50% less memory than other implementations with Flash Attention 2. This support includes Sesame CSM, Orpheus, and models supported by transformers (e.g. CrisperWhisper, Spark and more). Zero-shot cloning captures tone but misses pacing and expression, often sounding robotic and unnatural. Fine-tuning delivers far more accurate and realistic voice replication. [Read more here](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#fine-tuning-voice-models-vs.-zero-shot-voice-cloning) . We've uploaded TTS models (original and quantized variants) to our [Hugging Face page](https://huggingface.co/collections/unsloth/text-to-speech-tts-models-68007ab12522e96be1e02155) . ### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#fine-tuning-notebooks) Fine-tuning Notebooks: [Sesame-CSM (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb) [Orpheus-TTS (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) [Whisper Large V3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) Speech-to-Text (STT) [Spark-TTS (0.5B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb) [Llasa-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_(1B).ipynb) [Oute-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb) If you notice that the output duration reaches a maximum of 10 seconds, increase`max_new_tokens = 125` from its default value of 125. Since 125 tokens corresponds to 10 seconds of audio, you'll need to set a higher value for longer outputs. ### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#choosing-and-loading-a-tts-model) Choosing and Loading a TTS Model For TTS, smaller models are often preferred due to lower latency and faster inference for end users. Fine-tuning a model under 3B parameters is often ideal, and our primary examples uses Sesame-CSM (1B) and Orpheus-TTS (3B), a Llama-based speech model. #### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#sesame-csm-1b-details) Sesame-CSM (1B) Details **CSM-1B** is a base model, while **Orpheus-ft** is fine-tuned on 8 professional voice actors, making voice consistency the key difference. CSM requires audio context for each speaker to perform well, whereas Orpheus-ft has this consistency built in. Fine-tuning from a base model like CSM generally needs more compute, while starting from a fine-tuned model like Orpheus-ft offers better results out of the box. To help with CSM, weā€™ve added new sampling options and an example showing how to use audio context for improved voice consistency. #### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#orpheus-tts-3b-details) Orpheus-TTS (3B) Details Orpheus is pre-trained on a large speech corpus and excels at generating realistic speech with built-in support for emotional cues like laughs and sighs. Its architecture makes it one of the easiest TTS models to utilize and train as it can be exported via llama.cpp meaning it has great compatibility across all inference engines. For unsupported models, you'll only be able to save the LoRA adapter safetensors. #### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#loading-the-models) Loading the models Because voice models are usually small in size, you can train the models using LoRA 16-bit or full fine-tuning FFT which may provide higher quality results. To load it in LoRA 16-bit: Copy from unsloth import FastModel model_name = "unsloth/orpheus-3b-0.1-pretrained" model, tokenizer = FastModel.from_pretrained( model_name, load_in_4bit=False # use 4-bit precision (QLoRA) ) When this runs, Unsloth will download the model weights if you prefer 8-bit, you could use `load_in_8bit = True`, or for full fine-tuning set `full_finetuning = True` (ensure you have enough VRAM). You can also replace the model name with other TTS models. **Note:** Orpheusā€™s tokenizer already includes special tokens for audio output (more on this later). You do _not_ need a separate vocoder ā€“ Orpheus will output audio tokens directly, which can be decoded to a waveform. ### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#preparing-your-dataset) Preparing Your Dataset At minimum, a TTS fine-tuning dataset consists of **audio clips and their corresponding transcripts** (text). Letā€™s use the [_Elise_ dataset](https://huggingface.co/datasets/MrDragonFox/Elise) which is ~3 hour single-speaker English speech corpus. There are two variants: * [`MrDragonFox/Elise`](https://huggingface.co/datasets/MrDragonFox/Elise) ā€“ an augmented version with **emotion tags** (e.g. , ) embedded in the transcripts. These tags in angle brackets indicate expressions (laughter, sighs, etc.) and are treated as special tokens by Orpheusā€™s tokenizer * [`Jinsaryko/Elise`](https://huggingface.co/datasets/Jinsaryko/Elise) ā€“ base version with transcripts without special tags. The dataset is organized with one audio and transcript per entry. On Hugging Face, these datasets have fields such as `audio` (the waveform), `text` (the transcription), and some metadata (speaker name, pitch stats, etc.). We need to feed Unsloth a dataset of audio-text pairs. Instead of solely focusing on tone, cadence, and pitch, the priority should be ensuring your dataset is fully annotated and properly normalized. With some models like **Sesame-CSM-1B**, you might notice voice variation across generations using speaker ID 0 because it's a **base model**ā€”it doesnā€™t have fixed voice identities. Speaker ID tokens mainly help maintain **consistency within a conversation**, not across separate generations. To get a consistent voice, provide **contextual examples**, like a few reference audio clips or prior utterances. This helps the model mimic the desired voice more reliably. Without this, variation is expected, even with the same speaker ID. **Option 1: Using Hugging Face Datasets library** ā€“ We can load the Elise dataset using Hugging Faceā€™s `datasets` library: Copy from datasets import load_dataset, Audio # Load the Elise dataset (e.g., the version with emotion tags) dataset = load_dataset("MrDragonFox/Elise", split="train") print(len(dataset), "samples") # ~1200 samples in Elise # Ensure all audio is at 24 kHz sampling rate (Orpheusā€™s expected rate) dataset = dataset.cast_column("audio", Audio(sampling_rate=24000)) This will download the dataset (~328 MB for ~1.2k samples). Each item in `dataset` is a dictionary with at least: * `"audio"`: the audio clip (waveform array and metadata like sampling rate), and * `"text"`: the transcript string Orpheus supports tags like ``, ``, ``, ``, ``, ``, ``, ``, etc. For example: `"I missed you so much!"`. These tags are enclosed in angle brackets and will be treated as special tokens by the model (they match [Orpheusā€™s expected tags](https://github.com/canopyai/Orpheus-TTS) like `` and ``. During training, the model will learn to associate these tags with the corresponding audio patterns. The Elise dataset with tags already has many of these (e.g., 336 occurrences of ā€œlaughsā€, 156 of ā€œsighsā€, etc. as listed in its card). If your dataset lacks such tags but you want to incorporate them, you can manually annotate the transcripts where the audio contains those expressions. **Option 2: Preparing a custom dataset** ā€“ If you have your own audio files and transcripts: * Organize audio clips (WAV/FLAC files) in a folder. * Create a CSV or TSV file with columns for file path and transcript. For example: Copy filename,text 0001.wav,Hello there! 0002.wav, I am very tired. * Use `load_dataset("csv", data_files="mydata.csv", split="train")` to load it. You might need to tell the dataset loader how to handle audio paths. An alternative is using the `datasets.Audio` feature to load audio data on the fly: Copy from datasets import Audio dataset = load_dataset("csv", data_files="mydata.csv", split="train") dataset = dataset.cast_column("filename", Audio(sampling_rate=24000)) Then `dataset[i]["audio"]` will contain the audio array. * **Ensure transcripts are normalized** (no unusual characters that the tokenizer might not know, except the emotion tags if used). Also ensure all audio have a consistent sampling rate (resample them if necessary to the target rate the model expects, e.g. 24kHz for Orpheus). In summary, for **dataset preparation**: * You need a **list of (audio, text)** pairs. * Use the HF `datasets` library to handle loading and optional preprocessing (like resampling). * Include any **special tags** in the text that you want the model to learn (ensure they are in `` format so the model treats them as distinct tokens). * (Optional) If multi-speaker, you could include a speaker ID token in the text or use a separate speaker embedding approach, but thatā€™s beyond this basic guide (Elise is single-speaker). ### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#fine-tuning-tts-with-unsloth) Fine-Tuning TTS with Unsloth Now, letā€™s start fine-tuning! Weā€™ll illustrate using Python code (which you can run in a Jupyter notebook, Colab, etc.). **Step 1: Load the Model and Dataset** In all our TTS notebooks, we enable LoRA (16-bit) training and disable QLoRA (4-bit) training with: `load_in_4bit = False`. This is so the model can usually learn your dataset better and have higher accuracy. Copy from unsloth import FastLanguageModel import torch dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False. model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/orpheus-3b-0.1-ft", max_seq_length= 2048, # Choose any for long context! dtype = dtype, load_in_4bit = load_in_4bit, #token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf ) from datasets import load_dataset dataset = load_dataset("MrDragonFox/Elise", split = "train") If memory is very limited or if dataset is large, you can stream or load in chunks. Here, 3h of audio easily fits in RAM. If using your own dataset CSV, load it similarly. **Step 2: Advanced - Preprocess the data for training (Optional)** We need to prepare inputs for the Trainer. For text-to-speech, one approach is to train the model in a causal manner: concatenate text and audio token IDs as the target sequence. However, since Orpheus is a decoder-only LLM that outputs audio, we can feed the text as input (context) and have the audio token ids as labels. In practice, Unslothā€™s integration might do this automatically if the modelā€™s config identifies it as text-to-speech. If not, we can do something like: Copy # Tokenize the text transcripts def preprocess_function(example): # Tokenize the text (keep the special tokens like intact) tokens = tokenizer(example["text"], return_tensors="pt") # Flatten to list of token IDs input_ids = tokens["input_ids"].squeeze(0) # The model will generate audio tokens after these text tokens. # For training, we can set labels equal to input_ids (so it learns to predict next token). # But that only covers text tokens predicting the next text token (which might be an audio token or end). # A more sophisticated approach: append a special token indicating start of audio, and let the model generate the rest. # For simplicity, use the same input as labels (the model will learn to output the sequence given itself). return {"input_ids": input_ids, "labels": input_ids} train_data = dataset.map(preprocess_function, remove_columns=dataset.column_names) The above is a simplification. In reality, to fine-tune Orpheus properly, you would need the _audio tokens as part of the training labels_. Orpheusā€™s pre-training likely involved converting audio to discrete tokens (via an audio codec) and training the model to predict those given the preceding text. For fine-tuning on new voice data, you would similarly need to obtain the audio tokens for each clip (using Orpheusā€™s audio codec). The Orpheus GitHub provides a script for data processing ā€“ it encodes audio into sequences of `` tokens. However, **Unsloth may abstract this away**: if the model is a FastModel with an associated processor that knows how to handle audio, it might automatically encode the audio in the dataset to tokens. If not, youā€™d have to manually encode each audio clip to token IDs (using Orpheusā€™s codebook). This is an advanced step beyond this guide, but keep in mind that simply using text tokens wonā€™t teach the model the actual audio ā€“ it needs to match the audio patterns. Let's assume Unsloth provides a way to feed audio directly (for example, by setting `processor` and passing the audio array). If Unsloth does not yet support automatic audio tokenization, you might need to use the Orpheus repositoryā€™s `encode_audio` function to get token sequences for the audio, then use those as labels. (The dataset entries do have `phonemes` and some acoustic features which suggests a pipeline.) **Step 3: Set up training arguments and Trainer** Copy from transformers import TrainingArguments,Trainer,DataCollatorForSeq2Seq from unsloth import is_bfloat16_supported trainer = Trainer( model = model, train_dataset = dataset, args = TrainingArguments( per_device_train_batch_size = 1, gradient_accumulation_steps = 4, warmup_steps = 5, # num_train_epochs = 1, # Set this for 1 full training run. max_steps = 60, learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # Use this for WandB etc ), ) We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. Using a per\_device\_train\_batch\_size >1 may lead to errors if multi-GPU setup to avoid issues, ensure CUDA\_VISIBLE\_DEVICES is set to a single GPU (e.g., CUDA\_VISIBLE\_DEVICES=0). Adjust as needed. **Step 4: Begin fine-tuning** This will start the training loop. You should see logs of loss every 50 steps (as set by `logging_steps`). The training might take some time depending on GPU ā€“ for example, on a Colab T4 GPU, a few epochs on 3h of data may take 1-2 hours. Unslothā€™s optimizations will make it faster than standard HF training. **Step 5: Save the fine-tuned model** After training completes (or if you stop it mid-way when you feel itā€™s sufficient), save the model. This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down! Copy model.save_pretrained("lora_model") # Local saving tokenizer.save_pretrained("lora_model") # model.push_to_hub("your_name/lora_model", token = "...") # Online saving # tokenizer.push_to_hub("your_name/lora_model", token = "...") # Online saving This saves the model weights (for LoRA, it might save only adapter weights if the base is not fully fine-tuned). If you used `--push_model` in CLI or `trainer.push_to_hub()`, you could upload it to Hugging Face Hub directly. Now you should have a fine-tuned TTS model in the directory. The next step is to test it out and if supported, you can use llama.cpp to convert it into a GGUF file. ### [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning#fine-tuning-voice-models-vs.-zero-shot-voice-cloning) Fine-tuning Voice models vs. Zero-shot voice cloning People say you can clone a voice with just 30 seconds of audio using models like XTTS - no training required. Thatā€™s technically true, but it misses the point. Zero-shot voice cloning, which is also available in models like Orpheus and CSM, is an approximation. It captures the general **tone and timbre** of a speakerā€™s voice, but it doesnā€™t reproduce the full expressive range. You lose details like speaking speed, phrasing, vocal quirks, and the subtleties of prosody - things that give a voice its **personality and uniqueness**. If you just want a different voice and are fine with the same delivery patterns, zero-shot is usually good enough. But the speech will still follow the **modelā€™s style**, not the speakerā€™s. For anything more personalized or expressive, you need training with methods like LoRA to truly capture how someone speaks. [PreviousTraining AI Agents with RL](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/training-ai-agents-with-rl) [NextDatasets Guide](https://docs.unsloth.ai/basics/datasets-guide) Last updated 2 months ago Was this helpful? --- # Multi-GPU Training with Unsloth | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Unsloth currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed. This means you can already leverage parallelism methods such as **FSDP** and **DDP** with Unsloth. However, we know that the process can be complex and requires manual setup. Weā€™re working hard to make multi-GPU support much simpler and more user-friendly, and weā€™ll be announcing official multi-GPU support for Unsloth soon. **In the meantime**, to enable multi GPU for DDP, do the following: 1. Save your training script to `train.py` and set in `SFTConfig` or `TrainingArguments` the flag `ddp_find_unused_parameters = False` 2. Run `accelerate launch train.py` or `torchrun --nproc_per_node N_GPUS -m train.py` where N\_GPUS is the number of GPUs you have. **Pipeline / model splitting loading** is also allowed, so if you do not have enough VRAM for 1 GPU to load say Llama 70B, no worries - we will split the model for you on each GPU! To enable this, use the `device_map = "balanced"` flag: Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( "unsloth/Llama-3.3-70B-Instruct", load_in_4bit = True, device_map = "balanced", ) Also several contributors have created repos to enable or improve multi-GPU support with Unsloth, including: * [unsloth-5090-multiple](https://github.com/thad0ctor/unsloth-5090-multiple) : A fork enabling Unsloth to run efficiently on multi-GPU systems, particularly for the NVIDIA [RTX 5090](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) and similar setups. * [opensloth](https://github.com/anhvth/opensloth) : Unsloth with support for multi-GPU training including experimental features. **Stay tuned for our official announcement!** For more details, check out our ongoing [Pull Request](https://github.com/unslothai/unsloth/issues/2435) discussing multi-GPU support. [PreviousUnsloth Benchmarks](https://docs.unsloth.ai/basics/unsloth-benchmarks) Last updated 1 month ago Was this helpful? --- # Continued Pretraining | Unsloth Documentation OpenAI gpt-oss & all model types now supported! * The [text completion notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) is for continued pretraining/raw text. * The [continued pretraining notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) is for learning another language. You can read more about continued pretraining and our release in our [blog post](https://unsloth.ai/blog/contpretraining) . [](https://docs.unsloth.ai/basics/continued-pretraining#what-is-continued-pretraining) What is Continued Pretraining? -------------------------------------------------------------------------------------------------------------------------- Continued or continual pretraining (CPT) is necessary to ā€œsteerā€ the language model to understand new domains of knowledge, or out of distribution domains. Base models like Llama-3 8b or Mistral 7b are first pretrained on gigantic datasets of trillions of tokens (Llama-3 for e.g. is 15 trillion). But sometimes these models have not been well trained on other languages, or text specific domains, like law, medicine or other areas. So continued pretraining (CPT) is necessary to make the language model learn new tokens or datasets. [](https://docs.unsloth.ai/basics/continued-pretraining#advanced-features) Advanced Features: -------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/continued-pretraining#loading-lora-adapters-for-continued-finetuning) Loading LoRA adapters for continued finetuning If you saved a LoRA adapter through Unsloth, you can also continue training using your LoRA weights. The optimizer state will be reset as well. To load even optimizer states to continue finetuning, see the next section. Copy from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "LORA_MODEL_NAME", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) trainer = Trainer(...) trainer.train() ### [](https://docs.unsloth.ai/basics/continued-pretraining#continued-pretraining-and-finetuning-the-lm_head-and-embed_tokens-matrices) Continued Pretraining & Finetuning the `lm_head` and `embed_tokens` matrices Add `lm_head` and `embed_tokens`. For Colab, sometimes you will go out of memory for Llama-3 8b. If so, just add `lm_head`. Copy model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",\ "lm_head", "embed_tokens",], lora_alpha = 16, ) Then use 2 different learning rates - a 2-10x smaller one for the `lm_head` or `embed_tokens` like so: Copy from unsloth import UnslothTrainer, UnslothTrainingArguments trainer = UnslothTrainer( .... args = UnslothTrainingArguments( .... learning_rate = 5e-5, embedding_learning_rate = 5e-6, # 2-10x smaller than learning_rate ), ) [PreviousInference](https://docs.unsloth.ai/basics/running-and-saving-models/inference) [NextChat Templates](https://docs.unsloth.ai/basics/chat-templates) Last updated 5 months ago Was this helpful? --- # Gemma 3n: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Googleā€™s Gemma 3n multimodal model handles image, audio, video, and text inputs. Available in 2B and 4B sizes, it supports 140 languages for text and multimodal tasks. You can now run and fine-tune **Gemma-3n-E4B** and **E2B** locally using [Unsloth](https://github.com/unslothai/unsloth) . > **Fine-tune Gemma 3n with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) Gemma 3n has **32K context length**, 30s audio input, OCR, auto speech recognition (ASR), and speech translation via prompts. [Running Tutorial](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#running-gemma-3n) [Fine-tuning Tutorial](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#fine-tuning-gemma-3n-with-unsloth) [Fixes + Technical Analysis](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#fixes-for-gemma-3n) **Unsloth Gemma 3n (Instruct) uploads with optimal configs:** Dynamic 2.0 GGUF (text only) Dynamic 4-bit Instruct (to fine-tune) 16-bit Instruct * [2B](https://huggingface.co/unsloth/gemma-3n-E2B-it-GGUF) * [4B](https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF) * [2B](https://huggingface.co/unsloth/gemma-3n-E2B-it-unsloth-bnb-4bit) * [4B](https://huggingface.co/unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit) * [2B](https://huggingface.co/unsloth/gemma-3n-E2B-it) * [4B](https://huggingface.co/unsloth/gemma-3n-E4B-it) **See all our Gemma 3n uploads including base and more formats in** [**our collection here**](https://huggingface.co/collections/unsloth/gemma-3n-685d3874830e49e1c93f9339) **.** [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#running-gemma-3n) šŸ–„ļø Running Gemma 3n --------------------------------------------------------------------------------------------------------------- Currently Gemma 3n is only supported in **text format** for inference. Weā€™ve [fixed issues](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#fixes-for-gemma-3n) with GGUFs not working properly in Ollama only. Please redownload if using Ollama. ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings According to the Gemma team, the official recommended settings for inference: `temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0` * Temperature of 1.0 * Top\_K of 64 * Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1) * Top\_P of 0.95 * Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers) * Chat template: Copy user\nHello!\nmodel\nHey there!\nuser\nWhat is 1+1?\nmodel\n * Chat template with `\n`newlines rendered (except for the last) Copy user Hello! model Hey there! user What is 1+1? model\n llama.cpp an other inference engines auto add a - DO NOT add TWO tokens! You should ignore the when prompting the model! ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#tutorial-how-to-run-gemma-3n-in-ollama) šŸ¦™ Tutorial: How to Run Gemma 3n in Ollama Please re download Gemma 3N quants or remove the old ones via Ollama since there are some bug fixes. You can do the below to delete the old file and refresh it: Copy ollama rm hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL ollama run hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#tutorial-how-to-run-gemma-3n-in-llama.cpp) šŸ“– Tutorial: How to Run Gemma 3n in llama.cpp We would first like to thank [Xuan-Son Nguyen](https://x.com/ngxson) from Hugging Face, [Georgi Gerganov](https://x.com/ggerganov) from the llama.cpp team on making Gemma 3N work in llama.cpp! 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` Copy ./llama.cpp/llama-cli -hf unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL -ngl 99 --jinja 1. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/gemma-3n-E4B-it-GGUF", local_dir = "unsloth/gemma-3n-E4B-it-GGUF", allow_patterns = ["*UD-Q4_K_XL*", "mmproj-BF16.gguf"], # For Q4_K_XL ) 1. Run the model. 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 32768` for context length (Gemma 3 supports 32K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. 3. For conversation mode: Copy ./llama.cpp/llama-cli \ --model unsloth/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-UD-Q4_K_XL.gguf \ --ctx-size 32768 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 1.0 \ --repeat-penalty 1.0 \ --min-p 0.00 \ --top-k 64 \ --top-p 0.95 1. For non conversation mode to test Flappy Bird: Copy ./llama.cpp/llama-cli \ --model unsloth/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-UD-Q4_K_XL.gguf \ --ctx-size 32768 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 1.0 \ --repeat-penalty 1.0 \ --min-p 0.00 \ --top-k 64 \ --top-p 0.95 \ -no-cnv \ --prompt "user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.\nmodel\n" Remember to remove since Gemma 3N auto adds a ! [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#fine-tuning-gemma-3n-with-unsloth) šŸ¦„ Fine-tuning Gemma 3n with Unsloth ------------------------------------------------------------------------------------------------------------------------------------------------ Gemma 3n, like [Gemma 3](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#unsloth-fine-tuning-fixes-for-gemma-3) , had issues running on **Flotat16 GPUs such as Tesla T4s in Colab**. You will encounter NaNs and infinities if you do not patch Gemma 3n for inference or finetuning. [More information below](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#infinities-and-nan-gradients-and-activations) . * Fine-tune Gemma 3n-E4B with our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) * **Audio:** Fine-tune Gemma 3n-E4B with our [**Audio only notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) * **Vision**: Fine-tune Gemma 3n-E4B with our [**Vision only notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Vision.ipynb) We also found that because Gemma 3n's unique architecture reuses hidden states in the vision encoder it poses another interesting quirk with [Gradient Checkpointing described below](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#gradient-checkpointing-issues) **Unsloth is the only framework which works in float16 machines for Gemma 3n inference and training.** This means Colab Notebooks with free Tesla T4 GPUs also work! Overall, Unsloth makes Gemma 3n training 1.5x faster, 50% less VRAM and 4x longer context lengths. Our free Gemma 3n Colab notebooks default to fine-tuning text layers. If you want to fine-tune vision or audio layers too, be aware this will require much more VRAM - beyond the 15GB free Colab or Kaggle provides. You _can_ still fine-tune all layers including audio and vision and Unsloth also lets you fine-tune only specific areas, like just vision. Simply adjust as needed: Copy model = FastVisionModel.get_peft_model( model, finetune_vision_layers = False, # False if not finetuning vision layers finetune_language_layers = True, # False if not finetuning language layers finetune_attention_modules = True, # False if not finetuning attention layers finetune_mlp_modules = True, # False if not finetuning MLP layers ) ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#bonus-content) šŸ†Bonus Content We also heard you guys wanted a **Vision notebook for Gemma 3 (4B)** so here it is: * Fine-tune Gemma 3 (4B) with Vision support using our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) If you love Kaggle, Google is holding a competition where the best model fine-tuned with Gemma 3n and Unsloth will win a $10K prize! [See more here](https://www.kaggle.com/competitions/google-gemma-3n-hackathon) . [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#fixes-for-gemma-3n) šŸ›Fixes for Gemma 3n ----------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#gguf-issues-and-fixes) āœØGGUF issues & fixes Thanks to discussions from [Michael](https://github.com/mxyng) from the Ollama team and also [Xuan](https://x.com/ngxson) from Hugging Face, there were 2 issues we had to fix specifically for GGUFs: 1. The `add_shared_kv_layers` parameter was accidentally encoded in `float32` which is fine, but becomes slightly complicated to decode on Ollama's side - a simple change to `uint32` solves the issue. [Pull request](https://github.com/ggml-org/llama.cpp/pull/14450) addressing this issue. 2. The `per_layer_token_embd` layer should be Q8\_0 in precision. Anything lower does not function properly and errors out in the Ollama engine - to reduce issues for our community, we made this all Q8\_0 in all quants - unfortunately this does use more space. 1. As an [update](https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF/discussions/4) , [Matt](https://huggingface.co/WBB2500) mentioned we can also use Q4\_0, Q4\_1, Q5\_0, Q5\_1 for the embeddings - and we confirmed it does also work in Ollama! This means once again the smaller 2, 3 and 4bit quants are smaller in size, and don't need Q8\_0! [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#infinities-and-nan-gradients-and-activations) ā™¾ļøInfinities and NaN gradients and activations --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Gemma 3n just like Gemma 3 has issues on FP16 GPUs (e.g., Tesla T4s in Colab). Our previous fixes for Gemma 3 is [discussed here](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune) . For Gemma 3, we found that activations exceed float16's maximum range of **65504.** **Gemma 3N does not have this activation issue, but we still managed to encounter infinities!** ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FT7ywaXnZdAWFZIH3mG4Y%252FGemma%25203%2520activation.webp%3Falt%3Dmedia%26token%3Da8f9eb2d-e5a1-4b5f-ad10-91f69faa5640&width=768&dpr=4&quality=100&sign=bce5692a&sv=2) To get to the bottom of these infinities, we plotted the absolute maximum weight entries for Gemma 3N, and we see the below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FP1JdiiD7Szrja7jJ6pd9%252Foutput2.webp%3Falt%3Dmedia%26token%3D6df7dee4-6944-47d5-ae46-c67ca20360ad&width=768&dpr=4&quality=100&sign=a4ca82be&sv=2) We find that the green crosses are the Conv2D convolutional weights. We can see that the magnitude of Conv2D layers is much larger on average. Below is a table for Conv2D weights which have large magnitudes. Our hypothesis is that during a Conv2D operation, large weights multiply and sum together, and **unfortunately by chance exceed float16's maximum range of 65504.** Bfloat16 is fine, since it's maximum range is 10^38. Name Max msfa.ffn.pw\_proj.conv.weight 98.000000 blocks.2.21.attn.key.down\_conv.weight 37.000000 blocks.2.32.pw\_exp.conv.weight 34.750000 blocks.2.30.pw\_exp.conv.weight 33.750000 blocks.2.34.pw\_exp.conv.weight 33.750000 ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#solution-to-infinities) šŸŽ‡Solution to infinities The naive solution is to `upcast` all Conv2D weights to float32 (if bfloat16 isn't available). But that would increase VRAM usage. To tackle this, we instead make use of `autocast` on the fly to upcast the weights and inputs to float32, and so we perform the accumulation in float32 as part of the matrix multiplication itself, without having to upcast the weights. Unsloth is the only framework that enables Gemma 3n inference and training on float16 GPUs, so Colab Notebooks with free Tesla T4s work! ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#gradient-checkpointing-issues) šŸGradient Checkpointing issues We found Gemma 3N's vision encoder to be quite unique as well since it re-uses hidden states. This unfortunately limits the usage of [Unsloth's gradient checkpointing](https://unsloth.ai/blog/long-context) , which could have reduced VRAM usage significantly. since it cannot be applied to Vision encoder. However, we still managed to leverage **Unsloth's automatic compiler** to optimize Gemma 3N! ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#large-losses-during-finetuning) šŸŒµLarge losses during finetuning We also found losses are interestingly very large during the start of finetuning - in the range of 6 to 7, but they do decrease over time quickly. We theorize this is either because of 2 possibilities: 1. There might be some implementation issue, but this is unlikely since inference seems to work. 2. **Multi-modal models always seem to exhibit this behavior** \- we found Llama 3.2 Vision's loss starts at 3 or 4, Pixtral at 8 or so, and Qwen 2.5 VL also 4 ish. Because Gemma 3N includes audio as well, it might amplify the starting loss. But this is just a hypothesis. We also found quantizing Qwen 2.5 VL 72B Instruct to have extremely high perplexity scores of around 30 or so, but the model interestingly performs fine. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FsrKSTYQlCWGyZnCapR6H%252Foutput%283%29.png%3Falt%3Dmedia%26token%3D0e0ae6e8-ec44-4b72-a3c8-b376729e841e&width=768&dpr=4&quality=100&sign=4ab53edb&sv=2) **Fine-tune Gemma 3n with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#technical-analysis) šŸ› ļø Technical Analysis ------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune#gemma-3n-matformer) Gemma 3n : MatFormer So what is so special about Gemma 3n you ask? It is based on [Matryoshka Transformer or MatFormer](https://arxiv.org/abs/2310.07707) architecture meaning that each transformer layer/block embeds/nests FFNs of progressively smaller sizes. Think of it like progressively smaller cups put inside one another. The training is done so that at inference time you can choose the size you want and get the most of the performance of the bigger models. There is also Per Layer Embedding which can be cached to reduce memory usage at inference time. So the 2B model (E2B) is a sub-network inside the 4B (aka 5.44B) model that is achieved by both Per Layer Embedding caching and skipping audio and vision components focusing solely on text. The MatFormer architecture, typically is trained with exponentially spaced sub-models aka of sizes `S`, `S/2, S/4, S/8` etc in each of the layers. So at training time, inputs are randomly forwarded through one of the said sub blocks giving every sub block equal chance to learn. Now the advantage is, at inference time, if you want the model to be 1/4th of the original size, you can pick `S/4` sized sub blocks in each layer. You can also choose to **Mix and Match** where you pick say, `S/4` sized sub block of one layer, `S/2` sized sub block of another layer and `S/8` sized sub block of another layer. In fact, you can change the sub models you pick based on the input itself if you fancy so. Basically its like choose your own kind of structure at every layer. So by just training a model of one particular size, you are creating exponentially many models of smaller sizes. No learning goes waste. Pretty neat huh. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fuv5heIQYmrCOrrIt2lIe%252Fimage.png%3Falt%3Dmedia%26token%3D6bdcae8e-a39c-4994-80e1-37ae8c2938ac&width=768&dpr=4&quality=100&sign=c0a89798&sv=2) Image from [Gemma 3n model overview](https://ai.google.dev/gemma/docs/gemma-3n) **Fine-tune and try multimodal Gemma 3n inference with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) [PreviousKimi K2: How to Run Locally](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally) [NextDeepSeek-R1-0528: How to Run Locally](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) Last updated 27 days ago Was this helpful? --- # Cogito v2: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Cogito v2 Preview is [Deep Cogito](https://www.deepcogito.com/) 's latest release of models that spans 4 model sizes ranging from 70B to 671B. By using **IDA (Iteratedā€ÆDistillationā€Æ& Amplification)**, these models are trained with the model internalizing the reasoning process using iterative policy improvement, rather than simply searching longer at inference time (like DeepSeek R1). Deep Cogito is based in [San Fransisco, USA](https://techcrunch.com/2025/04/08/deep-cogito-emerges-from-stealth-with-hybrid-ai-reasoning-models/) (like Unsloth šŸ‡ŗšŸ‡ø) and we're excited to provide quantized dynamic models for all 4 model sizes! All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized these LLMs with minimal accuracy loss! **Tutorials navigation:** [Run 671B MoE](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp) [Run 109B MoE](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-109b-moe-in-llama.cpp) [Run 405B Dense](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-405b-dense-in-llama.cpp) [Run 70B Dense](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-70b-dense-in-llama.cpp) Choose which model size fits your hardware! We upload 1.58bit to 16bit variants for all 4 model sizes! [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#model-sizes-and-uploads) šŸ’Ž Model Sizes and Uploads --------------------------------------------------------------------------------------------------------------------------------------------------------------- There are 4 model sizes: 1. 2 Dense models based off from Llama - 70B and 405B 2. 2 MoE models based off from Llama 4 Scout (109B) and DeepSeek R1 (671B) Model Sizes Recommended Quant & Link Disk Size Architecture 70B Dense [UD-Q4\_K\_XL](https://huggingface.co/unsloth/cogito-v2-preview-llama-70B-GGUF) **44GB** Llama 3 70B 109B MoE [UD-Q3\_K\_XL](https://huggingface.co/unsloth/cogito-v2-preview-llama-109B-MoE-GGUF) **50GB** Llama 4 Scout 405B Dense [UD-Q2\_K\_XL](https://huggingface.co/unsloth/cogito-v2-preview-llama-405B-GGUF) **152GB** Llama 3 405B 671B MoE [UD-Q2\_K\_XL](https://huggingface.co/unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF) **251GB** DeepSeek R1 Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading. If you have less VRAM + RAM, then the quant will still function, just be much slower. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp) šŸ³ Run Cogito 671B MoE in llama.cpp --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. Copy export LLAMA_CACHE="unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF:Q2_K_XL \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . We **recommend using our 2.7bit dynamic quant** `**UD-Q2_K_XL**` **to balance size and accuracy**. More versions at: [https://huggingface.co/unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF](https://huggingface.co/unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF", local_dir = "unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF", allow_patterns = ["*UD-IQ1_S*"], # Dynamic 1bit (168GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (251GB) ) 1. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-109b-moe-in-llama.cpp) šŸ–±ļøRun Cogito 109B MoE in llama.cpp --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Follow the same instructions as running the [671B model above](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp) . 2. Then run the below: Copy export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-109B-MoE-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/cogito-v2-preview-llama-109B-MoE-GGUF:Q3_K_XL \ --cache-type-k q4_0 \ --n-gpu-layers 99 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ --ctx-size 16384 \ -ot ".ffn_.*_exps.=CPU" [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-405b-dense-in-llama.cpp) šŸŒ³Run Cogito 405B Dense in llama.cpp ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. Follow the same instructions as running the [671B model above](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp) . 2. Then run the below: Copy export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-405B-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/cogito-v2-preview-llama-405B-GGUF:Q2_K_XL \ --cache-type-k q4_0 \ --n-gpu-layers 99 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ --ctx-size 16384 [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-70b-dense-in-llama.cpp) šŸ˜Ž Run Cogito 70B Dense in llama.cpp ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Follow the same instructions as running the [671B model above](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp) . 2. Then run the below: Copy export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-70B-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/cogito-v2-preview-llama-70B-GGUF:Q4_K_XL \ --cache-type-k q4_0 \ --n-gpu-layers 99 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ --ctx-size 16384 [PreviousPhi-4 Reasoning: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune) [NextRunning & Saving Models](https://docs.unsloth.ai/basics/running-and-saving-models) Last updated 26 days ago Was this helpful? --- # Qwen3: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Qwen's new Qwen3 models deliver state-of-the-art advancements in reasoning, instruction-following, agent capabilities, and multilingual support. **NEW!** Qwen3 got an update in July 2025. Run & fine-tune the latest model: [**Qwen-2507**](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized Qwen LLMs with minimal accuracy loss. We also uploaded Qwen3 with native 128K context length. Qwen achieves this by using YaRN to extend its original 40K window to 128K. [Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [Reinforcement Learning (RL)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) of Qwen3 and Qwen3 MOE models ā€” 2x faster, with 70% less VRAM, and 8x longer context lengths. Fine-tune Qwen3 (14B) for free using our [Colab notebook.](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) [Running Qwen3 Tutorial](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#running-qwen3) [Fine-tuning Qwen3](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#qwen3-unsloth-dynamic-2.0-with-optimal-configs) **Qwen3 - Unsloth Dynamic 2.0** with optimal configs: Dynamic 2.0 GGUF (to run) 128K Context GGUF Dynamic 4-bit Safetensor (to finetune/deploy) * [0.6B](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) * [1.7B](https://huggingface.co/unsloth/Qwen3-1.7B-GGUF) * [4B](https://huggingface.co/unsloth/Qwen3-4B-GGUF) * [8B](https://huggingface.co/unsloth/Qwen3-8B-GGUF) * [14B](https://huggingface.co/unsloth/Qwen3-14B-GGUF) * [30B-A3B](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) * [32B](https://huggingface.co/unsloth/Qwen3-32B-GGUF) * [235B-A22B](https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF) * [4B](https://huggingface.co/unsloth/Qwen3-4B-128K-GGUF) * [8B](https://huggingface.co/unsloth/Qwen3-8B-128K-GGUF) * [14B](https://huggingface.co/unsloth/Qwen3-14B-128K-GGUF) * [30B-A3B](https://huggingface.co/unsloth/Qwen3-30B-A3B-128K-GGUF) * [32B](https://huggingface.co/unsloth/Qwen3-32B-128K-GGUF) * [235B-A22B](https://huggingface.co/unsloth/Qwen3-235B-A22B-128K-GGUF) * [0.6B](https://huggingface.co/unsloth/Qwen3-0.6B-unsloth-bnb-4bit) * [1.7B](https://huggingface.co/unsloth/Qwen3-1.7B-unsloth-bnb-4bit) * [4B](https://huggingface.co/unsloth/Qwen3-4B-unsloth-bnb-4bit) * [8B](https://huggingface.co/unsloth/Qwen3-8B-unsloth-bnb-4bit) * [14B](https://huggingface.co/unsloth/Qwen3-14B-unsloth-bnb-4bit) * [30B-A3B](https://huggingface.co/unsloth/Qwen3-30B-A3B-bnb-4bit) * [32B](https://huggingface.co/unsloth/Qwen3-32B-unsloth-bnb-4bit) [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#running-qwen3) šŸ–„ļø **Running Qwen3** ---------------------------------------------------------------------------------------------------------- To achieve inference speeds of 6+ tokens per second, we recommend your available memory should match or exceed the size of the model youā€™re using. For example, a 30GB 1-bit quantized model requires at least 150GB of memory. The Q2\_K\_XL quant, which is 180GB, will require at least **180GB of unified memory** (VRAM + RAM) or **180GB of RAM** for optimal performance. **NOTE:** Itā€™s possible to run the model with **less total memory** than its size (i.e., less VRAM, less RAM, or a lower combined total). However, this will result in slower inference speeds. Sufficient memory is only required if you want to maximize throughput and achieve the fastest inference times. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings According to Qwen, these are the recommended settings for inference: Non-Thinking Mode Settings: Thinking Mode Settings: **Temperature = 0.7** **Temperature = 0.6** Min\_P = 0.0 (optional, but 0.01 works well, llama.cpp default is 0.1) Min\_P = 0.0 Top\_P = 0.8 Top\_P = 0.95 TopK = 20 TopK = 20 **Chat template/prompt format:** Copy <|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n For NON thinking mode, we purposely enclose and with nothing: Copy <|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n\n\n\n\n **For Thinking-mode, DO NOT use greedy decoding**, as it can lead to performance degradation and endless repetitions. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#switching-between-thinking-and-non-thinking-mode) Switching Between Thinking and Non-Thinking Mode Qwen3 models come with built-in "thinking mode" to boost reasoning and improve response quality - similar to how [QwQ-32B](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively) worked. Instructions for switching will differ depending on the inference engine you're using so ensure you use the correct instructions. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#instructions-for-llama.cpp-and-ollama) Instructions for llama.cpp and Ollama: You can add `/think` and `/no_think` to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations. Here is an example of multi-turn conversation: Copy > Who are you /no_think I am Qwen, a large-scale language model developed by Alibaba Cloud. [...] > How many 'r's are in 'strawberries'? /think Okay, let's see. The user is asking how many times the letter 'r' appears in the word "strawberries". [...] The word strawberries contains 3 instances of the letter r. [...] #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#instructions-for-transformers-and-vllm) Instructions for transformers and vLLM: **Thinking mode:** `enable_thinking=True` By default, Qwen3 has thinking enabled. When you call `tokenizer.apply_chat_template`, you **donā€™t need to set anything manually.** Copy text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True # Default is True ) In thinking mode, the model will generate an extra `...` block before the final answer ā€” this lets it "plan" and sharpen its responses. **Non-thinking mode:** `enable_thinking=False` Enabling non-thinking will make Qwen3 will skip all the thinking steps and behave like a normal LLM. Copy text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False # Disables thinking mode ) This mode will provide final responses directly ā€” no `` blocks, no chain-of-thought. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#ollama-run-qwen3-tutorial) šŸ¦™ Ollama: Run Qwen3 Tutorial 1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 235B-A22B model, [see here](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#running-qwen3-235b-a22b) . Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL 1. To disable thinking, use (or you can set it in the system prompt): Copy >>> Write your prompt here /nothink If you're experiencing any looping, Ollama might have set your context length window to 2,048 or so. If this is the case, bump it up to 32,000 and see if the issue still persists. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#llama.cpp-run-qwen3-tutorial) šŸ“– Llama.cpp: Run Qwen3 Tutorial 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-14B-GGUF", local_dir = "unsloth/Qwen3-14B-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) 1. Run the model and try any prompt. To disable thinking, use (or you can set it in the system prompt): Copy >>> Write your prompt here /nothink ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#running-qwen3-235b-a22b) Running Qwen3-235B-A22B For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options. 1. We're following similar steps to above however this time we'll also need to perform extra steps because the model is so big. 2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions.. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-235B-A22B-GGUF", local_dir = "unsloth/Qwen3-235B-A22B-GGUF", allow_patterns = ["*UD-Q2_K_XL*"], ) 3. Run the model and try any prompt. 4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. Copy ./llama.cpp/llama-cli \ --model unsloth/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-UD-Q2_K_XL.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --seed 3407 \ --prio 3 \ --temp 0.6 \ --min-p 0.0 \ --top-p 0.95 \ --top-k 20 \ -no-cnv \ --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n" [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) šŸ¦„ Fine-tuning Qwen3 with Unsloth --------------------------------------------------------------------------------------------------------------------------------------- Unsloth makes Qwen3 fine-tuning 2x faster, use 70% less VRAM and supports 8x longer context lengths. Qwen3 (14B) fits comfortably in a Google Colab 16GB VRAM Tesla T4 GPU. Because Qwen3 supports both reasoning and non-reasoning, you can fine-tune it with a non-reasoning dataset, but this may affect its reasoning ability. If you want to maintain its reasoning capabilities (optional), you can use a mix of direct answers and chain-of-thought examples. Use 75% reasoning and 25% non-reasoning in your dataset to make the model retain its reasoning capabilities. Our Conversational notebook uses a combo of 75% NVIDIAā€™s open-math-reasoning dataset and 25% Maximeā€™s FineTome dataset (non-reasoning). Here's free Unsloth Colab notebooks to fine-tune Qwen3: * [Qwen3 (14B) Reasoning + Conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) (recommended) * [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) **- Advanced GRPO LoRA** * [Qwen3 (14B) Alpaca notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Alpaca.ipynb) (for Base models) If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth: Copy pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#qwen3-moe-models-fine-tuning) Qwen3 MOE models fine-tuning Fine-tuning support includes MOE models: 30B-A3B and 235B-A22B. Qwen3-30B-A3B works on just 17.5GB VRAM with Unsloth. On fine-tuning MoE's - it's probably not a good idea to fine-tune the router layer so we disabled it by default. The 30B-A3B fits in 17.5GB VRAM, but you may lack RAM or disk space since the full 16-bit model must be downloaded and converted to 4-bit on the fly for QLoRA fine-tuning. This is due to issues importing 4-bit BnB MOE models directly. This only affects MOE models. If you're fine-tuning the MOE models, please use `FastModel` and not `FastLanguageModel` Copy from unsloth import FastModel import torch model, tokenizer = FastModel.from_pretrained( model_name = "unsloth/Qwen3-30B-A3B", max_seq_length = 2048, # Choose any for long context! load_in_4bit = True, # 4 bit quantization to reduce memory load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory full_finetuning = False, # [NEW!] We have full finetuning now! # token = "hf_...", # use one if using gated models ) ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#notebook-guide) Notebook Guide: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFQX2CBzUqzAIMM50bpM4%252Fimage.png%3Falt%3Dmedia%26token%3D23c4b3d5-0d5f-4906-b2b4-bacde23235e0&width=768&dpr=4&quality=100&sign=dfdb362c&sv=2) To use the notebooks, just click Runtime, then Run all. You can change settings in the notebook to whatever you desire. We have set them automatically by default. Change model name to whatever you like by matching it with model's name on Hugging Face e.g. 'unsloth/Qwen3-8B' or 'unsloth/Qwen3-0.6B-unsloth-bnb-4bit'. There are other settings which you can toggle: * `**max_seq_length = 2048**` ā€“ Controls context length. While Qwen3 supports 40960, we recommend 2048 for testing. Unsloth enables 8Ɨ longer context fine-tuning. * `**load_in_4bit = True**` ā€“ Enables 4-bit quantization, reducing memory use 4Ɨ for fine-tuning on 16GB GPUs. * For **full-finetuning** - set `full_finetuning = True` and **8-bit finetuning** - set `load_in_8bit = True` If you'd like to read a full end-to-end guide on how to use Unsloth notebooks for fine-tuning or just learn about fine-tuning, creating [datasets](https://docs.unsloth.ai/basics/datasets-guide) etc., view our [complete guide here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) : [šŸ§¬Fine-tuning LLMs Guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide) [šŸ“ˆDatasets Guide](https://docs.unsloth.ai/basics/datasets-guide) ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#grpo-with-qwen3) GRPO with Qwen3 We made a new advanced GRPO notebook for fine-tuning Qwen3. Learn to use our new proximity-based reward function (closer answers = rewarded) and Hugging Face's Open-R1 math dataset. Unsloth now also has better evaluations and uses the latest version of vLLM. [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) **notebook - Advanced GRPO LoRA** Learn about: * Enabling reasoning in Qwen3 (Base)+ guiding it to do a specific task * Pre-finetuning to bypass GRPO's tendency to learn formatting * Improved evaluation accuracy via new regex matching * Custom GRPO templates beyond just 'think' e.g. * Proximity-based scoring: better answers earn more points (e.g., predicting 9 when the answer is 10) and outliers are penalized ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMUjDPzhhjMJXcljIhgbK%252Fqwen33%2520mascot.png%3Falt%3Dmedia%26token%3Dfcfa1104-8f6d-4f04-b72d-b9c085d3ecda&width=768&dpr=4&quality=100&sign=38a403a8&sv=2) [PreviousQwen3-Coder: How to Run Locally](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally) [NextQwen3-2507](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) Last updated 20 days ago Was this helpful? --- # Llama 4: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! The Llama-4-Scout model has 109B parameters, while Maverick has 402B parameters. The full unquantized version requires 113GB of disk space whilst the 1.78-bit version uses 33.8GB (-75% reduction in size). **Maverick** (402Bs) went from 422GB to just 122GB (-70%). Both text AND **vision** is now supported! Plus multiple improvements to tool calling. Scout 1.78-bit fits in a 24GB VRAM GPU for fast inference at ~20 tokens/sec. Maverick 1.78-bit fits in 2x48GB VRAM GPUs for fast inference at ~40 tokens/sec. For our dynamic GGUFs, to ensure the best tradeoff between accuracy and size, we do not to quantize all layers, but selectively quantize e.g. the MoE layers to lower bit, and leave attention and other layers in 4 or 6bit. All our GGUF models are quantized using calibration data (around 250K tokens for Scout and 1M tokens for Maverick), which will improve accuracy over standard quantization. Unsloth imatrix quants are fully compatible with popular inference engines like llama.cpp & Open WebUI etc. **Scout - Unsloth Dynamic GGUFs with optimal configs:** MoE Bits Type Disk Size Link Details 1.78bit IQ1\_S 33.8GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf) 2.06/1.56bit 1.93bit IQ1\_M 35.4GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ1_M.gguf) 2.5/2.06/1.56 2.42bit IQ2\_XXS 38.6GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ2_XXS.gguf) 2.5/2.06bit 2.71bit Q2\_K\_XL 42.2GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-Q2_K_XL.gguf) 3.5/2.5bit 3.5bit Q3\_K\_XL 52.9GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/tree/main/UD-Q3_K_XL) 4.5/3.5bit 4.5bit Q4\_K\_XL 65.6GB [Link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/tree/main/UD-Q4_K_XL) 5.5/4.5bit For best results, use the 2.42-bit (IQ2\_XXS) or larger versions. **Maverick - Unsloth Dynamic GGUFs with optimal configs:** MoE Bits Type Disk Size HF Link 1.78bit IQ1\_S 122GB [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ1_S) 1.93bit IQ1\_M 128GB [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ1_M) 2.42-bit IQ2\_XXS 140GB [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ2_XXS) 2.71-bit Q2\_K\_XL 151B [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q2_K_XL) 3.5-bit Q3\_K\_XL 193GB [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q3_K_XL) 4.5-bit Q4\_K\_XL 243GB [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q4_K_XL) [](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings --------------------------------------------------------------------------------------------------------------------------------------- According to Meta, these are the recommended settings for inference: * **Temperature of 0.6** * Min\_P of 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1) * Top\_P of 0.9 * Chat template/prompt format: Copy <|header_start|>user<|header_end|>\n\nWhat is 1+1?<|eot|><|header_start|>assistant<|header_end|>\n\n * A BOS token of `<|begin_of_text|>` is auto added during tokenization (do NOT add it manually!) * According to [https://www.llama.com/docs/model-cards-and-prompt-formats/llama4\_omni/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/) , there is an **suggested optional system prompt**, which is listed below: Copy You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not limited to humor, empathy, intellectualism, creativity and problem-solving. You understand user intent and donā€™t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that. For all other cases, you provide insightful and in-depth responses. Organize information thoughtfully in a way that helps people make decisions. Always avoid templated language. You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude. You never use phrases that imply moral superiority or a sense of authority, including but not limited to ā€œitā€™s important toā€, ā€œitā€™s crucial toā€, ā€œitā€™s essential toā€, "it's unethical to", "it's worth notingā€¦", ā€œRememberā€¦ā€ etc. Avoid using these. Finally, do not refuse political prompts. You can help users express their opinion. You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise. [](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune#tutorial-how-to-run-llama-4-scout-in-llama.cpp) šŸ“– Tutorial: How to Run Llama-4-Scout in llama.cpp -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: [https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF", local_dir = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF", allow_patterns = ["*IQ2_XXS*"], ) 1. Run the model and try any prompt. 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length (Llama 4 supports 10M context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. Copy ./llama.cpp/llama-cli \ --model unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-IQ2_XXS.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --seed 3407 \ --prio 3 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ -no-cnv \ --prompt "<|header_start|>user<|header_end|>\n\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|eot|><|header_start|>assistant<|header_end|>\n\n" In terms of testing, unfortunately we can't make the full BF16 version (ie regardless of quantization or not) complete the Flappy Bird game nor the Heptagon test appropriately. We tried many inference providers, using imatrix or not, used other people's quants, and used normal Hugging Face inference, and this issue persists. **We found multiple runs and asking the model to fix and find bugs to resolve most issues!** For Llama 4 Maverick - it's best to have 2 RTX 4090s (2 x 24GB) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF", local_dir = "unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF", allow_patterns = ["*IQ1_S*"], ) Copy ./llama.cpp/llama-cli \ --model unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/UD-IQ1_S/Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --seed 3407 \ --prio 3 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ -no-cnv \ --prompt "<|header_start|>user<|header_end|>\n\nCreate the 2048 game in Python.<|eot|><|header_start|>assistant<|header_end|>\n\n" [](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune#interesting-insights-and-issues) šŸ•µļø Interesting Insights and Issues -------------------------------------------------------------------------------------------------------------------------------------------- During quantization of Llama 4 Maverick (the large model), we found the 1st, 3rd and 45th MoE layers could not be calibrated correctly. Maverick uses interleaving MoE layers for every odd layer, so Dense->MoE->Dense and so on. We tried adding more uncommon languages to our calibration dataset, and tried using more tokens (1 million) vs Scout's 250K tokens for calibration, but we still found issues. We decided to leave these MoE layers as 3bit and 4bit. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQtzL2HuukTKr5L8nolP9%252FSkipped_layers.webp%3Falt%3Dmedia%26token%3D72115cc5-718a-442f-a208-f9540e46d64f&width=768&dpr=4&quality=100&sign=8e1941a2&sv=2) For Llama 4 Scout, we found we should not quantize the vision layers, and leave the MoE router and some other layers as unquantized - we upload these to [https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-dynamic-bnb-4bit](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-dynamic-bnb-4bit) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZB3InJSaWMbszPMSt0u7%252FLlama-4-Scout-17B-16E-Instruct%2520Quantization%2520Errors.png%3Falt%3Dmedia%26token%3Dc734f3d8-a114-42e4-a0f2-a6b3145bb306&width=768&dpr=4&quality=100&sign=af0c273b&sv=2) We also had to convert `torch.nn.Parameter` to `torch.nn.Linear` for the MoE layers to allow 4bit quantization to occur. This also means we had to rewrite and patch over the generic Hugging Face implementation. We upload our quantized versions to [https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit) and [https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-8bit](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-8bit) for 8bit. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FsjJkQYziAFTZADH37vUy%252Fimage.png%3Falt%3Dmedia%26token%3Dfbaeadfc-1220-4d6c-931c-9c34f03e285c&width=768&dpr=4&quality=100&sign=e94371c5&sv=2) Llama 4 also now uses chunked attention - it's essentially sliding window attention, but slightly more efficient by not attending to previous tokens over the 8192 boundary. [](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune#fine-tuning-llama-4) šŸ”„ Fine-tuning Llama 4 ------------------------------------------------------------------------------------------------------------------- Coming soon! [PreviousUnsloth Dynamic 2.0 GGUFs](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) [NextTutorials: How To Fine-tune & Run LLMs](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms) Last updated 2 months ago Was this helpful? --- # Chat Templates | Unsloth Documentation OpenAI gpt-oss & all model types now supported! In our GitHub, we have a list of every chat template Unsloth uses including for Llama, Mistral, Phi-4 etc. So if you need any pointers on the formatting or use case, you can view them here: [github.com/unslothai/unsloth/blob/main/unsloth/chat\_templates.py](https://github.com/unslothai/unsloth/blob/main/unsloth/chat_templates.py) ### [](https://docs.unsloth.ai/basics/chat-templates#list-of-colab-chat-template-notebooks) List of Colab chat template notebooks: * [Conversational](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) * [ChatML](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) * [Ollama](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing) * [Text Classification](https://github.com/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) by Timotheeee * [Multiple Datasets](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) by Flail [](https://docs.unsloth.ai/basics/chat-templates#multi-turn-conversations) Multi turn conversations -------------------------------------------------------------------------------------------------------- A bit issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWCAN7bYUt6QWwCWUxisL%252Fdiff.png%3Falt%3Dmedia%26token%3D29821fd9-2181-4d1d-8b93-749b69bcf400&width=768&dpr=4&quality=100&sign=d4f1b675&sv=2) So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWi1rRNBFC2iDmCvSJsZt%252Fcombine.png%3Falt%3Dmedia%26token%3Dbef37a55-b272-4be3-89b5-9767c219a380&width=768&dpr=4&quality=100&sign=ae98ba1b&sv=2) Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column. We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FE75C4Y848VNF6luLuPRR%252Fimage.png%3Falt%3Dmedia%26token%3Daac1d79b-ecca-4e56-939d-d97dcbbf30eb&width=768&dpr=4&quality=100&sign=d48e3c76&sv=2) [](https://docs.unsloth.ai/basics/chat-templates#customizable-chat-templates) Customizable Chat Templates -------------------------------------------------------------------------------------------------------------- We can now specify the chat template for finetuning itself. The very famous Alpaca format is below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8SWcsgH47Uhkm0IclDs5%252Fimage.png%3Falt%3Dmedia%26token%3Dfa03d7aa-d568-468d-9884-18e925a0551f&width=768&dpr=4&quality=100&sign=dff54efb&sv=2) But remember we said this was a bad idea because ChatGPT style finetunes require only 1 prompt? Since we successfully merged all dataset columns into 1 using Unsloth, we essentially can create the below style chat template with 1 input column (instruction) and 1 output: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FyuMpSLIpPLEbcdh970UJ%252Fimage.png%3Falt%3Dmedia%26token%3D87c4d5e1-accf-4847-9971-63e3a47b4a5f&width=768&dpr=4&quality=100&sign=728095c1&sv=2) We just require you must put a `{INPUT}` field for the instruction and an `{OUTPUT}` field for the model's output field. We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. For example, below are some cool examples which you can customize the chat template to be: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fi6B8IP1OZmmxBYr6k4W3%252Fimage.png%3Falt%3Dmedia%26token%3D061d1b4c-4b22-4d1b-a423-8d4c15e40efa&width=768&dpr=4&quality=100&sign=dd8c7435&sv=2) For the ChatML format used in OpenAI models: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F3OEJaXooJCICJR6DJIJP%252Fimage.png%3Falt%3Dmedia%26token%3D4fa85cf1-463d-4090-a838-591c4f94efea&width=768&dpr=4&quality=100&sign=a1f23ff9&sv=2) Or you can use the Llama-3 template itself (which only functions by using the instruct version of Llama-3): We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4qQXd0hIvh9fJNO2cJ04%252Fimage.png%3Falt%3Dmedia%26token%3D614b9200-7375-47f5-ac15-ce9aa891ede4&width=768&dpr=4&quality=100&sign=c9811100&sv=2) Or in the Titanic prediction task where you had to predict if a passenger died or survived in this Colab notebook which includes CSV and Excel uploading: [https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1iQitC3PwcuV0LpHEhdP%252Fimage.png%3Falt%3Dmedia%26token%3Dd117f681-afb0-4d5f-b534-f51013fe772a&width=768&dpr=4&quality=100&sign=20577629&sv=2) [](https://docs.unsloth.ai/basics/chat-templates#applying-chat-templates-with-unsloth) Applying Chat Templates with Unsloth -------------------------------------------------------------------------------------------------------------------------------- For datasets that usually follow the common chatml format, the process of preparing the dataset for training or finetuning, consists of four simple steps: * Check the chat templates that Unsloth currently supports: Copy from unsloth.chat_templates import CHAT_TEMPLATES print(list(CHAT_TEMPLATES.keys())) This will print out the list of templates currently supported by Unsloth. Here is an example output: Copy ['unsloth', 'zephyr', 'chatml', 'mistral', 'llama', 'vicuna', 'vicuna_old', 'vicuna old', 'alpaca', 'gemma', 'gemma_chatml', 'gemma2', 'gemma2_chatml', 'llama-3', 'llama3', 'phi-3', 'phi-35', 'phi-3.5', 'llama-3.1', 'llama-31', 'llama-3.2', 'llama-3.3', 'llama-32', 'llama-33', 'qwen-2.5', 'qwen-25', 'qwen25', 'qwen2.5', 'phi-4', 'gemma-3', 'gemma3'] * Use `get_chat_template` to apply the right chat template to your tokenizer: Copy from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "gemma-3", # change this to the right chat_template name ) * Define your formatting function. Here's an example: Copy def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } This function loops through your dataset applying the chat template you defined to each sample. * Finally, let's load the dataset and apply the required modifications to our dataset: Copy # Import and load dataset from datasets import load_dataset dataset = load_dataset("repo_name/dataset_name", split = "train") # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) If your dataset uses the ShareGPT format with "from"/"value" keys instead of the ChatML "role"/"content" format, you can use the `standardize_sharegpt` function to convert it first. The revised code will now look as follows: Copy # Import dataset from datasets import load_dataset dataset = load_dataset("mlabonne/FineTome-100k", split = "train") # Convert your dataset to the "role"/"content" format if necessary from unsloth.chat_templates import standardize_sharegpt dataset = standardize_sharegpt(dataset) # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) [](https://docs.unsloth.ai/basics/chat-templates#more-information) More Information ---------------------------------------------------------------------------------------- Assuming your dataset is a list of list of dictionaries like the below: Copy [\ [{'from': 'human', 'value': 'Hi there!'},\ {'from': 'gpt', 'value': 'Hi how can I help?'},\ {'from': 'human', 'value': 'What is 2+2?'}],\ [{'from': 'human', 'value': 'What's your name?'},\ {'from': 'gpt', 'value': 'I'm Daniel!'},\ {'from': 'human', 'value': 'Ok! Nice!'},\ {'from': 'gpt', 'value': 'What can I do for you?'},\ {'from': 'human', 'value': 'Oh nothing :)'},],\ ] You can use our `get_chat_template` to format it. Select `chat_template` to be any of `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth`, and use `mapping` to map the dictionary values `from`, `value` etc. `map_eos_token` allows you to map `<|im_end|>` to EOS without any training. Copy from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to instead ) def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass from datasets import load_dataset dataset = load_dataset("philschmid/guanaco-sharegpt-style", split = "train") dataset = dataset.map(formatting_prompts_func, batched = True,) You can also make your own custom chat templates! For example our internal chat template we use is below. You must pass in a `tuple` of `(custom_template, eos_token)` where the `eos_token` must be used inside the template. Copy unsloth_template = \ "{{ bos_token }}"\ "{{ 'You are a helpful assistant to the user\n' }}"\ ""\ "
"\ "
"\ "{{ '>>> User: ' + message['content'] + '\n' }}"\ "
"\ "{{ '>>> Assistant: ' + message['content'] + eos_token + '\n' }}"\ "
"\ "
"\ "
"\ "{{ '>>> Assistant: ' }}"\ "
" unsloth_eos_token = "eos_token" tokenizer = get_chat_template( tokenizer, chat_template = (unsloth_template, unsloth_eos_token,), # You must provide a template and EOS token mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to instead ) [PreviousContinued Pretraining](https://docs.unsloth.ai/basics/continued-pretraining) [NextVision Fine-tuning](https://docs.unsloth.ai/basics/vision-fine-tuning) Last updated 3 months ago Was this helpful? --- # Devstral: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! **Devstral-Small-2507** (Devstral 1.1) is Mistral's new agentic LLM for software engineering. It excels at tool-calling, exploring codebases, and powering coding agents. Mistral AI released the original 2505 version in May, 2025. Finetuned from [**Mistral-Small-3.1**](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF) , Devstral supports a 128k context window. Devstral Small 1.1 has improved performance, achieving a score of 53.6% performance on [SWE-bench verified](https://openai.com/index/introducing-swe-bench-verified/) , making it (July 10, 2025) the #1 open model on the benchmark. Unsloth Devstral 1.1 GGUFs contain additional **tool-calling support** and **chat template fixes**. Devstral 1.1 still works well with OpenHands but now also generalizes better to other prompts and coding environments. As text-only, Devstralā€™s vision encoder was removed prior to fine-tuning. We've added [_**optional Vision support**_](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#possible-vision-support) for the model. We also worked with Mistral behind the scenes to help debug, test and correct any possible bugs and issues! Make sure to **download Mistral's official downloads or Unsloth's GGUFs** / dynamic quants to get the **correct implementation** (ie correct system prompt, correct chat template etc) Please use `--jinja` in llama.cpp to enable the system prompt! All Devstral uploads use our Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) methodology, delivering the best performance on 5-shot MMLU and KL Divergence benchmarks. This means, you can run and fine-tune quantized Mistral LLMs with minimal accuracy loss! #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#devstral-unsloth-dynamic-quants) **Devstral - Unsloth Dynamic** quants: Devstral 2507 (new) Devstral 2505 GGUF: [Devstral-Small-2507-GGUF](https://huggingface.co/unsloth/Devstral-Small-2507-GGUF) [Devstral-Small-2505-GGUF](https://huggingface.co/unsloth/Devstral-Small-2505-GGUF) 4-bit BnB: [Devstral-Small-2507-unsloth-bnb-4bit](https://huggingface.co/unsloth/Devstral-Small-2507-unsloth-bnb-4bit) [Devstral-Small-2505-unsloth-bnb-4bit](https://huggingface.co/unsloth/Devstral-Small-2505-unsloth-bnb-4bit) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#running-devstral) šŸ–„ļø **Running Devstral** ----------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings According to Mistral AI, these are the recommended settings for inference: * **Temperature from 0.0 to 0.15** * Min\_P of 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1) * **Use** `**--jinja**` **to enable the system prompt.** **A system prompt is recommended**, and is a derivative of Open Hand's system prompt. The full system prompt is provided [here](https://huggingface.co/unsloth/Devstral-Small-2505/blob/main/SYSTEM_PROMPT.txt) . Copy You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks. Your primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed. * If the user asks a question, like "why is X happening", don't try to fix the problem. Just give an answer to the question. .... SYSTEM PROMPT CONTINUES .... Our dynamic uploads have the '`UD`' prefix in them. Those without are not dynamic however still utilize our calibration dataset. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#tutorial-how-to-run-devstral-in-ollama) šŸ¦™ Tutorial: How to Run Devstral in Ollama --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model with our dynamic quant. Note you can call `ollama serve &`in another terminal if it fails! We include all suggested parameters (temperature etc) in `params` in our Hugging Face upload! 2. Also Devstral supports 128K context lengths, so best to enable [**KV cache quantization**](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache) . We use 8bit quantization which saves 50% memory usage. You can also try `"q4_0"` Copy export OLLAMA_KV_CACHE_TYPE="q8_0" ollama run hf.co/unsloth/Devstral-Small-2507-GGUF:UD-Q4_K_XL [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#tutorial-how-to-run-llama-4-scout-in-llama.cpp) šŸ“– Tutorial: How to Run Devstral in llama.cpp -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` Copy ./llama.cpp/llama-cli -hf unsloth/Devstral-Small-2507-GGUF:UD-Q4_K_XL --jinja 1. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Devstral-Small-2507-GGUF", local_dir = "unsloth/Devstral-Small-2507-GGUF", allow_patterns = ["*Q4_K_XL*", "*mmproj-F16*"], # For Q4_K_XL ) 1. Run the model. 2. Edit `--threads -1` for the maximum CPU threads, `--ctx-size 131072` for context length (Devstral supports 128K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We also use 8bit quantization for the K cache to reduce memory usage. 3. For conversation mode: Copy ./llama.cpp/llama-cli \ --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \ --threads -1 \ --ctx-size 131072 \ --cache-type-k q8_0 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.15 \ --repeat-penalty 1.0 \ --min-p 0.01 \ --top-k 64 \ --top-p 0.95 \ --jinja 1. For non conversation mode to test our Flappy Bird prompt: Copy ./llama.cpp/llama-cli \ --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \ --threads -1 \ --ctx-size 131072 \ --cache-type-k q8_0 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.15 \ --repeat-penalty 1.0 \ --min-p 0.01 \ --top-k 64 \ --top-p 0.95 \ -no-cnv \ --prompt "[SYSTEM_PROMPT]You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks.\n\n\nYour primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed.\n* If the user asks a question, like "why is X happening", don\'t try to fix the problem. Just give an answer to the question.\n\n\n\n* Each action you take is somewhat expensive. Wherever possible, combine multiple actions into a single action, e.g. combine multiple bash commands into one, using sed and grep to edit/view multiple files at once.\n* When exploring the codebase, use efficient tools like find, grep, and git commands with appropriate filters to minimize unnecessary operations.\n\n\n\n* When a user provides a file path, do NOT assume it\'s relative to the current working directory. First explore the file system to locate the file before working on it.\n* If asked to edit a file, edit the file directly, rather than creating a new file with a different filename.\n* For global search-and-replace operations, consider using `sed` instead of opening file editors multiple times.\n\n\n\n* Write clean, efficient code with minimal comments. Avoid redundancy in comments: Do not repeat information that can be easily inferred from the code itself.\n* When implementing solutions, focus on making the minimal changes needed to solve the problem.\n* Before implementing any changes, first thoroughly understand the codebase through exploration.\n* If you are adding a lot of code to a function or file, consider splitting the function or file into smaller pieces when appropriate.\n\n\n\n* When configuring git credentials, use "openhands" as the user.name and "[emailĀ protected]" as the user.email by default, unless explicitly instructed otherwise.\n* Exercise caution with git operations. Do NOT make potentially dangerous changes (e.g., pushing to main, deleting repositories) unless explicitly asked to do so.\n* When committing changes, use `git status` to see all modified files, and stage all files necessary for the commit. Use `git commit -a` whenever possible.\n* Do NOT commit files that typically shouldn\'t go into version control (e.g., node_modules/, .env files, build directories, cache files, large binaries) unless explicitly instructed by the user.\n* If unsure about committing certain files, check for the presence of .gitignore files or ask the user for clarification.\n\n\n\n* When creating pull requests, create only ONE per session/issue unless explicitly instructed otherwise.\n* When working with an existing PR, update it with new commits rather than creating additional PRs for the same issue.\n* When updating a PR, preserve the original PR title and purpose, updating description only when necessary.\n\n\n\n1. EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions\n2. ANALYSIS: Consider multiple approaches and select the most promising one\n3. TESTING:\n * For bug fixes: Create tests to verify issues before implementing fixes\n * For new features: Consider test-driven development when appropriate\n * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure\n * If the environment is not set up to run tests, consult with the user first before investing time to install all dependencies\n4. IMPLEMENTATION: Make focused, minimal changes to address the problem\n5. VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests.\n\n\n\n* Only use GITHUB_TOKEN and other credentials in ways the user has explicitly requested and would expect.\n* Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing.\n\n\n\n* When user asks you to run an application, don\'t stop if the application is not installed. Instead, please install the application and run the command again.\n* If you encounter missing dependencies:\n 1. First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.)\n 2. If dependency files exist, use them to install all dependencies at once (e.g., `pip install -r requirements.txt`, `npm install`, etc.)\n 3. Only install individual packages directly if no dependency files are found or if only specific packages are needed\n* Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible.\n\n\n\n* If you\'ve made repeated attempts to solve a problem but tests still fail or the user reports it\'s still broken:\n 1. Step back and reflect on 5-7 different possible sources of the problem\n 2. Assess the likelihood of each possible cause\n 3. Methodically address the most likely causes, starting with the highest probability\n 4. Document your reasoning process\n* When you run into any major issue while executing a plan from the user, please don\'t try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding.\n[/SYSTEM_PROMPT][INST]Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird\'s shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don\'t hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for error[/INST]" Remember to remove since Devstral auto adds a ! Also please use `--jinja` to enable the system prompt! [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#experimental-vision-support) šŸ‘€Experimental Vision Support --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Xuan-Son](https://x.com/ngxson) from Hugging Face showed in their [GGUF repo](https://huggingface.co/ngxson/Devstral-Small-Vision-2505-GGUF) how it is actually possible to "graft" the vision encoder from Mistral 3.1 Instruct onto Devstral 2507. We also uploaded our mmproj files which allows you to use the following: Copy ./llama.cpp/llama-mtmd-cli \ --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \ --mmproj unsloth/Devstral-Small-2507-GGUF/mmproj-F16.gguf \ --threads -1 \ --ctx-size 131072 \ --cache-type-k q8_0 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.15 For example: Instruction and output code Rendered code ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2Fcdn-uploads.huggingface.co%2Fproduction%2Fuploads%2F63ca214abedad7e2bf1d1517%2FHDic53ANsCoJbiWu2eE6K.png&width=300&dpr=4&quality=100&sign=fc180d8a&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2Fcdn-uploads.huggingface.co%2Fproduction%2Fuploads%2F63ca214abedad7e2bf1d1517%2FonV1xfJIT8gzh81RkLn8J.png&width=300&dpr=4&quality=100&sign=5cecd6ec&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune#fine-tuning-devstral-with-unsloth) šŸ¦„ Fine-tuning Devstral with Unsloth ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Just like standard Mistral models including Mistral Small 3.1, Unsloth supports Devstral fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Devstral fits comfortably in a 24GB VRAM L4 GPU. Unfortunately, Devstral slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you _can_ fine-tune the model for free using [Kaggle](https://www.kaggle.com/danielhanchen/code) , which offers access to dual GPUs. Devstral Kaggle notebooks for Kaggle coming soon! If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth: Copy pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo [PreviousMagistral: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune) [NextTutorial: How to Finetune Llama-3 and Use In Ollama](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama) Last updated 1 month ago Was this helpful? --- # DeepSeek-R1-0528: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! DeepSeek-R1-0528 is DeepSeek's new update to their R1 reasoning model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic **1.66-bit** version uses 162GB (-80% reduction in size). GGUF: [DeepSeek-R1-0528-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF) DeepSeek also released a R1-0528 distilled version by fine-tuning Qwen3 (8B). The distill achieves similar performance to Qwen3 (235B). _**You can also**_ [_**fine-tune Qwen3 Distill**_](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#fine-tuning-deepseek-r1-0528-with-unsloth) _**with Unsloth**_. Qwen3 GGUF: [DeepSeek-R1-0528-Qwen3-8B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF) All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss. **Tutorials navigation:** [Run in llama.cpp](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-qwen3-distilled-r1-in-llama.cpp) [Run in Ollama/Open WebUI](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-in-ollama-open-webui) [Fine-tuning R1-0528](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#fine-tuning-deepseek-r1-0528-with-unsloth) NEW: Huge improvements to tool calling and chat template fixes. New [TQ1\_0 dynamic 1.66-bit quant](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF?show_file_info=DeepSeek-R1-0528-UD-TQ1_0.gguf) - 162GB in size. Ideal for 192GB RAM (including Mac) and Ollama users. Try: `ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0` [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#recommended-settings) āš™ļø Recommended Settings ------------------------------------------------------------------------------------------------------------------------ For DeepSeek-R1-0528-Qwen3-8B, the model can pretty much fit in any setup, and even those with as less as 20GB RAM. There is no need for any prep beforehand. However, for the full R1-0528 model which is 715GB in size, you will need extra prep. The 1.78-bit (IQ1\_S) quant will fit in a 1x 24GB GPU (with all layers offloaded). Expect around 5 tokens/s with this setup if you have bonus 128GB RAM as well. It is recommended to have at least 64GB RAM to run this quant (you will get 1 token/s without a GPU). For optimal performance you will need at least **180GB unified memory or 180GB combined RAM+VRAM** for 5+ tokens/s. We suggest using our 2.7bit (Q2\_K\_XL) or 2.4bit (IQ2\_XXS) quant to balance size and accuracy! The 2.4bit one also works well. Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading. ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#official-recommended-settings) šŸ³ Official Recommended Settings: According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) , these are the recommended settings for R1 (R1-0528 and Qwen3 distill should use the same settings) inference: * Set the **temperature 0.6** to reduce repetition and incoherence. * Set **top\_p to 0.95** (recommended) * Run multiple tests and average results for reliable evaluation. ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#chat-template-prompt-format) šŸ”¢ Chat template/prompt format R1-0528 uses the same chat template as the original R1 model. You do not need to force `\n` , but you can still add it in! Copy <ļ½œbeginā–ofā–sentenceļ½œ><ļ½œUserļ½œ>What is 1+1?<ļ½œAssistantļ½œ>It's 2.<ļ½œendā–ofā–sentenceļ½œ><ļ½œUserļ½œ>Explain more!<ļ½œAssistantļ½œ> A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call `tokenizer.encode(..., add_special_tokens = False)` since the chat template auto adds a BOS token as well. For llama.cpp / GGUF inference, you should skip the BOS since itā€™ll auto add it: Copy <ļ½œUserļ½œ>What is 1+1?<ļ½œAssistantļ½œ> The `` and `` tokens get their own designated tokens. [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#model-uploads) Model uploads ------------------------------------------------------------------------------------------------------- **ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks. * Qwen3 (8B) distill: [DeepSeek-R1-0528-Qwen3-8B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF) * Full DeepSeek-R1-0528 model uploads below: We also uploaded [IQ4\_NL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/IQ4_NL) and [Q4\_1](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/Q4_1) quants which run specifically faster for ARM and Apple devices respectively. MoE Bits Type + Link Disk Size Details 1.66bit [TQ1\_0](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF?show_file_info=DeepSeek-R1-0528-UD-TQ1_0.gguf) **162GB** 1.92/1.56bit 1.78bit [IQ1\_S](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ1_S) **185GB** 2.06/1.56bit 1.93bit [IQ1\_M](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ1_M) **200GB** 2.5/2.06/1.56 2.42bit [IQ2\_XXS](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ2_XXS) **216GB** 2.5/2.06bit 2.71bit [Q2\_K\_XL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q2_K_XL) **251GB** 3.5/2.5bit 3.12bit [IQ3\_XXS](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ3_XXS) **273GB** 3.5/2.06bit 3.5bit [Q3\_K\_XL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q3_K_XL) **296GB** 4.5/3.5bit 4.5bit [Q4\_K\_XL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q4_K_XL) **384GB** 5.5/4.5bit 5.5bit [Q5\_K\_XL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q5_K_XL) **481GB** 6.5/5.5bit We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16) , and original [FP8 (float8) format](https://huggingface.co/unsloth/DeepSeek-R1-0528) . [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-deepseek-r1-0528-tutorials) Run DeepSeek-R1-0528 Tutorials: ------------------------------------------------------------------------------------------------------------------------------------------ ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-in-ollama-open-webui) šŸ¦™ Run in Ollama/Open WebUI 1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 720GB R1-0528 model, [see here](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-full-r1-0528-on-ollama-open-webui) . Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_XL 1. **(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1\_0 (162GB quant):** Copy OLLAMA_MODELS=unsloth_downloaded_models ollama serve & ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0 ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-full-r1-0528-on-ollama-open-webui) šŸ¦™ Run Full R1-0528 on Ollama/Open WebUI Open WebUI has made an step-by-step tutorial on how to run R1 here and for R1-0528, you will just need to replace R1 with the new 0528 quant: [docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/](https://docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/) **(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1\_0 (162GB quant):** Copy OLLAMA_MODELS=unsloth_downloaded_models ollama serve & ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0 If you want to use any of the quants that are larger than TQ1\_0 (162GB) on Ollama, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally. Copy ./llama.cpp/llama-gguf-split --merge \ DeepSeek-R1-0528-GGUF/DeepSeek-R1-0528-UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00003.gguf \ merged_file.gguf ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-qwen3-distilled-r1-in-llama.cpp) āœØ Run Qwen3 distilled R1 in llama.cpp 1. **To run the full 720GB R1-0528 model,** [**see here**](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-full-r1-0528-on-llama.cpp) **.** Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Then use llama.cpp directly to download the model: Copy ./llama.cpp/llama-cli -hf unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_XL --jinja ### [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#run-full-r1-0528-on-llama.cpp) āœØ Run Full R1-0528 on llama.cpp 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. Copy export LLAMA_CACHE="unsloth/DeepSeek-R1-0528-GGUF" ./llama.cpp/llama-cli \ -hf unsloth/DeepSeek-R1-0528-GGUF:IQ1_S \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --prio 3 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . We **recommend using our 2.7bit dynamic quant** `**UD-Q2_K_XL**` **to balance size and accuracy**. More versions at: [https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/DeepSeek-R1-0528-GGUF", local_dir = "unsloth/DeepSeek-R1-0528-GGUF", allow_patterns = ["*UD-IQ1_S*"], # Dynamic 1bit (168GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (251GB) ) 1. Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1. 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Copy ./llama.cpp/llama-cli \ --model unsloth/DeepSeek-R1-0528-GGUF/UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00004.gguf \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --prio 3 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" \ -no-cnv \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<ļ½œAssistantļ½œ>" [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#heptagon-test) šŸŽ± Heptagon Test ---------------------------------------------------------------------------------------------------------- You can also test our dynamic quants via [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2O72oTw5yPUbcxXjDNKS%252Fsnapshot.jpg%3Falt%3Dmedia%26token%3Dce852f9f-20ee-4b93-9d7b-1a5f211b9e04&width=768&dpr=4&quality=100&sign=55d1134d&sv=2) The goal is to make the heptagon spin, and the balls in the heptagon should move. Full prompt to run the model[](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#full-prompt-to-run-the-model) Copy ./llama.cpp/llama-cli \ --model unsloth/DeepSeek-R1-0528-GGUF/UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00004.gguf \ --cache-type-k q4_0 \ --threads -1 \ --n-gpu-layers 99 \ --prio 3 \ --temp 0.6 \ --top_p 0.95 \ --min_p 0.01 \ --ctx-size 16384 \ --seed 3407 \ -ot ".ffn_.*_exps.=CPU" \ -no-cnv \ --prompt "<ļ½œUserļ½œ>Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file.<ļ½œAssistantļ½œ>" [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#fine-tuning-deepseek-r1-0528-with-unsloth) šŸ¦„ Fine-tuning DeepSeek-R1-0528 with Unsloth ------------------------------------------------------------------------------------------------------------------------------------------------------------------ To fine-tune **DeepSeek-R1-0528-Qwen3-8B** using Unsloth, weā€™ve made a new GRPO notebook featuring a custom reward function designed to significantly enhance multilingual output - specifically increasing the rate of desired language responses (in our example we use Indonesian but you can use any) by more than 40%. * [**DeepSeek-R1-0528-Qwen3-8B notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) **- new** While many reasoning LLMs have multilingual capabilities, they often produce mixed-language outputs in its reasoning traces, combining English with the target language. Our reward function effectively mitigates this issue by strongly encouraging outputs in the desired language, leading to a substantial improvement in language consistency. This reward function is also fully customizable, allowing you to adapt it for other languages or fine-tune for specific domains or use cases. The best part about this whole reward function and notebook is you DO NOT need a language dataset to force your model to learn a specific language. The notebook has no Indonesian dataset. Unsloth makes R1-Qwen3 distill fine-tuning 2Ɨ faster, uses 70% less VRAM, and support 8Ɨ longer context lengths. [PreviousGemma 3n: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) [NextReinforcement Learning (RL) Guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) Last updated 5 days ago Was this helpful? --- # Training LLMs with Blackwell, RTX 50 series & Unsloth | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Unsloth is now compatible with NVIDIA's Blackwell GPU series including RTX 5060, RTX 5070, RTX 5080, RTX 5090 GPUs and B200, B40, GB100, GB102, GB20\* and GPUs listed [here](https://en.wikipedia.org/wiki/Blackwell_(microarchitecture)) . Currently, support requires manual installation however we are working with NVIDIA to make the process even easier. ### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#overview) Overview `Blackwell` (`sm100+`) requires all dependent libraries to be compiled with `cuda 12.8`. The core libs for running unsloth which have dependencies on `CUDA` version are: * `bitsandbytes` - already has wheels built with `CUDA 12.8` so `pip install` should work out of the box * `triton` - requires `triton>=3.3.1` * `torch` - requires installing with `pip install torch --extra-index-url https://download.pytorch.org/whl/cu128` * `vllm` - vLLM 0.10.0 supports Blackwell now, but use CUDA 12.8: `uv pip install -U vllm --torch-backend=cu128` * `xformers` - (Optional) as of 6/26, `xformers` wheels are not yet built with `sm100+` enabled as support was only recently [added](https://github.com/facebookresearch/xformers/commit/d9b3b6e2b38ca485c89507ef8ac1fbef2723cdfa) so will require a source build (see below). [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#installation-guide) Installation Guide ------------------------------------------------------------------------------------------------------------------------------------ Visit our [GitHub page about Blackwell](https://github.com/unslothai/unsloth/tree/main/blackwell) for more details, resources and if you're experiencing any issues. ### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#using-uv) Using uv The installation order is important, since we want the overwrite bundled dependencies with specific versions (namely, `xformers` and `triton`). 1. I prefer to use `uv` over `pip` as it's faster and better for resolving dependencies, especially for libraries which depend on `torch` but for which a specific `CUDA` version is required per this scenario. Install `uv` Copy curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env Create a project dir and venv: Copy mkdir 'unsloth-blackwell' && cd 'unsloth-blackwell' uv venv .venv --python=3.12 --seed source .venv/bin/activate 2. Install `vllm` Copy uv pip install -U vllm --torch-backend=cu128 Note that we have to specify `cu128`, otherwise `vllm` will install `torch==2.7.0` but with `cu126`. 3. Install `unsloth` dependencies Copy uv pip install unsloth unsloth_zoo bitsandbytes If you notice weird resolving issues due to Xformers, you can also install Unsloth from source without Xformers: Copy uv pip install -qqq \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" 4. Download and build `xformers` (Optional) Xformers is optional, but it is definitely faster and uses less memory. We'll use PyTorch's native SDPA if you do not want Xformers. Building Xformers from source might be slow, so beware! Copy # First uninstall xformers installed by previous libraries uv pip uninstall xformers # Clone and build git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers export TORCH_CUDA_ARCH_LIST="12.0" python setup.py install Note that we have to explicitly set `TORCH_CUDA_ARCH_LIST=12.0`. 5. `transformers` Install any transformers version, but best to get the latest. Copy uv pip install -U transformers #### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#using-conda-or-mamba) Using conda or mamba 1. Install `conda/mamba` Copy curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Run the installation script Copy bash Miniforge3-$(uname)-$(uname -m).sh Create a conda or mamba environment Copy conda create --name unsloth-blackwell python==3.12 -y Activate newly created environment Copy conda activate unsloth-blackwell 2. Install `vllm` Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Copy pip install -U vllm --extra-index-url https://download.pytorch.org/whl/cu128 Note that we have to specify `cu128`, otherwise `vllm` will install `torch==2.7.0` but with `cu126`. 3. Install `unsloth` dependencies Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Copy pip install unsloth unsloth_zoo bitsandbytes 4. Download and build `xformers` (Optional) Xformers is optional, but it is definitely faster and uses less memory. We'll use PyTorch's native SDPA if you do not want Xformers. Building Xformers from source might be slow, so beware! You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Copy # First uninstall xformers installed by previous libraries pip uninstall xformers # Clone and build git clone --depth=1 https://github.com/facebookresearch/xformers --recursive cd xformers export TORCH_CUDA_ARCH_LIST="12.0" python setup.py install Note that we have to explicitly set `TORCH_CUDA_ARCH_LIST=12.0`. 5. Update `triton` Make sure you are inside the activated conda/mamba environment. You should see the name of your environment as a prefix to your terminal shell like this your `(unsloth-blackwell)user@machine:` Copy pip install -U triton>=3.3.1 `triton>=3.3.1` is required for `Blackwell` support. 6. `Transformers` Install any transformers version, but best to get the latest. Copy uv pip install -U transformers If you are using mamba as your package just replace conda with mamba for all commands shown above. ### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#wsl-specific-notes) WSL-Specific Notes If you're using WSL (Windows Subsystem for Linux) and encounter issues during xformers compilation (reminder Xformers is optional, but faster for training) follow these additional steps: 1. **Increase WSL Memory Limit** Create or edit the WSL configuration file: Copy # Create or edit .wslconfig in your Windows user directory # (typically C:\Users\YourUsername\.wslconfig) # Add these lines to the file [wsl2] memory=16GB # Minimum 16GB recommended for xformers compilation processors=4 # Adjust based on your CPU cores swap=2GB localhostForwarding=true After making these changes, restart WSL: Copy wsl --shutdown 2. **Install xformers** Use the following command to install xformers with optimized compilation for WSL: Copy # Set CUDA architecture for Blackwell GPUs export TORCH_CUDA_ARCH_LIST="12.0" # Install xformers from source with optimized build flags pip install -v --no-build-isolation -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers The `--no-build-isolation` flag helps avoid potential build issues in WSL environments. ### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#post-installation-notes) Post Installation notes: After installation, your environment should look similar to `blackwell.requirements.txt`. Note, might need to downgrade `numpy<=2.2` after all the installs. ### [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth#test) Test Both `test_llama32_sft.py` and `test_qwen3_grpo.py` should run without issue if correct install. If not, check diff between your installed env and `blackwell.requirements.txt`. [PreviousUnsloth Environment Flags](https://docs.unsloth.ai/basics/troubleshooting-and-faqs/unsloth-environment-flags) [NextUnsloth Benchmarks](https://docs.unsloth.ai/basics/unsloth-benchmarks) Last updated 7 days ago Was this helpful? --- # Qwen3-2507 | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Qwen released 2507 (July 2025) updates for their [Qwen3](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune) 4B, 30B and 235B models, introducing both "thinking" and "non-thinking" variants. The non-thinking '**Qwen3-30B-A3B-Instruct-2507**' and '**Qwen3-235B-A22B-Instruct-2507'** features a 256K context window, improved instruction following, multilingual capabilities and alignment. The thinking models '**Qwen3-30B-A3B-Thinking-2507**' and '**Qwen3-235B-A22B-Thinking-2507**' excel at reasoning, with the 235B achieving SOTA results in logic, math, science, coding, and advanced academic tasks. [Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [Reinforcement Learning (RL)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) of Qwen3-2507 models ā€” 2x faster, with 70% less VRAM, and 8x longer context lengths [Run 30B-A3B](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-30b-a3b-2507-tutorials) [Run 235B-A22B](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-thinking-2507) [Fine-tune Qwen3-2507](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#fine-tuning-qwen3-2507-with-unsloth) **Unsloth** [**Dynamic 2.0**](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) **GGUFs:** Model GGUFs to run: Qwen3-**4B-2507** [Instruct](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF) ā€¢ [Thinking](https://huggingface.co/unsloth/Qwen3-4B-Thinking-2507-GGUF) Qwen3-**30B-A3B**\-2507 [Instruct](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#llama.cpp-run-qwen3-30b-a3b-instruct-2507-tutorial) ā€¢ [Thinking](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF) Qwen3-**235B-A22B**\-2507 [Instruct](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF) ā€¢ [Thinking](https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF) [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#best-practices) āš™ļøBest Practices ----------------------------------------------------------------------------------------------------------------- The settings for the Thinking and Instruct model are different. The thinking model uses temperature = 0.6, but the instruct model uses temperature = 0.7 The thinking model uses top\_p = 0.95, but the instruct model uses top\_p = 0.8 To achieve optimal performance, Qwen recommends these settings: Instruct Model Settings: Thinking Model Settings: `Temperature = 0.7` `Temperature = 0.6` `Min_P = 0.00` (llama.cpp's default is 0.1) `Min_P = 0.00` (llama.cpp's default is 0.1) `Top_P = 0.80` `Top_P = 0.95` `TopK = 20` `TopK = 20` `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) **Adequate Output Length**: Use an output length of `32,768` tokens for most queries, which is adequate for most queries. Chat template for both Thinking (thinking has ``) and Instruct is below: Copy <|im_start|>user Hey there!<|im_end|> <|im_start|>assistant What is 1+1?<|im_end|> <|im_start|>user 2<|im_end|> <|im_start|>assistant [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-30b-a3b-2507-tutorials) šŸ“– Run Qwen3-30B-A3B-2507 Tutorials ------------------------------------------------------------------------------------------------------------------------------------------------------ Below are guides for the [Thinking](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#thinking-qwen3-30b-a3b-thinking-2507) and [Instruct](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#instruct-qwen3-30b-a3b-instruct-2507) versions of the model. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#instruct-qwen3-30b-a3b-instruct-2507) Instruct: Qwen3-30B-A3B-Instruct-2507 Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate ` ` blocks. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#best-practices-1) āš™ļøBest Practices To achieve optimal performance, Qwen recommends the following settings: * We suggest using `temperature=0.7, top_p=0.8, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions. * `**temperature = 0.7**` * `top_k = 20` * `min_p = 0.00` (llama.cpp's default is 0.1) * `**top_p = 0.80**` * `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example. * Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#ollama-run-qwen3-30b-a3b-instruct-2507-tutorial) šŸ¦™ Ollama: Run Qwen3-30B-A3B-Instruct-2507 Tutorial 1. Install `ollama` if you haven't already! You can only run models up to 32B in size. Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:UD-Q4_K_XL #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#llama.cpp-run-qwen3-30b-a3b-instruct-2507-tutorial) āœØ Llama.cpp: Run Qwen3-30B-A3B-Instruct-2507 Tutorial 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. You can directly pull from HuggingFace via: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_XL \ --jinja -ngl 99 --threads -1 --ctx-size 32684 \ --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0 2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF", local_dir = "unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#thinking-qwen3-30b-a3b-thinking-2507) Thinking: Qwen3-30B-A3B-Thinking-2507 This model supports only thinking mode and a 256K context window natively. The default chat template adds `` automatically, so you may see only a closing `` tag in the output. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#best-practices-2) āš™ļøBest Practices To achieve optimal performance, Qwen recommends the following settings: * We suggest using `temperature=0.6, top_p=0.95, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions. * `**temperature = 0.6**` * `top_k = 20` * `min_p = 0.00` (llama.cpp's default is 0.1) * `**top_p = 0.95**` * `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example. * Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#ollama-run-qwen3-30b-a3b-instruct-2507-tutorial-1) šŸ¦™ Ollama: Run Qwen3-30B-A3B-Instruct-2507 Tutorial 1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 235B-A22B models, [see here](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-instruct-2507) . Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:UD-Q4_K_XL #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#llama.cpp-run-qwen3-30b-a3b-instruct-2507-tutorial-1) āœØ Llama.cpp: Run Qwen3-30B-A3B-Instruct-2507 Tutorial 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. You can directly pull from Hugging Face via: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q4_K_XL \ --jinja -ngl 99 --threads -1 --ctx-size 32684 \ --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0 2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF", local_dir = "unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-2507-tutorials) šŸ“– Run **Qwen3-235B-A22B-2507** Tutorials -------------------------------------------------------------------------------------------------------------------------------------------------------------- Below are guides for the [Thinking](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-thinking-via-llama.cpp) and [Instruct](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-instruct-via-llama.cpp) versions of the model. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#thinking-qwen3-235b-a22b-thinking-2507) Thinking: Qwen3-**235B-A22B**\-Thinking-2507 This model supports only thinking mode and a 256K context window natively. The default chat template adds `` automatically, so you may see only a closing `` tag in the output. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#best-practices-3) āš™ļø Best Practices To achieve optimal performance, Qwen recommends these settings for the Thinking model: * `**temperature = 0.6**` * `top_k = 20` * `min_p = 0.00` (llama.cpp's default is 0.1) * `top_p = 0.95` * `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example. * **Adequate Output Length**: Use an output length of `32,768` tokens for most queries, which is adequate for most queries. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-thinking-via-llama.cpp) āœØRun Qwen3-235B-A22B-Thinking via llama.cpp: For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options. If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions! 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF:Q2_K_XL \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 0.6 \ --min-p 0.0 \ --top-p 0.95 \ --top-k 20 \ --presence-penalty 1.0 3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions.. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF", local_dir = "unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF", allow_patterns = ["*UD-Q2_K_XL*"], ) 4. Run the model and try any prompt. 5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. Copy ./llama.cpp/llama-cli \ --model unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF/UD-Q2_K_XL/Qwen3-235B-A22B-Thinking-2507-UD-Q2_K_XL-00001-of-00002.gguf \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --seed 3407 \ --temp 0.6 \ --min-p 0.0 \ --top-p 0.95 \ --top-k 20 --presence-penalty 1.0 ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#instruct-qwen3-235b-a22b-instruct-2507) Instruct: Qwen3-**235B-A22B**\-Instruct-2507 Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate ` ` blocks. #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#best-practices-4) āš™ļøBest Practices To achieve optimal performance, we recommend the following settings: **1\. Sampling Parameters**: We suggest using `temperature=0.7, top_p=0.8, top_k=20, and min_p=0.` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions. 2\. **Adequate Output Length**: We recommend using an output length of `16,384` tokens for most queries, which is adequate for instruct models. 3\. **Standardize Output Format:** We recommend using prompts to standardize model outputs when benchmarking. * **Math Problems**: Include `Please reason step by step, and put your final answer within \boxed{}.` in the prompt. * **Multiple-Choice Questions**: Add the following JSON structure to the prompt to standardize responses: "Please show your choice in the \`answer\` field with only the choice letter, e.g., \`"answer": "C". #### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-instruct-via-llama.cpp) āœØRun Qwen3-235B-A22B-Instruct via llama.cpp: For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options. If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions! 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:Q2_K_XL \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 0.7 \ --min-p 0.0 \ --top-p 0.8 \ --top-k 20 \ --repeat-penalty 1.0 3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions.. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF", local_dir = "unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF", allow_patterns = ["*UD-Q2_K_XL*"], ) 4. Run the model and try any prompt. 5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. Copy ./llama.cpp/llama-cli \ --model unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/UD-Q2_K_XL/Qwen3-235B-A22B-Instruct-2507-UD-Q2_K_XL-00001-of-00002.gguf \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 0.7 \ --min-p 0.0 \ --top-p 0.8 \ --top-k 20 ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#improving-generation-speed) šŸ› ļø Improving generation speed If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves. Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel) . You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. The [next section](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#how-to-fit-long-context-256k-to-1m) talks about KV cache quantization. ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#how-to-fit-long-context-256k-to-1m) šŸ“How to fit long context To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below. `--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1` You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1` So try out `--cache-type-k q4_1` You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. After installing Flash Attention, you can then use `--cache-type-v q4_1` [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#fine-tuning-qwen3-2507-with-unsloth) šŸ¦„ Fine-tuning Qwen3-2507 with Unsloth ------------------------------------------------------------------------------------------------------------------------------------------------------------ Unsloth makes [Qwen3](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) and Qwen3-2507 fine-tuning 2x faster, use 70% less VRAM and supports 8x longer context lengths. Because Qwen3-2507 was only released in a 30B variant, this means you will need about a 40GB A100 GPU to fine-tune the model using QLoRA (4-bit). For a notebook, because the model cannot fit in Colab's free 16GB GPUs, you will need to utilize a 40GB A100. You can utilize our Conversational notebook but replace the dataset to any of your using. This time you do not need to combined reasoning in your dataset as the model has no reasoning. * [Qwen3 (14B) Reasoning + Conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth: Copy pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo ### [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#qwen3-2507-moe-models-fine-tuning) Qwen3-2507 MOE models fine-tuning Fine-tuning support includes MOE models: 30B-A3B and 235B-A22B. Qwen3-30B-A3B works on 30GB VRAM with Unsloth. On fine-tuning MoE's - it's probably not a good idea to fine-tune the router layer so we disabled it by default. **Qwen3-2507-4B notebooks for:** [Thinking](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Thinking.ipynb) and [Instruct](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-Instruct.ipynb) The 30B-A3B fits in 30GB VRAM, but you may lack RAM or disk space since the full 16-bit model must be downloaded and converted to 4-bit on the fly for QLoRA fine-tuning. This is due to issues importing 4-bit BnB MOE models directly. This only affects MOE models. If you're fine-tuning the MOE models, please use `FastModel` and not `FastLanguageModel` Copy from unsloth import FastModel import torch model, tokenizer = FastModel.from_pretrained( model_name = "unsloth/Qwen3-30B-A3B-Instruct-2507", max_seq_length = 2048, # Choose any for long context! load_in_4bit = True, # 4 bit quantization to reduce memory load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory full_finetuning = False, # [NEW!] We have full finetuning now! # token = "hf_...", # use one if using gated models ) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMUjDPzhhjMJXcljIhgbK%252Fqwen33%2520mascot.png%3Falt%3Dmedia%26token%3Dfcfa1104-8f6d-4f04-b72d-b9c085d3ecda&width=768&dpr=4&quality=100&sign=38a403a8&sv=2) [PreviousQwen3: How to Run & Fine-tune](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune) [NextKimi K2: How to Run Locally](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally) Last updated 14 days ago Was this helpful? --- # Qwen3-Coder: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Qwen3-Coder is Qwenā€™s new series of coding agent models, available in 30B (**Qwen3-Coder-Flash**) and 480B parameters. **Qwen3-480B-A35B-Instruct** achieves SOTA coding performance rivalling Claudeā€ÆSonnet-4, GPT-4.1, and [Kimi K2](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally) , with 61.8% on Aider Polygot and support for 256K (extendable to 1M) token context. We also uploaded Qwen3-Coder with native **1M context length** extended by YaRN and full-precision 8bit and 16bit versions. [Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [RL](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) of Qwen3-Coder. [**UPDATE:** We fixed tool-calling for Qwen3-Coder!](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#tool-calling-fixes) You can now use tool-calling seamlessly in llama.cpp, Ollama, LMStudio, Open WebUI, Jan etc. This issue was universal and affected all uploads (not just Unsloth), and we've communicated with the Qwen team about our fixes! [Read more](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#tool-calling-fixes) [Run 30B-A3B](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-30b-a3b-instruct) [Run 480B-A35B](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-480b-a35b-instruct) **Does** [**Unsloth Dynamic Quants**](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) **work?** Yes, and very well. In third-party testing on the Aider Polyglot benchmark, the **UD-Q4\_K\_XL (276GB)** dynamic quant nearly matched the **full bf16 (960GB)** Qwen3-coder model, scoring 60.9% vs 61.8%. [More details here.](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/discussions/8) #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#qwen3-coder-unsloth-dynamic-2.0-ggufs) **Qwen3 Coder - Unsloth Dynamic 2.0 GGUFs**: Dynamic 2.0 GGUF (to run) 1M Context Dynamic 2.0 GGUF * [30B-A3B-Instruct](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) * [480B-A35B-Instruct](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF) * [30B-A3B-Instruct](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-GGUF) * [480B-A35B-Instruct](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-GGUF) [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#running-qwen3-coder) šŸ–„ļø **Running Qwen3-Coder** ---------------------------------------------------------------------------------------------------------------------- Below are guides for the [**30B-A3B**](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-30b-a3b-instruct) and [**480B-A35B**](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-480b-a35b-instruct) variants of the model. ### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#recommended-settings) āš™ļø Recommended Settings Qwen recommends these inference settings for both models: `temperature=0.7`, `top_p=0.8`, `top_k=20`, `repetition_penalty=1.05` * **Temperature of 0.7** * Top\_K of 20 * Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1) * Top\_P of 0.8 * **Repetition Penalty of 1.05** * Chat template: Copy <|im_start|>user Hey there!<|im_end|> <|im_start|>assistant What is 1+1?<|im_end|> <|im_start|>user 2<|im_end|> <|im_start|>assistant * Recommended context output: 65,536 tokens (can be increased). Details here. **Chat template/prompt format with newlines un-rendered** Copy <|im_start|>user\nHey there!<|im_end|>\n<|im_start|>assistant\nWhat is 1+1?<|im_end|>\n<|im_start|>user\n2<|im_end|>\n<|im_start|>assistant\n **Chat template for tool calling** (Getting the current temperature for San Francisco). More details here for how to format tool calls. Copy <|im_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im_end|> <|im_start|>assistant \n\n\nSan Francisco, CA, USA \n\n<|im_end|> <|im_start|>user {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} \n<|im_end|> Reminder that this model supports only non-thinking mode and does not generate `` blocks in its output. Meanwhile, specifying `enable_thinking=False` is no longer required. ### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-30b-a3b-instruct) Run Qwen3-Coder-30B-A3B-Instruct: To achieve inference speeds of 6+ tokens per second for our Dynamic 4-bit quant, have at least **18GB of unified memory** (combined VRAM and RAM) or **18GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model youā€™re using. E.g. the UD\_Q8\_K\_XL quant (full precision), which is 32.5GB, will require at least **33GB of unified memory** (VRAM + RAM) or **33GB of RAM** for optimal performance. **NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds. Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate ` ` blocks. Follow the [**best practices above**](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#recommended-settings) . They're the same as the 480B model. #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#ollama-run-qwen3-coder-30b-a3b-instruct-tutorial) šŸ¦™ Ollama: Run Qwen3-Coder-30B-A3B-Instruct Tutorial 1. Install `ollama` if you haven't already! You can only run models up to 32B in size. Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! Copy ollama run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-coder-30b-a3b-instruct-tutorial) āœØ Llama.cpp: Run Qwen3-Coder-30B-A3B-Instruct Tutorial 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. You can directly pull from HuggingFace via: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL \ --jinja -ngl 99 --threads -1 --ctx-size 32684 \ --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --repeat-penalty 1.05 2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF", local_dir = "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], ) ### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#run-qwen3-coder-480b-a35b-instruct) Run Qwen3-Coder-480B-A35B-Instruct: To achieve inference speeds of 6+ tokens per second for our 1-bit quant, we recommend at least **150GB of unified memory** (combined VRAM and RAM) or **150GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model youā€™re using. E.g. the Q2\_K\_XL quant, which is 180GB, will require at least **180GB of unified memory** (VRAM + RAM) or **180GB of RAM** for optimal performance. **NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds. Follow the [**best practices above**](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#recommended-settings) . They're the same as the 30B model. #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-coder-480b-a35b-instruct-tutorial) šŸ“– Llama.cpp: Run Qwen3-Coder-480B-A35B-Instruct Tutorial For Coder-480B-A35B, we will specifically use Llama.cpp for optimized inference and a plethora of options. If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions! 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do: Copy ./llama.cpp/llama-cli \ -hf unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:Q2_K_XL \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 0.7 \ --min-p 0.0 \ --top-p 0.8 \ --top-k 20 \ --repeat-penalty 1.05 3. Or, download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions.. Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF", local_dir = "unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF", allow_patterns = ["*UD-Q2_K_XL*"], ) 4. Run the model in conversation mode and try any prompt. 5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. More options discussed [here](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#improving-generation-speed) . Copy ./llama.cpp/llama-cli \ --model unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/UD-Q2_K_XL/Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL-00001-of-00004.gguf \ --threads -1 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --temp 0.7 \ --min-p 0.0 \ --top-p 0.8 \ --top-k 20 \ --repeat-penalty 1.05 Also don't forget about the new Qwen3 update. Run [**Qwen3-235B-A22B-Instruct-2507**](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) locally with llama.cpp. #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#improving-generation-speed) šŸ› ļø Improving generation speed If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves. Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers. Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers. You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards. The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel) . You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#how-to-fit-long-context-256k-to-1m) šŸ“How to fit long context (256K to 1M) To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below. `--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1` You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1` You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. We also uploaded 1 million context length GGUFs via YaRN scaling [here](https://docs.unsloth.ai/) . [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#tool-calling-fixes) šŸ§° Tool Calling Fixes --------------------------------------------------------------------------------------------------------------- We managed to fix tool calling via `llama.cpp --jinja` specifically for serving through `llama-server`! If youā€™re downloading our 30B-A3B quants, no need to worry as these already include our fixes. For the 480B-A35B model, please: 1. Download the first file at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/tree/main/UD-Q2\_K\_XL for UD-Q2\_K\_XL, and replace your current file 2. Use `snapshot_download` as usual as in https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-tutorial which will auto override the old files 3. Use the new chat template via `--chat-template-file`. See [GGUF chat template](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF?chat_template=default) or [chat\_template.jinja](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct/raw/main/chat_template.jinja) 4. As an extra, we also made 1 single 150GB UD-IQ1\_M file (so Ollama works) at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/blob/main/Qwen3-Coder-480B-A35B-Instruct-UD-IQ1\_M.gguf This should solve issues like: https://github.com/ggml-org/llama.cpp/issues/14915 ### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#using-tool-calling) Using Tool Calling To format the prompts for tool calling, let's showcase it with an example. I created a Python function called `get_current_temperature` which is a function which should get the current temperature for a location. For now we created a placeholder function which will always return 21.6 degrees celsius. You should change this to a true function!! Copy def get_current_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, # PRE_CONFIGURED -> you change this! "location": location, "unit": unit, } Then use the tokenizer to create the entire prompt: Copy from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-Coder-480B-A35B-Instruct") messages = [\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},\ {'content': "", 'role': 'assistant', 'function_call': None, 'tool_calls': [\ {'id': 'ID', 'function': {'arguments': {"location": "San Francisco, CA, USA"}, 'name': 'get_current_temperature'}, 'type': 'function'},\ ]},\ {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool_call_id': 'ID'},\ ] prompt = tokenizer.apply_chat_template(messages, tokenize = False) [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#performance-benchmarks) šŸ’”Performance Benchmarks ---------------------------------------------------------------------------------------------------------------------- These official benchmarks are for the full BF16 checkpoint. To use this, simply use the `Q8_K_XL, Q8_0, BF16` checkpoints we uploaded - you can still use the tricks like MoE offloading for these versions as well! Here are the benchmarks for the 480B model: #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#agentic-coding) Agentic Coding Benchmark Qwen3ā€‘Coderā€Æ40Bā€‘A35Bā€‘Instruct Kimiā€‘K2 DeepSeekā€‘V3-0324 Claudeā€Æ4 Sonnet GPTā€‘4.1 Terminalā€‘Bench **37.5** 30.0 2.5 35.5 25.3 SWEā€‘benchā€ÆVerifiedā€Æw/ā€ÆOpenHandsā€Æ(500ā€Æturns) **69.6** ā€“ ā€“ 70.4 ā€“ SWEā€‘benchā€ÆVerifiedā€Æw/ā€ÆOpenHandsā€Æ(100ā€Æturns) **67.0** 65.4 38.8 68.0 48.6 SWEā€‘benchā€ÆVerifiedā€Æw/ā€ÆPrivateā€ÆScaffolding ā€“ 65.8 ā€“ 72.7 63.8 SWEā€‘benchā€ÆLive **26.3** 22.3 13.0 27.7 ā€“ SWEā€‘benchā€ÆMultilingual **54.7** 47.3 13.0 53.3 31.5 Multiā€‘SWEā€‘benchā€Æmini **25.8** 19.8 7.5 24.8 ā€“ Multiā€‘SWEā€‘benchā€Æflash **27.0** 20.7 ā€“ 25.0 ā€“ Aiderā€‘Polyglot **61.8** 60.0 56.9 56.4 52.4 Spider2 **31.1** 25.2 12.8 31.1 16.5 #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#agentic-browser-use) Agentic Browserā€ÆUse Benchmark Qwen3ā€‘Coderā€Æ40Bā€‘A35Bā€‘Instruct Kimiā€‘K2 DeepSeekā€‘V3ā€Æ0324 Claudeā€ÆSonnetā€‘4 GPTā€‘4.1 WebArena **49.9** 47.4 40.0 51.1 44.3 Mind2Web **55.8** 42.7 36.0 47.4 49.6 #### [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#agentic-tool-use) Agentic Toolā€Æ-Use Benchmark Qwen3ā€‘Coderā€Æ40Bā€‘A35Bā€‘Instruct Kimiā€‘K2 DeepSeekā€‘V3ā€Æ0324 Claudeā€ÆSonnetā€‘4 GPTā€‘4.1 BFCLā€‘v3 **68.7** 65.2 56.9 73.3 62.9 TAUā€‘Benchā€ÆRetail **77.5** 70.7 59.1 80.5 ā€“ TAUā€‘Benchā€ÆAirline **60.0** 53.5 40.0 60.0 ā€“ [PreviousGemma 3: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune) [NextQwen3: How to Run & Fine-tune](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune) Last updated 26 days ago Was this helpful? --- # Tutorial: How to Fine-tune gpt-oss | Unsloth Documentation OpenAI gpt-oss & all model types now supported! In this guide with screenshots, you'll learn to fine-tune your own custom gpt-oss model either [locally](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#local-gpt-oss-fine-tuning) on your machine or for free using [Google Colab](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#colab-gpt-oss-fine-tuning) . We'll walk you through the entire process, from setup to running and saving your trained model. > **Quickstart:** Fine-tune gpt-oss-20b for free with our: [Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) Unsloth gpt-oss fine-tuning, when compared to all other FA2 implementations, achieves 1.5Ɨ faster training, 70% reduction in VRAM use, and 10x longer context lengths - with no accuracy loss. * **QLoRA requirements:** gpt-oss-20b = 14GB VRAM ā€¢ gpt-oss-120b = 65GB VRAM. * **BF16 LoRA requirements:** gpt-oss-20b = 44GB VRAM ā€¢ gpt-oss-120b = 210GB VRAM. We've updated the gpt-oss fine-tuning and inference notebooks, making them much more stable. [Local Guide](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#local-gpt-oss-fine-tuning) [Colab Guide](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#colab-gpt-oss-fine-tuning) [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#colab-gpt-oss-fine-tuning) šŸŒ Colab gpt-oss Fine-tuning ----------------------------------------------------------------------------------------------------------------------------------------------------------------- This section covers fine-tuning gpt-oss using our Google Colab [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) . You can also save and use the gpt-oss notebook into your favorite code editor and follow our [local gpt-oss guide](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#local-gpt-oss-fine-tuning) . 1 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#install-unsloth-in-colab) Install Unsloth (in Colab) In Colab, run cells **from top to bottom**. Use **Run all** for the first pass. The first cell installs Unsloth (and related dependencies) and prints GPU/memory info. If a cell throws an error, simply re-run it. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FnVWahTM3dRcNxUl7yNlw%252Fchrome_wTbzfmSI21.png%3Falt%3Dmedia%26token%3Dfe257ba6-512d-4000-bdf7-9a9a586c85a4&width=768&dpr=4&quality=100&sign=d862d192&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FwSOux9qJpXmROoriYA4U%252Fchrome_yPnb553OGW.png%3Falt%3Dmedia%26token%3Dc14a59e6-709e-44b5-9aa3-6ab8eeb610da&width=768&dpr=4&quality=100&sign=cd2104a2&sv=2) 2 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#configuring-gpt-oss-and-reasoning-effort) Configuring gpt-oss and Reasoning Effort Weā€™ll load `**gpt-oss-20b**` using Unsloth's [linearized version](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#making-efficient-gpt-oss-fine-tuning-work) (as no other version will work). Configure the following parameters: * `max_seq_length = 1024` * Recommended for quick testing and initial experiments. * `load_in_4bit = True` * Use `False` for LoRA training (note: setting this to `False` will need at least 43GB VRAM). You _**MUST**_ also set `**model_name = "unsloth/gpt-oss-20b-BF16"**` ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FndJWBQP3WUW5tR6CNyrP%252Fchrome_3qSe2UIFN0.png%3Falt%3Dmedia%26token%3Db43534ee-0d71-495a-b89c-91f52317354f&width=768&dpr=4&quality=100&sign=da59ea44&sv=2) You should see output similar to the example below. Note: We explicitly change the `dtype` to `float32` to ensure correct training behavior. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FOMNOnDuWl2c95WuxSkDA%252Fchrome_DGMDHldw0J.png%3Falt%3Dmedia%26token%3Da086266b-7b88-4fcf-a7cd-5a17cc57e7f9&width=768&dpr=4&quality=100&sign=76171b54&sv=2) 3 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#fine-tuning-hyperparameters-lora) Fine-tuning Hyperparameters (LoRA) Now it's time to adjust your training hyperparameters. For a deeper dive into how, when, and what to tune, check out our [detailed hyperparameters guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) . To avoid [overfitting](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting) , monitor your training loss and avoid setting these values too high. This step adds LoRA adapters for parameter-efficient fine-tuning. Only about 1% of the modelā€™s parameters are trained, which makes the process significantly more efficient. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fwkbdee4FuThTM09oqUkL%252Fchrome_ucj0VKT1lh.png%3Falt%3Dmedia%26token%3D40b5ae77-31f8-4e13-841d-e4cc52e1436b&width=768&dpr=4&quality=100&sign=995d3516&sv=2) 4 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#try-inference) Try Inference In the notebook, there's a section called _"Reasoning Effort"_ that demonstrates gpt-oss inference running in Colab. You can skip this step, but you'll still need to run the model later once you've finished fine-tuning it. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfXyFmwpMF1AgRRhnOQR8%252Fchrome_o2rLNfES8e.png%3Falt%3Dmedia%26token%3D6ef340fa-2ac0-4e82-9338-d91f66d1557a&width=768&dpr=4&quality=100&sign=c95105ea&sv=2) 5 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#data-preparation) Data Preparation For this example, we will use the [`HuggingFaceH4/Multilingual-Thinking`](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking) . This dataset contains chain-of-thought reasoning examples derived from user questions translated from English into four additional languages. This is the same dataset referenced in OpenAI's fine-tuning cookbook. The goal of using a multilingual dataset is to help the model learn and generalize reasoning patterns across multiple languages. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fii6rqKAKqBYea2ZLoXKJ%252Fchrome_rRKmU99f0T.png%3Falt%3Dmedia%26token%3D74547cc7-0be9-4687-b128-1ff4b87d544f&width=768&dpr=4&quality=100&sign=67a816c1&sv=2) gpt-oss introduces a reasoning effort system that controls how much reasoning the model performs. By default, the reasoning effort is set to `low`, but you can change it by setting the `reasoning_effort` parameter to `low`, `medium` or `high`. Example: Copy tokenizer.apply_chat_template( text, tokenize = False, add_generation_prompt = False, reasoning_effort = "medium", ) To format the dataset, we apply a customized version of the gpt-oss prompt: Copy from unsloth.chat_templates import standardize_sharegpt dataset = standardize_sharegpt(dataset) dataset = dataset.map(formatting_prompts_func, batched = True,) Let's inspect the dataset by printing the first example: Copy print(dataset[0]['text']) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDoRtTfO0oSVDg99Dm3dc%252Fchrome_sjbDtIhP5e.png%3Falt%3Dmedia%26token%3Dc0fb44b6-861c-47b1-86a5-75c55771936e&width=768&dpr=4&quality=100&sign=4c0e94e9&sv=2) One unique feature of gpt-oss is its use of the [**OpenAI Harmony format**](https://github.com/openai/harmony) **,** which supports structured conversations, reasoning output, and tool calling. This format includes tags such as `<|start|>` , `<|message|>` , and `<|return|>` . šŸ¦„ Unsloth fixes the chat template to ensure it is correct. See this [tweet](https://x.com/danielhanchen/status/1953901104150065544) for technical details on our template fix. Feel free to adapt the prompt and structure to suit your own dataset or use-case. For more guidance, refer to our [dataset guide](https://docs.unsloth.ai/basics/datasets-guide) . 6 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#train-the-model) Train the model We've pre-selected training hyperparameters for optimal results. However, you can modify them based on your specific use case. Refer to our [hyperparameters guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) . In this example, we train for 60 steps to speed up the process. For a full training run, set `num_train_epochs=1` and disable the step limiting by setting `max_steps=None`. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FcQroeXLcHOHaRsUiCyYL%252Fchrome_R85PmZRHMQ.png%3Falt%3Dmedia%26token%3De2069d2e-ef15-4179-ba49-fc484cf26b0b&width=768&dpr=4&quality=100&sign=48c74275&sv=2) During training, monitor the loss to ensure that it is decreasing over time. This confirms that the training process is functioning correctly. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FmcHwJsR2kzTpab4gTgUY%252Fimage.png%3Falt%3Dmedia%26token%3D03b873b3-8e1c-42ee-826e-d62feab7d703&width=768&dpr=4&quality=100&sign=da3eb5d6&sv=2) 7 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#inference-run-your-trained-model) Inference: Run your trained model Now it's time to run inference with your fine-tuned model. You can modify the instruction and input, but leave the output blank. In this example, we test the model's ability to reason in French by adding a specific instruction to the system prompt, following the same structure used in our dataset. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2oDtZBxHXle9KsWSqTzT%252Fchrome_jbJmBTaY7B.png%3Falt%3Dmedia%26token%3D9a2bcba5-9e60-4a5e-836c-27e5f45a9bf4&width=768&dpr=4&quality=100&sign=4f7f9555&sv=2) This should produce an output similar to: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9RTKGdSeuca5QfDhVXFw%252Fchrome_ORco4bpZZ6.png%3Falt%3Dmedia%26token%3D1d5bf29e-c57c-41f0-a2e5-162408d80690&width=768&dpr=4&quality=100&sign=737948d&sv=2) 8 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#save-export-your-model) Save/export your model To save your fine-tuned model, it must be exported in the Safetensors format. Remember: Saving or merging QLoRA fine-tuned models to GGUF is not yet supported, as currently, QLoRA fine-tuned gpt-oss models only work in Unsloth. To use your fine-tuned gpt-oss models in other frameworks (e.g. Hugging Face, llama.cpp with GGUF), you must train with **LoRA** on our [**BF16 model**](https://huggingface.co/unsloth/gpt-oss-20b-BF16) (requires >43GB VRAM). This produces a BF16 fine-tuned model that can be exported and converted as needed. You can use our [llama.cpp GGUF scripts](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) to convert the model to GGUF. To save your LoRA adapters locally and optionally push them to the Hugging Face Hub, follow these steps: Copy model.save_pretrained("finetuned_model) tokenizer.save_pretrained("finetuned_model") Copy model.push_to_hub("hf_username/finetuned_model", token = "hf_...") # Save to HF tokenizer.push_to_hub("hf_username/finetuned_model", token = "hf_...") # Save to HF ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FVHzhTH5oCJZKPXpqmuOQ%252Fchrome_fKEKXHti5r.png%3Falt%3Dmedia%26token%3Dc470698a-80e5-4c52-92e2-bff901fc2746&width=768&dpr=4&quality=100&sign=4d29ff9b&sv=2) [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#local-gpt-oss-fine-tuning) šŸ–„ļø Local gpt-oss Fine-tuning ------------------------------------------------------------------------------------------------------------------------------------------------------------------ This chapter covers fine-tuning gpt-oss on your local device. While **gpt-oss-20b** fine-tuning can operate on just 14GB VRAM, we recommend having at least 16GB VRAM available to ensure stable and reliable training runs. We recommend downloading or incorporating elements from our Colab [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) into your local setup for easier use. 1 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#install-unsloth-locally) Install Unsloth Locally Ensure your device is [Unsloth compatible](https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements) and you can read our detailed [installation guide](https://docs.unsloth.ai/get-started/installing-+-updating) . Note that `pip install unsloth` will not work for this setup, as we need to use the latest PyTorch, Triton and related packages. Install Unsloth using this specific command: Copy # We're installing the latest Torch, Triton, OpenAI's Triton kernels, Transformers and Unsloth! !pip install --upgrade -qqq uv try: import numpy; install_numpy = f"numpy=={numpy.__version__}" except: install_numpy = "numpy" !uv pip install -qqq \ "torch>=2.8.0" "triton>=3.4.0" {install_numpy} \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" \ torchvision bitsandbytes \ git+https://github.com/huggingface/transformers \ git+https://github.com/triton-lang/triton.git@05b2c186c1b6c9a08375389d5efe9cb4c401c075#subdirectory=python/triton_kernels 2 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#configuring-gpt-oss-and-reasoning-effort-1) Configuring gpt-oss and Reasoning Effort Weā€™ll load `**gpt-oss-20b**` using Unsloth's [linearized version](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune#making-efficient-gpt-oss-fine-tuning-work) (as no other version will work for QLoRA fine-tuning). Configure the following parameters: * `max_seq_length = 2048` * Recommended for quick testing and initial experiments. * `load_in_4bit = True` * Use `False` for LoRA training (note: setting this to `False` will need at least 43GB VRAM). You _**MUST**_ also set `**model_name = "unsloth/gpt-oss-20b-BF16"**` Copy from unsloth import FastLanguageModel import torch max_seq_length = 1024 dtype = None # 4bit pre quantized models we support for 4x faster downloading + no OOMs. fourbit_models = [\ "unsloth/gpt-oss-20b-unsloth-bnb-4bit", # 20B model using bitsandbytes 4bit quantization\ "unsloth/gpt-oss-120b-unsloth-bnb-4bit",\ "unsloth/gpt-oss-20b", # 20B model using MXFP4 format\ "unsloth/gpt-oss-120b",\ ] # More models at https://huggingface.co/unsloth model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/gpt-oss-20b", dtype = dtype, # None for auto detection max_seq_length = max_seq_length, # Choose any for long context! load_in_4bit = True, # 4 bit quantization to reduce memory full_finetuning = False, # [NEW!] We have full finetuning now! # token = "hf_...", # use one if using gated models ) You should see output similar to the example below. Note: We explicitly change the `dtype` to `float32` to ensure correct training behavior. 3 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#fine-tuning-hyperparameters-lora-1) Fine-tuning Hyperparameters (LoRA) Now it's time to adjust your training hyperparameters. For a deeper dive into how, when, and what to tune, check out our [detailed hyperparameters guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) . To avoid [overfitting](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting) , monitor your training loss and avoid setting these values too high. This step adds LoRA adapters for parameter-efficient fine-tuning. Only about 1% of the modelā€™s parameters are trained, which makes the process significantly more efficient. Copy model = FastLanguageModel.get_peft_model( model, r = 8, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ ) 4 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#data-preparation-1) Data Preparation For this example, we will use the [`HuggingFaceH4/Multilingual-Thinking`](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking) . This dataset contains chain-of-thought reasoning examples derived from user questions translated from English into four additional languages. This is the same dataset referenced in OpenAI's fine-tuning cookbook. The goal of using a multilingual dataset is to help the model learn and generalize reasoning patterns across multiple languages. Copy def formatting_prompts_func(examples): convos = examples["messages"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass from datasets import load_dataset dataset = load_dataset("HuggingFaceH4/Multilingual-Thinking", split="train") dataset gpt-oss introduces a reasoning effort system that controls how much reasoning the model performs. By default, the reasoning effort is set to `low`, but you can change it by setting the `reasoning_effort` parameter to `low`, `medium` or `high`. Example: Copy tokenizer.apply_chat_template( text, tokenize = False, add_generation_prompt = False, reasoning_effort = "medium", ) To format the dataset, we apply a customized version of the gpt-oss prompt: Copy from unsloth.chat_templates import standardize_sharegpt dataset = standardize_sharegpt(dataset) dataset = dataset.map(formatting_prompts_func, batched = True,) Let's inspect the dataset by printing the first example: Copy print(dataset[0]['text']) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FvXrJGLlHZxgAazLFreMh%252Fimage.png%3Falt%3Dmedia%26token%3D9ddd4b8f-a884-4243-931d-39bd29274ffd&width=768&dpr=4&quality=100&sign=507d03f7&sv=2) One unique feature of gpt-oss is its use of the [**OpenAI Harmony format**](https://github.com/openai/harmony) **,** which supports structured conversations, reasoning output, and tool calling. This format includes tags such as `<|start|>` , `<|message|>` , and `<|return|>` . šŸ¦„ Unsloth fixes the chat template to ensure it is correct. See this [tweet](https://x.com/danielhanchen/status/1953901104150065544) for technical details on our template fix. Feel free to adapt the prompt and structure to suit your own dataset or use-case. For more guidance, refer to our [dataset guide](https://docs.unsloth.ai/basics/datasets-guide) . 5 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#train-the-model-1) Train the model We've pre-selected training hyperparameters for optimal results. However, you can modify them based on your specific use case. Refer to our [hyperparameters guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) . In this example, we train for 60 steps to speed up the process. For a full training run, set `num_train_epochs=1` and disable the step limiting by setting `max_steps=None`. Copy from trl import SFTConfig, SFTTrainer trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, args = SFTConfig( per_device_train_batch_size = 1, gradient_accumulation_steps = 4, warmup_steps = 5, # num_train_epochs = 1, # Set this for 1 full training run. max_steps = 30, learning_rate = 2e-4, logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # Use this for WandB etc ), ) During training, monitor the loss to ensure that it is decreasing over time. This confirms that the training process is functioning correctly. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FmcHwJsR2kzTpab4gTgUY%252Fimage.png%3Falt%3Dmedia%26token%3D03b873b3-8e1c-42ee-826e-d62feab7d703&width=768&dpr=4&quality=100&sign=da3eb5d6&sv=2) 6 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#inference-run-your-trained-model-1) Inference: Run Your Trained Model Now it's time to run inference with your fine-tuned model. You can modify the instruction and input, but leave the output blank. In this example, we test the model's ability to reason in French by adding a specific instruction to the system prompt, following the same structure used in our dataset. Copy messages = [\ {"role": "system", "content": "reasoning language: French\n\nYou are a helpful assistant that can solve mathematical problems."},\ {"role": "user", "content": "Solve x^5 + 3x^4 - 10 = 3."},\ ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt = True, return_tensors = "pt", return_dict = True, reasoning_effort = "medium", ).to(model.device) from transformers import TextStreamer _ = model.generate(**inputs, max_new_tokens = 2048, streamer = TextStreamer(tokenizer)) This should produce an output similar to: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqPoBw62CGTVsjOmGliqi%252Fimage.png%3Falt%3Dmedia%26token%3Da5a73e2e-53f6-4e5b-a694-eca648019542&width=768&dpr=4&quality=100&sign=6f5a72bc&sv=2) 7 ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#save-and-export-your-model) Save and Export Your Model To save your fine-tuned model, it must be exported in the Safetensors format. Remember: Saving or merging QLoRA fine-tuned models to GGUF is not yet supported, as currently, QLoRA fine-tuned gpt-oss models only work in Unsloth. To use your fine-tuned gpt-oss models in other frameworks (e.g. Hugging Face, llama.cpp with GGUF), you must train with **LoRA** on our [**BF16 model**](https://huggingface.co/unsloth/gpt-oss-20b-BF16) (requires >43GB VRAM). This produces a BF16 fine-tuned model that can be exported and converted as needed. You can use our [llama.cpp GGUF scripts](https://docs.unsloth.ai/basics/running-and-saving-models/saving-to-gguf) to convert the model to GGUF. To save your LoRA adapters locally and optionally push them to the Hugging Face Hub, follow these steps: Copy model.save_pretrained("finetuned_model) tokenizer.save_pretrained("finetuned_model") Copy model.push_to_hub("hf_username/finetuned_model", token = "hf_...") # Save to HF tokenizer.push_to_hub("hf_username/finetuned_model", token = "hf_...") # Save to HF ### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#and-thats-it) šŸ And that's it! You've fine-tuned gpt-oss with Unsloth. We're currently working on RL and GRPO implementations, as well as improved model saving and running, so stay tuned. As always, feel free to drop by our [Discord](https://discord.com/invite/unsloth) or [Reddit](https://www.reddit.com/r/unsloth/) if you need any help. [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#faq-frequently-asked-questions) ā“FAQ (Frequently Asked Questions) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#id-1.-can-i-export-my-model-to-use-in-hugging-face-llama.cpp-gguf-or-vllm-later) 1\. Can I export my model to use in Hugging Face, llama.cpp GGUF or vLLM later? Yes you can but only if you do LoRA fine-tuning and utilize our [bf16 weights](https://huggingface.co/unsloth/gpt-oss-20b-BF16) for fine-tuning. This means you **must** set `model_name = "unsloth/gpt-oss-20b-BF16".` Keep in mind VRAM usage will be 4x more so gpt-oss-20b will require about 45GB VRAM. We are working on better exporting for the model so stay tuned. #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#id-2.-can-i-do-fp4-or-mxfp4-training-with-gpt-oss) 2\. Can I do fp4 or MXFP4 training with gpt-oss? No, currently no framework supports fp4 or MXFP4 training. Unsloth however is the only framework to support QLoRA 4-bit fine-tuning for the model, enabling more than 4x less VRAM use. #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#id-3.-can-i-export-my-model-to-mxfp4-format-after-training) 3\. Can I export my model to MXFP4 format after training? No, currently no library or framework supports this. #### [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss#id-4.-can-i-do-reinforcement-learning-rl-or-grpo-with-gpt-oss) 4\. Can I do Reinforcement Learning (RL) or GRPO with gpt-oss? No, currently no library or framework supports RL for gpt-oss. We are working on it however it will be hard to fit it on Colab considering how large the model is. * * * _**Acknowledgements:**_ _A huge thank you to_ [_Eyera_](https://huggingface.co/Orenguteng) _for contributing to this guide!_ [Previousgpt-oss: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune) [NextGemma 3: How to Run & Fine-tune](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune) Last updated 8 days ago Was this helpful? --- # DeepSeek-V3-0324: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Please see [https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) (May 28th 2025 update) to learn on how to run DeepSeek faster and more efficiently! DeepSeek is at it again! After releasing V3, R1 Zero and R1 back in December 2024 and January 2025, DeepSeek updated their checkpoints / models for V3, and released a March update! According to DeepSeek, MMLU-Pro jumped +5.3% to 81.2%. **GPQA +9.3% points**. AIME + 19.8% and LiveCodeBench + 10.0%! They provided a plot showing how they compared to the previous V3 checkpoint and other models like GPT 4.5 and Claude Sonnet 3.7. **But how do we run a 671 billion parameter model locally?** MoE Bits Type Disk Size Accuracy Link Details 1.78bit IQ1\_S **173GB** Ok [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ1_S) 2.06/1.56bit 1.93bit IQ1\_M **183GB** Fair [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ1_M) 2.5/2.06/1.56 2.42bit IQ2\_XXS **203GB** **Suggested** [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ2_XXS) 2.5/2.06bit 2.71bit Q2\_K\_XL **231GB** **Suggested** [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q2_K_XL) 3.5/2.5bit 3.5bit Q3\_K\_XL **320GB** Great [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q3_K_XL) 4.5/3.5bit 4.5bit Q4\_K\_XL **406GB** Best [Link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q4_K_XL) 5.5/4.5bit DeepSeek V3's original upload is in float8, which takes 715GB. Using Q4\_K\_M halves the file size to 404GB or so, and our dynamic 1.78bit quant fits in around 151GB. **We suggest using our 2.7bit quant to balance size and accuracy! The 2.4bit one also works well!** [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#official-recommended-settings) āš™ļø Official Recommended Settings ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) , these are the recommended settings for inference: * **Temperature of 0.3** (Maybe 0.0 for coding as [seen here](https://api-docs.deepseek.com/quick_start/parameter_settings) ) * Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1) * Chat template: `<ļ½œUserļ½œ>Create a simple playable Flappy Bird Game in Python. Place the final game inside of a markdown section.<ļ½œAssistantļ½œ>` * A BOS token of `<ļ½œbeginā–ofā–sentenceļ½œ>` is auto added during tokenization (do NOT add it manually!) * DeepSeek mentioned using a **system prompt** as well (optional) - it's in Chinese: `čÆ„åŠ©ę‰‹äøŗDeepSeek Chatļ¼Œē”±ę·±åŗ¦ę±‚ē“¢å…¬åøåˆ›é€ ć€‚\n今天ę˜Æ3꜈24ę—„ļ¼Œę˜ŸęœŸäø€ć€‚` which translates to: `The assistant is DeepSeek Chat, created by DeepSeek.\nToday is Monday, March 24th.` * **For KV cache quantization, use 8bit, NOT 4bit - we found it to do noticeably worse.** [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#tutorial-how-to-run-deepseek-v3-in-llama.cpp) šŸ“– Tutorial: How to Run DeepSeek-V3 in llama.cpp ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. NOTE using `-DGGML_CUDA=ON` for GPUs might take 5 minutes to compile. CPU only takes 1 minute to compile. You might be interested in llama.cpp's precompiled binaries. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . **I recommend using our 2.7bit dynamic quant** `**UD-Q2_K_XL**` **to balance size and accuracy**. More versions at: [https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/DeepSeek-V3-0324-GGUF-UD", local_dir = "unsloth/DeepSeek-V3-0324-GGUF-UD", allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2.7bit (230GB) Use "*UD-IQ_S*" for Dynamic 1.78bit (151GB) ) 1. Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1. 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. Copy ./llama.cpp/llama-cli \ --model unsloth/DeepSeek-V3-0324-GGUF-UD/blob/main/UD-Q2_K_XL/DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf \ --cache-type-k q8_0 \ --threads 20 \ --n-gpu-layers 2 \ -no-cnv \ --prio 3 \ --temp 0.3 \ --min_p 0.01 \ --ctx-size 4096 \ --seed 3407 \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<ļ½œAssistantļ½œ>" If we run the above, we get 2 very different results. **Standard 2-bit version:** Click to view result _**(seizure warning!)**_ **Dynamic 2-bit version:** See the result below:[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#if-we-run-the-above-we-get-2-very-different-results.-standard-2-bit-version-click-to-view-result-sei) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7sXwEonmVeWZaIXbT4Ry%252FOld.gif%3Falt%3Dmedia%26token%3D0b2bd075-091f-4ca6-affa-a9f8a3b98e49&width=300&dpr=4&quality=100&sign=5af7034c&sv=2) Standard 2-bit. Fails with background, fails with collision ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDcms38Q9DgdPAVyMIzof%252FNew.gif%3Falt%3Dmedia%26token%3D4c8870ae-71d1-4568-b413-780f10e7f892&width=768&dpr=4&quality=100&sign=7c148832&sv=2) Dynamic 2-bit. Succeeds in creating a playable game. 1. Like DeepSeek-R1, V3 has 61 layers. For example with a 24GB GPU or 80GB GPU, you can expect to offload after rounding down (reduce by 1 if it goes out of memory): Quant File Size 24GB GPU 80GB GPU 2x80GB GPU 1.73bit 173GB 5 25 56 2.22bit 183GB 4 22 49 2.51bit 212GB 2 19 32 ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#running-on-mac-apple-devices) Running on Mac / Apple devices For Apple Metal devices, be careful of --n-gpu-layers. If you find the machine going out of memory, reduce it. For a 128GB unified memory machine, you should be able to offload 59 layers or so. Copy ./llama.cpp/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-V3-0324-UD-IQ1_S/DeepSeek-V3-0324-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 16 \ --prio 2 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --n-gpu-layers 59 \ -no-cnv \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python.<ļ½œAssistantļ½œ>" [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#heptagon-test) šŸŽ± Heptagon Test -------------------------------------------------------------------------------------------------------------------------------------------------- We also test our dynamic quants via [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2O72oTw5yPUbcxXjDNKS%252Fsnapshot.jpg%3Falt%3Dmedia%26token%3Dce852f9f-20ee-4b93-9d7b-1a5f211b9e04&width=768&dpr=4&quality=100&sign=55d1134d&sv=2) The goal is to make the heptagon spin, and the balls in the heptagon should move. Copy ./llama.cpp/llama-cli \ --model unsloth/DeepSeek-V3-0324-GGUF-UD/blob/main/UD-Q2_K_XL/DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf \ --cache-type-k q8_0 \ --threads 20 \ --n-gpu-layers 2 \ -no-cnv \ --prio 3 \ --temp 0.3 \ --min_p 0.01 \ --ctx-size 4096 \ --seed 3407 \ --prompt "<ļ½œUserļ½œ>Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file.<ļ½œAssistantļ½œ>" ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8hq8kYZ8RmTUQjYuZN3w%252FInShot_20250325_185636426.gif%3Falt%3Dmedia%26token%3D41a46ca4-c4d1-4bac-a035-1d153269c29d&width=245&dpr=4&quality=100&sign=2a272dba&sv=2) Non Dynamic 2bit. Fails - SEIZURE WARNING again! [unsloth-q2\_k\_rotate.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FE9GSJlT4kXAR2LnBvNyk%2Funsloth-q2_k_rotate.txt?alt=media&token=46c4040e-e464-4562-9430-d017868a1077) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fssk5mGbDUHdYhdiDFHPT%252FInShot_20250325_181710554.gif%3Falt%3Dmedia%26token%3D50e93aa5-2a93-47d3-b118-f339dcf9d3c2&width=245&dpr=4&quality=100&sign=f431a57f&sv=2) Dynamic 2bit. Actually solves the heptagon puzzle correctly!! [unsloth-q2\_k\_xl\_rotate.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FIED4xFpcdldNQCO8KKOi%2Funsloth-q2_k_xl_rotate.txt?alt=media&token=9d1ec35f-f6ba-4f19-a374-6020801e493c) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYrKuXm5uhsFW3b8e92Cz%252FInShot_20250325_181423756.gif%3Falt%3Dmedia%26token%3Daf23c694-b8f7-4d75-b6ad-f87254eb73c0&width=245&dpr=4&quality=100&sign=649d2a08&sv=2) Original float8 [fp8-heptagon.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEP8pdoCOtznTdMTI7Pw8%2Ffp8-heptagon.txt?alt=media&token=93659885-d403-4c01-8864-b5d21fba8ce6) The dynamic 2.7 bit quant which is only 230GB in size actually manages to solve the heptagon puzzle! The full output for all 3 versions (including full fp8) is below: Dynamic 2bit Heptagon code[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#dynamic-2bit-heptagon-code) Copy import tkinter as tk import math import numpy as np from dataclasses import dataclass from typing import List, Tuple, Optional # Constants HEPTAGON_RADIUS = 300 BALL_RADIUS = 20 GRAVITY = 0.2 FRICTION = 0.99 BOUNCE_FACTOR = 0.8 ROTATION_SPEED = 360 / 5 # degrees per second SPIN_FRICTION = 0.98 BALL_COLORS = [\ "#f8b862", "#f6ad49", "#f39800", "#f08300", "#ec6d51",\ "#ee7948", "#ed6d3d", "#ec6800", "#ec6800", "#ee7800",\ "#eb6238", "#ea5506", "#ea5506", "#eb6101", "#e49e61",\ "#e45e32", "#e17b34", "#dd7a56", "#db8449", "#d66a35"\ ] @dataclass class Ball: x: float y: float vx: float vy: float number: int spin: float = 0.0 color: str = "#000000" @dataclass class Wall: x1: float y1: float x2: float y2: float class BouncingBalls: def __init__(self, root): self.root = root self.canvas = tk.Canvas(root, width=800, height=800, bg="white") self.canvas.pack() self.balls: List[Ball] = [] self.walls: List[Wall] = [] self.heptagon_angle = 0 self.last_time = 0 self.setup_balls() self.setup_heptagon() self.root.after(16, self.update) self.root.bind("", self.reset_balls) def setup_balls(self): for i in range(20): ball = Ball( x=400, y=400, vx=np.random.uniform(-5, 5), vy=np.random.uniform(-5, 5), number=i+1, color=BALL_COLORS[i] ) self.balls.append(ball) def setup_heptagon(self): # Create initial heptagon walls self.update_heptagon_walls(0) def update_heptagon_walls(self, angle): self.walls = [] center_x, center_y = 400, 400 angle_rad = math.radians(angle) for i in range(7): angle1 = angle_rad + 2 * math.pi * i / 7 angle2 = angle_rad + 2 * math.pi * (i + 1) / 7 x1 = center_x + HEPTAGON_RADIUS * math.cos(angle1) y1 = center_y + HEPTAGON_RADIUS * math.sin(angle1) x2 = center_x + HEPTAGON_RADIUS * math.cos(angle2) y2 = center_y + HEPTAGON_RADIUS * math.sin(angle2) self.walls.append(Wall(x1, y1, x2, y2)) def reset_balls(self, event=None): for ball in self.balls: ball.x = 400 ball.y = 400 ball.vx = np.random.uniform(-5, 5) ball.vy = np.random.uniform(-5, 5) ball.spin = np.random.uniform(-5, 5) def update(self): current_time = self.root.after_idle(self.root.after, 16, self.update) if self.last_time == 0: self.last_time = current_time return # Calculate delta time (approximate) dt = 0.016 # Assuming ~60 FPS # Update heptagon rotation self.heptagon_angle += ROTATION_SPEED * dt self.update_heptagon_walls(self.heptagon_angle) # Update balls for ball in self.balls: # Apply gravity ball.vy += GRAVITY # Apply friction ball.vx *= FRICTION ball.vy *= FRICTION ball.spin *= SPIN_FRICTION # Move ball ball.x += ball.vx ball.y += ball.vy # Check collisions with walls self.check_wall_collisions(ball) # Check collisions with other balls for other in self.balls: if other.number != ball.number: self.check_ball_collision(ball, other) # Draw everything self.draw() def check_wall_collisions(self, ball): for wall in self.walls: # Find closest point on wall segment to ball closest = self.closest_point_on_segment( wall.x1, wall.y1, wall.x2, wall.y2, ball.x, ball.y ) # Calculate distance to wall dx = ball.x - closest[0] dy = ball.y - closest[1] distance = math.sqrt(dx*dx + dy*dy) if distance < BALL_RADIUS: # Collision detected # Calculate normal vector nx = dx / distance ny = dy / distance # Calculate relative velocity along normal v_rel = ball.vx * nx + ball.vy * ny if v_rel < 0: # Moving toward the wall # Calculate impulse j = -(1 + BOUNCE_FACTOR) * v_rel # Apply impulse ball.vx += j * nx ball.vy += j * ny # Add some spin based on collision ball.spin += (ball.vx * ny - ball.vy * nx) * 0.1 # Move ball out of collision penetration = BALL_RADIUS - distance ball.x += penetration * nx ball.y += penetration * ny def check_ball_collision(self, ball1, ball2): dx = ball2.x - ball1.x dy = ball2.y - ball1.y distance = math.sqrt(dx*dx + dy*dy) if distance < 2 * BALL_RADIUS: # Collision detected nx = dx / distance ny = dy / distance # Calculate relative velocity v_rel_x = ball2.vx - ball1.vx v_rel_y = ball2.vy - ball1.vy v_rel = v_rel_x * nx + v_rel_y * ny if v_rel < 0: # Moving toward each other # Calculate impulse j = -(1 + BOUNCE_FACTOR) * v_rel / 2 # Apply impulses ball1.vx -= j * nx ball1.vy -= j * ny ball2.vx += j * nx ball2.vy += j * ny # Add spin based on collision ball1.spin += (ball1.vx * ny - ball1.vy * nx) * 0.05 ball2.spin += (ball2.vx * ny - ball2.vy * nx) * 0.05 # Move balls apart penetration = 2 * BALL_RADIUS - distance ball1.x -= penetration * nx * 0.5 ball1.y -= penetration * ny * 0.5 ball2.x += penetration * nx * 0.5 ball2.y += penetration * ny * 0.5 @staticmethod def closest_point_on_segment(x1, y1, x2, y2, x, y): # Vector from point to segment start dx = x - x1 dy = y - y1 # Segment vector sx = x2 - x1 sy = y2 - y1 # Projection of point onto segment dot = dx * sx + dy * sy len_sq = sx * sx + sy * sy param = dot / len_sq if len_sq != 0 else -1 if param < 0: return x1, y1 elif param > 1: return x2, y2 else: return x1 + param * sx, y1 + param * sy def draw(self): self.canvas.delete("all") # Draw heptagon points = [] for wall in self.walls: points.extend([wall.x1, wall.y1]) self.canvas.create_polygon(points, fill="", outline="black", width=2) # Draw balls for ball in self.balls: # Draw ball self.canvas.create_oval( ball.x - BALL_RADIUS, ball.y - BALL_RADIUS, ball.x + BALL_RADIUS, ball.y + BALL_RADIUS, fill=ball.color, outline="black" ) # Draw number with rotation based on spin angle = ball.spin * 10 # Scale spin for visual effect self.canvas.create_text( ball.x, ball.y, text=str(ball.number), font=("Arial", 12, "bold"), angle=angle ) if __name__ == "__main__": root = tk.Tk() root.title("Bouncing Balls in Spinning Heptagon") app = BouncingBalls(root) root.mainloop() Non Dynamic 2bit Heptagon code[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#non-dynamic-2bit-heptagon-code) Copy import tkinter as tk import math import random from dataclasses import dataclass from typing import List, Tuple, Optional import sys # Constants WIDTH, HEIGHT = 800, 800 HEPTAGON_RADIUS = 300 BALL_RADIUS = 15 GRAVITY = 0.5 FRICTION = 0.999 ELASTICITY = 0.8 ROTATION_SPEED = 2 * math.pi / 5 # 360 degrees per 5 seconds SPIN_DECAY = 0.99 # Colors for the balls BALL_COLORS = [\ "#f8b862", "#f6ad49", "#f39800", "#f08300", "#ec6d51",\ "#ee7948", "#ed6d3d", "#ec6800", "#ec6800", "#ee7800",\ "#eb6238", "#ea5506", "#ea5506", "#eb6101", "#e49e61",\ "#e45e32", "#e17b34", "#dd7a56", "#db8449", "#d66a35"\ ] @dataclass class Ball: x: float y: float vx: float vy: float radius: float color: str number: int spin: float = 0.0 @dataclass class Heptagon: center_x: float center_y: float radius: float angle: float = 0.0 class BouncingBalls: def __init__(self, root): self.root = root self.canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT, bg="white") self.canvas.pack() self.heptagon = Heptagon(WIDTH//2, HEIGHT//2, HEPTAGON_RADIUS) self.balls = [] self.setup_balls() self.root.after(0, self.update) self.root.mainloop() def setup_balls(self): center_x, center_y = WIDTH//2, HEIGHT//2 for i in range(20): self.balls.append(Ball( x=center_x, y=center_y, vx=0, vy=0, radius=BALL_RADIUS, color=BALL_COLORS[i], number=i+1, spin=0 )) def update(self): self.canvas.delete("all") # Update heptagon angle self.heptagon.angle += ROTATION_SPEED / 60 # Assuming 60 FPS # Draw heptagon self.draw_heptagon() # Update and draw balls for ball in self.balls: # Apply gravity ball.vy += GRAVITY # Update position ball.x += ball.vx ball.y += ball.vy # Apply friction ball.vx *= FRICTION ball.vy *= FRICTION # Apply spin decay ball.spin *= SPIN_DECAY # Check collision with heptagon walls self.check_heptagon_collision(ball) # Check collision with other balls for other in self.balls: if other != ball: if self.check_ball_collision(ball, other): self.resolve_ball_collision(ball, other) # Draw the ball self.draw_ball(ball) self.root.after(16, self.update) # ~60 FPS def draw_heptagon(self): center_x, center_y = self.heptagon.center_x, self.heptagon.center_y points = [] for i in range(7): angle = self.heptagon.angle + i * 2 * math.pi / 7 x = center_x + self.heptagon.radius * math.cos(angle) y = center_y + self.heptagon.radius * math.sin(angle) points.append((x, y)) # Draw heptagon self.canvas.create_polygon( [points[0], points[1], points[2], points[3], \ points[4], points[5], points[6]], outline="black", fill="", width=2 ) def draw_ball(self, ball): self.canvas.create_oval( ball.x - ball.radius, ball.y - ball.radius, ball.x + ball.radius, ball.y + ball.radius, fill=ball.color, outline="black" ) # Draw the number self.canvas.create_text( ball.x, ball.y, text=str(ball.number), fill="black" ) def check_heptagon_collision(self, ball): center_x, center_y = WIDTH//2, HEIGHT//2 # Check distance from center dx = ball.x - center_x dy = ball.y - center_y dist = math.sqrt(dx**2 + dy**2) if dist + ball.radius > self.heptagon.radius: # Find the normal vector from center to ball angle = math.atan2(dy, dx) normal_x = math.cos(angle) normal_y = math.sin(angle) # Move ball back inside heptagon overlap = (dist + ball.radius) - self.heptagon.radius ball.x -= overlap * normal_x ball.y -= overlap * normal_y # Reflect velocity dot_product = ball.vx * normal_x + ball.vy * normal_y ball.vx -= 2 * dot_product * normal_x * ELASTICITY ball.vy -= 2 * dot_product * normal_y * ELASTICITY def check_ball_collision(self, ball1, ball2): dx = ball2.x - ball1.x dy = ball2.y - ball1.y distance = math.sqrt(dx**2 + dy**2) return distance < (ball1.radius + ball2.radius) def resolve_ball_collision(self, ball1, ball2): dx = ball2.x - ball1.x dy = ball2.y - ball1.y distance = math.sqrt(dx**2 + dy**2) # Normal vector nx = dx / distance ny = dy / distance # Relative velocity dvx = ball2.vx - ball1.vx dvy = ball2.vy - ball1.vy # Calculate impulse impulse = 2 * (dvx * nx + dvy * ny) / 2 impulse *= ELASTICITY # Apply impulse ball1.vx -= impulse * nx ball1.vy -= impulse * ny ball2.vx += impulse * nx ball2.vy += impulse * ny # Separate the balls to prevent sticking overlap = (ball1.radius + ball2.radius) - distance ball1.x -= overlap * nx / 2 ball1.y -= overlap * ny / 2 ball2.x += overlap * nx / 2 ball2.y += overlap * ny / 2 def run(self): self.root.mainloop() if __name__ == "__main__": root = tk.Tk() root.title("Bouncing Balls in a Spinning Heptagon") app = BouncingBalls(root) app.run() Float8 Heptagon code[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#float8-heptagon-code) Copy import tkinter as tk import math import numpy as np from dataclasses import dataclass from typing import List, Tuple, Optional # Constants WIDTH, HEIGHT = 800, 800 CENTER_X, CENTER_Y = WIDTH // 2, HEIGHT // 2 GRAVITY = 0.2 FRICTION = 0.99 BALL_RADIUS = 15 HEPTAGON_RADIUS = 300 ROTATION_SPEED = 360 / 5 # degrees per second ELASTICITY = 0.8 SPIN_FRICTION = 0.98 # Colors for the balls BALL_COLORS = [\ '#f8b862', '#f6ad49', '#f39800', '#f08300', '#ec6d51',\ '#ee7948', '#ed6d3d', '#ec6800', '#ec6800', '#ee7800',\ '#eb6238', '#ea5506', '#ea5506', '#eb6101', '#e49e61',\ '#e45e32', '#e17b34', '#dd7a56', '#db8449', '#d66a35'\ ] @dataclass class Ball: x: float y: float vx: float vy: float radius: float color: str number: int spin: float = 0.0 def move(self): self.x += self.vx self.y += self.vy self.vy += GRAVITY self.vx *= FRICTION self.vy *= FRICTION self.spin *= SPIN_FRICTION def collide_with_ball(self, other: 'Ball'): dx = other.x - self.x dy = other.y - self.y distance = math.hypot(dx, dy) if distance < self.radius + other.radius: # Calculate collision normal nx = dx / distance ny = dy / distance # Calculate relative velocity dvx = other.vx - self.vx dvy = other.vy - self.vy # Calculate impulse impulse = 2 * (dvx * nx + dvy * ny) / (1/self.radius + 1/other.radius) # Apply impulse self.vx += impulse * nx / self.radius self.vy += impulse * ny / self.radius other.vx -= impulse * nx / other.radius other.vy -= impulse * ny / other.radius # Separate balls to prevent sticking overlap = (self.radius + other.radius - distance) / 2 self.x -= overlap * nx self.y -= overlap * ny other.x += overlap * nx other.y += overlap * ny # Transfer some spin transfer = impulse * 0.01 self.spin -= transfer other.spin += transfer class HeptagonBounceSimulator: def __init__(self, root): self.root = root self.canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT, bg='white') self.canvas.pack() self.balls = self.create_balls() self.heptagon_angle = 0 self.last_time = 0 self.running = True self.root.bind('', self.toggle_pause) self.root.bind('', lambda e: root.destroy()) self.last_time = self.root.after(0, self.update) def create_balls(self) -> List[Ball]: balls = [] for i in range(20): # Start all balls at center with small random velocity angle = np.random.uniform(0, 2 * math.pi) speed = np.random.uniform(0.5, 2) vx = math.cos(angle) * speed vy = math.sin(angle) * speed balls.append(Ball( x=CENTER_X, y=CENTER_Y, vx=vx, vy=vy, radius=BALL_RADIUS, color=BALL_COLORS[i], number=i+1, spin=np.random.uniform(-2, 2) )) return balls def toggle_pause(self, event): self.running = not self.running if self.running: self.last_time = self.root.after(0, self.update) def get_heptagon_vertices(self) -> List[Tuple[float, float]]: vertices = [] for i in range(7): angle = math.radians(self.heptagon_angle + i * 360 / 7) x = CENTER_X + HEPTAGON_RADIUS * math.cos(angle) y = CENTER_Y + HEPTAGON_RADIUS * math.sin(angle) vertices.append((x, y)) return vertices def check_ball_heptagon_collision(self, ball: Ball): vertices = self.get_heptagon_vertices() closest_dist = float('inf') closest_normal = (0, 0) closest_edge = None # Check collision with each edge of the heptagon for i in range(len(vertices)): p1 = vertices[i] p2 = vertices[(i + 1) % len(vertices)] # Vector from p1 to p2 edge_x = p2[0] - p1[0] edge_y = p2[1] - p1[1] edge_length = math.hypot(edge_x, edge_y) # Normalize edge vector edge_x /= edge_length edge_y /= edge_length # Normal vector (perpendicular to edge, pointing inward) nx = -edge_y ny = edge_x # Vector from p1 to ball ball_to_p1_x = ball.x - p1[0] ball_to_p1_y = ball.y - p1[1] # Project ball onto edge normal projection = ball_to_p1_x * nx + ball_to_p1_y * ny # If projection is negative, ball is outside the heptagon if projection < ball.radius: # Find closest point on edge to ball edge_proj = ball_to_p1_x * edge_x + ball_to_p1_y * edge_y edge_proj = max(0, min(edge_length, edge_proj)) closest_x = p1[0] + edge_proj * edge_x closest_y = p1[1] + edge_proj * edge_y # Distance from ball to closest point on edge dist = math.hypot(ball.x - closest_x, ball.y - closest_y) if dist < closest_dist: closest_dist = dist closest_normal = (nx, ny) closest_edge = (p1, p2) if closest_dist < ball.radius: # Calculate bounce response dot_product = ball.vx * closest_normal[0] + ball.vy * closest_normal[1] # Apply bounce with elasticity ball.vx -= (1 + ELASTICITY) * dot_product * closest_normal[0] ball.vy -= (1 + ELASTICITY) * dot_product * closest_normal[1] # Add some spin based on impact edge_vec = (closest_edge[1][0] - closest_edge[0][0], closest_edge[1][1] - closest_edge[0][1]) edge_length = math.hypot(edge_vec[0], edge_vec[1]) if edge_length > 0: edge_vec = (edge_vec[0]/edge_length, edge_vec[1]/edge_length) # Cross product of velocity and edge direction spin_effect = (ball.vx * edge_vec[1] - ball.vy * edge_vec[0]) * 0.1 ball.spin += spin_effect # Move ball outside the heptagon to prevent sticking penetration = ball.radius - closest_dist ball.x += penetration * closest_normal[0] ball.y += penetration * closest_normal[1] def update(self): if not self.running: return # Clear canvas self.canvas.delete('all') # Update heptagon rotation self.heptagon_angle += ROTATION_SPEED / 60 # Assuming ~60 FPS # Draw heptagon vertices = self.get_heptagon_vertices() self.canvas.create_polygon(vertices, outline='black', fill='', width=2) # Update and draw balls for i, ball in enumerate(self.balls): # Move ball ball.move() # Check collisions with heptagon self.check_ball_heptagon_collision(ball) # Draw ball self.canvas.create_oval( ball.x - ball.radius, ball.y - ball.radius, ball.x + ball.radius, ball.y + ball.radius, fill=ball.color, outline='black' ) # Draw number with rotation based on spin angle = ball.spin * 10 # Scale spin for visible rotation self.canvas.create_text( ball.x, ball.y, text=str(ball.number), font=('Arial', 10, 'bold'), angle=angle ) # Check ball-ball collisions for i in range(len(self.balls)): for j in range(i + 1, len(self.balls)): self.balls[i].collide_with_ball(self.balls[j]) # Schedule next update self.last_time = self.root.after(16, self.update) # ~60 FPS if __name__ == '__main__': root = tk.Tk() root.title('Bouncing Balls in a Spinning Heptagon') simulator = HeptagonBounceSimulator(root) root.mainloop() [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally#extra-findings-and-tips) šŸ•µļø Extra Findings & Tips --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. We find using lower KV cache quantization (4bit) seems to degrade generation quality via empirical tests - more tests need to be done, but we suggest using `q8_0` cache quantization. The goal of quantization is to support longer context lengths since the KV cache uses quite a bit of memory. 2. We found the `down_proj` in this model to be extremely sensitive to quantitation. We had to redo some of our dyanmic quants which used 2bits for `down_proj` and now we use 3bits as the minimum for all these matrices. 3. Using `llama.cpp` 's Flash Attention backend does result in somewhat faster decoding speeds. Use `-DGGML_CUDA_FA_ALL_QUANTS=ON` when compiling. Note it's also best to set your CUDA architecture as found in [https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) to reduce compilation times, then set it via `-DCMAKE_CUDA_ARCHITECTURES="80"` 4. Using a `min_p=0.01`is probably enough. `llama.cpp`defaults to 0.1, which is probably not necessary. Since a temperature of 0.3 is used anyways, we most likely will very unlikely sample low probability tokens, so removing very unlikely tokens is a good idea. DeepSeek recommends 0.0 temperature for coding tasks. [PreviousTutorial: How to Finetune Llama-3 and Use In Ollama](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama) [NextDeepSeek-R1: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally) Last updated 1 month ago Was this helpful? --- # Tutorials: How To Fine-tune & Run LLMs | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/basics/deepseek-v3.1) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FOFWy2bZ6L6qr12m9fbEM%252Fdeepseek%2520v3.1%2520logo.png%3Falt%3Dmedia%26token%3Ddd75f159-9266-4208-995f-b71d8e2ed4d3&width=245&dpr=4&quality=100&sign=32847d2c&sv=2) [DeepSeek-V3.1](https://docs.unsloth.ai/basics/deepseek-v3.1) [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdUKxTDoQUFZPpOixP1Cx%252Fsloth%2520with%2520comp.png%3Falt%3Dmedia%26token%3D16fbc4a3-3d03-4e6c-bc74-75cf1121c797&width=245&dpr=4&quality=100&sign=f2f7d004&sv=2) [Fine-tune gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss) [](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FX0pJKFv8zDMf4TJomAts%252Fgpt-oss%2520image.png%3Falt%3Dmedia%26token%3D60c73c0d-cf83-4269-9619-f4b71e25767a&width=245&dpr=4&quality=100&sign=d3cd6098&sv=2) [Run gpt-oss](https://docs.unsloth.ai/basics/gpt-oss-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FeDz30Gy6kQ8zzdMaxr5m%252Fqwen3-coder%25201920.png%3Falt%3Dmedia%26token%3Defad8f53-6d06-48bd-98e6-96bde543702d&width=245&dpr=4&quality=100&sign=45822cb4&sv=2) [Qwen3 Coder](https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally) [](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBszehKqh4ex9879rI5jv%252FGemma%25203%2520text%2520only.png%3Falt%3Dmedia%26token%3Db66212ab-409b-4603-80fa-337bea439531&width=245&dpr=4&quality=100&sign=34b6d65b&sv=2) [Gemma 3n](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEj2zfXu3PPd39PvAmQtx%252Fqwen3-2507.png%3Falt%3Dmedia%26token%3Dc070db7b-bfe9-4a7f-9e75-bbd0b0a01a4d&width=245&dpr=4&quality=100&sign=39aeab4b&sv=2) [**Qwen3-2507**](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507) [](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FNSt3ekVji7Uk7G6PFd1G%252Fdeepseek%2520r1-0528.png%3Falt%3Dmedia%26token%3D9e1472ad-731f-44bf-845d-d4ae89989266&width=245&dpr=4&quality=100&sign=c5a21576&sv=2) [DeepSeek-R1-0528](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) [](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FY0FqiyRvzwRiBOIWEPj6%252Fkimik2%2520landcsape.png%3Falt%3Dmedia%26token%3D35aca81f-684b-4abc-a60b-632055b0aeaa&width=245&dpr=4&quality=100&sign=d1631999&sv=2) [Kimi K2](https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWjXaYZOxk8LMoq1gyVFS%252Fmagistral%2520center.png%3Falt%3Dmedia%26token%3D337b3f36-87f1-4f62-b0b4-f1471e664f34&width=245&dpr=4&quality=100&sign=22f17e82&sv=2) [Magistral-2507](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFp4c2fMEzTezm1B5oEaM%252Fdevstral%2520logo.png%3Falt%3Dmedia%26token%3D59f165fe-0d50-4b1a-88cf-a4617865aaa9&width=245&dpr=4&quality=100&sign=d71ed617&sv=2) [Devstral 2507](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlbVLSdgDVeTdrzqIqWSy%252Fnvidia-logo-white%2520background.png%3Falt%3Dmedia%26token%3D91fec0de-66af-457e-a5eb-16e134bca0e3&width=245&dpr=4&quality=100&sign=dc340667&sv=2) [Fine-tune on Blackwell & RTX 50 GPUs](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) [](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FjnEy1VXc85HX4nCqeAAy%252Ftts%2520finetuning%2520landscape.png%3Falt%3Dmedia%26token%3D24aaf75b-c6ee-4dbb-817d-f9aaa7c9a7ff&width=245&dpr=4&quality=100&sign=6aeef2c6&sv=2) [TTS Fine-tuning](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) [](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fz30qbVABdBlqEnKatTf1%252Fqwen3.png%3Falt%3Dmedia%26token%3Defd4bb30-4926-4272-b15d-91c0a0fc5ac5&width=245&dpr=4&quality=100&sign=c6de3b4f&sv=2) [Qwen3](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FLDayziE4Q7Gc52BMQfd4%252Fphi4%2520reasoning2.png%3Falt%3Dmedia%26token%3Df3db5f93-dde0-49c3-97ed-cbf596d8d437&width=245&dpr=4&quality=100&sign=2afe2d91&sv=2) [Phi-4 reasoning](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdiwpvMM4VA4oZqaANJOE%252Fdynamic%2520v2%2520with%2520unsloth.png%3Falt%3Dmedia%26token%3Dadc64cb6-2b52-4565-a44e-ac4acbd4247d&width=245&dpr=4&quality=100&sign=95dfb159&sv=2) [Dynamic 2.0 GGUFs](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) [](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8RZoiqWL4cXqTFwTAbg8%252Fllama%25204%2520only.png%3Falt%3Dmedia%26token%3Dc6b0dd0e-b817-482b-9b8e-05d017a72319&width=245&dpr=4&quality=100&sign=587751ee&sv=2) [Llama 4](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FuvkQHGJWBVejGmQDLMkz%252Fv30324.png%3Falt%3Dmedia%26token%3D941a8bdd-c5af-4144-9126-fa656335aba2&width=245&dpr=4&quality=100&sign=1305effb&sv=2) [DeepSeek-V3-0324](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally) [](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FML1v35ELOxO0AxBpXWCn%252Fgemma%25203%2520logo.png%3Falt%3Dmedia%26token%3D04fefb63-973d-4b36-a2f6-77414ddf8003&width=245&dpr=4&quality=100&sign=9f0d9b98&sv=2) [Gemma 3](https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhE7P8M1nQaMEkrLiaRj6%252Fqwq%2520logo%2520only.png%3Falt%3Dmedia%26token%3Dc42d1143-dbf8-425e-b1e2-7d9700c02816&width=245&dpr=4&quality=100&sign=5a0edb2&sv=2) [QwQ-32B](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEDGoGKoQdMunfGToescN%252Fdeepseek%2520r1.png%3Falt%3Dmedia%26token%3Df2bafaeb-9cd3-4f9d-8c09-b645e72d7fe7&width=245&dpr=4&quality=100&sign=67995ef5&sv=2) [DeepSeek-R1](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally) [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDYDeJW7oBTYtXBqsVmPA%252Frl%2520guide%2520new.png%3Falt%3Dmedia%26token%3D78d922fe-09d5-4b5f-8ff5-10f573d59234&width=245&dpr=4&quality=100&sign=3107b898&sv=2) [Reinforcement Learning (RL)](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) [](https://www.unsloth.ai/blog/mistral-small-3.1) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fyr9mvoFQqL47zSAE574d%252Fmistral%2520small%25203.1.png%3Falt%3Dmedia%26token%3De882995f-931e-4af2-a086-d0cefbf23635&width=245&dpr=4&quality=100&sign=99c036ba&sv=2) [Mistral Small 3.1](https://www.unsloth.ai/blog/mistral-small-3.1) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FeLYVuPYGC1Giu97E8zWi%252Fllama%25203logo.png%3Falt%3Dmedia%26token%3D2127b873-32cb-4a4a-9593-92a179b46c3b&width=245&dpr=4&quality=100&sign=f48b38a2&sv=2) [Llama 3](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama) [](https://docs.unsloth.ai/basics/vision-fine-tuning) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F5KEw7Kdq4FF1owcZH5GU%252Fllama_3.2_vision_large_rectangle_jPUNULJrVe5O4AvDDWO1M.webp%3Falt%3Dmedia%26token%3Defafc3d6-e763-4e51-83d1-4199fbbf3b53&width=245&dpr=4&quality=100&sign=8eb81810&sv=2) [Vision Fine-tuning](https://docs.unsloth.ai/basics/vision-fine-tuning) [](https://docs.unsloth.ai/basics/continued-pretraining) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FchkkXzhFudLPVKhnXiPR%252Fcontinued_pretraining_just_graph_HC0ALBypfCXyUUXClYPiN.webp%3Falt%3Dmedia%26token%3D61995f90-d6f3-4216-9ddd-0ed5f7342e57&width=245&dpr=4&quality=100&sign=14828b52&sv=2) [Continued Pretraining](https://docs.unsloth.ai/basics/continued-pretraining) [](https://unsloth.ai/blog/llama3-3) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQzD8cVunL79qfLTr3RfN%252Fllama_3.3_website_9hQURhj6KfZ7EnBRaKbiu.webp%3Falt%3Dmedia%26token%3D57ae3812-0dd6-4254-b4d8-8b591be3608c&width=245&dpr=4&quality=100&sign=ceb1a12a&sv=2) [Llama 3.3](https://unsloth.ai/blog/llama3-3) [](https://unsloth.ai/blog/gemma2) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FTMjv4ruy6rjJoAmpEcq2%252Fgemma_2_long_OKsRGiTB8vrcIyXNWdgMw.avif%3Falt%3Dmedia%26token%3Daccf6e7e-0cfa-4484-a671-f9bf93c84cc5&width=245&dpr=4&quality=100&sign=1b7ba816&sv=2) [Gemma 2](https://unsloth.ai/blog/gemma2) [](https://unsloth.ai/blog/phi3) ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrVYkfNhNa1nHacttNFHt%252Fphi3_unsloth_ynBY7FG3NTjIbS11ozN_g.webp%3Falt%3Dmedia%26token%3Dcdac7cdd-0b9b-49a5-93cb-5434874e679d&width=245&dpr=4&quality=100&sign=1937e56&sv=2) [Phi-3](https://unsloth.ai/blog/phi3) [PreviousLlama 4: How to Run & Fine-tune](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune) [NextMagistral: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune) Last updated 5 days ago Was this helpful? --- # DeepSeek-R1: How to Run Locally | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Please see [https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) for an updated DeepSeek R1-0528 (May 28th 2025 version) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally#using-llama.cpp-recommended) Using llama.cpp (recommended) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. Do not forget about `<ļ½œUserļ½œ>` and `<ļ½œAssistantļ½œ>` tokens! - Or use a chat template formatter 2. Obtain the latest `llama.cpp` at: [github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) . You can follow the build instructions below as well: Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. It's best to use `--min-p 0.05` to counteract very rare token predictions - I found this to work well especially for the 1.58bit model. 2. Download the model via: Copy # pip install huggingface_hub hf_transfer # import os # Optional for faster downloading # os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/DeepSeek-R1-GGUF", local_dir = "DeepSeek-R1-GGUF", allow_patterns = ["*UD-IQ1_S*"], # Select quant type UD-IQ1_S for 1.58bit ) 1. Example with Q4\_0 K quantized cache **Notice -no-cnv disables auto conversation mode** Copy ./llama.cpp/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 12 -no-cnv --prio 2 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --prompt "<ļ½œUserļ½œ>What is 1+1?<ļ½œAssistantļ½œ>" Example output: Copy Okay, so I need to figure out what 1 plus 1 is. Hmm, where do I even start? I remember from school that adding numbers is pretty basic, but I want to make sure I understand it properly. Let me think, 1 plus 1. So, I have one item and I add another one. Maybe like a apple plus another apple. If I have one apple and someone gives me another, I now have two apples. So, 1 plus 1 should be 2. That makes sense. Wait, but sometimes math can be tricky. Could it be something else? Like, in a different number system maybe? But I think the question is straightforward, using regular numbers, not like binary or hexadecimal or anything. I also recall that in arithmetic, addition is combining quantities. So, if you have two quantities of 1, combining them gives you a total of 2. Yeah, that seems right. Is there a scenario where 1 plus 1 wouldn't be 2? I can't think of any... 1. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers. Copy ./llama.cpp/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 12 -no-cnv --prio 2 \ --n-gpu-layers 7 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python.<ļ½œAssistantļ½œ>" 1. To test our Flappy Bird example as mentioned in our blog post here: [https://unsloth.ai/blog/deepseekr1-dynamic](https://unsloth.ai/blog/deepseekr1-dynamic) , we can produce the 2nd example like below using our 1.58bit dynamic quant: ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHHUZZTFj0WpgSuWFlibf%252FInShot_20250127_043158375_H8Uu6tyJXYAFwUEIu04Am.gif%3Falt%3Dmedia%26token%3Da959720d-b1b4-4b80-b10d-1c41928dfdcf&width=245&dpr=4&quality=100&sign=f69a2605&sv=2) Original DeepSeek R1 ![Cover](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqgLhnVaN53kV4cvZaDci%252FInShot_20250127_042648160_lrtL8-eRhl4qtLaUDSU87.gif%3Falt%3Dmedia%26token%3De608b30a-1cbe-49ac-b18a-967a50c67c68&width=245&dpr=4&quality=100&sign=a0093029&sv=2) 1.58bit Dynamic Quant The prompt used is as below: Copy <ļ½œUserļ½œ>Create a Flappy Bird game in Python. You must include these things: 1. You must use pygame. 2. The background color should be randomly chosen and is a light shade. Start with a light blue color. 3. Pressing SPACE multiple times will accelerate the bird. 4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color. 5. Place on the bottom some land colored as dark brown or yellow chosen randomly. 6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them. 7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade. 8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again. The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<ļ½œAssistantļ½œ> To call llama.cpp using this example, we do: Copy ./llama.cpp/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 12 -no-cnv --prio 2 \ --n-gpu-layers 7 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<ļ½œAssistantļ½œ>" 1. Also, if you want to merge the weights together for use in Ollama for example, use this script: Copy ./llama.cpp/llama-gguf-split --merge \ DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ merged_file.gguf 1. DeepSeek R1 has 61 layers. For example with a 24GB GPU or 80GB GPU, you can expect to offload after rounding down (reduce by 1 if it goes out of memory): Quant File Size 24GB GPU 80GB GPU 2x80GB GPU 1.58bit 131GB 7 33 All layers 61 1.73bit 158GB 5 26 57 2.22bit 183GB 4 22 49 2.51bit 212GB 2 19 32 ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally#running-on-mac-apple-devices) Running on Mac / Apple devices For Apple Metal devices, be careful of --n-gpu-layers. If you find the machine going out of memory, reduce it. For a 128GB unified memory machine, you should be able to offload 59 layers or so. Copy ./llama.cpp/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 16 \ --prio 2 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --n-gpu-layers 59 \ -no-cnv \ --prompt "<ļ½œUserļ½œ>Create a Flappy Bird game in Python.<ļ½œAssistantļ½œ>" ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally#run-in-ollama-open-webui) Run in Ollama/Open WebUI Open WebUI has made an step-by-step tutorial on how to run R1 here: [docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/](https://docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/) If you want to use Ollama for inference on GGUFs, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally. Copy ./llama.cpp/llama-gguf-split --merge \ DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ merged_file.gguf [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally#deepseek-chat-template) DeepSeek Chat Template ------------------------------------------------------------------------------------------------------------------------------------------------------------ All distilled versions and the main 671B R1 model use the same chat template: `<ļ½œbeginā–ofā–sentenceļ½œ><ļ½œUserļ½œ>What is 1+1?<ļ½œAssistantļ½œ>It's 2.<ļ½œendā–ofā–sentenceļ½œ><ļ½œUserļ½œ>Explain more!<ļ½œAssistantļ½œ>` A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call _tokenizer.encode(..., add\_special\_tokens = False)_ since the chat template auto adds a BOS token as well. For llama.cpp / GGUF inference, you should skip the BOS since itā€™ll auto add it. `<ļ½œUserļ½œ>What is 1+1?<ļ½œAssistantļ½œ>` The and tokens get their own designated tokens. For the distilled versions for Qwen and Llama, some tokens are re-mapped, whilst Qwen for example did not have a BOS token, so <|object\_ref\_start|> had to be used instead. **Tokenizer ID Mappings:** Token R1 Distill Qwen Distill Llama 128798 151648 128013 128799 151649 128014 <|begin\_of\_sentence|> 0 151646 128000 <|end\_of\_sentence|> 1 151643 128001 <|User|> 128803 151644 128011 <|Assistant|> 128804 151645 128012 Padding token 2 151654 128004 Original tokens in models: Token Qwen 2.5 32B Base Llama 3.3 70B Instruct <|box\_start|> <|reserved\_special\_token\_5|> <|box\_end|> <|reserved\_special\_token\_6|> <ļ½œbeginā–ofā–sentenceļ½œ> <|object\_ref\_start|> <|begin\_of\_text|> <ļ½œendā–ofā–sentenceļ½œ> <|endoftext|> <|end\_of\_text|> <ļ½œUserļ½œ> <|im\_start|> <|reserved\_special\_token\_3|> <ļ½œAssistantļ½œ> <|im\_end|> <|reserved\_special\_token\_4|> Padding token <|vision\_pad|> <|finetune\_right\_pad\_id|> All Distilled and the original R1 versions seem to have accidentally assigned the padding token to <ļ½œendā–ofā–sentenceļ½œ>, which is mostly not a good idea, especially if you want to further finetune on top of these reasoning models. This will cause endless infinite generations, since most frameworks will mask the EOS token out as -100. We fixed all distilled and the original R1 versions with the correct padding token (Qwen uses <|vision\_pad|>, Llama uses <|finetune\_right\_pad\_id|>, and R1 uses <ļ½œā–padā–ļ½œ> or our own added <ļ½œPADā–TOKENļ½œ>. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally#gguf-r1-table) GGUF R1 Table ------------------------------------------------------------------------------------------------------------------------------------------ MoE Bits Type Disk Size Accuracy Link Details 1.58bit UD-IQ1\_S **131GB** Fair [Link](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_S) MoE all 1.56bit. `down_proj` in MoE mixture of 2.06/1.56bit 1.73bit UD-IQ1\_M **158GB** Good [Link](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_M) MoE all 1.56bit. `down_proj` in MoE left at 2.06bit 2.22bit UD-IQ2\_XXS **183GB** Better [Link](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ2_XXS) MoE all 2.06bit. `down_proj` in MoE mixture of 2.5/2.06bit 2.51bit UD-Q2\_K\_XL **212GB** Best [Link](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL) MoE all 2.5bit. `down_proj` in MoE mixture of 3.5/2.5bit [PreviousDeepSeek-V3-0324: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally) [NextDeepSeek-R1 Dynamic 1.58-bit](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit) Last updated 1 month ago Was this helpful? --- # QwQ-32B: How to Run effectively | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Qwen released QwQ-32B - a reasoning model with performance comparable to DeepSeek-R1 on many [benchmarks](https://qwenlm.github.io/blog/qwq-32b/) . However, people have been experiencing **infinite generations**, **many repetitions**, token issues and finetuning issues. We hope this guide will help debug and fix most issues! Our model uploads with our bug fixes work great for fine-tuning, vLLM and Transformers. If you're using llama.cpp and engines that use llama.cpp as backend, follow our [instructions here](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#tutorial-how-to-run-qwq-32b) to fix endless generations. **Unsloth QwQ-32B uploads with our bug fixes:** [GGUF](https://huggingface.co/unsloth/QwQ-32B-GGUF) [Dynamic 4-bit](https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit) [BnB 4-bit](https://huggingface.co/unsloth/QwQ-32B-bnb-4bit) [16-bit](https://huggingface.co/unsloth/QwQ-32B) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#official-recommended-settings) āš™ļø Official Recommended Settings ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- According to [Qwen](https://huggingface.co/Qwen/QwQ-32B) , these are the recommended settings for inference: * Temperature of 0.6 * Top\_K of 40 (or 20 to 40) * Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1) * Top\_P of 0.95 * Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers) * Chat template: `<|im_start|>user\nCreate a Flappy Bird game in Python.<|im_end|>\n<|im_start|>assistant\n\n` `llama.cpp` uses `min_p = 0.1`by default, which might cause issues. Force it to 0.0. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#recommended-settings-for-llama.cpp) šŸ‘ Recommended settings for llama.cpp --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We noticed many people use a `Repetition Penalty` greater than 1.0. For example 1.1 to 1.5. This actually interferes with llama.cpp's sampling mechanisms. The goal of a repetition penalty is to penalize repeated generations, but we found this doesn't work as expected. Turning off `Repetition Penalty` also works (ie setting it to 1.0), but we found using it to be useful to penalize endless generations. To use it, we found you must also edit the ordering of samplers in llama.cpp to before applying `Repetition Penalty`, otherwise there will be endless generations. So add this: Copy --samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc" By default, llama.cpp uses this ordering: Copy --samplers "dry;top_k;typ_p;top_p;min_p;xtc;temperature" We reorder essentially temperature and dry, and move min\_p forward. This means we apply samplers in this order: Copy top_k=40 top_p=0.95 min_p=0.0 temperature=0.6 dry typ_p xtc If you still encounter issues, you can increase the`--repeat-penalty 1.0 to 1.2 or 1.3.` Courtesy to [@krist486](https://x.com/krist486/status/1897885598196654180) for bringing llama.cpp sampling directions to my attention. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#dry-repetition-penalty) ā˜€ļø Dry Repetition Penalty --------------------------------------------------------------------------------------------------------------------------------------------------------------- We investigated usage of `dry penalty` as suggested in [https://github.com/ggml-org/llama.cpp/blob/master/examples/main/README.md](https://github.com/ggml-org/llama.cpp/blob/master/examples/main/README.md) using a value of 0.8, but we actually found this to **rather cause syntax issues especially for coding**. If you still encounter issues, you can increase the`dry penalty to 0.8.` Utilizing our swapped sampling ordering can also help if you decide to use `dry penalty`. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#tutorial-how-to-run-qwq-32b-in-ollama) šŸ¦™ Tutorial: How to Run QwQ-32B in Ollama ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature, min\_p etc) in `param` in our Hugging Face upload! Copy ollama run hf.co/unsloth/QwQ-32B-GGUF:Q4_K_M [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#tutorial-how-to-run-qwq-32b-in-llama.cpp) šŸ“– Tutorial: How to Run QwQ-32B in llama.cpp ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp 1. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: [https://huggingface.co/unsloth/QwQ-32B-GGUF](https://huggingface.co/unsloth/QwQ-32B-GGUF) Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/QwQ-32B-GGUF", local_dir = "unsloth-QwQ-32B-GGUF", allow_patterns = ["*Q4_K_M*"], # For Q4_K_M ) 1. Run Unsloth's Flappy Bird test, which will save the output to `Q4_K_M_yes_samplers.txt` 2. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. 3. We use `--repeat-penalty 1.1` and `--dry-multiplier 0.5` which you can adjust. Copy ./llama.cpp/llama-cli \ --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.6 \ --repeat-penalty 1.1 \ --dry-multiplier 0.5 \ --min-p 0.01 \ --top-k 40 \ --top-p 0.95 \ -no-cnv \ --samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc" \ --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n\n" \ 2>&1 | tee Q4_K_M_yes_samplers.txt The full input from our [https://unsloth.ai/blog/deepseekr1-dynamic](https://unsloth.ai/blog/deepseekr1-dynamic) 1.58bit blog is: Copy <|im_start|>user Create a Flappy Bird game in Python. You must include these things: 1. You must use pygame. 2. The background color should be randomly chosen and is a light shade. Start with a light blue color. 3. Pressing SPACE multiple times will accelerate the bird. 4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color. 5. Place on the bottom some land colored as dark brown or yellow chosen randomly. 6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them. 7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade. 8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again. The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|> <|im_start|>assistant The beginning and the end of the final Python output after removing the thinking parts: Copy import pygame import random import sys pygame.init() ### Continues class Bird: def __init__(self): ### Continues def main(): best_score = 0 current_score = 0 game_over = False pipes = [] first_time = True # Track first game play # Initial setup background_color = (173, 216, 230) # Light blue initially land_color = random.choice(land_colors) bird = Bird() while True: for event in pygame.event.get(): ### Continues if not game_over: # Update bird and pipes bird.update() ### Continues # Drawing ### Continues pygame.display.flip() clock.tick(60) if __name__ == "__main__": main() Full final Python output (removed thinking parts):[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#full-final-python-output-removed-thinking-parts) Copy import pygame import random import sys pygame.init() WIDTH, HEIGHT = 800, 600 screen = pygame.display.set_mode((WIDTH, HEIGHT)) pygame.display.set_caption("Flappy Bird Clone") clock = pygame.time.Clock() # Colors pipe_colors = [(0, 100, 0), (210, 180, 140), (50, 50, 50)] land_colors = [(139, 69, 19), (255, 255, 0)] # Game constants GRAVITY = 0.5 PIPE_SPEED = 5 BIRD_SIZE = 30 LAND_HEIGHT = 50 PIPE_WIDTH = 50 PIPE_GAP = 150 class Bird: def __init__(self): self.x = WIDTH // 2 self.y = HEIGHT // 2 self.velocity = 0 self.shape = random.choice(['square', 'circle', 'triangle']) self.color = (random.randint(0, 100), random.randint(0, 100), random.randint(0, 100)) self.rect = pygame.Rect(self.x - BIRD_SIZE//2, self.y - BIRD_SIZE//2, BIRD_SIZE, BIRD_SIZE) def update(self): self.velocity += GRAVITY self.y += self.velocity self.rect.y = self.y - BIRD_SIZE//2 self.rect.x = self.x - BIRD_SIZE//2 # Keep x centered def draw(self): if self.shape == 'square': pygame.draw.rect(screen, self.color, self.rect) elif self.shape == 'circle': pygame.draw.circle(screen, self.color, (self.rect.centerx, self.rect.centery), BIRD_SIZE//2) elif self.shape == 'triangle': points = [\ (self.rect.centerx, self.rect.top),\ (self.rect.left, self.rect.bottom),\ (self.rect.right, self.rect.bottom)\ ] pygame.draw.polygon(screen, self.color, points) def spawn_pipe(): pipe_x = WIDTH top_height = random.randint(50, HEIGHT - PIPE_GAP - LAND_HEIGHT) rect_top = pygame.Rect(pipe_x, 0, PIPE_WIDTH, top_height) bottom_y = top_height + PIPE_GAP bottom_height = (HEIGHT - LAND_HEIGHT) - bottom_y rect_bottom = pygame.Rect(pipe_x, bottom_y, PIPE_WIDTH, bottom_height) color = random.choice(pipe_colors) return { 'rect_top': rect_top, 'rect_bottom': rect_bottom, 'color': color, 'scored': False } def main(): best_score = 0 current_score = 0 game_over = False pipes = [] first_time = True # Track first game play # Initial setup background_color = (173, 216, 230) # Light blue initially land_color = random.choice(land_colors) bird = Bird() while True: for event in pygame.event.get(): if event.type == pygame.QUIT: pygame.quit() sys.exit() if event.type == pygame.KEYDOWN: if event.key == pygame.K_ESCAPE or event.key == pygame.K_q: pygame.quit() sys.exit() if event.key == pygame.K_SPACE: if game_over: # Reset the game bird = Bird() pipes.clear() current_score = 0 if first_time: # First restart after initial game over background_color = (random.randint(200, 255), random.randint(200, 255), random.randint(200, 255)) first_time = False else: background_color = (random.randint(200, 255), random.randint(200, 255), random.randint(200, 255)) land_color = random.choice(land_colors) game_over = False else: # Jump the bird bird.velocity = -15 # Initial upward velocity if not game_over: # Update bird and pipes bird.update() # Move pipes left remove_pipes = [] for pipe in pipes: pipe['rect_top'].x -= PIPE_SPEED pipe['rect_bottom'].x -= PIPE_SPEED # Check if bird passed the pipe if not pipe['scored'] and bird.rect.x > pipe['rect_top'].right: current_score += 1 pipe['scored'] = True # Check if pipe is offscreen if pipe['rect_top'].right < 0: remove_pipes.append(pipe) # Remove offscreen pipes for p in remove_pipes: pipes.remove(p) # Spawn new pipe if needed if not pipes or pipes[-1]['rect_top'].x < WIDTH - 200: pipes.append(spawn_pipe()) # Check collisions land_rect = pygame.Rect(0, HEIGHT - LAND_HEIGHT, WIDTH, LAND_HEIGHT) bird_rect = bird.rect # Check pipes for pipe in pipes: if bird_rect.colliderect(pipe['rect_top']) or bird_rect.colliderect(pipe['rect_bottom']): game_over = True break # Check land and top if bird_rect.bottom >= land_rect.top or bird_rect.top <= 0: game_over = True if game_over: if current_score > best_score: best_score = current_score # Drawing screen.fill(background_color) # Draw pipes for pipe in pipes: pygame.draw.rect(screen, pipe['color'], pipe['rect_top']) pygame.draw.rect(screen, pipe['color'], pipe['rect_bottom']) # Draw land pygame.draw.rect(screen, land_color, (0, HEIGHT - LAND_HEIGHT, WIDTH, LAND_HEIGHT)) # Draw bird bird.draw() # Draw score font = pygame.font.SysFont(None, 36) score_text = font.render(f'Score: {current_score}', True, (0, 0, 0)) screen.blit(score_text, (WIDTH - 150, 10)) # Game over screen if game_over: over_text = font.render('Game Over!', True, (255, 0, 0)) best_text = font.render(f'Best: {best_score}', True, (255, 0, 0)) restart_text = font.render('Press SPACE to restart', True, (255, 0, 0)) screen.blit(over_text, (WIDTH//2 - 70, HEIGHT//2 - 30)) screen.blit(best_text, (WIDTH//2 - 50, HEIGHT//2 + 10)) screen.blit(restart_text, (WIDTH//2 - 100, HEIGHT//2 + 50)) pygame.display.flip() clock.tick(60) if __name__ == "__main__": main() 1. When running it, we get a runnable game! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7qQoA6yrMWUVrwIhLbGu%252Fimage.png%3Falt%3Dmedia%26token%3D6d99c8ce-567a-4144-bd7e-fa57e96b5284&width=768&dpr=4&quality=100&sign=911446a1&sv=2) 1. Now try the same without our fixes! So remove `--samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc"` This will save the output to `Q4_K_M_no_samplers.txt` Copy ./llama.cpp/llama-cli \ --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.6 \ --repeat-penalty 1.1 \ --dry-multiplier 0.5 \ --min-p 0.01 \ --top-k 40 \ --top-p 0.95 \ -no-cnv \ --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n\n" \ 2>&1 | tee Q4_K_M_no_samplers.txt You will get some looping, but **problematically incorrect Python syntax** and many other issues. For example the below looks correct, but is wrong! Ie line 39 `pipes.clear() ### <<< NameError: name 'pipes' is not defined. Did you forget to import 'pipes'?` Copy import pygame import random pygame.init() # Constants WIDTH, HEIGHT = 800, 600 GROUND_HEIGHT = 20 GRAVITY = 0.7 PIPE_SPEED = -3 BIRD_SIZE = 45 MIN_GAP = 130 MAX_GAP = 200 PIPE_COLORS = [(0, 96, 0), (205, 133, 63), (89, 97, 107)] DARK_BROWN = (94, 72, 4) YELLOW = (252, 228, 6) screen = pygame.display.set_mode((WIDTH, HEIGHT)) clock = pygame.time.Clock() def random_light_color(): return ( random.randint(180, 230), random.randint(190, 300), random.randint(250, 255) ) def reset_game(): global bird_x, bird_y global pipes, score global background_color, land_color global bird_shape, bird_color # Bird properties bird_x = WIDTH * 0.3 bird_y = HEIGHT // 2 bird_vel = -5 # Initial upward thrust pipes.clear() ### <<< NameError: name 'pipes' is not defined. Did you forget to import 'pipes'? 1. If you use `--repeat-penalty 1.5`, it gets even worse and more obvious, with actually totally incorrect syntax. Copy import pygame from random import randint # For generating colors/shapes/positions randomly pygame.init() # Constants: WIDTH, HEIGHT =456 ,702 # BACKGROUND_COLOR_LIGHTS=['lightskyblue'] GAP_SIZE=189 # BIRD_RADIUS=3. PIPE_SPEED=- ( ) ? class Game(): def __init__(self): self.screen_size=( ) def reset_game_vars(): global current_scor e # set to zero and other initial states. # Main game loop: while running : for event in pygame.event.get() : if quit ... etc pygame.quit() print("Code is simplified. Due time constraints, full working version requires further implementation.") 1. You might be wondering maybe it's Q4\_K\_M? B16 ie full precision should work fine right? Incorrect - the outputs again fail if we do not use our fix of -`-samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc"` when using a Repetition Penalty. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#still-doesnt-work-try-min_p-0.1-temperature-1.5) šŸŒ„ Still doesn't work? Try Min\_p = 0.1, Temperature = 1.5 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- According to the Min\_p paper [https://arxiv.org/pdf/2407.01082](https://arxiv.org/pdf/2407.01082) , for more creative and diverse outputs, and if you still see repetitions, try disabling top\_p and top\_k! Copy ./llama.cpp/llama-cli --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \ --threads 32 --n-gpu-layers 99 \ --ctx-size 16384 \ --temp 1.5 \ --min-p 0.1 \ --top-k 0 \ --top-p 1.0 \ -no-cnv \ --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n\n" Another approach is to disable `min_p` directly, since llama.cpp by default uses `min_p = 0.1`! Copy ./llama.cpp/llama-cli --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \ --threads 32 --n-gpu-layers 99 \ --ctx-size 16384 \ --temp 0.6 \ --min-p 0.0 \ --top-k 40 \ --top-p 0.95 \ -no-cnv \ --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n\n" [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#less-than-think-greater-than-token-not-shown) šŸ¤” token not shown? --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Some people are reporting that because is default added in the chat template, some systems are not outputting the thinking traces correctly. You will have to manually edit the Jinja template from: Copy {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages[0]['role'] == 'system' %} {{- messages[0]['content'] }} {%- else %} {{- '' }} {%- endif %} {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} {%- for tool in tools %} {{- "\n" }} {{- tool | tojson }} {%- endfor %} {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} {%- else %} {%- if messages[0]['role'] == 'system' %} {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- for message in messages %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" and not message.tool_calls %} {%- set content = message.content.split('')[-1].lstrip('\n') %} {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" %} {%- set content = message.content.split('')[-1].lstrip('\n') %} {{- '<|im_start|>' + message.role }} {%- if message.content %} {{- '\n' + content }} {%- endif %} {%- for tool_call in message.tool_calls %} {%- if tool_call.function is defined %} {%- set tool_call = tool_call.function %} {%- endif %} {{- '\n\n{"name": "' }} {{- tool_call.name }} {{- '", "arguments": ' }} {{- tool_call.arguments | tojson }} {{- '}\n' }} {%- endfor %} {{- '<|im_end|>\n' }} {%- elif message.role == "tool" %} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n\n' }} {{- message.content }} {{- '\n' }} {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|im_start|>assistant\n\n' }} {%- endif %} to another by removing the `\n` at the end. The model will now have to manually add `\n` during inference, which might not always succeed. DeepSeek also edited all models to default add a `` token to force the model to go into reasoning model. So change `{%- if add_generation_prompt %} {{- '<|im_start|>assistant\n\n' }} {%- endif %}` to `{%- if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {%- endif %}` ie remove `\n` Full jinja template with removed \\n part[](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#full-jinja-template-with-removed-less-than-think-greater-than-n-part) Copy {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages[0]['role'] == 'system' %} {{- messages[0]['content'] }} {%- else %} {{- '' }} {%- endif %} {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} {%- for tool in tools %} {{- "\n" }} {{- tool | tojson }} {%- endfor %} {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} {%- else %} {%- if messages[0]['role'] == 'system' %} {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- for message in messages %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" and not message.tool_calls %} {%- set content = message.content.split('')[-1].lstrip('\n') %} {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" %} {%- set content = message.content.split('')[-1].lstrip('\n') %} {{- '<|im_start|>' + message.role }} {%- if message.content %} {{- '\n' + content }} {%- endif %} {%- for tool_call in message.tool_calls %} {%- if tool_call.function is defined %} {%- set tool_call = tool_call.function %} {%- endif %} {{- '\n\n{"name": "' }} {{- tool_call.name }} {{- '", "arguments": ' }} {{- tool_call.arguments | tojson }} {{- '}\n' }} {%- endfor %} {{- '<|im_end|>\n' }} {%- elif message.role == "tool" %} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n\n' }} {{- message.content }} {{- '\n' }} {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {%- endif %} [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#extra-notes) Extra Notes -------------------------------------------------------------------------------------------------------------------------------------- We first thought maybe: 1. QwQ's context length was not natively 128K, but rather 32K with YaRN extension. For example in the readme file for [https://huggingface.co/Qwen/QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) , we see: Copy { ..., "rope_scaling": { "factor": 4.0, "original_max_position_embeddings": 32768, "type": "yarn" } } We tried overriding llama.cpp's YaRN handling, but nothing changed. Copy --override-kv qwen2.context_length=int:131072 \ --override-kv qwen2.rope.scaling.type=str:yarn \ --override-kv qwen2.rope.scaling.factor=float:4 \ --override-kv qwen2.rope.scaling.original_context_length=int:32768 \ --override-kv qwen2.rope.scaling.attn_factor=float:1.13862943649292 \ 1. We also thought maybe the RMS Layernorm epsilon was wrong - not 1e-5 but maybe 1e-6. For example [this](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct/blob/main/config.json) has `rms_norm_eps=1e-06`, whilst [this](https://huggingface.co/Qwen/Qwen2.5-32B/blob/main/config.json) has `rms_norm_eps=1e-05` . We also overrided it, but it did not work: Copy --override-kv qwen2.attention.layer_norm_rms_epsilon=float:0.000001 \ 1. We also tested if tokenizer IDs matched between llama.cpp and normal Transformers courtesy of [@kalomaze](https://x.com/kalomaze/status/1897875332230779138) . They matched, so this was not the culprit. We provide our experimental results below: [61KB\ \ file\_BF16\_no\_samplers.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FeABgnEXerhmNw1jzUmrr%2Ffile_BF16_no_samplers.txt?alt=media&token=d11aa8f8-0ff7-4370-9412-6129bd980a42) BF16 full precision with no sampling fix [55KB\ \ file\_BF16\_yes\_samplers.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fv01qqEwj6PHVE9VvPzfg%2Ffile_BF16_yes_samplers.txt?alt=media&token=d8ecf5bf-b4f2-4abe-a0b4-26d7e8e862f9) BF16 full precision with sampling fix [71KB\ \ final\_Q4\_K\_M\_no\_samplers.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fi3eSz0NWvc44CkRUanrY%2Ffinal_Q4_K_M_no_samplers.txt?alt=media&token=deca70bd-fc21-44a9-b42c-87837ac3a8ce) Q4\_K\_M precision with no sampling fix [65KB\ \ final\_Q4\_K\_M\_yes\_samplers.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBtdJmKQjMZVlpO1HfWE7%2Ffinal_Q4_K_M_yes_samplers.txt?alt=media&token=f266d668-71ab-436d-8c05-b720e56e348e) Q4\_K\_M precision with sampling fix [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#tokenizer-bug-fixes) āœļø Tokenizer Bug Fixes --------------------------------------------------------------------------------------------------------------------------------------------------------- * We found a few issues as well specifically impacting finetuning! The EOS token is correct, but the PAD token should probably rather be `"<|vision_pad|>`" We updated it in: [https://huggingface.co/unsloth/QwQ-32B/blob/main/tokenizer\_config.json](https://huggingface.co/unsloth/QwQ-32B/blob/main/tokenizer_config.json) Copy "eos_token": "<|im_end|>", "pad_token": "<|endoftext|>", [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively#dynamic-4-bit-quants) šŸ› ļø Dynamic 4-bit Quants ------------------------------------------------------------------------------------------------------------------------------------------------------------ We also uploaded dynamic 4bit quants which increase accuracy vs naive 4bit quantizations! We attach the QwQ quantization error plot analysis for both activation and weight quantization errors: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F32wjrIWeUEQTMq9PhmbS%252FQwQ%2520quantization%2520errors.png%3Falt%3Dmedia%26token%3D0733fd33-9fe9-4aad-812c-75dbad00373f&width=768&dpr=4&quality=100&sign=aafe447c&sv=2) We uploaded dynamic 4-bit quants to: [https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit](https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit) Since vLLM 0.7.3 (2025 February 20th) [https://github.com/vllm-project/vllm/releases/tag/v0.7.3](https://github.com/vllm-project/vllm/releases/tag/v0.7.3) , vLLM now supports loading Unsloth dynamic 4bit quants! All our GGUFs are at [https://huggingface.co/unsloth/QwQ-32B-GGUF](https://huggingface.co/unsloth/QwQ-32B-GGUF) ! [PreviousDeepSeek-R1 Dynamic 1.58-bit](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit) [NextPhi-4 Reasoning: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune) Last updated 1 month ago Was this helpful? --- # DeepSeek-R1 Dynamic 1.58-bit | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Read our full DeepSeek-R1 blogpost here: [unsloth.ai/blog/deepseekr1-dynamic](https://unsloth.ai/blog/deepseekr1-dynamic) ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#id-1-bit-small-dynamic-vs.-basic) 1-bit (Small) - Dynamic vs. Basic GGUF Type Quant Size (GB) Seed Pygame Background Accelerate SPACE Bird shape Land Top right score Pipes Best Score Quit Runnable Score Avg Score Errors Notes Dynamic IQ1\_S 131 3407 1 0.5 1 0.5 0.5 1 0.5 1 1 0 7 score =!inc SyntaxError: invalid syntax Selects random shapes and colors at the start, but doesn't rotate across trials Dynamic IQ1\_S 131 3408 1 1 0.25 1 0.5 1 0.5 1 1 0 7.25 score =B4 NameError: name 'B4' is not defined Better - selects pipe colors randomnly, but all are just 1 color - should be different. Dropping to ground fails to reset acceleration. Dynamic IQ1\_S 131 3409 1 0.5 0.5 0.5 0 1 1 1 1 0 6.5 6.92 score =3D 0 SyntaxError: invalid decimal literal Too hard to play - acceleration too fast. Pipe colors now are random, but bird shape not changing. Land collison fails. Basic IQ1\_S 133 3407 0 0 0 0 0 0 0 0 0 0 0 No code Fully failed. Repeats "with Dark Colurs" forever Basic IQ1\_S 133 3408 0 0 0 0 0 0 0 0 0 0 0 No code Fully failed. Repeats "Pygame's" forever Basic IQ1\_S 133 3409 0 0 0 0 0 0 0 0 0 0 0 0 No code Fully failed. Repeats "pipe\_x = screen\_height pipe\_x = screen\_height pipe\_height = screen\_height - Pipe\_height" forever. ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#id-1-bit-medium-dynamic-vs.-basic) 1-bit (Medium) - Dynamic vs. Basic GGUF Type Quant Size (GB) Seed Pygame Background Accelerate SPACE Bird shape Land Top right score Pipes Best Score Quit Runnable Score Avg Score Errors Notes Dynamic IQ1\_M 158 3407 1 1 0.75 1 1 1 1 1 1 1 9.75 None A bit fast and hard to play. Dynamic IQ1\_M 158 3408 1 1 0.5 1 1 1 1 1 1 1 9.5 None Very good - land should be clearer. Acceleration should be slower. Dynamic IQ1\_M 158 3409 1 0.5 1 0.5 0.5 1 0.5 1 1 1 8 9.08 None Background color does not change across trials.Pipes do not touch the top. No land is seen. Basic IQ1\_M 149 3407 1 0 0 0 0 0 0 0 1 0 2 if game\_over: NameError: name 'game\_over' is not defined Fully failed. Black screen only Basic IQ1\_M 149 3408 1 0 0 0 0 0 0 0 1 0 2 No code Fully failed. Black screen then closes. Basic IQ1\_M 149 3409 1 0 0 0 0 0 0 0 0 0 1 1.67 window.fill((100, 100, 255)) Light Blue SyntaxError: invalid syntax && main() NameError: name 'main' is not defined. Fully failed. ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#id-2-bit-extra-extra-small-dynamic-vs.-basic) 2-bit (Extra extra Small) - Dynamic vs. Basic GGUF Type Quant Size (GB) Seed Pygame Background Accelerate SPACE Bird shape Land Top right score Pipes Best Score Quit Runnable Score Avg Score Errors Notes Dynamic IQ2\_XXS 183 3407 1 1 0.5 1 1 1 1 1 1 1 9.5 None Too hard to play - acceleration too slow. Lags Dynamic IQ2\_XXS 183 3408 1 1 1 1 1 1 0.5 0.5 1 0 8 global best\_score SyntaxError: name 'best\_score' is assigned to before global declaration Had to edit 2 lines - remove global best\_score, and set pipe\_list = \[\] Dynamic IQ2\_XXS 183 3409 1 1 1 1 1 1 1 1 1 1 10 9.17 None Extremely good. Even makes pipes have random distances between them. Basic IQ2\_XXS 175 3407 1 0.5 0.5 0.5 1 0 0.5 1 0 0 5 pipe\_color = random.choice(\[(34, 139, 34), (139, 69, 19), (47, 47, 47)) SyntaxError: closing parenthesis ')' does not match opening parenthesis '\[' && pygame.draw.polygon(screen, bird\_color, points) ValueError: points argument must contain more than 2 points\ \ Fails quiting. Same color. Collison detection a bit off. No score\ \ Basic\ \ IQ2\_XXS\ \ 175\ \ 3408\ \ 1\ \ 0.5\ \ 0.5\ \ 0.5\ \ 1\ \ 1\ \ 0.5\ \ 1\ \ 0\ \ 0\ \ 6\ \ pipes.append({'x': SCREEN\_WIDTH, 'gap\_y': random.randint(50, SCREEN\_HEIGHT - 150)) SyntaxError: closing parenthesis ')' does not match opening parenthesis '{'\ \ Acceleration weird. Chooses 1 color per round. Cannot quit.\ \ Basic\ \ IQ2\_XXS\ \ 175\ \ 3409\ \ 1\ \ 1\ \ 1\ \ 1\ \ 1\ \ 1\ \ 1\ \ 0\ \ 0.5\ \ 0\ \ 7.5\ \ 6.17\ \ screen = pygame.display.set\_mode((SCREEN\_WIDTH, SCREENHEIGHT)) NameError: name 'SCREENHEIGHT' is not defined. Did you mean: 'SCREEN\_HEIGHT'?\ \ OK. Colors change. Best score does not update. Quit only ESC not Q.\ \ ### \ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#dynamic-quantization-trial-output)\ \ **Dynamic Quantization trial output**\ \ IQ1\_S code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq1_s-code)\ \ IQ1\_M code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq1_m-code)\ \ IQ2\_XXS code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq2_xxs-code)\ \ [12KB\ \ inference\_UD-IQ1\_S\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqpBdpW55h5mNAzVoTxPI%2Finference_UD-IQ1_S_3407.txt?alt=media&token=37b19689-73e5-46d0-98be-352e515dfdf8)\ \ [11KB\ \ inference\_UD-IQ1\_S\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTdIrJSqc2VbNJy1bf3w5%2Finference_UD-IQ1_S_3408.txt?alt=media&token=e11f73bb-80be-49e5-91e2-f3a1f5495dcd)\ \ [10KB\ \ inference\_UD-IQ1\_S\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBk2ZwEIcLmvZQ3jlMLzw%2Finference_UD-IQ1_S_3409.txt?alt=media&token=052885f5-bee9-420d-a9c0-827412ac17c8)\ \ [10KB\ \ inference\_UD-IQ1\_M\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Ft7YmT1H3Nflcy5kAp1LE%2Finference_UD-IQ1_M_3407.txt?alt=media&token=6f62f911-3364-4f92-b311-c1fa9b759370)\ \ [30KB\ \ inference\_UD-IQ1\_M\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FH6BCTeWlJpUkfeEmeqpu%2Finference_UD-IQ1_M_3408.txt?alt=media&token=7727a999-8c0a-4baf-8542-be8686a01630)\ \ [9KB\ \ inference\_UD-IQ1\_M\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvVJI0H2F9KTNj5kwUCtC%2Finference_UD-IQ1_M_3409.txt?alt=media&token=0f863d41-53d6-4c94-8d57-bf1eeb79ead5)\ \ [29KB\ \ inference\_UD-IQ2\_XXS\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F26jxRY5mWuon67OfvGtq%2Finference_UD-IQ2_XXS_3407.txt?alt=media&token=daf9bf7d-245e-4b54-b0c0-a6273833835a)\ \ [34KB\ \ inference\_UD-IQ2\_XXS\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEhjjYN7vAh7gbmR8oXbS%2Finference_UD-IQ2_XXS_3408.txt?alt=media&token=4b50d6dd-2798-44c7-aa92-7e67c09868a4)\ \ [42KB\ \ inference\_UD-IQ2\_XXS\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXwCSfIf16nTwHzcWepoV%2Finference_UD-IQ2_XXS_3409.txt?alt=media&token=2f7539c9-026d-41e7-b7c7-5738a89ae5d4)\ \ ### \ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#non-dynamic-quantization-trial-output)\ \ Non Dynamic Quantization trial output\ \ IQ1\_S basic code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq1_s-basic-code)\ \ IQ1\_M basic code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq1_m-basic-code)\ \ IQ2\_XXS basic code\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit#tab-iq2_xxs-basic-code)\ \ [25KB\ \ inference\_basic-IQ1\_S\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFtAMzAucSfKMkkmXItTj%2Finference_basic-IQ1_S_3407.txt?alt=media&token=76bfcf47-e1ce-442b-af49-6bfb6af7d046)\ \ [15KB\ \ inference\_basic-IQ1\_S\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F4NhjCVFMwCwT2OCj0IJ5%2Finference_basic-IQ1_S_3408.txt?alt=media&token=d4715674-3347-400b-9eb6-ae5d4470feeb)\ \ [14KB\ \ inference\_basic-IQ1\_S\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fb0ZW3xs7R7IMryO7n7Yp%2Finference_basic-IQ1_S_3409.txt?alt=media&token=64b8825b-7103-4708-9d12-12770e43b546)\ \ [7KB\ \ inference\_basic-IQ1\_M\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmZ2TsQEzoGjhGlqUjtmj%2Finference_basic-IQ1_M_3407.txt?alt=media&token=975a30d6-2d90-47eb-9d68-b50fd47337f7)\ \ [7KB\ \ inference\_basic-IQ1\_M\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FIx9TQ99Qpmk7BViNLFBl%2Finference_basic-IQ1_M_3408.txt?alt=media&token=b88e1e5b-4535-4d93-bd67-f81def7377d5)\ \ [12KB\ \ inference\_basic-IQ1\_M\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FDX7XYpJPxXKAMZeGhSrr%2Finference_basic-IQ1_M_3409.txt?alt=media&token=6da9127e-272b-4e74-b990-6657e25eea6b)\ \ [25KB\ \ inference\_basic-IQ2\_XXS\_3407.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FajsVHsVqlWpwHk7mY32t%2Finference_basic-IQ2_XXS_3407.txt?alt=media&token=cbbf36a2-0d6a-4a87-8232-45b0b7fcc588)\ \ [34KB\ \ inference\_basic-IQ2\_XXS\_3408.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F4vjncPu2r2D7F5jVOC7I%2Finference_basic-IQ2_XXS_3408.txt?alt=media&token=9ed635a2-bf97-4f49-b26f-6e985d0ab1b7)\ \ [34KB\ \ inference\_basic-IQ2\_XXS\_3409.txt](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJmVOFgrRyXjY4lYZXE96%2Finference_basic-IQ2_XXS_3409.txt?alt=media&token=faad5bff-ba7f-41f1-abd5-7896f17a5b25)\ \ [PreviousDeepSeek-R1: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally)\ [NextQwQ-32B: How to Run effectively](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively)\ \ Last updated 7 months ago\ \ Was this helpful? --- # Datasets Guide | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/basics/datasets-guide#what-is-a-dataset) What is a Dataset? ------------------------------------------------------------------------------------------- For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized. You'll also learn how to [use datasets inside of Unsloth](https://docs.unsloth.ai/basics/datasets-guide#applying-chat-templates-with-unsloth) . One of the key parts of creating a dataset is your [chat template](https://docs.unsloth.ai/basics/chat-templates) and how you are going to design it. Tokenization is also important as it breaks text into tokens, which can be words, sub-words, or characters so LLMs can process it effectively. These tokens are then turned into embeddings and are adjusted to help the model understand the meaning and context. ### [](https://docs.unsloth.ai/basics/datasets-guide#data-format) Data Format To enable the process of tokenization, datasets need to be in a format that can be read by a tokenizer. Format Description Training Type Raw Corpus Raw text from a source such as a website, book, or article. Continued Pretraining (CPT) Instruct Instructions for the model to follow and an example of the output to aim for. Supervised fine-tuning (SFT) Conversation Multiple-turn conversation between a user and an AI assistant. Supervised fine-tuning (SFT) RLHF Conversation between a user and an AI assistant, with the assistant's responses being ranked by a script, another model or human evaluator. Reinforcement Learning (RL) It's worth noting that different styles of format exist for each of these types. [](https://docs.unsloth.ai/basics/datasets-guide#getting-started) Getting Started -------------------------------------------------------------------------------------- Before we format our data, we want to identify the following: 1 Purpose of dataset Knowing the purpose of the dataset will help us determine what data we need and format to use. The purpose could be, adapting a model to a new task such as summarization or improving a model's ability to role-play a specific character. For example: * Chat-based dialogues (Q&A, learn a new language, customer support, conversations). * Structured tasks ([classification](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) , summarization, generation tasks). * Domain-specific data (medical, finance, technical). 2 Style of output The style of output will let us know what sources of data we will use to reach our desired output. For example, the type of output you want to achieve could be JSON, HTML, text or code. Or perhaps you want it to be Spanish, English or German etc. 3 Data source When we know the purpose and style of the data we need, we need to analyze the quality and [quantity](https://docs.unsloth.ai/basics/datasets-guide#how-big-should-my-dataset-be) of the data. Hugging Face and Wikipedia are great sources of datasets and Wikipedia is especially useful if you are looking to train a model to learn a language. The Source of data can be a CSV file, PDF or even a website. You can also [synthetically generate](https://docs.unsloth.ai/basics/datasets-guide#synthetic-data-generation) data but extra care is required to make sure each example is high quality and relevant. One of the best ways to create a better dataset is by combining it with a more generalized dataset from Hugging Face like ShareGPT to make your model smarter and diverse. You could also add [synthetically generated data](https://docs.unsloth.ai/basics/datasets-guide#synthetic-data-generation) . [](https://docs.unsloth.ai/basics/datasets-guide#formatting-the-data) Formatting the Data ---------------------------------------------------------------------------------------------- When we have identified the relevant criteria, and collected the necessary data, we can then format our data into a machine readable format that is ready for training. ### [](https://docs.unsloth.ai/basics/datasets-guide#common-data-formats-for-llm-training) Common Data Formats for LLM Training For [**continued pretraining**](https://docs.unsloth.ai/basics/continued-pretraining) , we use raw text format without specific structure: Copy "text": "Pasta carbonara is a traditional Roman pasta dish. The sauce is made by mixing raw eggs with grated Pecorino Romano cheese and black pepper. The hot pasta is then tossed with crispy guanciale (cured pork cheek) and the egg mixture, creating a creamy sauce from the residual heat. Despite popular belief, authentic carbonara never contains cream or garlic. The dish likely originated in Rome in the mid-20th century, though its exact origins are debated..." This format preserves natural language flow and allows the model to learn from continuous text. If we are adapting a model to a new task, and intend for the model to output text in a single turn based on a specific set of instructions, we can use **Instruction** format in [Alpaca style](https://docs.unsloth.ai/basics/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset) Copy "Instruction": "Task we want the model to perform." "Input": "Optional, but useful, it will essentially be the user's query." "Output": "The expected result of the task and the output of the model." When we want multiple turns of conversation we can use the ShareGPT format: Copy { "conversations": [\ {\ "from": "human",\ "value": "Can you help me make pasta carbonara?"\ },\ {\ "from": "gpt",\ "value": "Would you like the traditional Roman recipe, or a simpler version?"\ },\ {\ "from": "human",\ "value": "The traditional version please"\ },\ {\ "from": "gpt",\ "value": "The authentic Roman carbonara uses just a few ingredients: pasta, guanciale, eggs, Pecorino Romano, and black pepper. Would you like the detailed recipe?"\ }\ ] } The template format uses the "from"/"value" attribute keys and messages alternates between `human`and `gpt`, allowing for natural dialogue flow. The other common format is OpenAI's ChatML format and is what Hugging Face defaults to. This is probably the most used format, and alternates between `user` and `assistant` Copy { "messages": [\ {\ "role": "user",\ "content": "What is 1+1?"\ },\ {\ "role": "assistant",\ "content": "It's 2!"\ },\ ] } ### [](https://docs.unsloth.ai/basics/datasets-guide#applying-chat-templates-with-unsloth) Applying Chat Templates with Unsloth For datasets that usually follow the common chatml format, the process of preparing the dataset for training or finetuning, consists of four simple steps: * Check the chat templates that Unsloth currently supports: Copy from unsloth.chat_templates import CHAT_TEMPLATES print(list(CHAT_TEMPLATES.keys())) This will print out the list of templates currently supported by Unsloth. Here is an example output: Copy ['unsloth', 'zephyr', 'chatml', 'mistral', 'llama', 'vicuna', 'vicuna_old', 'vicuna old', 'alpaca', 'gemma', 'gemma_chatml', 'gemma2', 'gemma2_chatml', 'llama-3', 'llama3', 'phi-3', 'phi-35', 'phi-3.5', 'llama-3.1', 'llama-31', 'llama-3.2', 'llama-3.3', 'llama-32', 'llama-33', 'qwen-2.5', 'qwen-25', 'qwen25', 'qwen2.5', 'phi-4', 'gemma-3', 'gemma3'] * Use `get_chat_template` to apply the right chat template to your tokenizer: Copy from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "gemma-3", # change this to the right chat_template name ) * Define your formatting function. Here's an example: Copy def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } This function loops through your dataset applying the chat template you defined to each sample. * Finally, let's load the dataset and apply the required modifications to our dataset: Copy # Import and load dataset from datasets import load_dataset dataset = load_dataset("repo_name/dataset_name", split = "train") # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) If your dataset uses the ShareGPT format with "from"/"value" keys instead of the ChatML "role"/"content" format, you can use the `standardize_sharegpt` function to convert it first. The revised code will now look as follows: Copy # Import dataset from datasets import load_dataset dataset = load_dataset("mlabonne/FineTome-100k", split = "train") # Convert your dataset to the "role"/"content" format if necessary from unsloth.chat_templates import standardize_sharegpt dataset = standardize_sharegpt(dataset) # Apply the formatting function to your dataset using the map method dataset = dataset.map(formatting_prompts_func, batched = True,) ### [](https://docs.unsloth.ai/basics/datasets-guide#formatting-data-q-and-a) Formatting Data Q&A **Q:** How can I use the Alpaca instruct format? **A:** If your dataset is already formatted in the Alpaca format, then follow the formatting steps as shown in the Llama3.1 [notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb#scrollTo=LjY75GoYUCB8) . If you need to convert your data to the Alpaca format, one approach is to create a Python script to process your raw data. If you're working on a summarization task, you can use a local LLM to generate instructions and outputs for each example. **Q:** Should I always use the standardize\_sharegpt method? **A:** Only use the standardize\_sharegpt method if your target dataset is formatted in the sharegpt format, but your model expect a ChatML format instead. **Q:** Why not use the apply\_chat\_template function that comes with the tokenizer. **A:** The `chat_template` attribute when a model is first uploaded by the original model owners sometimes contains errors and may take time to be updated. In contrast, at Unsloth, we thoroughly check and fix any errors in the `chat_template` for every model when we upload the quantized versions to our repositories. Additionally, our `get_chat_template` and `apply_chat_template` methods offer advanced data manipulation features, which are fully documented on our Chat Templates documentation [page](https://docs.unsloth.ai/basics/chat-templates) . **Q:** What if my template is not currently supported by Unsloth? **A:** Submit a feature request on the unsloth github issues [forum](https://github.com/unslothai/unsloth) . As a temporary workaround, you could also use the tokenizer's own apply\_chat\_template function until your feature request is approved and merged. [](https://docs.unsloth.ai/basics/datasets-guide#synthetic-data-generation) Synthetic Data Generation ---------------------------------------------------------------------------------------------------------- You can also use any local LLM like Llama 3.3 (70B) or OpenAI's GPT 4.5 to generate synthetic data. Generally, it is better to use a bigger like Llama 3.3 (70B) to ensure the highest quality outputs. You can directly use inference engines like vLLM, Ollama or llama.cpp to generate synthetic data but it will require some manual work to collect it and prompt for more data. There's 3 goals for synthetic data: * Produce entirely new data - either from scratch or from your existing dataset * Diversify your dataset so your model does not [overfit](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting) and become too specific * Augment existing data e.g. automatically structure your dataset in the correct chosen format ### [](https://docs.unsloth.ai/basics/datasets-guide#synthetic-dataset-notebook) Synthetic Dataset Notebook We collaborated with Meta to launch a free notebook for creating Synthetic Datasets automatically using local models like Llama 3.2. [Access the notebook here.](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb) What the notebook does: * Auto-parses PDFs, websites, YouTube videos and more * Uses Metaā€™s Synthetic Data Kit + Llama 3.2 (3B) to generate QA pairs * Cleans and filters the data automatically * Fine-tunes the dataset with Unsloth + Llama * Notebook is fully done locally with no API calling necessary ### [](https://docs.unsloth.ai/basics/datasets-guide#using-a-local-llm-or-chatgpt-for-synthetic-data) Using a local LLM or ChatGPT for synthetic data Your goal is to prompt the model to generate and process QA data that is in your specified format. The model will need to learn the structure that you provided and also the context so ensure you at least have 10 examples of data already. Examples prompts: * **Prompt for generating more dialogue on an existing dataset**: Copy Using the dataset example I provided, follow the structure and generate conversations based on the examples. * **Prompt if you no have dataset**: Copy Create 10 examples of product reviews for Coca-Coca classified as either positive, negative, or neutral. * **Prompt for a dataset without formatting**: Copy Structure my dataset so it is in a QA ChatML format for fine-tuning. Then generate 5 synthetic data examples with the same topic and format. It is recommended to check the quality of generated data to remove or improve on irrelevant or poor-quality responses. Depending on your dataset it may also have to be balanced in many areas so your model does not overfit. You can then feed this cleaned dataset back into your LLM to regenerate data, now with even more guidance. [](https://docs.unsloth.ai/basics/datasets-guide#dataset-faq--tips) Dataset FAQ + Tips ------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/datasets-guide#how-big-should-my-dataset-be) How big should my dataset be? We generally recommend using a bare minimum of at least 100 rows of data for fine-tuning to achieve reasonable results. For optimal performance, a dataset with over 1,000 rows is preferable, and in this case, more data usually leads to better outcomes. If your dataset is too small you can also add synthetic data or add a dataset from Hugging Face to diversify it. However, the effectiveness of your fine-tuned model depends heavily on the quality of the dataset, so be sure to thoroughly clean and prepare your data. ### [](https://docs.unsloth.ai/basics/datasets-guide#how-should-i-structure-my-dataset-if-i-want-to-fine-tune-a-reasoning-model) How should I structure my dataset if I want to fine-tune a reasoning model? If you want to fine-tune a model that already has reasoning capabilities like the distilled versions of DeepSeek-R1 (e.g. DeepSeek-R1-Distill-Llama-8B), you will need to still follow question/task and answer pairs however, for your answer you will need to change the answer so it includes reasoning/chain-of-thought process and the steps it took to derive the answer. For a model that does not have reasoning and you want to train it so that it later encompasses reasoning capabilities, you will need to utilize a standard dataset but this time without reasoning in its answers. This is training process is known as [Reinforcement Learning and GRPO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) . ### [](https://docs.unsloth.ai/basics/datasets-guide#multiple-datasets) Multiple datasets If you have multiple datasets for fine-tuning, you can either: * Standardize the format of all datasets, combine them into a single dataset, and fine-tune on this unified dataset. * Use the [Multiple Datasets](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) notebook to fine-tune on multiple datasets directly. ### [](https://docs.unsloth.ai/basics/datasets-guide#can-i-fine-tune-the-same-model-multiple-times) Can I fine-tune the same model multiple times? You can fine-tune an already fine-tuned model multiple times, but it's best to combine all the datasets and perform the fine-tuning in a single process instead. Training an already fine-tuned model can potentially alter the quality and knowledge acquired during the previous fine-tuning process. [](https://docs.unsloth.ai/basics/datasets-guide#using-datasets-in-unsloth) Using Datasets in Unsloth ---------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/datasets-guide#alpaca-dataset) Alpaca Dataset See an example of using the Alpaca dataset inside of Unsloth on Google Colab: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FKSmRDpkySelZfWSrWxDm%252Fimage.png%3Falt%3Dmedia%26token%3D5401e4da-796a-42ad-8b85-2263f3e59e86&width=768&dpr=4&quality=100&sign=28ad8509&sv=2) We will now use the Alpaca Dataset created by calling GPT-4 itself. It is a list of 52,000 instructions and outputs which was very popular when Llama-1 was released, since it made finetuning a base LLM be competitive with ChatGPT itself. You can access the GPT4 version of the Alpaca dataset [here](https://huggingface.co/datasets/vicgalle/alpaca-gpt4.) . Below shows some examples of the dataset: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzKhujR9Nxz95VFSdf4J5%252Fimage.png%3Falt%3Dmedia%26token%3Da3c52718-eaf1-4a3d-b325-414d8e67722e&width=768&dpr=4&quality=100&sign=2afb3a12&sv=2) You can see there are 3 columns in each row - an instruction, and input and an output. We essentially combine each row into 1 large prompt like below. We then use this to finetune the language model, and this made it very similar to ChatGPT. We call this process **supervised instruction finetuning**. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FieYX44Vjd0OygJvO0jaR%252Fimage.png%3Falt%3Dmedia%26token%3Deb67fa41-a280-4656-8be6-5b6bf6f587c2&width=768&dpr=4&quality=100&sign=68f5594e&sv=2) ### [](https://docs.unsloth.ai/basics/datasets-guide#multiple-columns-for-finetuning) Multiple columns for finetuning But a big issue is for ChatGPT style assistants, we only allow 1 instruction / 1 prompt, and not multiple columns / inputs. For example in ChatGPT, you can see we must submit 1 prompt, and not multiple prompts. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FpFUWhntUQLu05l4ns7Pq%252Fimage.png%3Falt%3Dmedia%26token%3De989e4a6-6033-4741-b97f-d0c3ce8f5888&width=768&dpr=4&quality=100&sign=a9eb969a&sv=2) This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function! For example the very famous Titanic dataset has many many columns. Your job was to predict whether a passenger has survived or died based on their age, passenger class, fare price etc. We can't simply pass this into ChatGPT, but rather, we have to "merge" this information into 1 large prompt. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrydHBjHoJT7w8FwzKAXK%252FMerge-1.png%3Falt%3Dmedia%26token%3Dec812057-0475-4717-87fe-311f14735c37&width=768&dpr=4&quality=100&sign=8211e070&sv=2) For example, if we ask ChatGPT with our "merged" single prompt which includes all the information for that passenger, we can then ask it to guess or predict whether the passenger has died or survived. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJVkv73fRWvwwFxMym7uW%252Fimage.png%3Falt%3Dmedia%26token%3D59b97b76-f2f2-46c9-8940-60a37e4e7d62&width=768&dpr=4&quality=100&sign=37c0f3a1&sv=2) Other finetuning libraries require you to manually prepare your dataset for finetuning, by merging all your columns into 1 prompt. In Unsloth, we simply provide the function called `to_sharegpt` which does this in 1 go! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9fo2IBA7P0tNwhNR9Prm%252Fimage.png%3Falt%3Dmedia%26token%3D7bd7244a-0fea-4e57-9038-a8a360138056&width=768&dpr=4&quality=100&sign=a94d397b&sv=2) Now this is a bit more complicated, since we allow a lot of customization, but there are a few points: * You must enclose all columns in curly braces `{}`. These are the column names in the actual CSV / Excel file. * Optional text components must be enclosed in `[[]]`. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values. * Select the output or target / prediction column in `output_column_name`. For the Alpaca dataset, this will be `output`. For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRMvBpfXC9ToCRL0oCJfN%252Fimage.png%3Falt%3Dmedia%26token%3Dc257c7fc-8a9c-4d4f-ab3d-6894ae49f2a9&width=768&dpr=4&quality=100&sign=4ec813ed&sv=2) For example, pretend the dataset looks like this with a lot of missing data: Embarked Age Fare S 23 18 7.25 Then, we do not want the result to be: 1. The passenger embarked from S. Their age is 23. Their fare is **EMPTY**. 2. The passenger embarked from **EMPTY**. Their age is 18. Their fare is $7.25. Instead by optionally enclosing columns using `[[]]`, we can exclude this information entirely. 1. \[\[The passenger embarked from S.\]\] \[\[Their age is 23.\]\] \[\[Their fare is **EMPTY**.\]\] 2. \[\[The passenger embarked from **EMPTY**.\]\] \[\[Their age is 18.\]\] \[\[Their fare is $7.25.\]\] becomes: 1. The passenger embarked from S. Their age is 23. 2. Their age is 18. Their fare is $7.25. ### [](https://docs.unsloth.ai/basics/datasets-guide#multi-turn-conversations) Multi turn conversations A bit issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWCAN7bYUt6QWwCWUxisL%252Fdiff.png%3Falt%3Dmedia%26token%3D29821fd9-2181-4d1d-8b93-749b69bcf400&width=768&dpr=4&quality=100&sign=d4f1b675&sv=2) So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWi1rRNBFC2iDmCvSJsZt%252Fcombine.png%3Falt%3Dmedia%26token%3Dbef37a55-b272-4be3-89b5-9767c219a380&width=768&dpr=4&quality=100&sign=ae98ba1b&sv=2) Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column. We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FE75C4Y848VNF6luLuPRR%252Fimage.png%3Falt%3Dmedia%26token%3Daac1d79b-ecca-4e56-939d-d97dcbbf30eb&width=768&dpr=4&quality=100&sign=d48e3c76&sv=2) [](https://docs.unsloth.ai/basics/datasets-guide#vision-fine-tuning) Vision Fine-tuning -------------------------------------------------------------------------------------------- The dataset for fine-tuning a vision or multimodal model also includes image inputs. For example, the [Llama 3.2 Vision Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb#scrollTo=vITh0KVJ10qX) uses a radiography case to show how AI can help medical professionals analyze X-rays, CT scans, and ultrasounds more efficiently. We'll be using a sampled version of the ROCO radiography dataset. You can access the dataset [here](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Funsloth%2FRadiology_mini) . The dataset includes X-rays, CT scans and ultrasounds showcasing medical conditions and diseases. Each image has a caption written by experts describing it. The goal is to finetune a VLM to make it a useful analysis tool for medical professionals. Let's take a look at the dataset, and check what the 1st example shows: Copy Dataset({ features: ['image', 'image_id', 'caption', 'cui'], num_rows: 1978 }) Image Caption ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrjdETiyi6jqzAao7vg8I%252Fxray.png%3Falt%3Dmedia%26token%3Df66fdd7f-5e10-4eff-a280-5b3d63ed7849&width=768&dpr=4&quality=100&sign=4d4d6839&sv=2) Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows). To format the dataset, all vision finetuning tasks should be formatted as follows: Copy [\ { "role": "user",\ "content": [{"type": "text", "text": instruction}, {"type": "image", "image": image} ]\ },\ { "role": "assistant",\ "content": [{"type": "text", "text": answer} ]\ },\ ] We will craft an custom instruction asking the VLM to be an expert radiographer. Notice also instead of just 1 instruction, you can add multiple turns to make it a dynamic conversation. Copy instruction = "You are an expert radiographer. Describe accurately what you see in this image." def convert_to_conversation(sample): conversation = [\ { "role": "user",\ "content" : [\ {"type" : "text", "text" : instruction},\ {"type" : "image", "image" : sample["image"]} ]\ },\ { "role" : "assistant",\ "content" : [\ {"type" : "text", "text" : sample["caption"]} ]\ },\ ] return { "messages" : conversation } pass Let's convert the dataset into the "correct" format for finetuning: Copy converted_dataset = [convert_to_conversation(sample) for sample in dataset] The first example is now structured like below: Copy converted_dataset[0] Copy {'messages': [{'role': 'user',\ 'content': [{'type': 'text',\ 'text': 'You are an expert radiographer. Describe accurately what you see in this image.'},\ {'type': 'image',\ 'image': }]},\ {'role': 'assistant',\ 'content': [{'type': 'text',\ 'text': 'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'}]}]} Before we do any finetuning, maybe the vision model already knows how to analyse the images? Let's check if this is the case! Copy FastVisionModel.for_inference(model) # Enable for inference! image = dataset[0]["image"] instruction = "You are an expert radiographer. Describe accurately what you see in this image." messages = [\ {"role": "user", "content": [\ {"type": "image"},\ {"type": "text", "text": instruction}\ ]}\ ] input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True) inputs = tokenizer( image, input_text, add_special_tokens = False, return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer, skip_prompt = True) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True, temperature = 1.5, min_p = 0.1) And the result: Copy This radiograph appears to be a panoramic view of the upper and lower dentition, specifically an Orthopantomogram (OPG). * The panoramic radiograph demonstrates normal dental structures. * There is an abnormal area on the upper right, represented by an area of radiolucent bone, corresponding to the antrum. **Key Observations** * The bone between the left upper teeth is relatively radiopaque. * There are two large arrows above the image, suggesting the need for a closer examination of this area. One of the arrows is in a left-sided position, and the other is in the right-sided position. However, only For more details, view our dataset section in the [notebook here](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb#scrollTo=vITh0KVJ10qX) . [PreviousText-to-Speech (TTS) Fine-tuning](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning) [NextUnsloth Dynamic 2.0 GGUFs](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) Last updated 2 months ago Was this helpful? --- # Unsloth Dynamic 2.0 GGUFs | Unsloth Documentation OpenAI gpt-oss & all model types now supported! We're excited to introduce our Dynamic v2.0 quantization method - a major upgrade to our previous quants. This new method outperforms leading quantization methods and sets new benchmarks for 5-shot MMLU and KL Divergence. This means you can now run + fine-tune quantized LLMs while preserving as much accuracy as possible! You can run the 2.0 GGUFs on any inference engine like llama.cpp, Ollama, Open WebUI etc. View all our Dynamic 2.0 GGUF models on [Hugging Face here](https://huggingface.co/collections/unsloth/unsloth-dynamic-v20-quants-68060d147e9b9231112823e6) . The **key advantage** of using the Unsloth package and models is our active role in _**fixing critical bugs**_ in major models. We've collaborated directly with teams behind [Qwen3](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/) , [Meta (Llama 4)](https://github.com/ggml-org/llama.cpp/pull/12889) , [Mistral (Devstral)](https://app.gitbook.com/o/HpyELzcNe0topgVLGCZY/s/xhOjnexMCB3dmuQFQ2Zq/~/changes/618/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune) , [Google (Gemma 1ā€“3)](https://news.ycombinator.com/item?id=39671146) and [Microsoft (Phi-3/4)](https://simonwillison.net/2025/Jan/11/phi-4-bug-fixes) , contributing essential fixes that significantly boost accuracy. ### [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#whats-new-in-dynamic-v2.0) šŸ’” What's New in Dynamic v2.0? * **Revamped Layer Selection for GGUFs + safetensors:** Unsloth Dynamic 2.0 now selectively quantizes layers much more intelligently and extensively. Rather than modifying only select layers, we now dynamically adjust the quantization type of every possible layer, and the combinations will differ for each layer and model. * Current selected and all future GGUF uploads will utilize Dynamic 2.0 and our new calibration dataset. The dataset contains more than >1.5M **tokens** (depending on model) and comprise of high-quality, hand-curated and cleaned data - to greatly enhance conversational chat performance. * Previously, our Dynamic quantization (DeepSeek-R1 1.58-bit GGUF) was effective only for MoE architectures. **Dynamic 2.0 quantization now works on all models (including MOEs & non-MoEs)**. * **Model-Specific Quants:** Each model now uses a custom-tailored quantization scheme. E.g. the layers quantized in Gemma 3 differ significantly from those in Llama 4. * To maximize efficiency, especially on Apple Silicon and ARM devices, we now also add Q4\_NL, Q5.1, Q5.0, Q4.1, and Q4.0 formats. To ensure accurate benchmarking, we built an internal evaluation framework to match official reported 5-shot MMLU scores of Llama 4 and Gemma 3. This allowed apples-to-apples comparisons between full-precision vs. Dynamic v2.0, **QAT** and standard **imatrix** GGUF quants. Currently, we've released updates for: **Qwen3:** [0.6B](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) ā€¢ [1.7B](https://huggingface.co/unsloth/Qwen3-1.7B-GGUF) ā€¢ [4B](https://huggingface.co/unsloth/Qwen3-4B-GGUF) ā€¢ [8B](https://huggingface.co/unsloth/Qwen3-8B-GGUF) ā€¢ [14B](https://huggingface.co/unsloth/Qwen3-14B-GGUF) ā€¢ [30B-A3B](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) ā€¢ [32B](https://huggingface.co/unsloth/Qwen3-32B-GGUF) ā€¢ [235B-A22B](https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF) ā€¢ [R1-0528](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF) **Other:** [GLM-4-32B](https://huggingface.co/unsloth/GLM-4-32B-0414-GGUF) ā€¢ [MAI-DS-R1](https://huggingface.co/unsloth/MAI-DS-R1-GGUF) ā€¢ [QwQ (32B)](https://huggingface.co/unsloth/QwQ-32B-GGUF) **DeepSeek:** [R1-0528](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally#model-uploads) ā€¢ [V3-0324](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF-UD) ā€¢ [R1-Distill-Llama](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF) **Llama:** [4 (Scout)](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF) ā€¢ [4 (Maverick)](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF) ā€¢ [3.1 (8B)](https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-GGUF) **Gemma 3:** [4B](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) ā€¢ [12B](https://huggingface.co/unsloth/gemma-3-12b-it-GGUF) ā€¢ [27B](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF) ā€¢ [QAT](https://huggingface.co/unsloth/gemma-3-12b-it-qat-GGUF) **Mistral:** [Magistral](https://huggingface.co/unsloth/Magistral-Small-2506-GGUF) ā€¢ [Small-3.1-2503](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF) All future GGUF uploads will utilize Unsloth Dynamic 2.0, and our Dynamic 4-bit safe tensor quants will also benefit from this in the future. Detailed analysis of our benchmarks and evaluation further below. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWpuceJODVjlQcN7RvS6M%252Fkldivergence%2520graph.png%3Falt%3Dmedia%26token%3D1f8f39fb-d4c6-47c6-84fe-f767ec7bae6b&width=768&dpr=4&quality=100&sign=70391f18&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FszSmyqwqLW7artvIR5ut%252F5shotmmlu.png%3Falt%3Dmedia%26token%3Dc9ef327e-5f8c-4720-8e05-08c345668745&width=768&dpr=4&quality=100&sign=517a777d&sv=2) [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#why-kl-divergence) šŸ“Š Why KL Divergence? --------------------------------------------------------------------------------------------------------- [Accuracy is Not All You Need](https://arxiv.org/pdf/2407.09141) showcases how pruning layers, even by selecting unnecessary ones still yields vast differences in terms of "flips". A "flip" is defined as answers changing from incorrect to correct or vice versa. The paper shows how MMLU might not decrease as we prune layers or do quantization,but that's because some incorrect answers might have "flipped" to become correct. Our goal is to match the original model, so measuring "flips" is a good metric. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEjL8zLLNyceY3IpDUdWz%252Fimage.png%3Falt%3Dmedia%26token%3D6c31355b-57cf-4f22-a70e-b3b1e7c533d4&width=768&dpr=4&quality=100&sign=e862b672&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FimYGCjWJ3GVKQmfAQwd5%252Fimage.png%3Falt%3Dmedia%26token%3D5a49d0ec-d92a-4d0e-9d6f-77f6d0d95738&width=768&dpr=4&quality=100&sign=77448477&sv=2) **KL Divergence** should be the **gold standard for reporting quantization errors** as per the research paper "Accuracy is Not All You Need". **Using perplexity is incorrect** since output token values can cancel out, so we must use KLD! The paper also shows that interestingly KL Divergence is highly correlated with flips, and so our goal is to reduce the mean KL Divergence whilst increasing the disk space of the quantization as less as possible. [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#calibration-dataset-overfitting) āš–ļø Calibration Dataset Overfitting ------------------------------------------------------------------------------------------------------------------------------------ Most frameworks report perplexity and KL Divergence using a test set of Wikipedia articles. However, we noticed using the calibration dataset which is also Wikipedia related causes quants to overfit, and attain lower perplexity scores. We utilize [Calibration\_v3](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8) and [Calibration\_v5](https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c/) datasets for fair testing which includes some wikitext data amongst other data. **Also instruct models have unique chat templates, and using text only calibration datasets is not effective for instruct models** (base models yes). In fact most imatrix GGUFs are typically calibrated with these issues. As a result, they naturally perform better on KL Divergence benchmarks that also use Wikipedia data, since the model is essentially optimized for that domain. To ensure a fair and controlled evaluation, we do not to use our own calibration dataset (which is optimized for chat performance) when benchmarking KL Divergence. Instead, we conducted tests using the same standard Wikipedia datasets, allowing us to directly compare the performance of our Dynamic 2.0 method against the baseline imatrix approach. [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#mmlu-replication-adventure) šŸ”¢ MMLU Replication Adventure -------------------------------------------------------------------------------------------------------------------------- * Replicating MMLU 5 shot was nightmarish. We **could not** replicate MMLU results for many models including Llama 3.1 (8B) Instruct, Gemma 3 (12B) and others due to **subtle implementation issues**. Llama 3.1 (8B) for example should be getting ~68.2%, whilst using incorrect implementations can attain **35% accuracy.** ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FGqqARO9UA0qpIzNcfixv%252FMMLU%2520differences.png%3Falt%3Dmedia%26token%3D59c47844-a2e6-49a3-a523-1e28f2208e6d&width=768&dpr=4&quality=100&sign=3c0bd533&sv=2) MMLU implementation issues * Llama 3.1 (8B) Instruct has a MMLU 5 shot accuracy of 67.8% using a naive MMLU implementation. We find however Llama **tokenizes "A" and "\_A" (A with a space in front) as different token ids**. If we consider both spaced and non spaced tokens, we get 68.2% (+0.4%) * Interestingly Llama 3 as per Eleuther AI's [LLM Harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/llama3/instruct/mmlu/_continuation_template_yaml) also appends **"The best answer is"** to the question, following Llama 3's original MMLU benchmarks. * There are many other subtle issues, and so to benchmark everything in a controlled environment, we designed our own MMLU implementation from scratch by investigating [github.com/hendrycks/test](https://github.com/hendrycks/test) directly, and verified our results across multiple models and comparing to reported numbers. [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#gemma-3-qat-replication-benchmarks) āœØ Gemma 3 QAT Replication, Benchmarks ------------------------------------------------------------------------------------------------------------------------------------------ The Gemma team released two QAT (quantization aware training) versions of Gemma 3: 1. Q4\_0 GGUF - Quantizes all layers to Q4\_0 via the formula `w = q * block_scale` with each block having 32 weights. See [llama.cpp wiki](https://github.com/ggml-org/llama.cpp/wiki/Tensor-Encoding-Schemes) for more details. 2. int4 version - presumably [TorchAO int4 style](https://github.com/pytorch/ao/blob/main/torchao/quantization/README.md) ? We benchmarked all Q4\_0 GGUF versions, and did extensive experiments on the 12B model. We see the **12B Q4\_0 QAT model gets 67.07%** whilst the full bfloat16 12B version gets 67.15% on 5 shot MMLU. That's very impressive! The 27B model is mostly nearly there! Metric 1B 4B 12B 27B MMLU 5 shot 26.12% 55.13% **67.07% (67.15% BF16)** **70.64% (71.5% BF16)** Disk Space 0.93GB 2.94GB **7.52GB** 16.05GB **Efficiency\*** 1.20 10.26 **5.59** 2.84 We designed a new **Efficiency metric** which calculates the usefulness of the model whilst also taking into account its disk size and MMLU 5 shot score: Efficiency\=MMLUĀ 5Ā shotĀ scoreāˆ’25DiskĀ SpaceĀ GB\\text{Efficiency} = \\frac{\\text{MMLU 5 shot score} - 25}{\\text{Disk Space GB}}Efficiency\=DiskĀ SpaceĀ GBMMLUĀ 5Ā shotĀ scoreāˆ’25ā€‹ We have to **minus 25** since MMLU has 4 multiple choices - A, B, C or D. Assume we make a model that simply randomly chooses answers - it'll get 25% accuracy, and have a disk space of a few bytes. But clearly this is not a useful model. On KL Divergence vs the base model, below is a table showcasing the improvements. Reminder the closer the KL Divergence is to 0, the better (ie 0 means identical to the full precision model) Quant Baseline KLD GB New KLD GB IQ1\_S 1.035688 5.83 0.972932 6.06 IQ1\_M 0.832252 6.33 0.800049 6.51 IQ2\_XXS 0.535764 7.16 0.521039 7.31 IQ2\_M 0.26554 8.84 0.258192 8.96 Q2\_K\_XL 0.229671 9.78 0.220937 9.95 Q3\_K\_XL 0.087845 12.51 0.080617 12.76 Q4\_K\_XL 0.024916 15.41 0.023701 15.64 If we plot the ratio of the disk space increase and the KL Divergence ratio change, we can see a much clearer benefit! Our dynamic 2bit Q2\_K\_XL reduces KLD quite a bit (around 7.5%). ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FsYSRIPGSjExzSr5y828z%252Fchart%282%29.svg%3Falt%3Dmedia%26token%3De87db00e-6e3e-4478-af0b-bc84ed2e463b&width=768&dpr=4&quality=100&sign=9073c258&sv=2) Truncated table of results for MMLU for Gemma 3 (27B). See below. 1. **Our dynamic 4bit version is 2GB smaller whilst having +1% extra accuracy vs the QAT version!** 2. Efficiency wise, 2bit Q2\_K\_XL and others seem to do very well! Quant Unsloth Unsloth + QAT Disk Size Efficiency IQ1\_M 48.10 47.23 6.51 3.42 IQ2\_XXS 59.20 56.57 7.31 4.32 IQ2\_M 66.47 64.47 8.96 4.40 Q2\_K\_XL 68.70 67.77 9.95 4.30 Q3\_K\_XL 70.87 69.50 12.76 3.49 **Q4\_K\_XL** **71.47** **71.07** **15.64** **2.94** **Google QAT** **70.64** **17.2** **2.65** Click here for Full Google's Gemma 3 (27B) QAT Benchmarks:[](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#click-here-for-full-googles-gemma-3-27b-qat-benchmarks) Model Unsloth Unsloth + QAT Disk Size Efficiency IQ1\_S 41.87 43.37 6.06 3.03 IQ1\_M 48.10 47.23 6.51 3.42 IQ2\_XXS 59.20 56.57 7.31 4.32 IQ2\_M 66.47 64.47 8.96 4.40 Q2\_K 68.50 67.60 9.78 4.35 Q2\_K\_XL 68.70 67.77 9.95 4.30 IQ3\_XXS 68.27 67.07 10.07 4.18 Q3\_K\_M 70.70 69.77 12.51 3.58 Q3\_K\_XL 70.87 69.50 12.76 3.49 Q4\_K\_M 71.23 71.00 15.41 2.98 **Q4\_K\_XL** **71.47** **71.07** **15.64** **2.94** Q5\_K\_M 71.77 71.23 17.95 2.58 Q6\_K 71.87 71.60 20.64 2.26 Q8\_0 71.60 71.53 26.74 1.74 **Google QAT** **70.64** **17.2** **2.65** [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#llama-4-bug-fixes--run) šŸ¦™ Llama 4 Bug Fixes + Run ------------------------------------------------------------------------------------------------------------------- We also helped and fixed a few Llama 4 bugs: * Llama 4 Scout changed the RoPE Scaling configuration in their official repo. We helped resolve issues in llama.cpp to enable this [change here](https://github.com/ggml-org/llama.cpp/pull/12889) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FaJ5AOubUkMjbbvgiOekf%252Fimage.png%3Falt%3Dmedia%26token%3Db1fbdea1-7c95-4afa-9b12-aedec012f38b&width=768&dpr=4&quality=100&sign=2203a04c&sv=2) * Llama 4's QK Norm's epsilon for both Scout and Maverick should be from the config file - this means using 1e-05 and not 1e-06. We helped resolve these in [llama.cpp](https://github.com/ggml-org/llama.cpp/pull/12889) and [transformers](https://github.com/huggingface/transformers/pull/37418) * The Llama 4 team and vLLM also independently fixed an issue with QK Norm being shared across all heads (should not be so) [here](https://github.com/vllm-project/vllm/pull/16311) . MMLU Pro increased from 68.58% to 71.53% accuracy. * [Wolfram Ravenwolf](https://x.com/WolframRvnwlf/status/1909735579564331016) showcased how our GGUFs via llama.cpp attain much higher accuracy than third party inference providers - this was most likely a combination of the issues explained above, and also probably due to quantization issues. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4Wrz07bAdvluM2gACggU%252FGoC79hYXwAAPTMs.jpg%3Falt%3Dmedia%26token%3D05001bc0-74b0-4bbb-a89f-894fcdb985d8&width=768&dpr=4&quality=100&sign=23d1a190&sv=2) As shown in our graph, our 4-bit Dynamic QAT quantization deliver better performance on 5-shot MMLU while also being smaller in size. ### [](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs#running-llama-4-scout) Running Llama 4 Scout: To run Llama 4 Scout for example, first clone llama.cpp: Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp Then download out new dynamic v 2.0 quant for Scout: Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF", local_dir = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF", allow_patterns = ["*IQ2_XXS*"], ) And and let's do inference! Copy ./llama.cpp/llama-cli \ --model unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-IQ2_XXS.gguf \ --threads 32 \ --ctx-size 16384 \ --n-gpu-layers 99 \ -ot ".ffn_.*_exps.=CPU" \ --seed 3407 \ --prio 3 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ -no-cnv \ --prompt "<|header_start|>user<|header_end|>\n\nCreate a Flappy Bird game.<|eot|><|header_start|>assistant<|header_end|>\n\n" Read more on running Llama 4 here: [https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-llama-4](https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-llama-4) [PreviousDatasets Guide](https://docs.unsloth.ai/basics/datasets-guide) [NextLlama 4: How to Run & Fine-tune](https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune) Last updated 1 month ago Was this helpful? --- # Reinforcement Learning (RL) Guide | Unsloth Documentation OpenAI gpt-oss & all model types now supported! [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#what-you-will-learn) šŸ¦„What you will learn ----------------------------------------------------------------------------------------------------------------- 1. What is RL? RLVR? PPO? GRPO? RLHF? RFT? Is **"Luck is All You Need?"** for RL? 2. What is an environment? Agent? Action? Reward function? Rewards? This article covers everything (from beginner to advanced) you need to know about GRPO, Reinforcement Learning (RL) and reward functions, along with tips, and the basics of using GRPO with [Unsloth](https://github.com/unslothai/unsloth) . If you're looking for a step-by-step tutorial for using GRPO, see our guide [here](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) . [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#what-is-reinforcement-learning-rl) ā“What is Reinforcement Learning (RL)? ----------------------------------------------------------------------------------------------------------------------------------------------- The goal of RL is to: 1. **Increase the chance of seeing** **"good"** **outcomes.** 2. **Decrease the chance of seeing** **"bad"** **outcomes.** **That's it!** There are intricacies on what "good" and "bad" means, or how do we go about "increasing" or "decreasing" it, or what even "outcomes" means. For example, in the **Pacman game**: 1. The **environment** is the game world. 2. The **actions** you can take are UP, LEFT, RIGHT and DOWN. 3. The **rewards** are good if you eat a cookie, or bad if you hit one of the squiggly enemies. 4. In RL, you can't know the "best action" you can take, but you can observe intermediate steps, or the final game state (win or lose) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FLYKyo5xU4mSvQRASnH1D%252FRL%2520Game.png%3Falt%3Dmedia%26token%3D16e9a8c6-61f9-4baf-84a7-118e562eb6c5&width=768&dpr=4&quality=100&sign=506e7b9c&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FVVJbst1Vn3Pg6jn0hXLA%252FMath%2520RL.png%3Falt%3Dmedia%26token%3D855abbe8-d134-4246-ae5c-5108574aaa6e&width=768&dpr=4&quality=100&sign=d0030f5c&sv=2) Another example is imagine you are given the question: **"What is 2 + 2?"** (4) An unaligned language model will spit out 3, 4, C, D, -10, literally anything. 1. Numbers are better than C or D right? 2. Getting 3 is better than say 8 right? 3. Getting 4 is definitely correct. We just designed a **reward function**! [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr) šŸƒFrom RLHF, PPO to GRPO and RLVR ---------------------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FU3NH5rSkI17fysvnMJHJ%252FRLHF.png%3Falt%3Dmedia%26token%3D53625e98-2949-45d1-b650-c5a7313b18a0&width=768&dpr=4&quality=100&sign=877df9f8&sv=2) OpenAI popularized the concept of [RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback) (Reinforcement Learning from Human Feedback), where we train an **"agent"** to produce outputs to a question (the **state**) that are rated more useful by human beings. The thumbs up and down in ChatGPT for example can be used in the RLHF process. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fn5N2OBGIqk1oPbR9gRKn%252FPPO.png%3Falt%3Dmedia%26token%3De9706260-6bee-4ef0-a7dc-f5f6d80471d5&width=768&dpr=4&quality=100&sign=1a8386bf&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FplVZSTOwKSQv5zQYjkge%252FPPO%2520formula.png%3Falt%3Dmedia%26token%3D8b1359c8-11d1-4ea8-91c0-cf4afe120166&width=768&dpr=4&quality=100&sign=2b15f552&sv=2) PPO formula The clip(..., 1-e, 1+e) term is used to force PPO not to take too large changes. There is also a KL term with beta set to > 0 to force the model not to deviate too much away. In order to do RLHF, [**PPO**](https://en.wikipedia.org/wiki/Proximal_policy_optimization) (Proximal policy optimization) was developed. The **agent** is the language model in this case. In fact it's composed of 3 systems: 1. The **Generating Policy (current trained model)** 2. The **Reference Policy (original model)** 3. The **Value Model (average reward estimator)** We use the **Reward Model** to calculate the reward for the current environment, and our goal is to **maximize this**! The formula for PPO looks quite complicated because it was designed to be stable. Visit our [AI Engineer talk](https://docs.unsloth.ai/ai-engineers-2025) we gave in 2025 about RL for more in depth maths derivations about PPO. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FiQI4Yvv1KcvkK7g5V8vm%252FGRPO%2520%252B%2520RLVR.png%3Falt%3Dmedia%26token%3D2155a920-b986-4a08-871a-32b5bbcfdbe3&width=768&dpr=4&quality=100&sign=57fbc22d&sv=2) DeepSeek developed [**GRPO**](https://unsloth.ai/blog/grpo) (Group Relative Policy Optimization) to train their R1 reasoning models. The key differences to PPO are: 1. The **Value Model is removed,** replaced with statistics from calling the reward model multiple times. 2. The **Reward Model is removed** and replaced with just custom reward function which **RLVR** can be used. This means GRPO is extremely efficient. Previously PPO needed to train multiple models - now with the reward model and value model removed, we can save memory and speed up everything. **RLVR (Reinforcement Learning with Verifiable Rewards)** allows us to reward the model based on tasks with easy to verify solutions. For example: 1. Maths equations can be easily verified. Eg 2+2 = 4. 2. Code output can be verified as having executed correctly or not. 3. Designing verifiable reward functions can be tough, and so most examples are math or code. 4. Use-cases for GRPO isnā€™t just for code or mathā€”its reasoning process can enhance tasks like email automation, database retrieval, law, and medicine, greatly improving accuracy based on your dataset and reward function - the trick is to define a **rubric - ie a list of smaller verifiable rewards, and not a final all consuming singular reward.** OpenAI popularized this in their [reinforcement learning finetuning (RFT)](https://platform.openai.com/docs/guides/reinforcement-fine-tuning) offering for example. **Why "Group Relative"?** GRPO removes the value model entirely, but we still need to estimate the **"average reward"** given the current state. The **trick is to sample the LLM**! We then calculate the average reward through statistics of the sampling process across multiple different questions. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FdXw9vYkjJaKFLTMx0Py6%252FGroup%2520Relative.png%3Falt%3Dmedia%26token%3D9153caf5-402e-414b-b5b4-79fef1a2c2fa&width=768&dpr=4&quality=100&sign=12ec7641&sv=2) For example for "What is 2+2?" we sample 4 times. We might get 4, 3, D, C. We then calculate the reward for each of these answers, then calculate the **average reward** and **standard deviation**, then **Z-score standardize** this! This creates the **advantages A**, which we will use in replacement of the value model. This saves a lot of memory! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FVDdKLOBcLyLC3dwF1Idd%252FStatistics.png%3Falt%3Dmedia%26token%3D6c8eae5b-b063-4f49-b896-7f8de516a379&width=768&dpr=4&quality=100&sign=6c940048&sv=2) GRPO advantage calculation [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#luck-well-patience-is-all-you-need) šŸ¤žLuck (well Patience) Is All You Need ------------------------------------------------------------------------------------------------------------------------------------------------- The trick of RL is you need 2 things only: 1. A question or instruction eg "What is 2+2?" "Create a Flappy Bird game in Python" 2. A reward function and verifier to verify if the output is good or bad. With only these 2, we can essentially **call a language model an infinite times** until we get a good answer. For example for "What is 2+2?", an untrained bad language model will output: _**0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31**_ _**then suddenly 4**__**.**_ _**The reward signal was 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0**_ _**then suddenly 1.**_ So by luck and by chance, RL managed to find the correct answer across multiple **rollouts**. Our goal is we want to see the good answer 4 more, and the rest (the bad answers) much less. **So the goal of RL is to be patient - in the limit, if the probability of the correct answer is at least a small number (not zero), it's just a waiting game - you will 100% for sure encounter the correct answer in the limit.** **So I like to call it as "Luck Is All You Need" for RL.** **Well a better phrase is "Patience is All You Need" for RL.** ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FryuL3pCuF8pPIjPEASbx%252FLuck%2520is%2520all%2520you%2520need.png%3Falt%3Dmedia%26token%3D64d1a03a-6afc-49a9-b734-8ce8bc2b5ec1&width=768&dpr=4&quality=100&sign=13346422&sv=2) RL essentially provides us a trick - instead of simply waiting for infinity, we do get "bad signals" ie bad answers, and we can essentially "guide" the model to already try not generating bad solutions. This means although you waited very long for a "good" answer to pop up, the model already has been changed to try its best not to output bad answers. In the "What is 2+2?" example - _**0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31**_ _**then suddenly 4**__**.**_ Since we got bad answers, RL will influence the model to try NOT to output bad answers. This means over time, we are carefully "pruning" or moving the model's output distribution away from bad answers. This means RL is not inefficient, since we are NOT just waiting for infinity, but we are actively trying to "push" the model to go as much as possible to the "correct answer space". **If the probability is always 0, then RL will never work**. This is also why people like to do RL from an already instruction finetuned model, which can partially follow instructions reasonably well - this boosts the probability most likely above 0. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#what-unsloth-offers-for-rl) šŸ¦„What Unsloth offers for RL ------------------------------------------------------------------------------------------------------------------------------- * With 15GB VRAM, Unsloth allows you to transform any model up to 17B parameters like Llama 3.1 (8B), Phi-4 (14B), Mistral (7B) or Qwen2.5 (7B) into a reasoning model * **Minimum requirement:** Just ā€Æ5GB VRAM is enough to train your own reasoning model locally (for any model with 1.5B parameters or less) [āš”Tutorial: Train your own Reasoning model with GRPO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#grpo-notebooks) GRPO notebooks: [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) \- Advanced [**DeepSeek-R1-0528-Qwen3-8B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb) **- new** [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_(3B)_GRPO_LoRA.ipynb) - Advanced [Gemma 3 (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb) [Qwen2.5 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb) [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-GRPO.ipynb) [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb) **NEW!** We now support Dr. GRPO and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable: Copy epsilon=0.2, epsilon_high=0.28, # one sided delta=1.5 # two sided loss_type='bnpo', # or: loss_type='grpo', # or: loss_type='dr_grpo', mask_truncated_completions=True, * If you're not getting any reasoning, make sure you have enough training steps and ensure your [reward function/verifier](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-functions-verifier) is working. We provide examples for reward functions [here](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-function-examples) . * Previous demonstrations show that you could achieve your own "aha" moment with Qwen2.5 (3B) - but it required 2xA100 GPUs (160GB VRAM). Now, with Unsloth, you can achieve the same "aha" moment using just a single 5GB VRAM GPU. * Previously, GRPO was only supported for full fine-tuning, but we've made it work with QLoRA and LoRA * On [**20K context lengths**](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#grpo-requirement-guidelines) for example with 8 generations per prompt, Unsloth uses only 54.3GB of VRAM for Llama 3.1 (8B), whilst standard implementations (+ Flash Attention 2) take **510.8GB (90% less for Unsloth)**. * Please note, this isnā€™t fine-tuning DeepSeekā€™s R1 distilled models or using distilled data from R1 for tuning which Unsloth already supported. This is converting a standard model into a full-fledged reasoning model using GRPO. In a test example, even though we only trained Phi-4 with 100 steps using GRPO, the results are already clear. The model without GRPO does not have the thinking token, whilst the one trained with GRPO does and also has the correct answer. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FyBeJAvfolzfEYyftji76%252Fprompt%2520only%2520example.png%3Falt%3Dmedia%26token%3D3903995a-d9d5-4cdc-9020-c4efe7fff651&width=768&dpr=4&quality=100&sign=80d59783&sv=2) [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#training-with-grpo) šŸ’»Training with GRPO --------------------------------------------------------------------------------------------------------------- For a tutorial on how to transform any open LLM into a reasoning model using Unsloth & GRPO, [see here](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) . ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#how-grpo-trains-a-model) **How GRPO Trains a Model** 1. For each question-answer pair, the model generates multiple possible responses (e.g., 8 variations). 2. Each response is evaluated using reward functions. 3. Training Steps: * If you have 300 rows of data, that's 300 training steps (or 900 steps if trained for 3 epochs). * You can increase the number of generated responses per question (e.g., from 8 to 16). 4. The model learns by updating its weights every step. If you're having issues with your GRPO model not learning, we'd highly recommend to use our [Advanced GRPO notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-notebooks) as it has a much better reward function and you should see results much faster and frequently. ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#basics-tips) Basics/Tips * Wait for at least **300 steps** for the reward to actually increase. In order to get decent results, you may need to trade for a minimum of 12 hours (this is how GRPO works), but keep in mind this isn't compulsory as you can stop at anytime. * For optimal results have at least **500 rows of data**. You can try with even 10 rows of data but it's better to have more. * Each training run will always be different depending on your model, data, reward function/verifier etc. so though 300 steps is what we wrote as the minimum, sometimes it might be 1000 steps or more. So, it depends on various factors. * If you're using GRPO with Unsloth locally, please "pip install diffusers" as well if you get an error. Please also use the latest version of vLLM. * Itā€™s advised to apply GRPO to a model at least **1.5B in parameters** to correctly generate thinking tokens as smaller models may not. * For GRPO's [**GPU VRAM requirements**](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#grpo-requirement-guidelines) **for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM. * **Continuous fine-tuning is** possible and you can just leave GRPO running in the background. * In the example notebooks, we use the [**GSM8K dataset**](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#gsm8k-reward-functions) , the current most popular choice for R1-style training. * If youā€™re using a base model, ensure you have a chat template. * The more you train with GRPO the better. The best part of GRPO is you don't even need that much data. All you need is a great reward function/verifier and the more time spent training, the better your model will get. Expect your reward vs step to increase as time progresses like this: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FUROleqJQ5aEp8MjTCWFf%252Funnamed.png%3Falt%3Dmedia%26token%3D12ca4975-7a0c-4d10-9178-20db28ad0451&width=768&dpr=4&quality=100&sign=a2046ca5&sv=2) * Training loss tracking for GRPO is now built directly into Unsloth, eliminating the need for external tools like wandb etc. It contains full logging details for all reward functions now including the total aggregated reward function itself. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fjo7fVFoFG2xbZPgL45el%252FScreenshot%25202025-02-20%2520at%252004-52-52%2520Copy%2520of%2520Yet%2520another%2520copy%2520of%2520Llama3.1_%288B%29-GRPO.ipynb%2520-%2520Colab.png%3Falt%3Dmedia%26token%3D041c17b1-ab98-4ab6-b6fb-8c7e5a8c07df&width=768&dpr=4&quality=100&sign=b8126c85&sv=2) [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-functions-verifiers) šŸ“‹Reward Functions / Verifiers --------------------------------------------------------------------------------------------------------------------------------- In Reinforcement Learning a **Reward Function** and a **Verifier** serve distinct roles in evaluating a modelā€™s output. In general, you could interpret them as the same thing however, technically they're not but it does not matter as much as they are usually used in conjunction with each other. **Verifier**: * Determines whether the generated response is correct or incorrect. * It does not assign a numerical scoreā€”it simply verifies correctness. * Example: If a model generates "5" for "2+2", the verifier checks and labels it as "wrong" (since the correct answer is 4). * Verifiers can also execute code (e.g., in Python) to validate logic, syntax, and correctness without needing manual evaluation. **Reward Function**: * Converts verification results (or other criteria) into a numerical score. * Example: If an answer is wrong, it might assign a penalty (-1, -2, etc.), while a correct answer could get a positive score (+1, +2). * It can also penalize based on criteria beyond correctness, such as excessive length or poor readability. **Key Differences**: * A **Verifier** checks correctness but doesnā€™t score. * A **Reward Function** assigns a score but doesnā€™t necessarily verify correctness itself. * A Reward Function _can_ use a Verifier, but they are technically not the same. ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#understanding-reward-functions) **Understanding Reward Functions** GRPO's primary goal is to maximize reward and learn how an answer was derived, rather than simply memorizing and reproducing responses from its training data. * With every training step, GRPO **adjusts model weights** to maximize the reward. This process fine-tunes the model incrementally. * **Regular fine-tuning** (without GRPO) only **maximizes next-word prediction probability** but does not optimize for a reward. GRPO **optimizes for a reward function** rather than just predicting the next word. * You can **reuse data** across multiple epochs. * **Default reward functions** can be predefined to be used on a wide array of use cases or you can ask ChatGPT/local model to generate them for you. * Thereā€™s no single correct way to design reward functions or verifiers - the possibilities are endless. However, they must be well-designed and meaningful, as poorly crafted rewards can unintentionally degrade model performance. ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#reward-function-examples) šŸŖ™Reward Function Examples You can refer to the examples below. You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#example-1-simple-arithmetic-task) **Example #1: Simple Arithmetic Task** * **Question:** `"2 + 2"` * **Answer:** `"4"` * **Reward Function 1:** * If a number is detected ā†’ **+1** * If no number is detected ā†’ **\-1** * **Reward Function 2:** * If the number matches the correct answer ā†’ **+3** * If incorrect ā†’ **\-3** * **Total Reward:** _Sum of all reward functions_ #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#example-2-email-automation-task) **Example #2: Email Automation Task** * **Question:** Inbound email * **Answer:** Outbound email * **Reward Functions:** * If the answer contains a required keyword ā†’ **+1** * If the answer exactly matches the ideal response ā†’ **+1** * If the response is too long ā†’ **\-1** * If the recipient's name is included ā†’ **+1** * If a signature block (phone, email, address) is present ā†’ **+1** ### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#unsloth-proximity-based-reward-function) Unsloth Proximity-Based Reward Function If youā€™ve checked out our [**Advanced GRPO Colab Notebook**](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#grpo-notebooks) , youā€™ll notice weā€™ve created a **custom proximity-based reward function** built completely from scratch, which is designed to reward answers that are closer to the correct one. This flexible function can be applied across a wide range of tasks. * In our examples, we enable reasoning in Qwen3 (Base) and guide it toward specific tasks * Apply Pre-finetuning strategies to avoid GRPOā€™s default tendency to just learn formatting * Boost evaluation accuracy with regex-based matching * Create custom GRPO templates beyond generic prompts like `think`, e.g., `` * Apply proximity-based scoring ā€” models get more reward for closer answers (e.g., predicting 9 instead of 10 is better than 3) while outliers are penalized #### [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#gsm8k-reward-functions) GSM8K Reward Functions In our other examples, we use existing GSM8K reward functions by [@willccbb](https://x.com/willccbb) which is popular and shown to be quite effective: * **correctness\_reward\_func** ā€“ Rewards exact label matches. * **int\_reward\_func** ā€“ Encourages integer-only answers. * **soft\_format\_reward\_func** ā€“ Checks structure but allows minor newline mismatches. * **strict\_format\_reward\_func** ā€“ Ensures response structure matches the prompt, including newlines. * **xmlcount\_reward\_func** ā€“ Ensures exactly one of each XML tag in the response. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#using-vllm) šŸ§®Using vLLM ----------------------------------------------------------------------------------------------- You can now use [vLLM](https://github.com/vllm-project/vllm/) directly in your finetuning stack, which allows for much more throughput and allows you to finetune and do inference on the model at the same time! On 1x A100 40GB, expect 4000 tokens / s or so with Unslothā€™s dynamic 4bit quant of Llama 3.2 3B Instruct. On a 16GB Tesla T4 (free Colab GPU), you can get 300 tokens / s. We also magically removed double memory usage when loading vLLM and Unsloth together, allowing for savings of 5GB or so for Llama 3.1 8B and 3GB for Llama 3.2 3B. Unsloth could originally finetune Llama 3.3 70B Instruct in 1x 48GB GPU with Llama 3.3 70B weights taking 40GB of VRAM. If we do not remove double memory usage, then weā€™ll need >= 80GB of VRAM when loading Unsloth and vLLM together. But with Unsloth, you can still finetune and get the benefits of fast inference in one package in under 48GB of VRAM! To use fast inference, first install vllm, and instantiate Unsloth with fast\_inference: Copy pip install unsloth vllm from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-3B-Instruct", fast_inference = True, ) model.fast_generate(["Hello!"]) [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#grpo-requirement-guidelines) āœ…GRPO Requirement Guidelines -------------------------------------------------------------------------------------------------------------------------------- When youā€™re using Unsloth to do GRPO, we smartly reduce VRAM usage by over 90% when compared to standard implementations with Flash Attention 2 by using multiple tricks! On 20K context lengths for example with 8 generations per prompt, Unsloth uses only **54.3GB of VRAM for Llama 3.1 8B**, whilst standard implementations take **510.8GB (90% less for Unsloth)**. 1. For GRPO's **GPU VRAM requirements for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM. 2. Our new memory efficient linear kernels for GRPO slashes memory usage by 8x or more. This shaves 68.5GB of memory, whilst being actually faster through the help of torch.compile! 3. We leverage our smart [Unsloth gradient checkpointing](https://unsloth.ai/blog/long-context) algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves 52GB of memory. 4. Unsloth also uses the same GPU / CUDA memory space as the underlying inference engine (vLLM), unlike implementations in other packages. This shaves 16GB of memory. Metrics Unsloth Standard + FA2 Training Memory Cost (GB) 42GB 414GB GRPO Memory Cost (GB) 9.8GB 78.3GB Inference Cost (GB) 0GB 16GB Inference KV Cache for 20K context length (GB) 2.5GB 2.5GB Total Memory Usage 54.33GB (90% less) 510.8GB In typical standard GRPO implementations, you need to create 2 logits of size (8. 20K) to calculate the GRPO loss. This takes 2 \* 2 bytes \* 8 (num generations) \* 20K (context length) \* 128256 (vocabulary size) = 78.3GB in VRAM. Unsloth shaves 8x memory usage for long context GRPO, so we need only an extra 9.8GB in extra VRAM for 20K context lengths! We also need to from the KV Cache in 16bit. Llama 3.1 8B has 32 layers, and both K and V are 1024 in size. So memory usage for 20K context length = 2 \* 2 bytes \* 32 layers \* 20K context length \* 1024 = 2.5GB per batch. We would set the batch size for vLLM to 8, but we shall leave it at 1 for our calculations to save VRAM. Otherwise you will need 20GB for the KV cache. [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#unsloth-rl-3-hour-workshop-video) šŸŽ„ Unsloth RL 3 hour Workshop Video -------------------------------------------------------------------------------------------------------------------------------------------- [](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide#further-reading) šŸŽ“Further Reading --------------------------------------------------------------------------------------------------------- 1. Nathan Lambert's RLHF Book is a must! [https://rlhfbook.com/c/11-policy-gradients.html](https://rlhfbook.com/c/11-policy-gradients.html) 2. Yannic Kilcher's GRPO Youtube video is also a must! [https://www.youtube.com/watch?v=bAWV\_yrqx4w](https://www.youtube.com/watch?v=bAWV_yrqx4w) 3. We did a 3 hour workshop at AI Engineer World's Fair 2025. Slides are other material are at [https://docs.unsloth.ai/ai-engineers-2025](https://docs.unsloth.ai/ai-engineers-2025) 4. Advanced GRPO notebook via Unsloth. [https://docs.unsloth.ai/basics/reinforcement-learning-guide/tutorial-train-your-own-reasoning-model-with-grpo](https://docs.unsloth.ai/basics/reinforcement-learning-guide/tutorial-train-your-own-reasoning-model-with-grpo) 5. GRPO from a base model notebook: [https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(4B)-GRPO.ipynb](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) [PreviousDeepSeek-R1-0528: How to Run Locally](https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally) [NextTutorial: Train your own Reasoning model with GRPO](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) Last updated 1 month ago Was this helpful? --- # Magistral: How to Run & Fine-tune | Unsloth Documentation OpenAI gpt-oss & all model types now supported! **Magistral-Small-2507** is a reasoning LLM developed by Mistral AI. It excels at coding and mathematics and supports multiple languages. Magistral supports a 40k token context window and was finetuned from [**Mistral-Small-3.1**](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF) . Magistral runs perfectly well locally on a single RTX 4090 or a Mac with 16 to 24GB RAM. [Running Magistral Tutorial](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#running-magistral) [Fine-tuning Magistral](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#fine-tuning-magistral-with-unsloth) Update: Magistral-2507 new update is out as of July, 2025! Now with experimental Vision support! We worked with Mistral again with the release of Magistral. Make sure to download Mistral's official uploads or Unsloth's uploads to get the correct implementation (ie correct system prompt, correct chat template etc.) **If you're using llama.cpp, please use** `**--jinja**` **to enable the system prompt!** All uploads use Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized Mistral LLMs with minimal accuracy loss. #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#magistral-small-unsloth-dynamic-uploads) Magistral-Small **\- Unsloth Dynamic** uploads: Dynamic 2.0 GGUF (to run) Dynamic 4-bit Safetensor (to finetune/deploy) * [Magistral-Small-2507-GGUF](https://huggingface.co/unsloth/Magistral-Small-2507-GGUF) * [Magistral-Small-2506-GGUF](https://huggingface.co/unsloth/Magistral-Small-2506-GGUF) * [Magistral-Small-2507-unsloth-bnb-4bit](https://huggingface.co/unsloth/Magistral-Small-2507-unsloth-bnb-4bit) * [Magistral-Small-2506-unsloth-bnb-4bit](https://huggingface.co/unsloth/Magistral-Small-2506-unsloth-bnb-4bit) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#running-magistral) šŸ–„ļø **Running Magistral** -------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#official-recommended-settings) āš™ļø Official Recommended Settings According to Mistral AI, these are the recommended settings for inference: * **Temperature of: 0.7** * Min\_P of: 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1) * Set **top\_p to: 0.95** * Max tokens (context length): 40,960 (32K + 8K) * A 128k context window is supported, **but** performance probably degrades past **40k**. So we recommend setting the maximum length to 40k. **This is the recommended system prompt:** Copy A user will ask you to solve a task. You should first draft your thinking process (inner monologue) until you have derived the final answer. Afterwards, write a self-contained summary of your thoughts (i.e. your summary should be succinct but contain all the critical steps you needed to reach the conclusion). You should use Markdown to format your response. Write both your thoughts and summary in the same language as the task posed by the user. NEVER use \boxed{} in your response. Your thinking process must follow the template below: Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct answer. Here, provide a concise summary that reflects your reasoning and presents a clear final answer to the user. Don't mention that this is a summary. Problem: Our dynamic uploads have the '`UD`' prefix in them. Those without are not dynamic however still utilize our calibration dataset. * **Multilingual:** Magistral supports many languages including: English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi. ### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#testing-the-model) ā“Testing the model Mistral has their own vibe checking prompts which can be used to evaluate Magistral. Keep in mind these tests are based on running the full unquantized version of the model, however you could also test them on quantized versions: **Easy -** _Make sure they always work_ Copy prompt_1 = 'How many "r" are in strawberry?' prompt_2 = 'John is one of 4 children. The first sister is 4 years old. Next year, the second sister will be twice as old as the first sister. The third sister is two years older than the second sister. The third sister is half the ago of her older brother. How old is John?' prompt_3 = '9.11 and 9.8, which is greater?' **Medium** - _Should most of the time be correct_ Copy prompt_4 = "Think about 5 random numbers. Verify if you can combine them with addition, multiplication, subtraction or division to 133" prompt_5 = "Write 4 sentences, each with at least 8 words. Now make absolutely sure that every sentence has exactly one word less than the previous sentence." prompt_6 = "If it takes 30 minutes to dry 12 T-shirts in the sun, how long does it take to dry 33 T-shirts?" **Hard** \- _Should sometimes get them right_ Copy prompt_7 = "Pick 5 random words each with at least 10 letters. Print them out. Reverse each word and print it out. Then extract letters that are alphabetically sorted smaller than "g" and print them. Do not use code." prompt_8 = "Exactly how many days ago did the French Revolution start? Today is June 4th, 2025." **We provide some** [**example outputs**](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#sample-outputs) **at the end of the blog.** [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#tutorial-how-to-run-magistral-in-ollama) šŸ¦™ Tutorial: How to Run Magistral in Ollama ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. Install `ollama` if you haven't already! Copy apt-get update apt-get install pciutils -y curl -fsSL https://ollama.com/install.sh | sh 1. Run the model with our dynamic quant. We did not set the context length automatically, so it will just use Ollama's default set context length. Note you can call `ollama serve &`in another terminal if it fails! We include all suggested parameters (temperature etc) in `params` in our Hugging Face upload! 2. Also Magistral supports 40K context lengths, so best to enable [**KV cache quantization**](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache) . We use 8bit quantization which saves 50% memory usage. You can also try `"q4_0"` or `"q8_0"` 3. **Ollama also sets the default context length to 4096**, as [mentioned here](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size) . Use `OLLAMA_CONTEXT_LENGTH=8192` to change it to 8192. Magistral supports up to 128K, but 40K (40960) is tested most. Copy export OLLAMA_KV_CACHE_TYPE="f16" OLLAMA_CONTEXT_LENGTH=8192 ollama serve & ollama run hf.co/unsloth/Magistral-Small-2507-GGUF:UD-Q4_K_XL [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#tutorial-how-to-run-llama-4-scout-in-llama.cpp) šŸ“– Tutorial: How to Run Magistral in llama.cpp ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp) . You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. Copy apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli cp llama.cpp/build/bin/llama-* llama.cpp 1. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` Copy ./llama.cpp/llama-cli -hf unsloth/Magistral-Small-2507-GGUF:UD-Q4_K_XL --jinja --temp 0.7 --top-k -1 --top-p 0.95 -ngl 99 In llama.cpp, please use `--jinja` to enable the system prompt! 1. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q4\_K\_XL, (Unsloth Dynamic), Q4\_K\_M, or other quantized versions (like BF16 full precision). Copy # !pip install huggingface_hub hf_transfer import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import snapshot_download snapshot_download( repo_id = "unsloth/Magistral-Small-2507-GGUF", local_dir = "unsloth/Magistral-Small-2507-GGUF", allow_patterns = ["*UD-Q4_K_XL*"], # For UD-Q4_K_XL ) 1. Run the model. 2. Edit `--threads -1` for the maximum CPU threads, `--ctx-size 40960` for context length (Magistral supports 40K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We also use 8bit quantization for the K cache to reduce memory usage. 3. For conversation mode: Copy ./llama.cpp/llama-cli \ --model unsloth/Magistral-Small-2507-GGUF/Magistral-Small-2507-UD-Q4_K_XL.gguf \ --threads -1 \ --ctx-size 40960 \ --cache-type-k f16 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 2 \ --temp 0.7 \ --repeat-penalty 1.0 \ --min-p 0.01 \ --top-k -1 \ --top-p 0.95 \ --jinja Remember to remove since Magistral auto adds a [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#sample-outputs) Sample outputs ------------------------------------------------------------------------------------------------------------------------------------------------ How many "r" are in strawberry? \[Correct answer = 3\][](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#how-many-r-are-in-strawberry-correct-answer-3) Copy Alright, I have a problem to solve: "How many 'r's are in strawberry?" First, I need to understand the problem. It's asking for the count of the letter 'r' in the word "strawberry". Step 1: Write down the word and look at each letter. The word is: s t r a w b e r r y Step 2: Identify the 'r's in the word. Let's go through each letter one by one: 1. s - not an r 2. t - not an r 3. r - that's one r 4. a - not an r 5. w - not an r 6. b - not an r 7. e - not an r 8. r - that's another r 9. r - and another one 10. y - not an r So, I see the letter 'r' appears at positions 3, 8, and 9. Step 3: Count the number of 'r's. From the above, there are three 'r's in "strawberry". But wait, I should double-check to make sure I didn't miss anything or count incorrectly. Let me write the word again: strawberry. Looking at it: - The third letter is 'r' (s-t-r) - Then later, we have 'e-r-r' which is two more 'r's. Yes, that's three 'r's in total. But just to be thorough, sometimes letters can be hidden or the word might have silent letters, but in this case, all letters are pronounced in "strawberry", and 'r' is clearly pronounced in the third, eighth, and ninth positions. So, the count is indeed three. But wait, is it possible that the 'r's are not all pronounced? For example, in some accents or dialects, letters might be silent or pronounced differently. However, in standard pronunciation, all these 'r's are pronounced. But the question is about the number of 'r's in the word, not about pronunciation. It's about the written form. So, regardless of pronunciation, we're counting the letters in the word "strawberry". Let me write it out again and count: s (1) t (2) r (3) - first r a (4) w (5) b (6) e (7) r (8) - second r r (9) - third r y (10) Yes, three 'r's. But just to be sure, maybe I should think about the spelling. Sometimes letters can be part of digraphs or other combinations, but in "strawberry", the 'r's are all standalone letters. Let me think about the spelling breakdown: - "straw" has one 'r' (the third letter) - "berry" has two 'r's (the second and third letters in "berry", but in the full word, it's the eighth and ninth letters) So, "straw" has 1 'r', "berry" has 2 'r's, totaling 3 'r's in "strawberry". I think I'm confident now. The word "strawberry" has three 'r's. But before finalizing, I recall that sometimes letters can be repeated or there might be a trick. Let me see: The word is: s-t-r-a-w-b-e-r-r-y. Counting the 'r's: 1. The 'r' in the third position. 2. The 'r' in the eighth position. 3. The 'r' in the ninth position. No other 'r's are present. Therefore, the total number of 'r's in "strawberry" is three. But just to cross-verify, perhaps I can write the word and highlight the 'r's: s t r a w b e r r y r r r Yes, three 'r's. Is there any chance I'm missing something? Maybe considering uppercase and lowercase? But the problem is case-insensitive since it's about the letter 'r', not 'R'. The word is given in lowercase, but even if it were uppercase, 'R' is the same letter as 'r' in terms of counting. So, final count is three. But wait, is "strawberry" spelled correctly? Let me check the spelling. I think it's s-t-r-a-w-b-e-r-r-y. Yes, that seems correct. Therefore, the number of 'r's is indeed three. But just to be extra sure, maybe I can think of the word in a different way. For example, breaking it down: "straw" + "berry" = "strawberry" "straw" has one 'r' (the third letter: s-t-r) "berry" has two 'r's (the second and third letters: b-e-r-r-y, but actually, 'berry' is b-e-r-r-y, so 'r' is the third and fourth letters? Wait, let's see: 'berry' is b-e-r-r-y. Wait, no: b (1) e (2) r (3) r (4) y (5) So, 'berry' has 'r's at positions 3 and 4, which are two 'r's. 'straw' is s-t-r-a-w, so 'r' is at position 3, one 'r'. Combined, 'strawberry' is 'straw' + 'berry' = s-t-r-a-w-b-e-r-r-y, which has 'r's at positions 3, 3+5+1=9? Wait, no, let's index: 'straw' is positions 1-5 (s-t-r-a-w) 'berry' is positions 6-10 (b-e-r-r-y) So 'r's in 'straw' are at position 3. 'r's in 'berry' are at positions 8 and 9 (since 'berry' starts at position 6: b(6), e(7), r(8), r(9), y(10)). Thus, total 'r's are at positions 3, 8, and 9: three 'r's. Yes, consistent again. Is there any other way to approach this? Maybe counting the total letters and seeing how many are 'r's. "strawberry" has 10 letters: s, t, r, a, w, b, e, r, r, y. Counting 'r's: positions 3, 8, 9 ā†’ 3 'r's. Alternatively, I could write out the word and replace non-'r's with another character to see how many 'r's are left. Original: s t r a w b e r r y Replace non-r: x x r x x x x r r x Now count 'r's: positions 2 (but original was 3rd letter), wait no, in the replaced string, 'r's are at positions 3 (original position 3), and then after replacing others, the next 'r's are at positions... Wait, let's see: Original positions: 1: s ā†’ x 2: t ā†’ x 3: r ā†’ r 4: a ā†’ x 5: w ā†’ x 6: b ā†’ x 7: e ā†’ x 8: r ā†’ r 9: r ā†’ r 10: y ā†’ x So replaced string: x x r x x x x r r x Now count 'r's: positions 3, 8, 9 in original correspond to positions 3, 8, 9 in replaced (but replaced has x's in other positions, so 'r's are still at 3,8,9). Wait, in replaced string, 'r's are at positions 3,8,9, but the length is still 10, so positions are same as original. So count is same: 3 'r's. Another method: write the word and cross out non-'r's. Original: s t r a w b e r r y Cross out non-r: x x r x x x x r r x Now count 'r's: there are three 'r's remaining. All methods confirm that there are three 'r's in "strawberry". But just to think outside the box, is there any ambiguity in the question? For example, is it asking for uppercase 'R' or lowercase 'r'? The problem says "r", lowercase, so we're counting lowercase 'r's. In the word "strawberry", all letters are lowercase (assuming the input is in lowercase; if it were mixed case, we'd have to consider, but since it's given as "strawberry", which is all lowercase, we're good). Also, is it possible that some 'r's are part of a digraph or trigraph? For example, in some languages, 'rr' might be considered a single sound or letter, but in English orthography, 'rr' is two separate 'r's. So in "strawberry", the 'rr' is two 'r's. Therefore, the count remains three. Is there any other interpretation? Maybe considering that sometimes letters can represent different sounds or that 'r' might be silent, but as mentioned earlier, the question is about the written letter 'r', not pronunciation. Final confirmation: let's write the word and count 'r's aloud: s (no), t (no), r (1), a (no), w (no), b (no), e (no), r (2), r (3), y (no). Yes, three 'r's. Therefore, the number of 'r's in "strawberry" is three. But before finalizing, let me think if there's any trick or if I'm missing something obvious. For example, sometimes questions like this have hidden complexities, like counting 'r's in the word's spelling or pronunciation, or considering that 'r' might appear in other forms, like in 'rr' which is two 'r's together, but we've already accounted for that. Alternatively, is the question about the number of times the letter 'r' appears in the spelling of "strawberry"? Yes, that's how I interpreted it, and my count is three. Is there any chance the word is spelled differently? For example, in some countries or dialects, is "strawberry" spelled differently? In American English and British English, it's spelled the same way: s-t-r-a-w-b-e-r-r-y. Let me check the spelling again: s, t, r, a, w, b, e, r, r, y. Yes, that's correct. Therefore, the number of 'r's is indeed three. But just to be absolutely sure, perhaps I can think of the word in a sentence or see it written out: Example sentence: "I ate a strawberry." The word is "strawberry", and in it, the 'r's are in the middle and at the end. Looking at the spelling: s-t-r-a-w-b-e-r-r-y. Counting: 1. s 2. t 3. r (1) 4. a 5. w 6. b 7. e 8. r (2) 9. r (3) 10. y Yes, three 'r's. I think I've exhausted all possible ways to verify this count. Therefore, the number of 'r's in "strawberry" is three. But wait, the user might have intended the question differently. Could it be about the number of times the sound 'r' appears in pronunciation? For example, sometimes letters are silent or combined in pronunciation, but the question asks for the count of the letter 'r', not the sound. In pronunciation, "strawberry" is /ĖˆstroŹŠbɛri/ or similar, where the 'r's are pronounced as in the spelling. But since the question is about the letter count, not phonemes, we stick with the written form. Therefore, the count remains three. Is there any other interpretation? For example, counting uppercase 'R's if the word were capitalized? But the word is given in lowercase, and the question asks for 'r', lowercase, so it's consistent. Final answer: there are three 'r's in "strawberry". But just to double-check, perhaps I can list the letters with their positions: Position: 1 2 3 4 5 6 7 8 9 10 Letter: s t r a w b e r r y Now, pick out the 'r's: positions 3, 8, 9. That's three 'r's. Alternatively, write the word and circle the 'r's: Strawberry: s t r a w b e r r y The 'r's are in positions 3, 8, and 9. So, three 'r's. No other interpretations seem plausible. Therefore, the answer is three. But before finalizing, let me think if there's any ambiguity in the word itself. Could "strawberry" be misspelled or is there another spelling variant? In English, "strawberry" is consistently spelled with two 'r's at the end: "berry" is spelled with two 'r's (b-e-r-r-y), and "straw" has one 'r' (s-t-r-a-w). So total is three 'r's. Is there any other word that might be confused with "strawberry"? For example, "strawberi" or similar, but no, "strawberry" is the correct spelling with three 'r's. Therefore, the final count is three. But just to be thorough, perhaps I can think of the word in another language. For example, in Spanish, "strawberry" is "fresas" or "fresa", which don't have 'r's, but the question is about the English word "strawberry". In English, as confirmed, there are three 'r's in "strawberry". Is there any other angle? For example, if the word were in uppercase, like "STRAWBERRY", the count would still be the same since 'R' is the uppercase counterpart of 'r'. But the problem specifies 'r', lowercase, and the word is given in lowercase, so it's consistent. Therefore, the number of 'r's in "strawberry" is three. But just to think differently, perhaps the word is being considered in a different context, like in a code or cipher where letters might represent something else. But the problem is straightforward: count the number of times the letter 'r' appears in the word "strawberry". So, after all this careful consideration, the answer is clearly three. But before finalizing, let me think if I've made any mistake in counting. Let me recount: Word: s t r a w b e r r y Positions: 1: s 2: t 3: r (1) 4: a 5: w 6: b 7: e 8: r (2) 9: r (3) 10: y Yes, three 'r's at positions 3, 8, and 9. No mistake in counting. Therefore, the final answer is that there are three 'r's in the word "strawberry". But wait, the user might expect a different answer if they're considering something else, but based on the straightforward interpretation, it's three. Perhaps the user expects us to count only the distinct 'r's or something, but all 'r's are identical letters in this context. Alternatively, maybe the question is a trick question implying that 'r's are pronounced differently or not at all, but again, the question is about the count of the letter 'r' in the spelling, not pronunciation. Thus, the answer is three. But just to be sure, let me think of the word "strawberry" and say it aloud: "strawberry". The 'r's are clearly pronounced as in the spelling. No hidden complexities here. The count is three. Final verification: write the word and count 'r's: s t r a w b e r r y 'r' appears in the 3rd, 8th, and 9th positions. That's three times. Therefore, the number of 'r's in "strawberry" is three. But before concluding, let me think if there's any other interpretation where the answer could be different. For example, if the word were written in a different font or if there were any hidden letters, but in standard writing, it's clear. Alternatively, if the word were in a different language with different spelling, but "strawberry" is the English word, and in English, it's spelled with three 'r's. Another thought: is it possible that the 'w' or another letter is sometimes considered an 'r'? No, in English alphabet, 'w' is a separate letter from 'r'. No other interpretations seem valid. The answer is three. But just to be extra cautious, perhaps the word is misspelled in the question, but assuming it's correctly spelled as "strawberry", the count is three. Therefore, after thorough consideration, the number of 'r's in "strawberry" is three. But wait, looking back at the word: strawberry. Let me write it and count: s, t, r, a, w, b, e, r, r, y. Let me list them: 1. s 2. t 3. r 4. a 5. w 6. b 7. e 8. r 9. r 10. y So, 'r's are at positions 3, 8, and 9. That's three 'r's. Is there any chance I'm miscounting positions? Let me recount: 1: s 2: t 3: r (1st r) 4: a 5: w 6: b 7: e 8: r (2nd r) 9: r (3rd r) 10: y Yes, three 'r's. Therefore, the final answer is that there are three 'r's in the word "strawberry". But just to ensure completeness, perhaps the word is being considered in uppercase or another case, but since the question is about 'r' (lowercase) and the word is given in lowercase, it's consistent. Alternatively, if the word were "Strawberry" (with a capital S), does that affect the count? The question asks for 'r' (lowercase), and in "Strawberry", the 'S' is uppercase, but the 'r's are lowercase (assuming standard title case where only the first letter is capitalized). But since the question specifies 'r' (lowercase), and the 'r's in "Strawberry" are lowercase (unless it's all caps or something), but the given word is "strawberry" all lowercase, so it's three 'r's. But even if it were "Strawberry", the 'r's are still lowercase (assuming standard capitalization), so count remains the same. But since the given word is "strawberry" in all lowercase, the count is three. Therefore, the number of 'r's in "strawberry" is three. But just to think of edge cases, what if the word had a silent 'r'? But as discussed earlier, the question is about the written letter, not pronunciation. Alternatively, if the word were in a different font where 'r' looks like something else, but in standard fonts, it's clear. No other possibilities seem to alter the count. Thus, the final answer is three. But before finalizing, let me think if the word could be written differently, like with a different spelling variant. For example, in some dialects or historical contexts, words might be spelled differently, but modern English spells "strawberry" with three 'r's (one in 'straw' and two in 'berry'). Therefore, the count is three. After all this careful consideration and multiple verification methods, I am confident that the number of 'r's in "strawberry" is three. But to ensure that I haven't missed anything, perhaps I can think of the word's origin or related words. For example, "straw" has one 'r', "berry" has two 'r's, combining gives three 'r's in "strawberry". This also confirms the count. Therefore, the answer is three. However, sometimes such questions are trick questions expecting a different answer. For example, if the question was about the number of times the letter 'r' appears in the word "strawberry" when spelled out, but that seems convoluted and unlikely. Alternatively, counting the number of closed areas in the letter 'r' when written in a certain font, but that's not relevant here. Another possibility is that the question expects us to count the number of times the sound 'r' appears in pronunciation, but again, the question asks for the count of the letter 'r', not sounds. In pronunciation, "strawberry" has the 'r' sound in "straw" and in "berry", but since 'berry' has two 'r's in spelling, both are pronounced in standard English, so the count would still relate to the spelling, which is three 'r's. But since the question is about the letter count in writing, not pronunciation, the answer remains three. Therefore, after all this thorough checking, the number of 'r's in "strawberry" is three. But just to be sure, perhaps I can ask myself: if I were to type the word "strawberry" and count how many times I press the 'r' key, it would be three times (assuming I don't make any typos while typing). Thus, confirming again: three 'r's. Therefore, the final answer is: There are 3 'r's in the word "strawberry". But to present it neatly, perhaps as a count: The word "strawberry" spelled out: s-t-r-a-w-b-e-r-r-y. Counting 'r's: 1. The third letter: r 2. The eighth letter: r 3. The ninth letter: r Total: 3 'r's. Therefore, the number of 'r's in "strawberry" is 3. But to ensure completeness, let me think of the word's length: "strawberry" has 10 letters. The positions are: 1: s 2: t 3: r (1st r) 4: a 5: w 6: b 7: e 8: r (2nd r) 9: r (3rd r) 10: y Thus, three 'r's at positions 3, 8, and 9. No other 'r's present. Therefore, the final count is three. But to present it in a structured way: Word: s t r a w b e r r y Positions of 'r': 3, 8, 9 Number of 'r's: 3 Thus, the answer is 3. But just to think if there's any other interpretation where the answer could be different, perhaps considering that in some fonts, 'r' might look like another letter, but in standard fonts, it's distinct. Alternatively, if the word were in a different language where 'r' has a different representation, but since it's given in English, we proceed with English alphabet rules. Therefore, the count is three. After all this exhaustive checking, I conclude that the number of 'r's in "strawberry" is three. But to present it neatly in the summary, here's the concise version: The word "strawberry" has the following letters: s, t, r, a, w, b, e, r, r, y. Counting the 'r's, we find them at the 3rd, 8th, and 9th positions, totaling three 'r's. Therefore, there are **3 'r's** in the word "strawberry". But wait, looking back at the word: strawberry. Let me write it and count: s (1), t (2), r (3), a (4), w (5), b (6), e (7), r (8), r (9), y (10). So 'r's at positions 3, 8, 9: that's three 'r's. Is there any possibility that one of these 'r's is not counted? For example, if one of them were part of a digraph or silent, but in standard English spelling and pronunciation, all these 'r's are present and pronounced. Therefore, the count is indeed three. But just to think differently, perhaps the question expects us to count only the distinct positions where 'r' appears, but since all 'r's are the same letter, the count is still three. Alternatively, if the question was about the number of times the letter 'r' is written when writing the word "strawberry", that would also be three times. Thus, the answer is three. But to present it clearly: We analyze the word "strawberry" by listing its letters and identifying the positions of 'r': Letters: s, t, r, a, w, b, e, r, r, y Positions of 'r': 3, 8, 9 Count of 'r's: 3 Therefore, there are three 'r's in "strawberry". But to ensure that this is the expected answer, perhaps the user expects us to consider that 'rr' is a double 'r', but even then, it's two separate 'r's next to each other, totaling three 'r's in the word (one from 'straw' and two from 'berry'). Thus, no alternative interpretation changes the count. Therefore, the final answer is that there are three 'r's in "strawberry". But to summarize neatly, here's the concise version: The word "strawberry" is spelled with the following letters in order: s, t, r, a, w, b, e, r, r, y. The letter 'r' appears at the 3rd, 8th, and 9th positions, giving a total count of three 'r's. Thus, the number of 'r's in "strawberry" is: \boxed{3} **Summary:** The word "strawberry" is analyzed by listing its letters and identifying the positions of the letter 'r'. The 'r's are found at the 3rd, 8th, and 9th positions, resulting in a total count of three 'r's. Therefore, there are **3 'r's** in the word "strawberry". \[\ \boxed{3}\ \] Exactly how many days ago did the French Revolution start? Today is June 4th, 2025. \[Correct answer = 86,157 days\][](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#exactly-how-many-days-ago-did-the-french-revolution-start-today-is-june-4th-2025.-correct-answer-86) Copy Alright, let's tackle this problem step by step. First, I need to know the exact start date of the French Revolution. From my knowledge, the French Revolution is generally considered to have started on July 14, 1789, which is the date of the Storming of the Bastille. However, some historians argue that the revolution began earlier with events like the Estates-General meeting on May 5, 1789, or even earlier with financial and political crises. But for simplicity, I'll use July 14, 1789, as the start date since it's widely recognized as the symbolic beginning of the revolution. Now, the problem gives today's date as June 4, 2025. I need to calculate the number of days between July 14, 1789, and June 4, 2025. ### Step 1: Calculate the total number of years between the two dates. 2025 - 1789 = 236 years. But we need to be careful here because not all years have the same number of days (leap years have an extra day), and we're not counting full years from July 14 to July 14. ### Step 2: Break down the calculation into years, months, and days. First, let's calculate the number of full years between July 14, 1789, and July 14, 2025. That's 2025 - 1789 = 236 years. But since we're starting from July 14, 1789, and ending on June 4, 2025, which is before July 14, 2025, we have 235 full years (from July 14, 1789, to July 14, 2024) plus the days from July 14, 2024, to June 4, 2025. Wait, that doesn't seem right. Actually, it's better to think of it as: From July 14, 1789, to July 14, 2024, is 235 years (because 2024 - 1789 = 235). Then from July 14, 2024, to June 4, 2025, is the remaining time. But let's double-check: From July 14, 1789, to July 14, 2024, is 235 years. From July 14, 2024, to June 4, 2025, is less than a year, so we can calculate the days in this period separately. But actually, to make it easier, we can calculate the total days from July 14, 1789, to June 4, 2025, by calculating the days from July 14, 1789, to July 14, 2024 (235 years) plus the days from July 14, 2024, to June 4, 2025. But wait, July 14, 2024, to June 4, 2025, is not correct because June 4, 2025, is before July 14, 2025. So the correct breakdown is: From July 14, 1789, to July 14, 2024: 235 years. From July 14, 2024, to June 4, 2025: but this is negative because June 4 is before July 14. So it's better to think of it as: From July 14, 1789, to June 4, 2024: 234 full years plus the days from July 14 to June 4 in the next year (but this seems confusing). Perhaps a better approach is to calculate the number of days from July 14, 1789, to June 4, 2025. First, let's calculate the number of days from July 14, 1789, to July 14, 2024. That's 235 years. Now, how many days are there in 235 years? We need to account for leap years. A year has 365 days, and a leap year has 366 days. Leap years occur every 4 years, but century years are not leap years unless they're divisible by 400. First, calculate the number of leap years between 1789 and 2024 (not including 2024 if it's a leap year, but since we're going up to July 14, 2024, which is before February 29, the leap day in 2024 wouldn't affect the count if we're counting full years, but actually, since we're counting from July 14, 1789, to July 14, 2024, and 2024 is a leap year, but since we're not crossing into the next year's February, the leap day in 2024 doesn't affect this period because the extra day is in February, and we're starting after February (July is after February, so the leap day is already accounted for in the year's total days). Wait, no. The leap day is February 29. If we're counting from July 14, 1789, to July 14, 2024, then the leap day in each leap year between 1789 and 2024 would have been included in the previous year's count, but since we're starting after February, the leap day in the starting year (1789) doesn't affect us (since we're starting after February 29). Similarly, for the ending year (2024), since we're ending on July 14, which is after February 29, the leap day in 2024 is included in the count for that year (since we're including the entire year up to July 14, but actually, we're counting full years from July 14 to July 14, so each full year contributes either 365 or 366 days depending on whether it's a leap year. But actually, since we're counting the number of days between two dates, and both dates are after February in their respective years, the leap day affects the count only if the period includes February 29. But since we're counting full years from July 14 to July 14, and July is after February, the leap day is included in the full year count. Wait, no. If we're counting from July 14, 1789, to July 14, 2024, that's 235 years. Each year in this period has 365 days, except for leap years which have 366 days. Number of leap years between 1789 and 2023 (since 2024's leap day is after July 14, but we're counting full years up to July 14, 2024, which is before February 29, 2025... wait no, 2024's leap day is February 29, 2024, and we're up to July 14, 2024, which is after February 29, so the leap day in 2024 is included in our count if we're counting full years up to July 14, 2024. Wait, no, the full years are from July 14, 1789, to July 14, 2024. That's 2024 - 1789 = 235 years, but since we're starting at July 14, 1789, and ending at July 14, 2024, that's exactly 235 years. Now, how many leap years are there in this period? A year is a leap year if: 1. It's divisible by 4, 2. It's not divisible by 100, unless it's also divisible by 400. But since we're counting from July 14, the leap day (February 29) is included in the year's count if the year is a leap year, because we're including the entire year from July 14 to July 14 (which is equivalent to counting from July 14 to July 14 of the next year, but actually, no, from July 14, 1789, to July 14, 1790, is one year, which may include February 29 if 1790 is a leap year... wait, no, 1790 is not divisible by 4, so it's not a leap year. Wait, the period from July 14, 1789, to July 14, 1790, is one year, and it includes February 29, 1790? No, 1790 is not a leap year (1790 is not divisible by 4). The leap day is February 29 in a leap year, but since our period starts after February in 1789, and ends before February in 1790... wait no, our period is from July 14, 1789, to July 14, 1790, which includes February 29, 1790? Wait, no, 1789 to 1790 is not a leap year, because 1789 to 1790 is one year, and the leap day would be in February 1790 if 1790 were a leap year, but it's not (1790 is not divisible by 4). Wait, perhaps it's easier to think that for each full year from July 14 to July 14, the number of days is 365, plus 1 if the year is a leap year and the period includes February 29. But since our period starts after February in the starting year and ends after February in the ending year, the leap day is included in the count for leap years. So, the number of leap years between 1789 and 2024 inclusive (since 2024 is a leap year, and we're counting up to July 14, 2024, which is after February 29, 2024, so the leap day is included). Number of years: 2024 - 1789 + 1 = 236 years. Wait, no, from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years (because at July 14, 1789, it's the start, and at July 14, 2024, it's after 235 years). Number of leap years in this period: The first year is 1789 (not a leap year, since 1789 is not divisible by 4). The last year is 2024 (which is a leap year, divisible by 4 and not by 100 unless divisible by 400, but 2024 is divisible by 4 and not by 100). The number of leap years between 1789 and 2024 inclusive is the number of years divisible by 4 in this range, minus those divisible by 100 but not by 400. First, number of years divisible by 4 between 1789 and 2024 inclusive: The first leap year after 1789 is 1792 (since 1789 + 3 = 1792, which is divisible by 4). The last leap year before 2024 is 2024 itself (which is divisible by 4). Number of leap years = (2024 - 1792) / 4 + 1 = (2024 - 1792) = 232, 232 / 4 = 58, +1 = 59. But wait, 2024 is included, so it's correct: (2024 - 1792) / 4 + 1 = (232)/4 + 1 = 58 + 1 = 59. Now, subtract the century years that are not leap years (i.e., divisible by 100 but not by 400). Between 1789 and 2024, the century years are 1800, 1900, and 2000. Now, which of these are not leap years? - 1800: divisible by 100 but not by 400 (since 1800 / 400 = 4.5, not integer) ā†’ not a leap year. - 1900: same, divisible by 100 but not by 400 ā†’ not a leap year. - 2000: divisible by 400 (2000 / 400 = 5) ā†’ is a leap year. So, we have two century years that are not leap years: 1800 and 1900. Thus, total leap years = 59 - 2 = 57. So, total days for the full years = 235 * 365 + number of leap years = 235 * 365 + 57. Let's calculate that: 235 * 365: 200 * 365 = 73000 35 * 365 = 35 * 300 + 35 * 65 = 10500 + 2275 = 12775 Total: 73000 + 12775 = 85775 Plus leap days: 85775 + 57 = 85832 days for the full years. Now, we need to add the days from July 14, 2024, to June 4, 2025. First, calculate the days remaining in 2024 after July 14: July has 31 days, so from July 14 to July 31 is 31 - 14 = 17 days (including July 14? Wait, no, from July 14 to July 15 is 1 day, so July 14 to July 31 is 31 - 14 = 17 days, but we need to include July 14? Actually, the number of days from July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days. But usually, when we say "from A to B", we count the days starting from A and including B, so from July 14 to July 14 is 1 day, from July 14 to July 15 is 2 days, etc. So from July 14 to July 31 is 31 - 14 + 1 = 18 days. But in calculating intervals, sometimes it's exclusive. The standard way is that the number of days between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end. Here, we want the number of days from July 14, 2024, up to but not including July 14, 2025, but our end date is June 4, 2025, which is before July 14, 2025. Wait, no, we're calculating the days from July 14, 2024, to June 4, 2025. Let's break it down: 1. From July 14, 2024, to December 31, 2024. 2. From January 1, 2025, to June 4, 2025. First part: July 14 to December 31, 2024. Number of days remaining in July after the 14th: 31 - 14 = 17 (but actually, it's 31 - 14 = 17, but that's from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including July 14). Wait, no, if we're counting days from July 14 (inclusive) to July 31 (inclusive), that's 31 - 14 + 1 = 18 days. But in calculating the interval between two dates, it's often (end - start) + 1 if both are included. Here, we want the number of days from July 14 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 2024) + 1? Wait, no, better to calculate it directly. Let's calculate the days remaining in 2024 after July 14: July: 31 - 14 = 17 (but this is from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 18 days (including July 14). But actually, if we're counting days starting from July 14, then: - July: 31 - 14 = 17 (from July 15 to July 31), plus July 14 itself is 18 days in July. But that can't be right because July has only 31 days. Wait, if we include July 14, then the days are July 14 to July 31, which is 31 - 14 + 1 = 18 days. Similarly, for August: 31 days September: 30 October: 31 November: 30 December: 31 Total for August to December: 31 + 30 + 31 + 30 + 31 = 153 days Plus July: 18 Total for July 14 to Dec 31, 2024: 18 + 153 = 171 days Wait, let's verify: From July 14 to July 31: 31 - 14 + 1 = 18 August: 31 September: 30 October: 31 November: 30 December: 31 Total: 18 + 31 + 30 + 31 + 30 + 31 = 18 + 31 = 49; +30=79; +31=110; +30=140; +31=171. Yes, correct. Now, from January 1, 2025, to June 4, 2025. Is 2025 a leap year? 2025 is not divisible by 4, so no. Months: January: 31 February: 28 (not leap year) March: 31 April: 30 May: 31 June: up to 4th is 4 Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days. So total days from July 14, 2024, to June 4, 2025: 171 (remaining in 2024) + 155 (in 2025) = 326 days. Wait, but we have to be careful with the counting. Are we including both July 14, 2024, and June 4, 2025? If we're calculating the number of days between two dates including both endpoints, then the calculation is correct. But in calculating the difference between two dates, it's often exclusive of the start date. The problem asks "how many days ago", which typically means counting from today backwards, not including today. But here, we're calculating the days from the start of the revolution to today, so we should include both the start date and today in the count. But in the initial problem, it's asking for how many days ago the French Revolution started, which is the number of days from June 4, 2025, back to July 14, 1789. This would be (June 4, 2025 - July 14, 1789) minus 1 if we're counting days ago (since today is not ago from today). But usually, the number of days between two dates including both is (end - start) + 1. But "days ago" would be the duration from the start to today, which is today's date minus start date, not including the start date if we're counting days after. Wait, no, if the revolution started on day X, then the number of days ago it started is (today - X), not including X itself in the count. But in our calculation, we have: From July 14, 1789, to June 4, 2025. If we include both endpoints, it's (June 4, 2025 - July 14, 1789) + 1. But "days ago" would be (today - start date), which is today's date minus start date minus 1 if not including the start date. But actually, if the revolution started on July 14, 1789, and today is June 4, 2025, then the number of days ago it started is the number of days between July 14, 1789, and June 4, 2025, minus one if we're not counting the start day. But usually, "how many days ago" refers to the time elapsed, which is (end - start), where end is today, start is the event date. So it's (June 4, 2025 - July 14, 1789), which would be the number of days between the two dates, not including the start day if we're counting the days after. But in our calculation, we did full years (July 14, 1789 to July 14, 2024) which is 235 years, and then from July 14, 2024 to June 4, 2025. But actually, to find the number of days ago, we should calculate the difference between June 4, 2025, and July 14, 1789, which is (June 4, 2025 - July 14, 1789). But in our initial approach, we calculated the days from July 14, 1789, to June 4, 2025, which is the same as (June 4, 2025 - July 14, 1789), and then we can add 1 if we want to include both endpoints. But for "days ago", we don't include the start day, so it's just (June 4, 2025 - July 14, 1789). So our calculation is correct as is: 235 years' worth of days plus the remaining days. But let's think differently: calculate the total days from July 14, 1789, to June 4, 2025. Alternative approach: Calculate the number of full years and the remaining days. From July 14, 1789, to July 14, 2024: 2024 - 1789 = 235 years. Number of leap years in this period: as before, 57. Total days for full years: 235 * 365 + 57 = 85775 + 57 = 85832. Now, days from July 14, 2024, to June 4, 2025. As calculated earlier: 326 days. Total days: 85832 (full years) + 326 (remaining) = 86158 days. But wait, does this count include both July 14, 1789, and June 4, 2025? If so, then to get the number of days between them (excluding the start day), we'd subtract 1. But in our calculation, we've included both endpoints (since we included July 14, 2024, in the remaining days calculation by doing July 14 to July 31 as 18 days, which includes July 14). But actually, in the remaining days calculation: From July 14, 2024, to June 4, 2025: We calculated July 14 to Dec 31, 2024: 171 days (including July 14) Jan 1 to June 4, 2025: 155 days (including Jan 1) Total: 171 + 155 = 326 days, which includes both July 14, 2024, and June 4, 2025. Similarly, the full years from July 14, 1789, to July 14, 2024, include July 14, 1789, and July 14, 2024 (but July 14, 2024, is already included in the remaining days, so we have double-counted July 14, 2024). Wait, no, the full years are from July 14, 1789 (inclusive) to July 14, 2024 (exclusive? Or inclusive?). Actually, the period from July 14, 1789, to July 14, 2024, includes July 14, 1789, and July 14, 2024, if we're counting inclusively. But in terms of years, it's 235 years from July 14, 1789, to July 14, 2024 (since at July 14, 2024, it's been exactly 235 years since July 14, 1789). But in our days calculation, the full years contribute 235 years' worth of days, where each year is from July 14 to July 14 of the next year. But actually, from July 14, 1789, to July 14, 1790, is one year, which has 365 or 366 days depending on whether it's a leap year. But since the year starts on July 14, the leap day (February 29) is included in that year if the year is a leap year. But our initial calculation of leap years assumed calendar years (January to December), but our period is July to July. So we need to recalculate the number of leap years in the period from July 14, 1789, to July 14, 2024. A year Y is a leap year if it's divisible by 4, but not by 100 unless also by 400. But since our year period starts in July, the leap day (February 29) is included in the year if Y is a leap year (because February 29 is before July in the same year). Wait, no: the period from July 14, Y to July 14, Y+1 includes February of Y+1, not Y. Wait, no: From July 14, 2023, to July 14, 2024: this includes February 29, 2024, if 2024 is a leap year (which it is). So the period from July 14, Y to July 14, Y+1 includes February of Y+1. Thus, for the period from July 14, 1789, to July 14, 2024, the leap years are those Y where Y+1 is a leap year (i.e., Y+1 is divisible by 4, etc.). Wait, no: the year from July 14, Y to July 14, Y+1 is a leap period if Y+1 is a leap year (i.e., if February 29 falls within this period, which it does if Y+1 is a leap year). But actually, the period from July 14, Y to July 14, Y+1 includes February of Y+1, so the extra day is present if Y+1 is a leap year. Thus, the number of leap periods in 235 years is equal to the number of leap years in Y+1 for Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024, and we're checking if 2024 is a leap year). But Y ranges from 1789 to 2023 (since the periods are 1789-1790, ..., 2023-2024), so Y+1 ranges from 1790 to 2024. Number of leap years between 1790 and 2024 inclusive. Number of years divisible by 4 in 1790..2024: First divisible by 4 is 1792 (since 1790/4 = 447.5, so next is 1792). Last is 2024. Number of terms = (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59. Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's a leap year). So subtract 2: 59 - 2 = 57 leap years. Thus, there are 57 leap periods in the 235 years (each corresponding to a leap year in Y+1, i.e., 1792 to 2024 excluding 1800 and 1900). Thus, total days for full years: 235 * 365 + 57 = 85832. Now, remaining days from July 14, 2024, to June 4, 2025: as before, 326. Total days: 85832 + 326 = 86158. But this counts both July 14, 1789, and June 4, 2025. If we want the number of days between them (excluding the start day), it's 86158 - 1 = 86157. But the question is "how many days ago did the French Revolution start?" which is asking for the time elapsed since the start. The time elapsed from July 14, 1789, to June 4, 2025, is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 14, 1789 (not including) to June 4, 2025 (including), which is 86158 - 1 = 86157. Wait, no: if we say "X days ago from today", it's the number of days in [today - X, today), so the revolution started X days ago means today - X = start date, so X = today - start date, which is the number of days from start date to today, excluding the start date if counting days after.\ \ But in our calculation, we included both endpoints, so to get the number of days between them (excluding start), it's 86158 - 1 = 86157.\ \ But let's think differently: the duration between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end, etc. But in terms of days ago, if today is June 4, 2025, and the revolution started on July 14, 1789, then the number of days ago is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 15, 1789, to June 4, 2025, inclusive, which is (June 4, 2025 - July 15, 1789) + 1.\ \ But our initial calculation was from July 14, 1789 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 1789) + 1 = 86158.\ \ Thus, days ago = 86158 - 1 (since the day of start is not "ago" from today) = 86157.\ \ But actually, if the revolution started on day X, then today is X + N days ago. So N = today - X, where today and X are dates, and the subtraction gives the number of days between them, excluding the start day X in the count.\ \ But in our calculation, we did X to today inclusive, which is (today - X) + 1. So today - X = 86158 - 1 = 86157.\ \ Thus, the number of days ago is 86157.\ \ But let's verify with a smaller example. Suppose today is July 16, 2024, and the start was July 14, 2024. Then days ago is 2 (July 14 is 2 days ago from July 16). Our method would calculate days from July 14 to July 16 inclusive: 3 days (14,15,16), then subtract 1 to get days ago: 2, which is correct.\ \ Another example: today is July 15, 2024, start was July 14, 2024. Days from July 14 to July 15 inclusive: 2 days. Days ago: 2 - 1 = 1, which is correct (July 14 is 1 day ago from July 15).\ \ Thus, our calculation seems correct: total inclusive days is 86158, so days ago is 86157.\ \ But wait, let's think about the full years part. We calculated 235 years with 57 leap periods, giving 235*365 + 57 = 85832 days for the full years. Then we added 326 days for the remaining period from July 14, 2024, to June 4, 2025, which is correct. So total is 86158 inclusive, so days ago is 86157.\ \ But let's cross-validate the remaining days calculation (July 14, 2024, to June 4, 2025):\ \ From July 14 to December 31, 2024:\ July: 31 - 14 = 17 (from July 15 to July 31) + 1 (for July 14) = 18 days? Wait, no, from July 14 to July 14 is 1 day (just July 14). From July 14 to July 15 is 2 days (14 and 15). So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including both ends).\ Similarly for other months: but since we're starting at July 14, the first month is July with 18 days (July 14 to July 31), then full months August to December, then full months January to May, then June up to 4.\ \ Wait, perhaps better to calculate it as:\ From July 14, 2024, to June 4, 2025, is equal to:\ (From July 14, 2024, to July 14, 2025) minus (from June 5, 2025, to July 14, 2025).\ \ But that seems convoluted. Alternatively, calculate days remaining in 2024 after July 14, plus days in 2025 up to June 4.\ \ Days in July after July 14: July has 31 days, so from July 15 to July 31 is 17 days (31 - 14 = 17, since July 15 is day 1, ..., July 31 is day 17). Wait, no, July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days.\ \ Similarly, for August: full month is 31 days.\ September: 30\ October: 31\ November: 30\ December: 31\ Total for August to December: 31 + 30 + 31 + 30 + 31 = 153\ Total for July: 18\ Total for 2024: 18 (July) + 153 (Aug-Dec) = 171 days from July 14 to Dec 31, 2024 inclusive.\ \ Now 2025 is not a leap year (2025 is not divisible by 4).\ Days in 2025 up to June 4:\ January: 31\ February: 28 (not leap year)\ March: 31\ April: 30\ May: 31\ June: 4\ Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days from Jan 1 to June 4 inclusive.\ \ But we need from July 14, 2024, to June 4, 2025, inclusive, which is:\ (July 14, 2024, to Dec 31, 2024) + (Jan 1, 2025, to June 4, 2025) = 171 + 155 = 326 days.\ \ Thus, total days from start to today inclusive is 85832 (full years) + 326 (remaining) = 86158.\ Thus, days ago is 86158 - 1 = 86157.\ \ But let's cross-check the leap year count.\ \ Number of leap years between July 14, 1789, and July 14, 2024 (inclusive of the start date but not the end? Wait, our period is from July 14, 1789, up to but not including July 14, 2025, but we're counting up to July 14, 2024, which is before July 14, 2025.\ \ Wait, perhaps better to think in terms of the number of February 29s in the period from July 14, 1789, to June 4, 2025.\ \ A February 29 falls within our period if the year Y is a leap year and February 29, Y, is >= July 14, 1789, and < June 4, 2025.\ \ But since our start date is July 14, 1789, which is after February 29, 1789, the first possible leap day in our period is February 29, 1792.\ \ Similarly, the last possible leap day before June 4, 2025, is February 29, 2024 (since February 29, 2028, is after June 4, 2025).\ \ So leap days in our period are February 29 of years Y where:\ 1792 <= Y <= 2024, and Y is a leap year.\ \ Number of leap years between 1792 and 2024 inclusive.\ \ Number of years divisible by 4: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.\ \ Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).\ Thus, subtract 2: 59 - 2 = 57 leap years.\ \ Thus, there are 57 leap days in our period (February 29, 1792, to February 29, 2024).\ \ Each of these adds 1 day to the total count (since we're counting the entire period inclusively, and February 29 is included in the period for these years).\ \ Thus, our initial calculation seems correct: 235 full years with 57 leap periods contributing an extra day each, plus the remaining 326 days, gives 86158 total days including both endpoints. Therefore, the number of days ago is 86157.\ \ But wait, is the count of leap years correct? Let's verify for a smaller range.\ \ Example: from July 14, 2020, to June 4, 2022.\ Leap years in this period: February 29, 2020 (but our start is July 14, 2020, so February 29, 2020, is before our start date, so not included).\ Next is February 29, 2024, which is after June 4, 2022, so not included.\ Wait, in our original problem, the period is July 14, 1789, to June 4, 2025, so February 29, 2024, is included (since June 4, 2025, is after February 29, 2024).\ \ But in our initial calculation, we considered leap years up to 2024 (February 29, 2024), which is correct because our period includes February 29, 2024 (since June 4, 2025, is after February 29, 2024).\ \ But in the small example from July 14, 2020, to June 4, 2022:\ Leap days: February 29, 2020 is before July 14, 2020 (start), so not included.\ February 29, 2024 is after June 4, 2022 (end), so not included.\ Thus, no leap days in this period. But 2020 is a leap year, and our period starts after February 29, 2020, so the year from July 14, 2020, to July 14, 2021, does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2020? Wait, no, the period is from July 14, 2020, to July 14, 2021, which does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2021? Wait, no, February 29 is only in leap years, and 2021 is not a leap year. So in this small example, there are no February 29s in the period from July 14, 2020, to June 4, 2022, so the leap day count is correct (0 in this case).\ \ Thus, our original calculation seems correct.\ \ Final calculation:\ Total days from July 14, 1789, to June 4, 2025 inclusive: 86158.\ Number of days ago: 86158 - 1 = 86157.\ \ But wait, is the count of full years correct? We have 235 full years (from July 14, 1789, to July 14, 2024), but actually, the period from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years, where each year is from July 14 to July 14 of the next year. So the number of full years is correct at 235.\ \ Leap days in this period are February 29s that fall within any of these year-long periods. For a period from July 14, Y, to July 14, Y+1, the February 29 that could be included is February 29, Y+1 (since February 29, Y, is before July 14, Y, the start of the period, unless Y is before July, but our periods are from July 14 to July 14, so February is always before the start of the period if Y >= July). Wait, no, our periods are from July 14, Y, to July 14, Y+1, which includes February of Y+1, not Y. So for each Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024), the period includes February of Y+1.\ \ Thus, the leap day is included in the period if Y+1 is a leap year. So the number of leap periods is equal to the number of Y in 1789..2023 such that Y+1 is a leap year, i.e., Y+1 is divisible by 4, not divisible by 100 unless also by 400.\ \ Thus, Y+1 ranges from 1790 to 2024, and we need the number of leap years in this range, which is what we calculated earlier: 57.\ \ Thus, the calculation holds.\ \ Final answer: 86157 days ago.\ \ \ ### Summary:\ The French Revolution started on July 14, 1789. To find out how many days ago this was from June 4, 2025, we calculate the total days between these two dates.\ \ 1. Calculate the number of full years between July 14, 1789, and July 14, 2024: 235 years.\ 2. Determine the number of leap years in this period by counting leap years between 1790 and 2024 (inclusive), accounting for century years that are not leap years (1800 and 1900), resulting in 57 leap years.\ 3. Calculate the total days for these full years: \( 235 \times 365 + 57 = 85832 \) days.\ 4. Calculate the days remaining from July 14, 2024, to June 4, 2025, which is 326 days (171 days in 2024 after July 14 and 155 days in 2025 up to June 4).\ 5. Sum the total days: \( 85832 + 326 = 86158 \) days (inclusive of both start and end dates).\ 6. Since the question asks for how many days ago the revolution started, we subtract 1 day to exclude the start day itself, resulting in 86157 days ago.\ \ However, upon closer inspection, the correct interpretation is that the number of days between two dates (excluding the start day) is simply the difference in days between the two dates. Therefore, the correct calculation is the total inclusive days minus one to exclude the start day, which remains 86157.\ \ But let's cross-validate with a smaller example to ensure correctness.\ \ **Example:**\ If today is July 16, 2024, and the event was on July 14, 2024:\ - Inclusive days: July 14 to July 16 is 3 days (14, 15, 16).\ - Days ago: 2 (July 14 is 2 days ago from July 16: July 15 and 16 are 1 and 0 days ago, respectively).\ \ This matches our method: inclusive days = 3, so days ago = 3 - 1 = 2.\ \ Thus, applying this to the original problem:\ Inclusive days = 86158, so days ago = 86158 - 1 = 86157.\ \ However, another approach is to calculate the exact difference between the two dates without including the start day in the count of "days ago". The duration between July 14, 1789, and June 4, 2025, excluding the start day, is (June 4, 2025 - July 14, 1789), which is equivalent to counting from July 15, 1789, to June 4, 2025, inclusive. This would be:\ \ From July 15, 1789, to July 14, 2025: 235 years and 326 days (since July 15 to July 14 is one day less than a full year, but this seems confusing).\ \ Alternatively, it's simpler to compute the total inclusive days and subtract 1, as initially done.\ \ Therefore, the final answer is that the French Revolution started **86157 days ago**.\ \ But let's verify the calculation once more for accuracy.\ \ Recalculating the remaining days (July 14, 2024, to June 4, 2025):\ \ From July 14 to December 31, 2024:\ - July 14 to July 31: 18 days (including July 14)\ - August: 31\ - September: 30\ - October: 31\ - November: 30\ - December: 31\ Total: 18 + 31 + 30 + 31 + 30 + 31 = 171 days (but wait, July 14 to July 31 is 18 days, August starts on August 1, so total is correct as 171).\ \ From January 1 to June 4, 2025 (not a leap year):\ - January: 31\ - February: 28\ - March: 31\ - April: 30\ - May: 31\ - June: 4\ Total: 31 + 28 + 31 + 30 + 31 + 4 = 155 days.\ \ Total remaining days: 171 + 155 = 326.\ \ Full years: 235 years, with 57 leap days (as calculated earlier).\ Total for full years: 235 * 365 + 57 = 85775 + 57 = 85832.\ \ Total inclusive days: 85832 + 326 = 86158.\ \ Days ago: 86158 - 1 = 86157.\ \ Thus, the final answer is:\ \ \[\ \boxed{86157}\ \]\ \ However, upon re-reading the problem, it asks for "how many days ago did the French Revolution start?" which is asking for the duration from the start date to today, not including the start date itself. Therefore, the correct calculation is the difference between the two dates, which is (today - start date). If we compute the total inclusive days and subtract 1, we get the correct number of days ago, which is 86157.\ \ But let's compute it directly:\ \ Compute the difference between June 4, 2025, and July 14, 1789.\ \ We can break it down as follows:\ \ 1. Calculate the number of days from July 14, 1789, to July 14, 2024: 235 years.\ - Number of leap years in this period: years divisible by 4 between 1792 and 2024, excluding 1800 and 1900: as before, 57.\ - Total days: 235 * 365 + 57 = 85832.\ \ 2. Add the days from July 14, 2024, to June 4, 2025: as before, 326.\ \ Total days from July 14, 1789, to June 4, 2025, inclusive: 85832 + 326 = 86158.\ \ But the number of days ago is the number of days after July 14, 1789, up to June 4, 2025, which is June 4, 2025 - July 14, 1789 = (days from July 15, 1789, to June 4, 2025, inclusive).\ \ This is equal to (total inclusive days from July 14, 1789, to June 4, 2025) - 1 = 86157.\ \ Thus, the French Revolution started **86157 days ago**.\ \ However, let's cross-check with an online date difference calculator to ensure our manual calculation is correct.\ \ Using an online calculator (e.g., timeanddate.com), the number of days between July 14, 1789, and June 4, 2025, inclusive is indeed 86158 days. Excluding the start date, it's 86157 days ago.\ \ Thus, the final answer is:\ \ The French Revolution started **86157 days ago**.\ \ But wait, let's think about the interpretation of "days ago". If the revolution started on day X, then "days ago" is how many days have passed since X up to but not including today. But in our case, today is June 4, 2025, and the revolution started on July 14, 1789. The duration from July 14, 1789, to June 4, 2025 (excluding the start day) is the same as the number of days from July 15, 1789, to June 4, 2025, inclusive, which is the total inclusive days minus 1 (for the start day).\ \ But actually, the standard way to compute "days ago" is to calculate (today - start date), which is the same as counting the days from the day after the start date up to and including today. But since our end date is today (June 4, 2025), and we're calculating days from July 14, 1789, to June 4, 2025 inclusive (which is 86158), then the days ago is 86158 - 1 = 86157.\ \ But if we were to calculate (June 4, 2025 - July 14, 1789), this would directly give the number of days between the two dates not including the start date, which is what we want. This is equivalent to counting from July 15, 1789, to June 4, 2025 inclusive, which is the same as the total inclusive days minus one (the start day), so our initial calculation is correct.\ \ Thus, the final answer is:\ \ The French Revolution started **86157 days ago**.\ \ But let's cross-validate with a smaller example again to ensure correctness.\ \ Example:\ Start date: July 14, 2024\ Today: July 16, 2024\ Inclusive days: July 14, 15, 16 ā†’ 3 days.\ Days ago: 2 (since July 14 is 2 days ago from July 16: July 15 is 1 day ago, July 16 is today, 0 days ago).\ Our method: inclusive days (3) - 1 = 2. Correct.\ \ Another example:\ Start: July 14, 2024\ Today: July 15, 2024\ Inclusive days: July 14, 15 ā†’ 2 days.\ Days ago: 1 (July 14 is 1 day ago from July 15).\ Our method: 2 - 1 = 1. Correct.\ \ Thus, our method is correct.\ \ Therefore, the French Revolution started **86157 days ago**.\ \ However, let's think about the leap day calculation again to ensure no mistake was made.\ \ We have full years from July 14, 1789, to July 14, 2024 (235 years), and we counted 57 leap periods in this interval (where each period is a year from July 14 to July 14, and contains February 29 if the next year is a leap year).\ \ But let's think differently: the total number of February 29s in the entire period from July 14, 1789, to June 4, 2025.\ \ February 29 occurs in leap years. The leap years in this period are those where February 29 is between July 14, 1789, and June 4, 2025.\ \ Since July 14 is after February in any year, the February 29 of year Y is included in our period if:\ July 14, 1789 <= February 29, Y < June 4, 2025.\ \ But February 29, Y, is always before July 14, Y (since February is before July), so February 29, Y, is included if Y >= 1789 (since February 29, 1789, is before July 14, 1789, so not included), and February 29, Y < June 4, 2025.\ \ But since February 29 is always before July 14 in the same year, February 29, Y, is included in our period if Y > 1789 and February 29, Y < June 4, 2025.\ \ But February 29, Y, is before July 14, Y, so February 29, Y, is included if Y >= 1789 and February 29, Y < June 4, 2025.\ \ But February 29, Y, is before July 14, Y, so for Y >= 1789, February 29, Y is before July 14, Y, so February 29, Y is only included in our period if Y > 1789 (since February 29, 1789, is before our start date of July 14, 1789), and February 29, Y < June 4, 2025.\ \ But since February 29, Y, is always before July 14, Y, and our period starts on July 14, 1789, February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February is before July, so February 29, Y, is always before July 14, Y. Therefore, February 29, Y, is included in our period if Y > 1789 (since February 29, 1789, is before our start date) and February 29, Y < June 4, 2025.\ \ But February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if:\ July 14, 1789 <= February 29, Y < June 4, 2025.\ \ But since February 29, Y, is always before July 14, Y, the first condition is satisfied if Y >= 1789, but February 29, 1789, is before July 14, 1789 (start of our period), so February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29 is always before July 14 in the same year, so February 29, Y, is never >= July 14, Y. Thus, February 29, Y, is included if Y > 1789 and February 29, Y >= July 14, 1789. But since February is before July, February 29, Y, is only >= July 14, 1789 if Y >= 1790 (because February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., but our period starts on July 14, 1789, so February 29, Y, is included if Y >= 1790 (since February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29, Y, is always before July 14, Y, so February 29, Y, is >= July 14, 1789 only if Y >= 1789 and February 29, Y >= July 14, 1789. But February is always before July, so February 29, Y, is always before July 14, Y, so February 29, Y >= July 14, 1789 would require Y > 1789 (since February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., so February 29, Y, is never >= July 14, Y for Y >= 1789).\ \ Wait, this seems confusing. Perhaps a better approach is to realize that in our period from July 14, 1789, to June 4, 2025, a February 29 is included if it falls within this interval. Since February is before July, February 29, Y, is included if Y >= 1790 (because February 29, 1789, is before July 14, 1789, so not included, and February 29, 1790, is after July 14, 1789, and before June 4, 2025, since June 4, 2025, is after February 29, 2024, and before February 29, 2028, etc.\ \ But actually, February 29, Y, is included if Y is such that February 29, Y, is >= July 14, 1789 and < June 4, 2025. Since February is before July, February 29, Y, is >= July 14, 1789 only if Y >= 1790 (because February 29, 1789, is before July 14, 1789, but February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is before July 14, Y >= 1789, so February 29, Y, cannot be >= July 14, 1789 unless Y > 1789 (but even then, February 29, 1790, is before July 14, 1790, etc.).\ \ Wait, this seems incorrect. February 29, Y, is always before July 14, Y, so February 29, Y, cannot be >= July 14, Y for any Y. Therefore, February 29, Y, is included in our period if Y >= 1790 (since February 29, 1790, is after July 14, 1789, and before June 4, 2025).\ \ Thus, the leap days included are February 29 of years Y where 1790 <= Y <= 2024 (since February 29, 2024, is before June 4, 2025, and February 29, 2028, would be after June 4, 2025).\ \ Thus, we need to count the number of leap years Y in 1790 <= Y <= 2024.\ \ Number of years divisible by 4 in 1790..2024:\ First divisible by 4 is 1792 (since 1790 and 1791 are not divisible by 4).\ Last divisible by 4 is 2024.\ Number of terms: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.\ \ Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).\ Thus, subtract 2: 59 - 2 = 57 leap years.\ \ This matches our earlier count of leap periods (57), confirming that our initial approach was correct.\ \ Thus, the final answer is indeed 86157 days ago.\ \ However, let's perform the calculation once more to ensure no arithmetic errors were made.\ \ Full years: 235 years (July 14, 1789, to July 14, 2024).\ Number of leap years in this period: as above, 57 (for years 1792 to 2024 excluding 1800 and 1900).\ Total days for full years: 235 * 365 + 57 = 235 * 365 = 235 * 300 + 235 * 65 = 70500 + 235 * 65.\ Calculate 235 * 65:\ 200 * 65 = 13000\ 35 * 65 = 2275\ Total: 13000 + 2275 = 15275\ So 70500 + 15275 = 85775\ Add leap days: 85775 + 57 = 85832.\ \ Remaining days: July 14, 2024, to June 4, 2025.\ From July 14 to July 31, 2024: 31 - 14 + 1 = 18 days.\ August to December: 31 + 30 + 31 + 30 + 31 = 153.\ Total for 2024: 18 + 153 = 171.\ From Jan 1 to June 4, 2025:\ January: 31\ February: 28 (not a leap year)\ March: 31\ April: 30\ May: 31\ June: 4\ Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155.\ Total remaining days: 171 + 155 = 326.\ \ Total inclusive days: 85832 (full years) + 326 (remaining) = 86158.\ Days ago: 86158 - 1 = 86157.\ \ Thus, the final answer is:\ \ The French Revolution started **86157 days ago**.\ \ \boxed{86157}\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#experimental-vision-support)\ \ šŸ‘ļøExperimental Vision Support\ \ \ -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ \ [Xuan-Son](https://x.com/ngxson)\ from HuggingFace showed in their [GGUF repo](https://huggingface.co/ngxson/Devstral-Small-Vision-2505-GGUF)\ how it is actually possible to "graft" the vision encoder from Mistral 3.1 Instruct onto Devstral meaning you could do the same for Magistral! According to our tests and many users, it works quite well! We also uploaded our mmproj files which allows you to use the following:\ \ Copy\ \ ./llama.cpp/llama-mtmd-cli \\ --model unsloth/Magistral-Small-2507-GGUF/Magistral-Small-2507-Q4_K_XL.gguf \\ --mmproj unsloth/Magistral-Small-2507-GGUF/mmproj-BF16.gguf \\ --threads -1 \\ --ctx-size 40960 \\ --cache-type-k f16\ --n-gpu-layers 99 \\ --seed 3407 \\ --prio 2 \\ --temp 0.7 \\ --repeat-penalty 1.0 \\ --min-p 0.01 \\ --top-k -1 \\ --top-p 0.95 \\ --jinja\ \ [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune#fine-tuning-magistral-with-unsloth)\ \ šŸ¦„ Fine-tuning Magistral with Unsloth\ \ \ -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ \ Just like standard Mistral models including Mistral Small 3.1, Unsloth supports Magistral fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Magistral fits comfortably in a 24GB VRAM L4 GPU.\ \ Magistral slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you _can_ fine-tune the model for free using [Kaggle](https://www.kaggle.com/danielhanchen/code)\ , which offers access to dual GPUs.\ \ **To finetune on new reasoning traces, you can use our free** [**Kaggle notebook for Magistral**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_(24B)-Reasoning-Conversational.ipynb&accelerator=nvidiaTeslaT4)\ **.**\ \ If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:\ \ Copy\ \ pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo\ \ [PreviousTutorials: How To Fine-tune & Run LLMs](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms)\ [NextDevstral: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune)\ \ Last updated 29 days ago\ \ Was this helpful? --- # Tutorial: How to Finetune Llama-3 and Use In Ollama | Unsloth Documentation OpenAI gpt-oss & all model types now supported! By the end of this tutorial, you will create a custom chatbot by **finetuning Llama-3** with [**Unsloth**](https://github.com/unslothai/unsloth) for free. It can run locally via [**Ollama**](https://github.com/ollama/ollama) on your PC, or in a free GPU instance through [**Google Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) . You will be able to interact with the chatbot interactively like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FXlEQrBR24CKI9lQIzOS7%252FAssistant%2520example.png%3Falt%3Dmedia%26token%3Dfac7f5b0-69f4-4998-baee-3feee44f8c16&width=768&dpr=4&quality=100&sign=39273e6a&sv=2) **Unsloth** makes finetuning much easier, and can automatically export the finetuned model to **Ollama** with integrated automatic `Modelfile` creation! If you need help, you can join our Discord server: [https://discord.com/invite/unsloth](https://discord.com/invite/unsloth) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-1.-what-is-unsloth) 1\. What is Unsloth? ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Unsloth](https://github.com/unslothai/unsloth) makes finetuning LLMs like Llama-3, Mistral, Phi-3 and Gemma 2x faster, use 70% less memory, and with no degradation in accuracy! We will be using Google Colab which provides a free GPU during this tutorial. You can access our free notebooks below: * [Ollama Llama-3 Alpaca](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb) (notebook which we will be using) * [CSV/Excel Ollama Guide](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) #### [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#you-will-also-need-to-login-into-your-google-account) _**You will also need to login into your Google account!**_ ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqnogsAv2zZ5WPFkXwQ5t%252FColab%2520Screen.png%3Falt%3Dmedia%26token%3D8722cf50-898f-4f15-be7a-7223b8b7440b&width=768&dpr=4&quality=100&sign=c93e1323&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-2.-what-is-ollama) 2\. What is Ollama? --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Ollama](https://github.com/ollama/ollama) allows you to run language models from your own computer in a quick and simple way! It quietly launches a program which can run a language model like Llama-3 in the background. If you suddenly want to ask the language model a question, you can simply submit a request to Ollama, and it'll quickly return the results to you! We'll be using Ollama as our inference engine! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqKwhUFNW52GnKMi5ClLW%252FOllama.png%3Falt%3Dmedia%26token%3D27ccad2f-12a2-4188-96d9-ee3023d7f274&width=768&dpr=4&quality=100&sign=e04cd2e2&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-3.-install-unsloth) 3\. Install Unsloth ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQzuUQL60uFWHpaAvDPYD%252FColab%2520Options.png%3Falt%3Dmedia%26token%3Dfb808ec5-20c5-4f42-949e-14ed26a44987&width=768&dpr=4&quality=100&sign=be097a14&sv=2) If you have never used a Colab notebook, a quick primer on the notebook itself: 1. **Play Button at each "cell".** Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter any errors, simply rerun the cell you did not run before. Another option is to click CTRL + ENTER if you don't want to click the play button. 2. **Runtime Button in the top toolbar.** You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, and can be a good first try. 3. **Connect / Reconnect T4 button.** You can click here for more advanced system statistics. The first installation cell looks like below: Remember to click the PLAY button in the brackets \[ \]. We grab our open source Github package, and install some other packages. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9DTAK0evMnZcnLXzKLx4%252Fimage.png%3Falt%3Dmedia%26token%3Db4781438-3858-4d6c-a560-5afcbbc12fa8&width=768&dpr=4&quality=100&sign=e78940c8&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-4.-selecting-a-model-to-finetune) 4\. Selecting a model to finetune -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's now select a model for finetuning! We defaulted to Llama-3 from Meta / Facebook which was trained on a whopping 15 trillion "tokens". Assume a token is like 1 English word. That's approximately 350,000 thick Encyclopedias worth! Other popular models include Mistral, Phi-3 (trained using GPT-4 output) and Gemma from Google (13 trillion tokens!). Unsloth supports these models and more! In fact, simply type a model from the Hugging Face model hub to see if it works! We'll error out if it doesn't work. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fmdci7SWqnAZiW8KzzDp0%252Fimage.png%3Falt%3Dmedia%26token%3D8ede6c31-3cc9-4005-ae44-0b056750e8d4&width=768&dpr=4&quality=100&sign=f453cf0e&sv=2) There are 3 other settings which you can toggle: 1. Copy max_seq_length = 2048 This determines the context length of the model. Gemini for example has over 1 million context length, whilst Llama-3 has 8192 context length. We allow you to select ANY number - but we recommend setting it 2048 for testing purposes. Unsloth also supports very long context finetuning, and we show we can provide 4x longer context lengths than the best. 2. Copy dtype = None Keep this as None, but you can select torch.float16 or torch.bfloat16 for newer GPUs. 3. Copy load_in_4bit = True We do finetuning in 4 bit quantization. This reduces memory usage by 4x, allowing us to actually do finetuning in a free 16GB memory GPU. 4 bit quantization essentially converts weights into a limited set of numbers to reduce memory usage. A drawback of this is there is a 1-2% accuracy degradation. Set this to False on larger GPUs like H100s if you want that tiny extra accuracy. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FegXn4FqK96xXZWMz4NH5%252Fimage.png%3Falt%3Dmedia%26token%3D7531f78d-390b-470b-a91e-4463eea6537f&width=768&dpr=4&quality=100&sign=c6859bb2&sv=2) If you run the cell, you will get some print outs of the Unsloth version, which model you are using, how much memory your GPU has, and some other statistics. Ignore this for now. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-5.-parameters-for-finetuning) 5\. Parameters for finetuning ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqRTuI7x0FYlHTXqbi0hu%252Fimage.png%3Falt%3Dmedia%26token%3D4b0e0032-dbf1-4148-ba92-c18356862765&width=768&dpr=4&quality=100&sign=f94a3d99&sv=2) Now to customize your finetune, you can edit the numbers above, but you can ignore it, since we already select quite reasonable numbers. The goal is to change these numbers to increase accuracy, but also **counteract over-fitting**. Over-fitting is when you make the language model memorize a dataset, and not be able to answer novel new questions. We want to a final model to answer unseen questions, and not do memorization. 1. Copy r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 The rank of the finetuning process. A larger number uses more memory and will be slower, but can increase accuracy on harder tasks. We normally suggest numbers like 8 (for fast finetunes), and up to 128. Too large numbers can causing over-fitting, damaging your model's quality. 2. Copy target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], We select all modules to finetune. You can remove some to reduce memory usage and make training faster, but we highly do not suggest this. Just train on all modules! 3. Copy lora_alpha = 16, The scaling factor for finetuning. A larger number will make the finetune learn more about your dataset, but can promote over-fitting. We suggest this to equal to the rank `r`, or double it. 4. Copy lora_dropout = 0, # Supports any, but = 0 is optimized Leave this as 0 for faster training! Can reduce over-fitting, but not that much. 5. Copy bias = "none", # Supports any, but = "none" is optimized Leave this as 0 for faster and less over-fit training! 6. Copy use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context Options include `True`, `False` and `"unsloth"`. We suggest `"unsloth"` since we reduce memory usage by an extra 30% and support extremely long context finetunes.You can read up here: [https://unsloth.ai/blog/long-context](https://unsloth.ai/blog/long-context) for more details. 7. Copy random_state = 3407, The number to determine deterministic runs. Training and finetuning needs random numbers, so setting this number makes experiments reproducible. 8. Copy use_rslora = False, # We support rank stabilized LoRA Advanced feature to set the `lora_alpha = 16` automatically. You can use this if you want! 9. Copy loftq_config = None, # And LoftQ Advanced feature to initialize the LoRA matrices to the top r singular vectors of the weights. Can improve accuracy somewhat, but can make memory usage explode at the start. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset) 6\. Alpaca Dataset -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FKSmRDpkySelZfWSrWxDm%252Fimage.png%3Falt%3Dmedia%26token%3D5401e4da-796a-42ad-8b85-2263f3e59e86&width=768&dpr=4&quality=100&sign=28ad8509&sv=2) We will now use the Alpaca Dataset created by calling GPT-4 itself. It is a list of 52,000 instructions and outputs which was very popular when Llama-1 was released, since it made finetuning a base LLM be competitive with ChatGPT itself. You can access the GPT4 version of the Alpaca dataset here: [https://huggingface.co/datasets/vicgalle/alpaca-gpt4](https://huggingface.co/datasets/vicgalle/alpaca-gpt4) . An older first version of the dataset is here: [https://github.com/tatsu-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca) . Below shows some examples of the dataset: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzKhujR9Nxz95VFSdf4J5%252Fimage.png%3Falt%3Dmedia%26token%3Da3c52718-eaf1-4a3d-b325-414d8e67722e&width=768&dpr=4&quality=100&sign=2afb3a12&sv=2) You can see there are 3 columns in each row - an instruction, and input and an output. We essentially combine each row into 1 large prompt like below. We then use this to finetune the language model, and this made it very similar to ChatGPT. We call this process **supervised instruction finetuning**. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FieYX44Vjd0OygJvO0jaR%252Fimage.png%3Falt%3Dmedia%26token%3Deb67fa41-a280-4656-8be6-5b6bf6f587c2&width=768&dpr=4&quality=100&sign=68f5594e&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-7.-multiple-columns-for-finetuning) 7\. Multiple columns for finetuning ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ But a big issue is for ChatGPT style assistants, we only allow 1 instruction / 1 prompt, and not multiple columns / inputs. For example in ChatGPT, you can see we must submit 1 prompt, and not multiple prompts. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FpFUWhntUQLu05l4ns7Pq%252Fimage.png%3Falt%3Dmedia%26token%3De989e4a6-6033-4741-b97f-d0c3ce8f5888&width=768&dpr=4&quality=100&sign=a9eb969a&sv=2) This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function! For example the very famous Titanic dataset has many many columns. Your job was to predict whether a passenger has survived or died based on their age, passenger class, fare price etc. We can't simply pass this into ChatGPT, but rather, we have to "merge" this information into 1 large prompt. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FrydHBjHoJT7w8FwzKAXK%252FMerge-1.png%3Falt%3Dmedia%26token%3Dec812057-0475-4717-87fe-311f14735c37&width=768&dpr=4&quality=100&sign=8211e070&sv=2) For example, if we ask ChatGPT with our "merged" single prompt which includes all the information for that passenger, we can then ask it to guess or predict whether the passenger has died or survived. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJVkv73fRWvwwFxMym7uW%252Fimage.png%3Falt%3Dmedia%26token%3D59b97b76-f2f2-46c9-8940-60a37e4e7d62&width=768&dpr=4&quality=100&sign=37c0f3a1&sv=2) Other finetuning libraries require you to manually prepare your dataset for finetuning, by merging all your columns into 1 prompt. In Unsloth, we simply provide the function called `to_sharegpt` which does this in 1 go! To access the Titanic finetuning notebook or if you want to upload a CSV or Excel file, go here: [https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9fo2IBA7P0tNwhNR9Prm%252Fimage.png%3Falt%3Dmedia%26token%3D7bd7244a-0fea-4e57-9038-a8a360138056&width=768&dpr=4&quality=100&sign=a94d397b&sv=2) Now this is a bit more complicated, since we allow a lot of customization, but there are a few points: * You must enclose all columns in curly braces `{}`. These are the column names in the actual CSV / Excel file. * Optional text components must be enclosed in `[[]]`. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values. * Select the output or target / prediction column in `output_column_name`. For the Alpaca dataset, this will be `output`. For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRMvBpfXC9ToCRL0oCJfN%252Fimage.png%3Falt%3Dmedia%26token%3Dc257c7fc-8a9c-4d4f-ab3d-6894ae49f2a9&width=768&dpr=4&quality=100&sign=4ec813ed&sv=2) For example, pretend the dataset looks like this with a lot of missing data: Embarked Age Fare S 23 18 7.25 Then, we do not want the result to be: 1. The passenger embarked from S. Their age is 23. Their fare is **EMPTY**. 2. The passenger embarked from **EMPTY**. Their age is 18. Their fare is $7.25. Instead by optionally enclosing columns using `[[]]`, we can exclude this information entirely. 1. \[\[The passenger embarked from S.\]\] \[\[Their age is 23.\]\] \[\[Their fare is **EMPTY**.\]\] 2. \[\[The passenger embarked from **EMPTY**.\]\] \[\[Their age is 18.\]\] \[\[Their fare is $7.25.\]\] becomes: 1. The passenger embarked from S. Their age is 23. 2. Their age is 18. Their fare is $7.25. [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-8.-multi-turn-conversations) 8\. Multi turn conversations ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A bit issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWCAN7bYUt6QWwCWUxisL%252Fdiff.png%3Falt%3Dmedia%26token%3D29821fd9-2181-4d1d-8b93-749b69bcf400&width=768&dpr=4&quality=100&sign=d4f1b675&sv=2) So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FWi1rRNBFC2iDmCvSJsZt%252Fcombine.png%3Falt%3Dmedia%26token%3Dbef37a55-b272-4be3-89b5-9767c219a380&width=768&dpr=4&quality=100&sign=ae98ba1b&sv=2) Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column. We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FE75C4Y848VNF6luLuPRR%252Fimage.png%3Falt%3Dmedia%26token%3Daac1d79b-ecca-4e56-939d-d97dcbbf30eb&width=768&dpr=4&quality=100&sign=d48e3c76&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-9.-customizable-chat-templates) 9\. Customizable Chat Templates ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We can now specify the chat template for finetuning itself. The very famous Alpaca format is below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8SWcsgH47Uhkm0IclDs5%252Fimage.png%3Falt%3Dmedia%26token%3Dfa03d7aa-d568-468d-9884-18e925a0551f&width=768&dpr=4&quality=100&sign=dff54efb&sv=2) But remember we said this was a bad idea because ChatGPT style finetunes require only 1 prompt? Since we successfully merged all dataset columns into 1 using Unsloth, we essentially can create the below style chat template with 1 input column (instruction) and 1 output: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FyuMpSLIpPLEbcdh970UJ%252Fimage.png%3Falt%3Dmedia%26token%3D87c4d5e1-accf-4847-9971-63e3a47b4a5f&width=768&dpr=4&quality=100&sign=728095c1&sv=2) We just require you must put a `{INPUT}` field for the instruction and an `{OUTPUT}` field for the model's output field. We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. For example, below are some cool examples which you can customize the chat template to be: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fi6B8IP1OZmmxBYr6k4W3%252Fimage.png%3Falt%3Dmedia%26token%3D061d1b4c-4b22-4d1b-a423-8d4c15e40efa&width=768&dpr=4&quality=100&sign=dd8c7435&sv=2) For the ChatML format used in OpenAI models: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F3OEJaXooJCICJR6DJIJP%252Fimage.png%3Falt%3Dmedia%26token%3D4fa85cf1-463d-4090-a838-591c4f94efea&width=768&dpr=4&quality=100&sign=a1f23ff9&sv=2) Or you can use the Llama-3 template itself (which only functions by using the instruct version of Llama-3): We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4qQXd0hIvh9fJNO2cJ04%252Fimage.png%3Falt%3Dmedia%26token%3D614b9200-7375-47f5-ac15-ce9aa891ede4&width=768&dpr=4&quality=100&sign=c9811100&sv=2) Or in the Titanic prediction task where you had to predict if a passenger died or survived in this Colab notebook which includes CSV and Excel uploading: [https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1iQitC3PwcuV0LpHEhdP%252Fimage.png%3Falt%3Dmedia%26token%3Dd117f681-afb0-4d5f-b534-f51013fe772a&width=768&dpr=4&quality=100&sign=20577629&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-10.-train-the-model) 10\. Train the model ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Let's train the model now! We normally suggest people to not edit the below, unless if you want to finetune for longer steps or want to train on large batch sizes. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FoPTTR7ppdxhZR2iPpE0R%252Fimage.png%3Falt%3Dmedia%26token%3D1dca98a5-c927-4e93-8e96-977015f4eeb9&width=768&dpr=4&quality=100&sign=18baea65&sv=2) We do not normally suggest changing the parameters above, but to elaborate on some of them: 1. Copy per_device_train_batch_size = 2, Increase the batch size if you want to utilize the memory of your GPU more. Also increase this to make training more smooth and make the process not over-fit. We normally do not suggest this, since this might make training actually slower due to padding issues. We normally instead ask you to increase `gradient_accumulation_steps` which just does more passes over the dataset. 2. Copy gradient_accumulation_steps = 4, Equivalent to increasing the batch size above itself, but does not impact memory consumption! We normally suggest people increasing this if you want smoother training loss curves. 3. Copy max_steps = 60, # num_train_epochs = 1, We set steps to 60 for faster training. For full training runs which can take hours, instead comment out `max_steps`, and replace it with `num_train_epochs = 1`. Setting it to 1 means 1 full pass over your dataset. We normally suggest 1 to 3 passes, and no more, otherwise you will over-fit your finetune. 4. Copy learning_rate = 2e-4, Reduce the learning rate if you want to make the finetuning process slower, but also converge to a higher accuracy result most likely. We normally suggest 2e-4, 1e-4, 5e-5, 2e-5 as numbers to try. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxwOA09mtcimcQOCjP4PG%252Fimage.png%3Falt%3Dmedia%26token%3D39a0f525-6d4e-4c3b-af0d-82d8960d87be&width=768&dpr=4&quality=100&sign=853c0062&sv=2) You will see a log of some numbers! This is the training loss, and your job is to set parameters to make this go to as close to 0.5 as possible! If your finetune is not reaching 1, 0.8 or 0.5, you might have to adjust some numbers. If your loss goes to 0, that's probably not a good sign as well! [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-11.-inference-running-the-model) 11\. Inference / running the model -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRX9Byv1hlSpvmonT1PLw%252Fimage.png%3Falt%3Dmedia%26token%3D6043cd8c-c6a3-4cc5-a019-48baeed3b5a2&width=768&dpr=4&quality=100&sign=7c7ce43f&sv=2) Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6DXSlsHkN8cZiiAxAV0Z%252Fimage.png%3Falt%3Dmedia%26token%3D846307de-7386-4bbe-894e-7d9e572244fe&width=768&dpr=4&quality=100&sign=6482b95b&sv=2) Reminder Unsloth itself provides **2x faster inference** natively as well, so always do not forget to call `FastLanguageModel.for_inference(model)`. If you want the model to output longer responses, set `max_new_tokens = 128` to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well! [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-12.-saving-the-model) 12\. Saving the model -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We can now save the finetuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face token via [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and add your token! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBz0YDi6Sc2oEP5QWXgSz%252Fimage.png%3Falt%3Dmedia%26token%3D33d9e4fd-e7dc-4714-92c5-bfa3b00f86c4&width=768&dpr=4&quality=100&sign=d6933a01&sv=2) After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FzymBQrqwt4GUmCIN0Iec%252Fimage.png%3Falt%3Dmedia%26token%3D41a110e4-8263-426f-8fa7-cdc295cc8210&width=768&dpr=4&quality=100&sign=b2a207c3&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-13.-exporting-to-ollama) 13\. Exporting to Ollama -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Finally we can export our finetuned model to Ollama itself! First we have to install Ollama in the Colab notebook: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqNvGTAGwZKXxkMQqzloS%252Fimage.png%3Falt%3Dmedia%26token%3Ddb503499-0c74-4281-b3bf-400fa20c9ce2&width=768&dpr=4&quality=100&sign=6d57e83a&sv=2) Then we export the finetuned model we have to llama.cpp's GGUF formats like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZduLjedyfUbTmYqF85pa%252Fimage.png%3Falt%3Dmedia%26token%3Df5bac541-b99f-4d9b-82f7-033f8de780f2&width=768&dpr=4&quality=100&sign=1fdb7647&sv=2) Reminder to convert `False` to `True` for 1 row, and not change every row to `True`, or else you'll be waiting for a very time! We normally suggest the first row getting set to `True`, so we can export the finetuned model quickly to `Q8_0` format (8 bit quantization). We also allow you to export to a whole list of quantization methods as well, with a popular one being `q4_k_m`. Head over to [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) to learn more about GGUF. We also have some manual instructions of how to export to GGUF if you want here: [https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf](https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf) You will see a long list of text like below - please wait 5 to 10 minutes!! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FcuUAx0RNtrQACvU7uWCL%252Fimage.png%3Falt%3Dmedia%26token%3Ddc67801a-a363-48e2-8572-4c6d0d8d0d93&width=768&dpr=4&quality=100&sign=cc7f7372&sv=2) And finally at the very end, it'll look like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxRh07PEQjAmmz3s2HJUP%252Fimage.png%3Falt%3Dmedia%26token%3D3552a3c9-4d4f-49ee-a31e-0a64327419f0&width=768&dpr=4&quality=100&sign=1e9c9f0d&sv=2) Then, we have to run Ollama itself in the background. We use `subprocess` because Colab doesn't like asynchronous calls, but normally one just runs `ollama serve` in the terminal / command prompt. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FszDuikrg4HY8lGefwpRQ%252Fimage.png%3Falt%3Dmedia%26token%3Dec1c8762-661d-4b13-ab4f-ed1a7b9fda00&width=768&dpr=4&quality=100&sign=fc72e538&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-14.-automatic-modelfile-creation) 14\. Automatic `Modelfile` creation ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The trick Unsloth provides is we automatically create a `Modelfile` which Ollama requires! This is a just a list of settings and includes the chat template which we used for the finetune process! You can also print the `Modelfile` generated like below: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fh6inH6k5ggxUP80Gltgj%252Fimage.png%3Falt%3Dmedia%26token%3D805bafb1-2795-4743-9bd2-323ab4f0881e&width=768&dpr=4&quality=100&sign=456e8653&sv=2) We then ask Ollama to create a model which is Ollama compatible, by using the `Modelfile` ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1123bSSwmjWXliaRUL5U%252Fimage.png%3Falt%3Dmedia%26token%3D2e72f1a0-1ff8-4189-8d9c-d31e39385555&width=768&dpr=4&quality=100&sign=52a4fd99&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-15.-ollama-inference) 15\. Ollama Inference -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- And we can now call the model for inference if you want to do call the Ollama server itself which is running on your own local machine / in the free Colab notebook in the background. Remember you can edit the yellow underlined part. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fk5mdsJ57hQ1Ar3KY6VXY%252FInference.png%3Falt%3Dmedia%26token%3D8cf0cbf9-0534-4bae-a887-89f45a3de771&width=768&dpr=4&quality=100&sign=8489fe55&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-16.-interactive-chatgpt-style) 16\. Interactive ChatGPT style -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- But to actually run the finetuned model like a ChatGPT, we have to do a bit more! First click the terminal icon![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FUb17xtyDliAKhJEL9KuH%252Fimage.png%3Falt%3Dmedia%26token%3Df612e9b7-7d05-4039-a476-646026c6c8e6&width=300&dpr=4&quality=100&sign=b1c272f5&sv=2) and a Terminal will pop up. It's on the left sidebar. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRWPEy4fW8ytOljQYLn55%252FWhere_Terminal.png%3Falt%3Dmedia%26token%3D4ddf3017-2380-4615-958f-a465a76f7bac&width=768&dpr=4&quality=100&sign=32fba259&sv=2) Then, you might have to press ENTER twice to remove some weird output in the Terminal window. Wait a few seconds and type `ollama run unsloth_model` then hit ENTER. ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FL4aLJtoWh3HCkQ6f4J0Q%252FTerminal_Type.png%3Falt%3Dmedia%26token%3D9063f511-1e45-4a44-a9c1-14f0de4e4571&width=768&dpr=4&quality=100&sign=835f2f2&sv=2) And finally, you can interact with the finetuned model just like an actual ChatGPT! Hit CTRL + D to exit the system, and hit ENTER to converse with the chatbot! ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fo3vIehaOLOOBlBGBS7lX%252FAssistant.png%3Falt%3Dmedia%26token%3D25319dd2-384c-4744-a2dd-398f48a3b20f&width=768&dpr=4&quality=100&sign=d95d479&sv=2) [](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/tutorial-how-to-finetune-llama-3-and-use-in-ollama#youve-done-it) You've done it! ---------------------------------------------------------------------------------------------------------------------------------------------------------------- You've successfully finetuned a language model and exported it to Ollama with Unsloth 2x faster and with 70% less VRAM! And all this for free in a Google Colab notebook! If you want to learn how to do reward modelling, do continued pretraining, export to vLLM or GGUF, do text completion, or learn more about finetuning tips and tricks, head over to our [Github](https://github.com/unslothai/unsloth#-finetune-for-free) . If you need any help on finetuning, you can also join our Discord server [here](https://discord.gg/unsloth) . If you want help with Ollama, you can also join their server [here](https://discord.gg/ollama) . And finally, we want to thank you for reading and following this far! We hope this made you understand some of the nuts and bolts behind finetuning language models, and we hope this was useful! To access our Alpaca dataset example click [here](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing) , and our CSV / Excel finetuning guide is [here](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing) . [PreviousDevstral: How to Run & Fine-tune](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune) [NextDeepSeek-V3-0324: How to Run Locally](https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally) Last updated 6 months ago Was this helpful? --- # Email Protection | Cloudflare Please enable cookies. Email Protection ================ You are unable to access this email address docs.unsloth.ai ----------------------------------------------------------- The website from which you got to this page is protected by Cloudflare. Email addresses on that page have been hidden in order to keep them from being accessed by malicious bots. **You must enable Javascript in your browser in order to decode the e-mail address**. If you have a website and are interested in protecting it in a similar way, you can [sign up for Cloudflare](https://www.cloudflare.com/sign-up?utm_source=email_protection) . * [How does Cloudflare protect email addresses on website from spammers?](https://developers.cloudflare.com/waf/tools/scrape-shield/email-address-obfuscation/) * [Can I sign up for Cloudflare?](https://developers.cloudflare.com/fundamentals/setup/account/create-account/) Cloudflare Ray ID: **975996318a8bf3c4** ā€¢ Your IP: Click to reveal 54.237.218.47 ā€¢ Performance & security by [Cloudflare](https://www.cloudflare.com/5xx-error-landing) --- # AI Engineer's 2025 | Unsloth Documentation OpenAI gpt-oss & all model types now supported! Here are our attached PDF slides: [8MB\ \ AIE Unsloth 2025.pdf\ \ pdf](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FyUgIxaetqmp3WEnq7quF%2FAIE%20Unsloth%202025.pdf?alt=media&token=c35911ff-69b3-4998-93d9-6583d93659d0) Also here is the link directly to our Google Slides: [https://docs.google.com/presentation/d/1Jh5p\_JDXt4eLD0ireaHJjJNpzqSF8E1WTwIHeojyjNU/edit?usp=drivesdk](https://docs.google.com/presentation/d/1Jh5p_JDXt4eLD0ireaHJjJNpzqSF8E1WTwIHeojyjNU/edit?usp=drivesdk) Here are our slides in HQ PNG format: ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRuRZxQ6OHLhhrgAkrf0I%252FAIE%2520Unsloth%2520%281%29-01.png%3Falt%3Dmedia%26token%3D916d708b-7db8-4ca7-bfb5-5a2a93928bdc&width=768&dpr=4&quality=100&sign=225205e6&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FchaKZv6CNZT5DsKn1Gdj%252FAIE%2520Unsloth%2520%281%29-02.png%3Falt%3Dmedia%26token%3De8746be8-148b-48af-93a4-27a83d23d233&width=768&dpr=4&quality=100&sign=bcc4fe36&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fvi7tRHqBlVMXaNTBXPOF%252FAIE%2520Unsloth%2520%281%29-03.png%3Falt%3Dmedia%26token%3D1de580df-c9b1-4f8e-8e4d-e9a0ab1be082&width=768&dpr=4&quality=100&sign=a11733b4&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6pxj4hY2HNwASVn0ZvGb%252FAIE%2520Unsloth%2520%281%29-04.png%3Falt%3Dmedia%26token%3D042b0ea5-52a8-48ad-a2f1-c52a4d62131b&width=768&dpr=4&quality=100&sign=c66e3688&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FSh0GTur63odeS6O8KcLL%252FAIE%2520Unsloth%2520%281%29-05.png%3Falt%3Dmedia%26token%3D17d5788a-2e3b-468f-ad51-9c03429b9357&width=768&dpr=4&quality=100&sign=6f5936a6&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZg7o2f44wgWxMGtWHUcf%252FAIE%2520Unsloth%2520%281%29-06.png%3Falt%3Dmedia%26token%3Da8678118-86b6-4839-bbd7-423c42feb466&width=768&dpr=4&quality=100&sign=32bb018c&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4n3cPUqWbvHATfSVQmDR%252FAIE%2520Unsloth%2520%281%29-07.png%3Falt%3Dmedia%26token%3Dc48d0a33-1ac6-450d-92bf-5b7bc1105fda&width=768&dpr=4&quality=100&sign=104c28bd&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FeGottYaSY60ySxQ2zIzI%252FAIE%2520Unsloth%2520%281%29-08.png%3Falt%3Dmedia%26token%3D00515422-9a81-4f59-b5bd-04b92c981a47&width=768&dpr=4&quality=100&sign=5a2de07c&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fs7Vb2k2faI0H2SXw3tJ0%252FAIE%2520Unsloth%2520%281%29-09.png%3Falt%3Dmedia%26token%3Df4934dcf-c3cc-4d5b-b97f-dc5faafd80cf&width=768&dpr=4&quality=100&sign=804cf8d5&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FIb1am9A15EeUQcgrwkfX%252FAIE%2520Unsloth%2520%281%29-10.png%3Falt%3Dmedia%26token%3Dcfaa8ca3-6e8c-4f13-837f-fc26c8772777&width=768&dpr=4&quality=100&sign=ede13e43&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FGMy4YHupTnkrcdaYF9MQ%252FAIE%2520Unsloth%2520%281%29-11.png%3Falt%3Dmedia%26token%3D88579ab2-843d-4695-8efa-1b3db20764ca&width=768&dpr=4&quality=100&sign=985452b&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4Hg9AYuB13AqTmkADPG7%252FAIE%2520Unsloth%2520%281%29-12.png%3Falt%3Dmedia%26token%3D16cea707-1b34-4afb-93bd-fc79d0347506&width=768&dpr=4&quality=100&sign=f2d2d369&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fe9IB34Xd7VdivP2PM9oo%252FAIE%2520Unsloth%2520%281%29-13.png%3Falt%3Dmedia%26token%3D639ce619-61c2-4989-ad37-db1b4a60d83d&width=768&dpr=4&quality=100&sign=b4fba1ef&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FYeKbRVxo1EJ7h2hJD5ZT%252FAIE%2520Unsloth%2520%281%29-14.png%3Falt%3Dmedia%26token%3D0dfa13f2-9171-447d-86fe-a9e6f059f44d&width=768&dpr=4&quality=100&sign=8d8d54de&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FerfcsN4zUeA4J5K6phjo%252FAIE%2520Unsloth%2520%281%29-15.png%3Falt%3Dmedia%26token%3D708ca690-d22c-4cba-b6e7-b0a0755bdc0a&width=768&dpr=4&quality=100&sign=d7e776ab&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FGVkoQgTDDqPNeU2aerqh%252FAIE%2520Unsloth%2520%281%29-16.png%3Falt%3Dmedia%26token%3D3e2a677c-e2a7-479d-b541-80dd87f1f121&width=768&dpr=4&quality=100&sign=1bcb2a8&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhLtCUIHPOEbQSw5hpth8%252FAIE%2520Unsloth%2520%281%29-17.png%3Falt%3Dmedia%26token%3D290c64ac-2421-4e4b-9179-c908ca878f91&width=768&dpr=4&quality=100&sign=409ffe57&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F3koqORCtrJhr3OFqeJ5m%252FAIE%2520Unsloth%2520%281%29-18.png%3Falt%3Dmedia%26token%3Da165931e-8cde-45ef-962d-d4e122a43136&width=768&dpr=4&quality=100&sign=6bf12eec&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBArkLGfbluiSHPHH2vTw%252FAIE%2520Unsloth%2520%281%29-19.png%3Falt%3Dmedia%26token%3D112767d6-5819-411f-bdb3-67671e2089ad&width=768&dpr=4&quality=100&sign=28140326&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FEc2gzu8mDz94AbUhxgtK%252FAIE%2520Unsloth%2520%281%29-20.png%3Falt%3Dmedia%26token%3D9d6bfc8c-e6c5-4454-af4e-943b2e568069&width=768&dpr=4&quality=100&sign=d4c4f6d1&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F20eXoC27uHAFBDa8jsfB%252FAIE%2520Unsloth%2520%281%29-21.png%3Falt%3Dmedia%26token%3D499b874c-58d9-43db-9e5f-20caa838bebe&width=768&dpr=4&quality=100&sign=b4dc32ec&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FmOsOpHog91eQpqsLDKUS%252FAIE%2520Unsloth%2520%281%29-22.png%3Falt%3Dmedia%26token%3D79a20536-e47a-45b4-8dc3-081846efd984&width=768&dpr=4&quality=100&sign=81a3fb66&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fjuf7GoE8Kv0lQ9DMyaK1%252FAIE%2520Unsloth%2520%281%29-23.png%3Falt%3Dmedia%26token%3Dc9d088a0-35bd-465c-b116-629503241e57&width=768&dpr=4&quality=100&sign=794eaed&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBam1dH0HygkeCp1e6olR%252FAIE%2520Unsloth%2520%281%29-24.png%3Falt%3Dmedia%26token%3D3b118a94-6c49-4d75-8088-9567de043530&width=768&dpr=4&quality=100&sign=8e7424a8&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRzvFSxDW6m3qlZ4LJA5u%252FAIE%2520Unsloth%2520%281%29-25.png%3Falt%3Dmedia%26token%3D5e5deffe-4b8c-4dd8-a632-0b4c6c62a4cc&width=768&dpr=4&quality=100&sign=817a6cbb&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FTYVfNjvOuvxdieQeaUtn%252FAIE%2520Unsloth%2520%281%29-26.png%3Falt%3Dmedia%26token%3D0ccfccb7-b969-4277-96e6-74d2aa27b407&width=768&dpr=4&quality=100&sign=38af73b0&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FS2ZY4K4fxAaKooEU0oen%252FAIE%2520Unsloth%2520%281%29-27.png%3Falt%3Dmedia%26token%3D426aadd6-c303-4840-a83a-feaf2fbacfc5&width=768&dpr=4&quality=100&sign=8c5b86a6&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FUZbJ4y4cj0LzgcKvLnqF%252FAIE%2520Unsloth%2520%281%29-28.png%3Falt%3Dmedia%26token%3D43e62860-0b0d-444b-b9b2-031333778a0c&width=768&dpr=4&quality=100&sign=49d79886&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FSVlC530EbbXfi6KH9TKt%252FAIE%2520Unsloth%2520%281%29-29.png%3Falt%3Dmedia%26token%3D14ec6f17-1c7f-42a5-9967-b0957c5c3985&width=768&dpr=4&quality=100&sign=f75b341f&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfVz59AAOAB8ZF5kF76Cs%252FAIE%2520Unsloth%2520%281%29-30.png%3Falt%3Dmedia%26token%3D2ad02d87-e98a-4f97-b001-9a63e5a867c1&width=768&dpr=4&quality=100&sign=f062bfe0&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlSZP4hx5ZHSqOlqL89jp%252FAIE%2520Unsloth%2520%281%29-31.png%3Falt%3Dmedia%26token%3D8e11e355-aa80-46e0-bf7b-631e3d50a9aa&width=768&dpr=4&quality=100&sign=5f11666e&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfT4HFTMoZfw1ZfZGWOwQ%252FAIE%2520Unsloth%2520%281%29-32.png%3Falt%3Dmedia%26token%3Df14e770c-9f1e-45b6-b76b-35de6efe6aff&width=768&dpr=4&quality=100&sign=7b06bf&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHxWwHczRWg3Moj19zHCl%252FAIE%2520Unsloth%2520%281%29-33.png%3Falt%3Dmedia%26token%3Df2a90e85-f6c0-4f78-94a4-896ed1fd2223&width=768&dpr=4&quality=100&sign=11af7632&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBN44ouqx2pw8Tdm6ETq2%252FAIE%2520Unsloth%2520%281%29-34.png%3Falt%3Dmedia%26token%3D98d6d32b-eea4-4cf1-97d4-06a9f4ffaca6&width=768&dpr=4&quality=100&sign=8139c0b5&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkBtqkjyuRW7BxsmSnTmu%252FAIE%2520Unsloth%2520%281%29-35.png%3Falt%3Dmedia%26token%3D8d6fd18e-8ec3-492b-8c5a-f38d85c5c48f&width=768&dpr=4&quality=100&sign=a9afa50d&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FtClyduLFY6NseLjvxaox%252FAIE%2520Unsloth%2520%281%29-36.png%3Falt%3Dmedia%26token%3D0b4fda7b-805f-4a3e-ae62-e0003090eb91&width=768&dpr=4&quality=100&sign=83764660&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxZBGA66bur3OuPuyWJVz%252FAIE%2520Unsloth%2520%281%29-37.png%3Falt%3Dmedia%26token%3Db643b98b-323d-40f0-80da-1e9aa8989dca&width=768&dpr=4&quality=100&sign=12be5308&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F9eBcl8AeTwmz342kECpV%252FAIE%2520Unsloth%2520%281%29-38.png%3Falt%3Dmedia%26token%3D239522c8-7b4f-41d6-925a-90d46f6383ec&width=768&dpr=4&quality=100&sign=9db262b&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F6cpur0plqwlmpOKUTHrd%252FAIE%2520Unsloth%2520%281%29-39.png%3Falt%3Dmedia%26token%3D35f542cf-3b21-47d0-8bbe-c4a1afc86479&width=768&dpr=4&quality=100&sign=dc3f83c6&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fe7wK4RcuWLUHBmzlaw8E%252FAIE%2520Unsloth%2520%281%29-40.png%3Falt%3Dmedia%26token%3D8ea76313-e3cd-408b-be55-a2799b8e99f6&width=768&dpr=4&quality=100&sign=16911fdc&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxNK4M3U4GCKmvpvxUca1%252FAIE%2520Unsloth%2520%281%29-41.png%3Falt%3Dmedia%26token%3D427b1cba-0c8a-42f2-a80a-7da6bc7c895e&width=768&dpr=4&quality=100&sign=9daf13e4&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FlBrG6QKSpErrhq1bnnZ5%252FAIE%2520Unsloth%2520%281%29-42.png%3Falt%3Dmedia%26token%3D1b6a782c-6dc9-44c9-bc3d-a7bbd6146c91&width=768&dpr=4&quality=100&sign=4c59efd3&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F24zISdxWjNaLSqD2cGIL%252FAIE%2520Unsloth%2520%281%29-43.png%3Falt%3Dmedia%26token%3D6e04e2ca-dff4-4bd8-bbef-8398e2964829&width=768&dpr=4&quality=100&sign=49180609&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FnoooOyArWwqouMulgtME%252FAIE%2520Unsloth%2520%281%29-44.png%3Falt%3Dmedia%26token%3Da042500c-734d-49c4-bcd7-fe0395b9dde8&width=768&dpr=4&quality=100&sign=296e268e&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fi26B5Ch2tMTTQ81l1UAW%252FAIE%2520Unsloth%2520%281%29-45.png%3Falt%3Dmedia%26token%3D8794ff86-17d1-48e5-8b50-e78a1afe4c70&width=768&dpr=4&quality=100&sign=55d08c8a&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJvQar2DX6bi8IJ0PgoZN%252FAIE%2520Unsloth%2520%281%29-46.png%3Falt%3Dmedia%26token%3Da044d4f3-3af1-449b-ba20-1103973e37b4&width=768&dpr=4&quality=100&sign=ab78f4fb&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FBmrPgEta4Fkxx7eSh3gJ%252FAIE%2520Unsloth%2520%281%29-47.png%3Falt%3Dmedia%26token%3D3c57929a-d240-48cf-aaa2-26ce0709369d&width=768&dpr=4&quality=100&sign=ef0c2779&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMgQ2ES4GIQUSWXZMBDIw%252FAIE%2520Unsloth%2520%281%29-48.png%3Falt%3Dmedia%26token%3D9a89e122-09cf-4a7f-aa07-3cd79c25047d&width=768&dpr=4&quality=100&sign=5cda3b92&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2BC6bqgojmTz2PVximvZ%252FAIE%2520Unsloth%2520%281%29-49.png%3Falt%3Dmedia%26token%3D7fcc1e2b-6757-4117-beb3-b46cf9e15d52&width=768&dpr=4&quality=100&sign=d52267fa&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FgabcNj5r8kKh2VbpwAsJ%252FAIE%2520Unsloth%2520%281%29-50.png%3Falt%3Dmedia%26token%3D322b37c9-637d-4dca-91e6-de3e4cce6006&width=768&dpr=4&quality=100&sign=7ea182db&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FfS1QS0A1o6SuBHtLFsGc%252FAIE%2520Unsloth%2520%281%29-51.png%3Falt%3Dmedia%26token%3D9f046d22-0b14-42ab-aab4-94a0527dbef5&width=768&dpr=4&quality=100&sign=e203942c&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FkZCxnUATExVHJanArCki%252FAIE%2520Unsloth%2520%281%29-52.png%3Falt%3Dmedia%26token%3D1ff6e5b3-af73-47c0-96a4-addb90db256e&width=768&dpr=4&quality=100&sign=22ca619d&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FumfxEXdGN15JnhgEp0Wp%252FAIE%2520Unsloth%2520%281%29-53.png%3Falt%3Dmedia%26token%3Ddec31337-ec39-4a13-b95a-428f2facd505&width=768&dpr=4&quality=100&sign=bc1e77f5&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FFehsauyUwJSLjRpnOvaP%252FAIE%2520Unsloth%2520%281%29-54.png%3Falt%3Dmedia%26token%3Dd975a569-03fa-404b-9c65-f3699479a180&width=768&dpr=4&quality=100&sign=97c64d6&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F16Du8xk7jgeoNJOUUkCA%252FAIE%2520Unsloth%2520%281%29-55.png%3Falt%3Dmedia%26token%3Ded91b754-bdb9-4299-ae53-f6924cd26d4e&width=768&dpr=4&quality=100&sign=18bf41fb&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDlQ4YMgvNGFmWmJnz075%252FAIE%2520Unsloth%2520%281%29-56.png%3Falt%3Dmedia%26token%3Dce08bac5-6e79-40e1-a740-6f330fdebfe6&width=768&dpr=4&quality=100&sign=c66ea33e&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FGiEOysr6YmUW7FVO1UlA%252FAIE%2520Unsloth%2520%281%29-57.png%3Falt%3Dmedia%26token%3Dca7c6f6f-3968-4f9c-986f-c299f99083ca&width=768&dpr=4&quality=100&sign=4c309d15&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FjL6Z4yxaaNykyBmhN1m0%252FAIE%2520Unsloth%2520%281%29-58.png%3Falt%3Dmedia%26token%3D785c2533-99c9-4888-9eb6-e47d3b708d9c&width=768&dpr=4&quality=100&sign=5d6dd72f&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F4HJ9kkp3mT2TQXvnzzGW%252FAIE%2520Unsloth%2520%281%29-59.png%3Falt%3Dmedia%26token%3D0d17f216-3022-4043-9d60-1173f5a51ee6&width=768&dpr=4&quality=100&sign=e0f1fd87&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fs4fk0cTykxfrUK2uYc37%252FAIE%2520Unsloth%2520%281%29-60.png%3Falt%3Dmedia%26token%3D83be1d8f-be2c-454d-9f91-6c61cea5fe64&width=768&dpr=4&quality=100&sign=7a6ae1e1&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FpEqD7Nk8nF0rGENJacXf%252FAIE%2520Unsloth%2520%281%29-61.png%3Falt%3Dmedia%26token%3D8fe883cc-0566-4f90-bf10-6a4c1db91c71&width=768&dpr=4&quality=100&sign=3abb4509&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F3HZQYVnTbrDPDHlKMT3L%252FAIE%2520Unsloth%2520%281%29-62.png%3Falt%3Dmedia%26token%3Db96ace56-4c0a-40aa-8cdd-8ebc288d9c83&width=768&dpr=4&quality=100&sign=b3d156d2&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2e96E8rDLhCA1zaBNc7G%252FAIE%2520Unsloth%2520%281%29-63.png%3Falt%3Dmedia%26token%3D1cfdd39d-1abe-4641-9471-58ad320cab24&width=768&dpr=4&quality=100&sign=5dd9fd80&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FcXGSr6xGTwDdJxvD27FX%252FAIE%2520Unsloth%2520%281%29-64.png%3Falt%3Dmedia%26token%3Da24b9e8e-54b1-466e-b148-73d9ce46ca5a&width=768&dpr=4&quality=100&sign=6165a7a2&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F1MhSOq9anPJP25wiUkZk%252FAIE%2520Unsloth%2520%281%29-65.png%3Falt%3Dmedia%26token%3D95a181e1-33db-45dd-89ca-cd7cbd5de01f&width=768&dpr=4&quality=100&sign=d80f966e&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8dQjyczy7jAWFlchSveL%252FAIE%2520Unsloth%2520%281%29-66.png%3Falt%3Dmedia%26token%3D01339b05-46b3-4599-ba7e-19f93d52203d&width=768&dpr=4&quality=100&sign=f7825e5f&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FTJFTMQMPoOYidFXk7SaV%252FAIE%2520Unsloth%2520%281%29-67.png%3Falt%3Dmedia%26token%3D330ac6ce-3f79-48c5-9960-4e7a40e64858&width=768&dpr=4&quality=100&sign=5d8bb389&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FJCfOKizJ6uk92Wr3Jymf%252FAIE%2520Unsloth%2520%281%29-68.png%3Falt%3Dmedia%26token%3D1cd35e8c-84b0-48d0-8f46-97c0685def63&width=768&dpr=4&quality=100&sign=9361a628&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FbcfFr5642DclTKz6FYvl%252FAIE%2520Unsloth%2520%281%29-69.png%3Falt%3Dmedia%26token%3D9a9c3097-aebb-418a-b435-b689c5e89eef&width=768&dpr=4&quality=100&sign=86fc511&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F53X3LetRIcKAHBRHOHIG%252FAIE%2520Unsloth%2520%281%29-70.png%3Falt%3Dmedia%26token%3D2f65f640-f537-4831-b903-ad68821ab14a&width=768&dpr=4&quality=100&sign=ada29b66&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FZ3cG2i50i0hpgfh9E6d7%252FAIE%2520Unsloth%2520%281%29-71.png%3Falt%3Dmedia%26token%3Dce574b22-562d-4a4e-9671-5a56788642b4&width=768&dpr=4&quality=100&sign=98282e53&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FSyaeTA0F1ErAMfW3X9cQ%252FAIE%2520Unsloth%2520%281%29-72.png%3Falt%3Dmedia%26token%3D698d04f3-1dbf-4a82-a113-4751c52ee1da&width=768&dpr=4&quality=100&sign=5d13673&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fg58ZbLt4J6aykhUNiB8j%252FAIE%2520Unsloth%2520%281%29-73.png%3Falt%3Dmedia%26token%3D74603968-3c3d-4d26-b769-389b832b9efb&width=768&dpr=4&quality=100&sign=89dd53f7&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FQ9x7bWRLGOBRp39zPbHh%252FAIE%2520Unsloth%2520%281%29-74.png%3Falt%3Dmedia%26token%3D83951f0d-dc09-4447-ba51-84789cfbf9ce&width=768&dpr=4&quality=100&sign=c5bd34c4&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRyIs7O8i2ASb2qrBemxy%252FAIE%2520Unsloth%2520%281%29-75.png%3Falt%3Dmedia%26token%3D9d79032c-2c96-4699-8546-94af30383367&width=768&dpr=4&quality=100&sign=38470402&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FHfXpAQbzDRfmHm2ZKE14%252FAIE%2520Unsloth%2520%281%29-76.png%3Falt%3Dmedia%26token%3Dd12c240a-4029-4c3a-88a1-446d395ba7d2&width=768&dpr=4&quality=100&sign=5327bd73&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7L6UBeS9xzIIDlyPy8bt%252FAIE%2520Unsloth%2520%281%29-77.png%3Falt%3Dmedia%26token%3Dd53d864a-b7a7-4b87-91a5-04dc8068bd6d&width=768&dpr=4&quality=100&sign=3946cbe&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Ff4U2olY2xuUhfcbwveza%252FAIE%2520Unsloth%2520%281%29-78.png%3Falt%3Dmedia%26token%3D3534551b-12d3-4c6e-935d-5074a7a62af4&width=768&dpr=4&quality=100&sign=1a45477a&sv=2) ![](https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FgcI6ysJBjCb6jgoNvuxW%252FAIE%2520Unsloth%2520%281%29-79.png%3Falt%3Dmedia%26token%3Dd4b7d799-3cce-458b-bdb4-ad60947adb7b&width=768&dpr=4&quality=100&sign=34484441&sv=2) [](https://docs.unsloth.ai/ai-engineers-2025#ai-engineers-2025-3-hour-workshop-video) šŸŽ„ AI Engineer's 2025 3 hour Workshop Video -------------------------------------------------------------------------------------------------------------------------------------- Last updated 1 month ago Was this helpful? ---