# Table of Contents - [Qwen](#qwen) - [Qwen](#qwen) - [Qwen](#qwen) - [Qwen](#qwen) - [Qwen](#qwen) - [Qwen](#qwen) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [SFT - Qwen](#sft-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [SFT - Qwen](#sft-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [Example - Qwen](#example-qwen) - [SFT - Qwen](#sft-qwen) - [RL - Qwen](#rl-qwen) - [Function Calling - Qwen](#function-calling-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [ms-swift - Qwen](#ms-swift-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [ms-swift - Qwen](#ms-swift-qwen) - [HF Transformers Inference - Qwen](#hf-transformers-inference-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [Performance of Quantized Models - Qwen](#performance-of-quantized-models-qwen) - [Langchain - Qwen](#langchain-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [Langchain - Qwen](#langchain-qwen) - [Speed Benchmark - Qwen](#speed-benchmark-qwen) - [llama.cpp - Qwen](#llama-cpp-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [verl - Qwen](#verl-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [llama.cpp - Qwen](#llama-cpp-qwen) - [verl - Qwen](#verl-qwen) - [Axolotl - Qwen](#axolotl-qwen) - [Axolotl - Qwen](#axolotl-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [Performance of Quantized Models - Qwen](#performance-of-quantized-models-qwen) - [Axolotl - Qwen](#axolotl-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [LLaMA-Factory - Qwen](#llama-factory-qwen) - [Unsloth - Qwen](#unsloth-qwen) - [verl - Qwen](#verl-qwen) - [Qwen-Agent - Qwen](#qwen-agent-qwen) - [MS-SWIFT - Qwen](#ms-swift-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [MS-SWIFT - Qwen](#ms-swift-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [Langchain - Qwen](#langchain-qwen) - [Qwen2.5 Speed Benchmark - Qwen](#qwen2-5-speed-benchmark-qwen) - [LlamaIndex - Qwen](#llamaindex-qwen) - [MS-SWIFT - Qwen](#ms-swift-qwen) - [Langchain - Qwen](#langchain-qwen) - [Langchain - Qwen](#langchain-qwen) - [Langchain - Qwen](#langchain-qwen) - [Function Calling - Qwen](#function-calling-qwen) - [Function Calling - Qwen](#function-calling-qwen) - [Function Calling - Qwen](#function-calling-qwen) - [Function Calling - Qwen](#function-calling-qwen) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/index.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/index.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Welcome to Qwen![¶](https://qwen.readthedocs.io/en/v1.5/#welcome-to-qwen "Permalink to this heading") ====================================================================================================== ![Qwen](https://qwen.readthedocs.io/en/v1.5/_images/qwen1.5.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen1.5. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen1.5, has the following features: * 6 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, and 72B; * Base and chat models for each size, with the chat models aligned for human preferences; * Multilingual support of both base and chat models; * Stable support of 32K context length for models of all sizes * Supporting tool use, RAG, role play, and playing as AI agent; For more information, please visit our: * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen1.5 Collection](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! Documentation[¶](https://qwen.readthedocs.io/en/v1.5/#documentation "Permalink to this heading") ------------------------------------------------------------------------------------------------- Getting Started * [Installation](https://qwen.readthedocs.io/en/v1.5/getting_started/installation.html) * [Quickstart](https://qwen.readthedocs.io/en/v1.5/getting_started/quickstart.html) Inference * [Using Transformers to Chat](https://qwen.readthedocs.io/en/v1.5/inference/chat.html) Run Locally * [llama.cpp](https://qwen.readthedocs.io/en/v1.5/run_locally/llama.cpp.html) * [Ollama](https://qwen.readthedocs.io/en/v1.5/run_locally/ollama.html) Web UI * [Text Generation Web UI](https://qwen.readthedocs.io/en/v1.5/web_ui/text_generation_webui.html) Quantization * [AWQ](https://qwen.readthedocs.io/en/v1.5/quantization/awq.html) * [GPTQ](https://qwen.readthedocs.io/en/v1.5/quantization/gptq.html) * [GGUF](https://qwen.readthedocs.io/en/v1.5/quantization/gguf.html) Deployment * [vLLM](https://qwen.readthedocs.io/en/v1.5/deployment/vllm.html) * [SkyPilot](https://qwen.readthedocs.io/en/v1.5/deployment/skypilot.html) Training * [SFT](https://qwen.readthedocs.io/en/v1.5/training/SFT/index.html) * [Example](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html) * [LLaMA-Factory](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html) Framework * [Function Calling](https://qwen.readthedocs.io/en/v1.5/framework/function_call.html) * [Qwen-Agent](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html) * [LlamaIndex](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html) * [Langchain](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html) Benchmark * [HF Transformers Inference](https://qwen.readthedocs.io/en/v1.5/benchmark/hf_infer.html) --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/index.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/index.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Welcome to Qwen![¶](https://qwen.readthedocs.io/en/v2.0/#welcome-to-qwen "Link to this heading") ================================================================================================= ![Qwen](https://qwen.readthedocs.io/en/v2.0/_images/qwen2.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen2. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen2, has the following features: * 5 sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B; * Base and instruction-tuned models for each size, with the instruction-tuned models aligned for human preferences; * Multilingual support of both base and instruction-tuned models; * Stable support of 32K context length for models of all sizes and up to 128K tokens with Qwen2-7B-Instruct and Qwen2-72B-Instruct; * Supporting tool use, RAG, role play, and playing as AI agent; For more information, please visit our: * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen2 Collection](https://huggingface.co/collections/Qwen/qwen2-6659360b33528ced941e557f) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/index.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Welcome to Qwen![¶](https://qwen.readthedocs.io/en/v2.5/#welcome-to-qwen "Link to this heading") ================================================================================================= ![Qwen2.5](https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/logo/qwen2.5_logo.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen2.5. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen2.5, has the following features: * Dense, easy-to-use, decoder-only language models, available in **0.5B**, **1.5B**, **3B**, **7B**, **14B**, **32B**, and **72B** sizes, and base and instruct variants. * Pretrained on our latest large-scale dataset, encompassing up to **18T** tokens. * Significant improvements in instruction following, generating long texts (over 8K tokens), understanding structured data (e.g, tables), and generating structured outputs especially JSON. * More resilient to the diversity of system prompts, enhancing role-play implementation and condition-setting for chatbots. * Context length support up to **128K** tokens and can generate up to **8K** tokens. * Multilingual support for over **29** languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. For more information, please visit our: * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen2.5 Collection](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/index.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Welcome to Qwen![¶](https://qwen.readthedocs.io/en/stable/#welcome-to-qwen "Link to this heading") =================================================================================================== ![Qwen3](https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen3, has the following features: * **Dense and Mixture-of-Experts (MoE) models**, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. * **Seamless switching between thinking mode** (for complex logical reasoning, math, and coding) and **non-thinking mode** (for efficient, general-purpose chat) **within a single model**, ensuring optimal performance across various scenarios. * **Significantly enhancement in reasoning capabilities**, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. * **Superior human preference alignment**, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. * **Expertise in agent capabilities**, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. * **Support of 100+ languages and dialects** with strong capabilities for **multilingual instruction following** and **translation**. For more information, please visit our: * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen3 Collection](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/index.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Welcome to Qwen![¶](https://qwen.readthedocs.io/en/v3.0/#welcome-to-qwen "Link to this heading") ================================================================================================= ![Qwen3](https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen3, has the following features: * **Dense and Mixture-of-Experts (MoE) models**, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. * **Seamless switching between thinking mode** (for complex logical reasoning, math, and coding) and **non-thinking mode** (for efficient, general-purpose chat) **within a single model**, ensuring optimal performance across various scenarios. * **Significantly enhancement in reasoning capabilities**, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. * **Superior human preference alignment**, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. * **Expertise in agent capabilities**, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. * **Support of 100+ languages and dialects** with strong capabilities for **multilingual instruction following** and **translation**. For more information, please visit our: * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen3 Collection](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! --- # Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/index.rst.txt "View this page") Welcome to Qwen![¶](https://qwen.readthedocs.io/en/latest/#welcome-to-qwen "Link to this heading") =================================================================================================== ![Qwen3](https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png) Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Qwen3-2507[¶](https://qwen.readthedocs.io/en/latest/#qwen3-2507 "Link to this heading") ---------------------------------------------------------------------------------------- With input from the community and insights from further research, Instruct-only and Thinking-only models are coming back! The results are Qwen3-2507: **Qwen3-Instruct-2507** has the following features: * **Significant improvements** in general capabilities, including **instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage**. * **Substantial gains** in long-tail knowledge coverage across **multiple languages**. * **Markedly better alignment** with user preferences in **subjective and open-ended tasks**, enabling more helpful responses and higher-quality text generation. * **Enhanced capabilities** in **256K long-context understanding**, extensible to 1M. **Qwen3-Thinking-2507** has the following features: * **Significantly improved performance** on reasoning tasks, including logical reasoning, mathematics, science, coding, and academic benchmarks that typically require human expertise — achieving **state-of-the-art results among open-source thinking models**. * **Markedly better general capabilities**, such as instruction following, tool usage, text generation, and alignment with human preferences. * **Enhanced 256K long-context understanding** capabilities, extensible to 1M. Qwen3[¶](https://qwen.readthedocs.io/en/latest/#qwen3 "Link to this heading") ------------------------------------------------------------------------------ Qwen3, aka Qwen3-2504, has the following features: * **Dense and Mixture-of-Experts (MoE) models**, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. * **Seamless switching between thinking mode** (for complex logical reasoning, math, and coding) and **non-thinking mode** (for efficient, general-purpose chat) **within a single model**, ensuring optimal performance across various scenarios. * **Significantly enhancement in reasoning capabilities**, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. * **Superior human preference alignment**, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. * **Expertise in agent capabilities**, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. * **Support of 100+ languages and dialects** with strong capabilities for **multilingual instruction following** and **translation**. Resource & Links[¶](https://qwen.readthedocs.io/en/latest/#resource-links "Link to this heading") -------------------------------------------------------------------------------------------------- For more information, please visit our: * [Qwen Home Page](https://qwen.ai/) * [Chat with Qwen (with Deep Research and Web Dev)](https://chat.qwen.ai/) * [Blog](https://qwenlm.github.io/) * [GitHub](https://github.com/QwenLM) * [Hugging Face](https://huggingface.co/Qwen) * [ModelScope](https://modelscope.cn/organization/qwen) * [Qwen3 Collection](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) Join our community by joining our [Discord](https://discord.gg/yPEP2vHTu4) and [WeChat](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png) group. We are looking forward to seeing you there! --- # Unknown .. Qwen documentation master file, created by sphinx-quickstart on Tue Feb 20 21:16:41 2024. You can adapt this file completely to your liking, but it should at least contain the root \`toctree\` directive. Welcome to Qwen! ================ .. figure:: ./assets/logo/qwen1.5.png :width: 60% :align: center :alt: Qwen :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen1.5. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen1.5, has the following features: \* 6 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, and 72B; \* Base and chat models for each size, with the chat models aligned for human preferences; \* Multilingual support of both base and chat models; \* Stable support of 32K context length for models of all sizes \* Supporting tool use, RAG, role play, and playing as AI agent; For more information, please visit our: \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen1.5 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! Documentation ------------- .. toctree:: :maxdepth: 1 :caption: Getting Started getting\_started/installation getting\_started/quickstart .. toctree:: :maxdepth: 1 :caption: Inference inference/chat .. toctree:: :maxdepth: 1 :caption: Run Locally run\_locally/llama.cpp run\_locally/ollama .. toctree:: :maxdepth: 1 :caption: Web UI web\_ui/text\_generation\_webui .. toctree:: :maxdepth: 1 :caption: Quantization quantization/awq quantization/gptq quantization/gguf .. toctree:: :maxdepth: 1 :caption: Deployment deployment/vllm deployment/skypilot .. toctree:: :maxdepth: 2 :caption: Training training/SFT/index .. toctree:: :maxdepth: 1 :caption: Framework framework/function\_call framework/qwen\_agent framework/LlamaIndex framework/Langchain .. toctree:: :maxdepth: 1 :caption: Benchmark benchmark/hf\_infer --- # Unknown Welcome to Qwen! ================ .. figure:: ./assets/logo/qwen2.png :width: 60% :align: center :alt: Qwen :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen2. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen2, has the following features: \* 5 sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B; \* Base and instruction-tuned models for each size, with the instruction-tuned models aligned for human preferences; \* Multilingual support of both base and instruction-tuned models; \* Stable support of 32K context length for models of all sizes and up to 128K tokens with Qwen2-7B-Instruct and Qwen2-72B-Instruct; \* Supporting tool use, RAG, role play, and playing as AI agent; For more information, please visit our: \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen2 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting\_started/quickstart getting\_started/concepts .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/chat .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run\_locally/ollama run\_locally/mlx-lm run\_locally/llama.cpp .. toctree:: :maxdepth: 1 :caption: Web UI :hidden: web\_ui/text\_generation\_webui .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/vllm deployment/tgi deployment/skypilot .. toctree:: :maxdepth: 2 :caption: Training :hidden: training/SFT/index .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/function\_call framework/qwen\_agent framework/LlamaIndex framework/Langchain .. toctree:: :maxdepth: 1 :caption: Benchmark :hidden: benchmark/quantization\_benchmark benchmark/speed\_benchmark --- # Unknown Welcome to Qwen! ================ .. figure:: https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo\_qwen3.png :width: 60% :align: center :alt: Qwen3 :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen3, has the following features: - \*\*Dense and Mixture-of-Experts (MoE) models\*\*, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. - \*\*Seamless switching between thinking mode\*\* (for complex logical reasoning, math, and coding) and \*\*non-thinking mode\*\* (for efficient, general-purpose chat) \*\*within a single model\*\*, ensuring optimal performance across various scenarios. - \*\*Significantly enhancement in reasoning capabilities\*\*, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. - \*\*Superior human preference alignment\*\*, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. - \*\*Expertise in agent capabilities\*\*, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. - \*\*Support of 100+ languages and dialects\*\* with strong capabilities for \*\*multilingual instruction following\*\* and \*\*translation\*\*. For more information, please visit our: \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen3 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting\_started/quickstart getting\_started/concepts getting\_started/speed\_benchmark getting\_started/quantization\_benchmark .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/transformers .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run\_locally/llama.cpp run\_locally/ollama run\_locally/mlx-lm .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/sglang deployment/vllm deployment/tgi deployment/dstack deployment/skypilot deployment/openllm .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Training :hidden: training/llama\_factory training/ms\_swift training/verl training/axolotl .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/function\_call framework/qwen\_agent framework/LlamaIndex framework/Langchain --- # Unknown Welcome to Qwen! ================ .. figure:: https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo\_qwen3.png :width: 60% :align: center :alt: Qwen3 :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen3, has the following features: - \*\*Dense and Mixture-of-Experts (MoE) models\*\*, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. - \*\*Seamless switching between thinking mode\*\* (for complex logical reasoning, math, and coding) and \*\*non-thinking mode\*\* (for efficient, general-purpose chat) \*\*within a single model\*\*, ensuring optimal performance across various scenarios. - \*\*Significantly enhancement in reasoning capabilities\*\*, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. - \*\*Superior human preference alignment\*\*, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. - \*\*Expertise in agent capabilities\*\*, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. - \*\*Support of 100+ languages and dialects\*\* with strong capabilities for \*\*multilingual instruction following\*\* and \*\*translation\*\*. For more information, please visit our: \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen3 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting\_started/quickstart getting\_started/concepts getting\_started/speed\_benchmark getting\_started/quantization\_benchmark .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/transformers .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run\_locally/llama.cpp run\_locally/ollama run\_locally/mlx-lm .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/sglang deployment/vllm deployment/tgi deployment/dstack deployment/skypilot deployment/openllm .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Training :hidden: training/llama\_factory training/ms\_swift training/verl training/axolotl .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/function\_call framework/qwen\_agent framework/LlamaIndex framework/Langchain --- # Unknown Welcome to Qwen! ================ .. figure:: https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/logo/qwen2.5\_logo.png :width: 60% :align: center :alt: Qwen2.5 :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Now the large language models have been upgraded to Qwen2.5. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. The latest version, Qwen2.5, has the following features: - Dense, easy-to-use, decoder-only language models, available in \*\*0.5B\*\*, \*\*1.5B\*\*, \*\*3B\*\*, \*\*7B\*\*, \*\*14B\*\*, \*\*32B\*\*, and \*\*72B\*\* sizes, and base and instruct variants. - Pretrained on our latest large-scale dataset, encompassing up to \*\*18T\*\* tokens. - Significant improvements in instruction following, generating long texts (over 8K tokens), understanding structured data (e.g, tables), and generating structured outputs especially JSON. - More resilient to the diversity of system prompts, enhancing role-play implementation and condition-setting for chatbots. - Context length support up to \*\*128K\*\* tokens and can generate up to \*\*8K\*\* tokens. - Multilingual support for over \*\*29\*\* languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. For more information, please visit our: \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen2.5 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting\_started/quickstart getting\_started/concepts .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/chat .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run\_locally/ollama run\_locally/mlx-lm run\_locally/llama.cpp .. toctree:: :maxdepth: 1 :caption: Web UI :hidden: web\_ui/text\_generation\_webui .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/vllm deployment/tgi deployment/skypilot deployment/openllm .. toctree:: :maxdepth: 2 :caption: Training :hidden: training/SFT/index training/RL/index .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/function\_call framework/qwen\_agent framework/LlamaIndex framework/Langchain .. toctree:: :maxdepth: 1 :caption: Benchmark :hidden: benchmark/quantization\_benchmark benchmark/speed\_benchmark --- # Unknown Welcome to Qwen! ================ .. figure:: https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo\_qwen3.png :width: 60% :align: center :alt: Qwen3 :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Qwen3-2507 ---------- With input from the community and insights from further research, Instruct-only and Thinking-only models are coming back! The results are Qwen3-2507: \*\*Qwen3-Instruct-2507\*\* has the following features: - \*\*Significant improvements\*\* in general capabilities, including \*\*instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage\*\*. - \*\*Substantial gains\*\* in long-tail knowledge coverage across \*\*multiple languages\*\*. - \*\*Markedly better alignment\*\* with user preferences in \*\*subjective and open-ended tasks\*\*, enabling more helpful responses and higher-quality text generation. - \*\*Enhanced capabilities\*\* in \*\*256K long-context understanding\*\*, extensible to 1M. \*\*Qwen3-Thinking-2507\*\* has the following features: - \*\*Significantly improved performance\*\* on reasoning tasks, including logical reasoning, mathematics, science, coding, and academic benchmarks that typically require human expertise — achieving \*\*state-of-the-art results among open-source thinking models\*\*. - \*\*Markedly better general capabilities\*\*, such as instruction following, tool usage, text generation, and alignment with human preferences. - \*\*Enhanced 256K long-context understanding\*\* capabilities, extensible to 1M. Qwen3 ----- Qwen3, aka Qwen3-2504, has the following features: - \*\*Dense and Mixture-of-Experts (MoE) models\*\*, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. - \*\*Seamless switching between thinking mode\*\* (for complex logical reasoning, math, and coding) and \*\*non-thinking mode\*\* (for efficient, general-purpose chat) \*\*within a single model\*\*, ensuring optimal performance across various scenarios. - \*\*Significantly enhancement in reasoning capabilities\*\*, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. - \*\*Superior human preference alignment\*\*, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. - \*\*Expertise in agent capabilities\*\*, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. - \*\*Support of 100+ languages and dialects\*\* with strong capabilities for \*\*multilingual instruction following\*\* and \*\*translation\*\*. Resource & Links ---------------- For more information, please visit our: \* \`Qwen Home Page \`\_\_ \* \`Chat with Qwen (with Deep Research and Web Dev) \`\_\_ \* \`Blog \`\_\_ \* \`GitHub \`\_\_ \* \`Hugging Face \`\_\_ \* \`ModelScope \`\_\_ \* \`Qwen3 Collection \`\_\_ Join our community by joining our \`Discord \`\_\_ and \`WeChat \`\_\_ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting\_started/quickstart getting\_started/concepts getting\_started/speed\_benchmark getting\_started/quantization\_benchmark .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/transformers .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run\_locally/llama.cpp run\_locally/ollama run\_locally/mlx-lm .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/sglang deployment/vllm deployment/tgi deployment/dstack deployment/skypilot deployment/openllm .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Training :hidden: training/axolotl training/llama\_factory training/ms\_swift training/unsloth training/verl .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/qwen\_agent framework/function\_call framework/LlamaIndex framework/Langchain --- # SFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/training/SFT/index.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/training/SFT/index.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/training/SFT/index.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/training/SFT/index.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar SFT[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/index.html#sft "Permalink to this heading") ==================================================================================================== * [Example](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html) * [LLaMA-Factory](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html) --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/training/SFT/llama_factory.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/training/SFT/llama_factory.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LLaMA-Factory[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#llama-factory "Permalink to this heading") ================================================================================================================================ Here we provide a script for supervised finetuning Qwen1.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#installation "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------------ Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#data-preparation "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follow. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#training "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#merge-lora "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/llama_factory.html#conclusion "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # SFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/training/SFT/index.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/training/SFT/index.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/training/SFT/index.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/training/SFT/index.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar SFT[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/index.html#sft "Link to this heading") =============================================================================================== * [LLaMA-Factory](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html) --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/training/SFT/llama_factory.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/training/SFT/llama_factory.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LLaMA-Factory[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#llama-factory "Link to this heading") =========================================================================================================================== Here we provide a script for supervised finetuning Qwen2 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#data-preparation "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#training "Link to this heading") ----------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#merge-lora "Link to this heading") --------------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/v2.0/training/SFT/llama_factory.html#conclusion "Link to this heading") --------------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # Example - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/training/SFT/example.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/training/SFT/example.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Example[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#example "Permalink to this heading") ============================================================================================================== Here we provide a very simple script for supervised finetuning, which is revised from the training script in `` `Fastchat `` <[https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat) \>\`\_\_. The script is used to finetune Qwen with Hugging Face Trainer. You can check the script [here](https://github.com/QwenLM/Qwen1.5/blob/main/finetune.py) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, [LoRA](https://arxiv.org/abs/2106.09685) , and [Q-LoRA](https://arxiv.org/abs/2305.14314) . In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#installation "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------ Before you start, make sure you have installed the following packages: pip install peft deepspeed optimum accelerate Data Preparation[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#data-preparation "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------------- For data preparation, we advise you to organize the data in a jsonl file, where each line is a dictionary as demonstrated below: { "type": "chatml", "messages": \[\ {\ "role": "system",\ "content": "You are a helpful assistant."\ },\ {\ "role": "user",\ "content": "Tell me something about large language models."\ },\ {\ "role": "assistant",\ "content": "Large language models are a type of language model that is trained on a large corpus of text data. They are capable of generating human-like text and are used in a variety of natural language processing tasks..."\ }\ \], "source": "unknown" } { "type": "chatml", "messages": \[\ {\ "role": "system",\ "content": "You are a helpful assistant."\ },\ {\ "role": "user",\ "content": "What is your name?"\ },\ {\ "role": "assistant",\ "content": "My name is Qwen."\ }\ \], "source": "self-made" } Above are two examples of each data sample in the dataset. Each sample is a JSON object with the following fields: `type`, `messages` and `source`. `messages` is required while the others are optional for you to label your data format and data source. The `messages` field is a list of JSON objects, each of which has two fields: `role` and `content`. `role` can be `system`, `user`, or `assistant`. `content` is the text of the message. `source` is the source of the data, which can be `self-made`, `alpaca`, `open-hermes`, or any other string. To make the jsonl file, you can use `json` to save a list of dictionaries to the jsonl file: import json with open('data.jsonl', 'w') as f: for sample in samples: f.write(json.dumps(sample) + '\\n') Quickstart[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#quickstart "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------- For you to start finetuning quickly, we directly provide a shell script for you to run without paying attention to details. You need different hyperparameters for different types of training, e.g., single-GPU / multi-GPU training, full-parameter tuning, LoRA, or Q-LoRA. cd examples/sft bash finetune.sh \-m \-d \--deepspeed \[\--use\_lora True\] \[\--q\_lora True\] Specify the `` for your model, `` for your data, and `` for your deepspeed configuration. If you use LoRA or Q-LoRA, just add `--use_lora True` or `--q_lora True` based on your requirements. This is the simplest way to start finetuning. If you want to change more hyperparameters, you can dive into the script and modify those parameters. Advanced Usages[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#advanced-usages "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------------ In this section, we introduce the details of the scripts, including the core python script as well as the corresponding shell script. ### Shell Script[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#shell-script "Permalink to this heading") Before we introduce the python code, we provide a brief introduction to the shell script with commands. We provide some guidance inside the shell script and here we take `finetune.sh` as an example. To set up the environment variables for distributed training (or single-GPU training), specify the following variables: `GPUS_PER_NODE`, `NNODES`, `NODE_RANK`, `MASTER_ADDR`, and `MASTER_PORT`. No need to worry too much about them as we provide the default settings for you. In the command, you can pass in the argument `-m` and `-d` to specify the model path and data path, respectively. You can also pass in the argument `--deepspeed` to specify the deepspeed configuration file. We provide two configuration files for ZeRO2 and ZeRO3, and you can choose one based on your requirements. In most cases, we recommend using ZeRO3 for multi-GPU training except for Q-LoRA, where we recommend using ZeRO2. There are a series of hyperparameters to tune. Passing in `--bf16` or `--fp16` to specify the precision for mixed precision training. The other significant hyperparameters include: * `--output_dir`: the path of your output models or adapters. * `--num_train_epochs`: the number of training epochs. * `--gradient_accumulation_steps`: the number of gradient accumulation steps. * `--per_device_train_batch_size`: the batch size per GPU for training, and the total batch size is equalt to `per_device_train_batch_size` \\(\\times\\) `number_of_gpus` \\(\\times\\) `gradient_accumulation_steps`. * `--learning_rate`: the learning rate. * `--warmup_steps`: the number of warmup steps. * `--lr_scheduler_type`: the type of learning rate scheduler. * `--weight_decay`: the value of weight decay. * `--adam_beta2`: the value of \\(\\beta\_2\\) in Adam. * `--model_max_length`: the maximum sequence length. * `--use_lora`: whether to use LoRA. Adding `--q_lora` can enable Q-LoRA. * `--gradient_checkpointing`: whether to use gradient checkpointing. ### Python Script[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#python-script "Permalink to this heading") In this script, we mainly use `trainer` from HF and `peft` to train our models. We also use `deepspeed` to accelerate the training process. The script is very simple and easy to understand. @dataclass @dataclass class ModelArguments: model\_name\_or\_path: Optional\[str\] \= field(default\="Qwen/Qwen-7B") @dataclass class DataArguments: data\_path: str \= field( default\=None, metadata\={"help": "Path to the training data."} ) eval\_data\_path: str \= field( default\=None, metadata\={"help": "Path to the evaluation data."} ) lazy\_preprocess: bool \= False @dataclass class TrainingArguments(transformers.TrainingArguments): cache\_dir: Optional\[str\] \= field(default\=None) optim: str \= field(default\="adamw\_torch") model\_max\_length: int \= field( default\=8192, metadata\={ "help": "Maximum sequence length. Sequences will be right padded (and possibly truncated)." }, ) use\_lora: bool \= False @dataclass class LoraArguments: lora\_r: int \= 64 lora\_alpha: int \= 16 lora\_dropout: float \= 0.05 lora\_target\_modules: List\[str\] \= field( default\_factory\=lambda: \[\ "q\_proj",\ "k\_proj",\ "v\_proj",\ "o\_proj",\ "up\_proj",\ "gate\_proj",\ "down\_proj",\ \] ) lora\_weight\_path: str \= "" lora\_bias: str \= "none" q\_lora: bool \= False The classes for arguments allow you to specify hyperparameters for model, data, training, and additionally LoRA if you use LoRA or Q-LoRA to train your model. Specifically, `model-max-length` is a key hyperparameter that determines your maximum sequence length of your training data. `LoRAArguments` includes the hyperparameters for LoRA or Q-LoRA: * `lora_r`: the rank for LoRA; * `lora_alpha`: the alpha value for LoRA; * `lora_dropout`: the dropout rate for LoRA; * `lora_target_modules`: the target modules for LoRA. By default we tune all linear layers; * `lora_weight_path`: the path to the weight file for LoRA; * `lora_bias`: the bias for LoRA; * `q_lora`: whether to use Q-LoRA. def maybe\_zero\_3(param): if hasattr(param, "ds\_id"): assert param.ds\_status \== ZeroParamStatus.NOT\_AVAILABLE with zero.GatheredParameters(\[param\]): param \= param.data.detach().cpu().clone() else: param \= param.detach().cpu().clone() return param \# Borrowed from peft.utils.get\_peft\_model\_state\_dict def get\_peft\_state\_maybe\_zero\_3(named\_params, bias): if bias \== "none": to\_return \= {k: t for k, t in named\_params if "lora\_" in k} elif bias \== "all": to\_return \= {k: t for k, t in named\_params if "lora\_" in k or "bias" in k} elif bias \== "lora\_only": to\_return \= {} maybe\_lora\_bias \= {} lora\_bias\_names \= set() for k, t in named\_params: if "lora\_" in k: to\_return\[k\] \= t bias\_name \= k.split("lora\_")\[0\] + "bias" lora\_bias\_names.add(bias\_name) elif "bias" in k: maybe\_lora\_bias\[k\] \= t for k, t in maybe\_lora\_bias: if bias\_name in lora\_bias\_names: to\_return\[bias\_name\] \= t else: raise NotImplementedError to\_return \= {k: maybe\_zero\_3(v) for k, v in to\_return.items()} return to\_return def safe\_save\_model\_for\_hf\_trainer( trainer: transformers.Trainer, output\_dir: str, bias\="none" ): """Collects the state dict and dump to disk.""" \# check if zero3 mode enabled if deepspeed.is\_deepspeed\_zero3\_enabled(): state\_dict \= trainer.model\_wrapped.\_zero3\_consolidated\_16bit\_state\_dict() else: if trainer.args.use\_lora: state\_dict \= get\_peft\_state\_maybe\_zero\_3( trainer.model.named\_parameters(), bias ) else: state\_dict \= trainer.model.state\_dict() if trainer.args.should\_save and trainer.args.local\_rank \== 0: trainer.\_save(output\_dir, state\_dict\=state\_dict) The method `safe_save_model_for_hf_trainer`, which uses `get_peft_state_maybe_zero_3`, helps tackle the problems in saving models trained either with or without ZeRO3. def preprocess( messages, tokenizer: transformers.PreTrainedTokenizer, max\_len: int, ) \-> Dict: """Preprocesses the data for supervised fine-tuning.""" texts \= \[\] for i, msg in enumerate(messages): texts.append( tokenizer.apply\_chat\_template( msg, tokenize\=True, add\_generation\_prompt\=False, padding\=True, max\_length\=max\_len, truncation\=True, ) ) input\_ids \= torch.tensor(texts, dtype\=torch.int) target\_ids \= input\_ids.clone() target\_ids\[target\_ids \== tokenizer.pad\_token\_id\] \= IGNORE\_TOKEN\_ID attention\_mask \= input\_ids.ne(tokenizer.pad\_token\_id) return dict( input\_ids\=input\_ids, target\_ids\=target\_ids, attention\_mask\=attention\_mask ) For data preprocessing, we use `preprocess` to organize the data. Specifically, we apply our ChatML template to the texts. If you prefer other chat templates, you can use others, e.g., by still applying `apply_chat_template()` with another tokenizer. The chat template is stored in the `tokenizer_config.json` in the HF repo. Additionally, we pad the sequence of each sample to the maximum length for training. class SupervisedDataset(Dataset): """Dataset for supervised fine-tuning.""" def \_\_init\_\_( self, raw\_data, tokenizer: transformers.PreTrainedTokenizer, max\_len: int ): super(SupervisedDataset, self).\_\_init\_\_() rank0\_print("Formatting inputs...") messages \= \[example\["messages"\] for example in raw\_data\] data\_dict \= preprocess(messages, tokenizer, max\_len) self.input\_ids \= data\_dict\["input\_ids"\] self.target\_ids \= data\_dict\["target\_ids"\] self.attention\_mask \= data\_dict\["attention\_mask"\] def \_\_len\_\_(self): return len(self.input\_ids) def \_\_getitem\_\_(self, i) \-> Dict\[str, torch.Tensor\]: return dict( input\_ids\=self.input\_ids\[i\], labels\=self.labels\[i\], attention\_mask\=self.attention\_mask\[i\], ) class LazySupervisedDataset(Dataset): """Dataset for supervised fine-tuning.""" def \_\_init\_\_( self, raw\_data, tokenizer: transformers.PreTrainedTokenizer, max\_len: int ): super(LazySupervisedDataset, self).\_\_init\_\_() self.tokenizer \= tokenizer self.max\_len \= max\_len rank0\_print("Formatting inputs...Skip in lazy mode") self.tokenizer \= tokenizer self.raw\_data \= raw\_data self.cached\_data\_dict \= {} def \_\_len\_\_(self): return len(self.raw\_data) def \_\_getitem\_\_(self, i) \-> Dict\[str, torch.Tensor\]: if i in self.cached\_data\_dict: return self.cached\_data\_dict\[i\] ret \= preprocess(\[self.raw\_data\[i\]\["messages"\]\], self.tokenizer, self.max\_len) ret \= dict( input\_ids\=ret\["input\_ids"\]\[0\], labels\=ret\["target\_ids"\]\[0\], attention\_mask\=ret\["attention\_mask"\]\[0\], ) self.cached\_data\_dict\[i\] \= ret return ret def make\_supervised\_data\_module( tokenizer: transformers.PreTrainedTokenizer, data\_args, max\_len, ) \-> Dict: """Make dataset and collator for supervised fine-tuning.""" dataset\_cls \= ( LazySupervisedDataset if data\_args.lazy\_preprocess else SupervisedDataset ) rank0\_print("Loading data...") train\_data \= \[\] with open(data\_args.data\_path, "r") as f: for line in f: train\_data.append(json.loads(line)) train\_dataset \= dataset\_cls(train\_data, tokenizer\=tokenizer, max\_len\=max\_len) if data\_args.eval\_data\_path: eval\_data \= \[\] with open(data\_args.eval\_data\_path, "r") as f: for line in f: eval\_data.append(json.loads(line)) eval\_dataset \= dataset\_cls(eval\_data, tokenizer\=tokenizer, max\_len\=max\_len) else: eval\_dataset \= None return dict(train\_dataset\=train\_dataset, eval\_dataset\=eval\_dataset) Then we utilize `make_supervised_data_module` by using `SupervisedDataset` or `LazySupervisedDataset` to build the dataset. def train(): global local\_rank parser \= transformers.HfArgumentParser( (ModelArguments, DataArguments, TrainingArguments, LoraArguments) ) ( model\_args, data\_args, training\_args, lora\_args, ) \= parser.parse\_args\_into\_dataclasses() \# This serves for single-gpu qlora. if ( getattr(training\_args, "deepspeed", None) and int(os.environ.get("WORLD\_SIZE", 1)) \== 1 ): training\_args.distributed\_state.distributed\_type \= DistributedType.DEEPSPEED local\_rank \= training\_args.local\_rank device\_map \= None world\_size \= int(os.environ.get("WORLD\_SIZE", 1)) ddp \= world\_size != 1 if lora\_args.q\_lora: device\_map \= {"": int(os.environ.get("LOCAL\_RANK") or 0)} if ddp else "auto" if len(training\_args.fsdp) \> 0 or deepspeed.is\_deepspeed\_zero3\_enabled(): logging.warning("FSDP or ZeRO3 is incompatible with QLoRA.") model\_load\_kwargs \= { "low\_cpu\_mem\_usage": not deepspeed.is\_deepspeed\_zero3\_enabled(), } compute\_dtype \= ( torch.float16 if training\_args.fp16 else (torch.bfloat16 if training\_args.bf16 else torch.float32) ) \# Load model and tokenizer config \= transformers.AutoConfig.from\_pretrained( model\_args.model\_name\_or\_path, cache\_dir\=training\_args.cache\_dir, ) config.use\_cache \= False model \= AutoModelForCausalLM.from\_pretrained( model\_args.model\_name\_or\_path, config\=config, cache\_dir\=training\_args.cache\_dir, device\_map\=device\_map, quantization\_config\=BitsAndBytesConfig( load\_in\_4bit\=True, bnb\_4bit\_use\_double\_quant\=True, bnb\_4bit\_quant\_type\="nf4", bnb\_4bit\_compute\_dtype\=compute\_dtype, ) if training\_args.use\_lora and lora\_args.q\_lora else None, \*\*model\_load\_kwargs, ) tokenizer \= AutoTokenizer.from\_pretrained( model\_args.model\_name\_or\_path, cache\_dir\=training\_args.cache\_dir, model\_max\_length\=training\_args.model\_max\_length, padding\_side\="right", use\_fast\=False, ) if training\_args.use\_lora: lora\_config \= LoraConfig( r\=lora\_args.lora\_r, lora\_alpha\=lora\_args.lora\_alpha, target\_modules\=lora\_args.lora\_target\_modules, lora\_dropout\=lora\_args.lora\_dropout, bias\=lora\_args.lora\_bias, task\_type\="CAUSAL\_LM", ) if lora\_args.q\_lora: model \= prepare\_model\_for\_kbit\_training( model, use\_gradient\_checkpointing\=training\_args.gradient\_checkpointing ) model \= get\_peft\_model(model, lora\_config) \# Print peft trainable params model.print\_trainable\_parameters() if training\_args.gradient\_checkpointing: model.enable\_input\_require\_grads() \# Load data data\_module \= make\_supervised\_data\_module( tokenizer\=tokenizer, data\_args\=data\_args, max\_len\=training\_args.model\_max\_length ) \# Start trainer trainer \= Trainer( model\=model, tokenizer\=tokenizer, args\=training\_args, \*\*data\_module ) \# \`not training\_args.use\_lora\` is a temporary workaround for the issue that there are problems with \# loading the checkpoint when using LoRA with DeepSpeed. \# Check this issue https://github.com/huggingface/peft/issues/746 for more information. if ( list(pathlib.Path(training\_args.output\_dir).glob("checkpoint-\*")) and not training\_args.use\_lora ): trainer.train(resume\_from\_checkpoint\=True) else: trainer.train() trainer.save\_state() safe\_save\_model\_for\_hf\_trainer( trainer\=trainer, output\_dir\=training\_args.output\_dir, bias\=lora\_args.lora\_bias ) The `train` method is the key to the training. In general, it loads the tokenizer and model with `AutoTokenizer.from_pretrained()` and `AutoModelForCausalLM.from_pretrained()`. If we use LoRA, the method will initialize LoRA configuration with `LoraConfig`. If we apply Q-LoRA, we should use `prepare_model_for_kbit_training`. Note that for now it still does not support resume for LoRA. Then we leave the following efforts to `trainer` and have a cup of coffee! Next Step[¶](https://qwen.readthedocs.io/en/v1.5/training/SFT/example.html#next-step "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------ Now, you are able to use a very simple script to perform different types of SFT. Alternatively, you can use more advanced training libraries, such as [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) or [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , to enjoy more functionalities. To take a step forward, after SFT, you can consider RLHF to align your model to human preferences! Stay tuned for our next tutorial on RLHF! --- # SFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/training/SFT/index.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/training/SFT/index.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/training/SFT/index.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar SFT[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/index.html#sft "Link to this heading") =============================================================================================== * [LLaMA-Factory](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html) * [ms-swift](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html) --- # RL - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/training/RL/index.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/training/RL/index.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/training/RL/index.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar RL[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/index.html#rl "Link to this heading") ============================================================================================ * [ms-swift](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html) --- # Function Calling - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/framework/function_call.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/framework/function_call.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/framework/function_call.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/framework/function_call.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Function Calling[¶](https://qwen.readthedocs.io/en/v1.5/framework/function_call.html#function-calling "Permalink to this heading") =================================================================================================================================== We offer a wrapper for function calling over the dashscope API and the OpenAI API in [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) . Use Case[¶](https://qwen.readthedocs.io/en/v1.5/framework/function_call.html#use-case "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------- import json import os from qwen\_agent.llm import get\_chat\_model \# Example dummy function hard coded to return the same weather \# In production, this could be your backend API or an external API def get\_current\_weather(location, unit\='fahrenheit'): """Get the current weather in a given location""" if 'tokyo' in location.lower(): return json.dumps({ 'location': 'Tokyo', 'temperature': '10', 'unit': 'celsius' }) elif 'san francisco' in location.lower(): return json.dumps({ 'location': 'San Francisco', 'temperature': '72', 'unit': 'fahrenheit' }) elif 'paris' in location.lower(): return json.dumps({ 'location': 'Paris', 'temperature': '22', 'unit': 'celsius' }) else: return json.dumps({'location': location, 'temperature': 'unknown'}) def test(): llm \= get\_chat\_model({ \# Use the model service provided by DashScope: 'model': 'qwen-max', 'model\_server': 'dashscope', 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), \# Use the model service provided by Together.AI: \# 'model': 'Qwen/Qwen1.5-14B-Chat', \# 'model\_server': 'https://api.together.xyz', # api\_base \# 'api\_key': os.getenv('TOGETHER\_API\_KEY'), \# Use your own model service compatible with OpenAI API: \# 'model': 'Qwen/Qwen1.5-72B-Chat', \# 'model\_server': 'http://localhost:8000/v1', # api\_base \# 'api\_key': 'EMPTY', }) \# Step 1: send the conversation and available functions to the model messages \= \[{\ 'role': 'user',\ 'content': "What's the weather like in San Francisco?"\ }\] functions \= \[{\ 'name': 'get\_current\_weather',\ 'description': 'Get the current weather in a given location',\ 'parameters': {\ 'type': 'object',\ 'properties': {\ 'location': {\ 'type': 'string',\ 'description':\ 'The city and state, e.g. San Francisco, CA',\ },\ 'unit': {\ 'type': 'string',\ 'enum': \['celsius', 'fahrenheit'\]\ },\ },\ 'required': \['location'\],\ },\ }\] print('# Assistant Response 1:') responses \= \[\] for responses in llm.chat(messages\=messages, functions\=functions, stream\=True): print(responses) messages.extend(responses) \# extend conversation with assistant's reply \# Step 2: check if the model wanted to call a function last\_response \= messages\[\-1\] if last\_response.get('function\_call', None): \# Step 3: call the function \# Note: the JSON response may not always be valid; be sure to handle errors available\_functions \= { 'get\_current\_weather': get\_current\_weather, } \# only one function in this example, but you can have multiple function\_name \= last\_response\['function\_call'\]\['name'\] function\_to\_call \= available\_functions\[function\_name\] function\_args \= json.loads(last\_response\['function\_call'\]\['arguments'\]) function\_response \= function\_to\_call( location\=function\_args.get('location'), unit\=function\_args.get('unit'), ) print('# Function Response:') print(function\_response) \# Step 4: send the info for each function call and function response to the model messages.append({ 'role': 'function', 'name': function\_name, 'content': function\_response, }) \# extend conversation with function response print('# Assistant Response 2:') for responses in llm.chat( messages\=messages, functions\=functions, stream\=True, ): \# get a new response from the model where it can see the function response print(responses) if \_\_name\_\_ \== '\_\_main\_\_': test() --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/training/SFT/llama_factory.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LLaMA-Factory[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#llama-factory "Link to this heading") =========================================================================================================================== Here we provide a script for supervised finetuning Qwen2.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#data-preparation "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#training "Link to this heading") ----------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#merge-lora "Link to this heading") --------------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/llama_factory.html#conclusion "Link to this heading") --------------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # ms-swift - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/training/SFT/ms_swift.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar ms-swift[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#ms-swift "Link to this heading") ============================================================================================================ Introduction to ms-swift SFT[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#introduction-to-ms-swift-sft "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------- ms-swift is the official large model and multimodal model training and deployment framework provided by the ModelScope community. GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) The SFT script in ms-swift has the following features: * Flexible training options: single-GPU and multi-GPU support * Efficient tuning methods: full-parameter, LoRA, Q-LoRA, and Dora * Broad model compatibility: supports various LLM and MLLM architectures For detailed model compatibility, see: [Supported Models](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) Environment Setup[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#environment-setup "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------ 1. Follow the instructions of [ms-swift](https://github.com/modelscope/ms-swift) , and build the environment. 2. Optional packages for advanced features: pip install deepspeed \# For multi-GPU training pip install flash\-attn \--no\-build\-isolation Data Preparation[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#data-preparation "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------- ms-swift supports multiple dataset formats: \# Standard messages format {"messages": \[\ {"role": "system", "content": ""},\ {"role": "user", "content": ""},\ {"role": "assistant", "content": ""}\ \]} # ShareGPT conversation format {"system": "", "conversation": \[\ {"human": "", "assistant": ""},\ {"human": "", "assistant": ""}\ \]} # Instruction tuning format {"system": "", "instruction": "", "input": "", "output": ""} # Multimodal format (supports images, audio, video) {"messages": \[\ {"role": "user", "content": "Describe this image"},\ {"role": "assistant", "content": ""}\ \], "images": \["/path/to/image.jpg"\]} For complete dataset formatting guidelines, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) Pre-built datasets are available at: [Supported Datasets](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html#datasets) Training Examples[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#training-examples "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------ Single-GPU Training[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#single-gpu-training "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------- **LLM Example (Qwen2.5-7B-Instruct):** \# 19GB CUDA\_VISIBLE\_DEVICES\=0 \\ swift sft \\ \--model Qwen/Qwen2.5-7B-Instruct \\ \--dataset 'AI-ModelScope/alpaca-gpt4-data-zh' \\ \--train\_type lora \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--gradient\_accumulation\_steps 16 \\ \--learning\_rate 1e-4 \\ \--max\_length 2048 \\ \--eval\_steps 50 \\ \--save\_steps 50 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--system 'You are a helpful assistant.' \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--attn\_impl flash\_attn **MLLM Example (Qwen2.5-VL-7B-Instruct):** \# 18GB CUDA\_VISIBLE\_DEVICES\=0 \\ MAX\_PIXELS\=602112 \\ swift sft \\ \--model Qwen/Qwen2.5-VL-7B-Instruct \\ \--dataset 'AI-ModelScope/LaTeX\_OCR:human\_handwrite' \\ \--train\_type lora \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--gradient\_accumulation\_steps 16 \\ \--learning\_rate 1e-4 \\ \--max\_length 2048 \\ \--eval\_steps 200 \\ \--save\_steps 200 \\ \--save\_total\_limit 5 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 Multi-GPU Training[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#multi-gpu-training "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------- **LLM Example with DeepSpeed:** \# 18G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ nohup swift sft \\ \--model Qwen/Qwen2.5-7B-Instruct \\ \--dataset 'AI-ModelScope/alpaca-gpt4-data-zh' \\ \--train\_type lora \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--torch\_dtype bfloat16 \\ \--deepspeed zero2 \\ \--per\_device\_train\_batch\_size 1 \\ \--gradient\_accumulation\_steps 16 \\ \--learning\_rate 1e-4 \\ \--max\_length 2048 \\ \--num\_train\_epochs 1 \\ \--output\_dir output \\ \--attn\_impl flash\_attn **MLLM Example with DeepSpeed:** \# 17G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ MAX\_PIXELS\=602112 \\ nohup swift sft \\ \--model Qwen/Qwen2.5-VL-7B-Instruct \\ \--dataset 'AI-ModelScope/LaTeX\_OCR:human\_handwrite' \\ \--train\_type lora \\ \--deepspeed zero2 \\ \--per\_device\_train\_batch\_size 1 \\ \--gradient\_accumulation\_steps 8 \\ \--learning\_rate 2e-5 \\ \--max\_length 4096 \\ \--num\_train\_epochs 2 \\ \--output\_dir output \\ \--attn\_impl flash\_attn Model Export[¶](https://qwen.readthedocs.io/en/v2.5/training/SFT/ms_swift.html#model-export "Link to this heading") -------------------------------------------------------------------------------------------------------------------- **Merge LoRA Adapters:** swift export \\ \--adapters output/checkpoint-xxx \\ \--merge\_lora true **Push to ModelScope Hub:** swift export \\ \--adapters output/checkpoint-xxx \\ \--push\_to\_hub true \\ \--hub\_model\_id '/' \\ \--hub\_token '' --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/framework/qwen_agent.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/framework/qwen_agent.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen-Agent[¶](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html#qwen-agent "Permalink to this heading") ==================================================================================================================== [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant. Installation[¶](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html#installation "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------ git clone https://github.com/QwenLM/Qwen-Agent.git cd Qwen-Agent pip install \-e ./ Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/v1.5/framework/qwen_agent.html#developing-your-own-agent "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- Qwen-Agent provides atomic components such as LLMs and prompts, as well as high-level components such as Agents. The example below uses the Assistant component as an illustration, demonstrating how to add custom tools and quickly develop an agent that uses tools. import json import os import json5 import urllib.parse from qwen\_agent.agents import Assistant from qwen\_agent.tools.base import BaseTool, register\_tool llm\_cfg \= { \# Use the model service provided by DashScope: 'model': 'qwen-max', 'model\_server': 'dashscope', \# 'api\_key': 'YOUR\_DASHSCOPE\_API\_KEY', \# It will use the \`DASHSCOPE\_API\_KEY' environment variable if 'api\_key' is not set here. \# Use your own model service compatible with OpenAI API: \# 'model': 'Qwen/Qwen1.5-72B-Chat', \# 'model\_server': 'http://localhost:8000/v1', # api\_base \# 'api\_key': 'EMPTY', \# (Optional) LLM hyperparameters for generation: 'generate\_cfg': { 'top\_p': 0.8 } } system \= 'According to the user\\'s request, you first draw a picture and then automatically run code to download the picture ' + \\ 'and select an image operation from the given document to process the image' \# Add a custom tool named my\_image\_gen： @register\_tool('my\_image\_gen') class MyImageGen(BaseTool): description \= 'AI painting (image generation) service, input text description, and return the image URL drawn based on text information.' parameters \= \[{\ 'name': 'prompt',\ 'type': 'string',\ 'description': 'Detailed description of the desired image content, in English',\ 'required': True\ }\] def call(self, params: str, \*\*kwargs) \-> str: prompt \= json5.loads(params)\['prompt'\] prompt \= urllib.parse.quote(prompt) return json.dumps( {'image\_url': f'https://image.pollinations.ai/prompt/{prompt}'}, ensure\_ascii\=False) tools \= \['my\_image\_gen', 'code\_interpreter'\] \# code\_interpreter is a built-in tool in Qwen-Agent bot \= Assistant(llm\=llm\_cfg, system\_message\=system, function\_list\=tools, files\=\[os.path.abspath('doc.pdf')\]) messages \= \[\] while True: query \= input('user question: ') messages.append({'role': 'user', 'content': query}) response \= \[\] for response in bot.run(messages\=messages): print('bot response:', response) messages.extend(response) The framework also provides more atomic components for developers to combine. For additional showcases, please refer to [examples](https://github.com/QwenLM/Qwen-Agent/tree/main/examples) . --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/framework/LlamaIndex.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/framework/LlamaIndex.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LlamaIndex[¶](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#llamaindex "Permalink to this heading") ==================================================================================================================== To connect Qwen1.5. with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen1.5. Preparation[¶](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#preparation "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#set-parameters "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen1.5-Chat supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen 1.5 model families support a maximum of 32K context window size. import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen1.5 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen1.5-7B-Chat", tokenizer\_name\="Qwen/Qwen1.5-7B-Chat", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#build-index "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/v1.5/framework/LlamaIndex.html#rag "Permalink to this heading") ------------------------------------------------------------------------------------------------------ Now you can perform queries, and Qwen1.5 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # ms-swift - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/training/RL/ms_swift.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar ms-swift[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#ms-swift "Link to this heading") =========================================================================================================== Introduction to ms-swift[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#introduction-to-ms-swift "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------- ms-swift is the official large model and multimodal model training and deployment framework provided by the ModelScope community. GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) The RL script in ms-swift has the following features: * Support single-GPU and multi-GPU training * Support full-parameter tuning, LoRA, Q-LoRA, and Dora * Supports multiple RL algorithms including GRPO, DAPO, PPO, DPO, KTO, ORPO, CPO, and SimPO * Supports both large language models (LLM) and multimodal models (MLLM) For detailed support information, please refer to: [Supported Features](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html#pre-training-and-fine-tuning) Environment Setup[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#environment-setup "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------- 1. Follow the instructions of [ms-swift](https://github.com/modelscope/ms-swift) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install math\_verify\==0.5.2 pip install flash\-attn \--no\-build\-isolation pip install vllm Data Preparation[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#data-preparation "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- ms-swift has built-in preprocessing logic for several datasets, which can be directly used for training via the `--dataset` parameter. For supported datasets, please refer to: [Supported Datasets](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html#datasets) You can also use local custom datasets by providing the local dataset path to the `--dataset` parameter. Example Dataset Formats: \# llm {"messages": \[{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} # mllm {"messages": \[{"role": "user", "content": "What is the difference between the two images?"}\], "images": \["/xxx/x.jpg"\]} {"messages": \[{"role": "user", "content": "What is the difference between the two images?"}\], "images": \["/xxx/y.jpg", "/xxx/z.png"\]} Notes on Dataset Requirements 1. Reward Function Calculation: Depending on the reward function being used, additional columns may be required in the dataset. For example: > When using the built-in accuracy/cosine reward, the dataset must include a `solution` column to compute accuracy. The other columns in the dataset will also be passed to the kwargs of the reward function. 2. Customizing the Reward Function: To tailor the reward function to your specific needs, you can refer to the following resource: [external reward plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin) GRPO Training Examples[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#grpo-training-examples "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------- Single-GPU Configuration[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#single-gpu-configuration "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------- **LLM (Qwen2.5-7B):** \# 42G CUDA\_VISIBLE\_DEVICES\=0 \\ nohup swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen2.5-7B \\ \--vllm\_gpu\_memory\_utilization 0.5 \\ \--use\_vllm true \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--reward\_funcs accuracy format \\ \--train\_type lora \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--torch\_dtype bfloat16 \\ \--dataset 'AI-MO/NuminaMath-TIR' \\ \--max\_completion\_length 1024 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 4 \\ \--per\_device\_eval\_batch\_size 4 \\ \--learning\_rate 1e-5 \\ \--gradient\_accumulation\_steps 1 \\ \--eval\_steps 100 \\ \--save\_steps 100 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--max\_length 2048 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--dataset\_num\_proc 4 \\ \--num\_generations 4 \\ \--temperature 0.9 \\ \--system 'examples/train/grpo/prompt.txt' \\ \--log\_completions true **MLLM (Qwen2.5-VL-7B-Instruct):** \# 55G CUDA\_VISIBLE\_DEVICES\=0 \\ MAX\_PIXELS\=602112 \\ swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen2.5-VL-7B-Instruct \\ \--vllm\_gpu\_memory\_utilization 0.5 \\ \--use\_vllm true \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--external\_plugins examples/train/grpo/plugin/plugin.py \\ \--reward\_funcs external\_r1v\_acc format \\ \--train\_type lora \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--torch\_dtype bfloat16 \\ \--dataset 'lmms-lab/multimodal-open-r1-8k-verified' \\ \--vllm\_max\_model\_len 4196 \\ \--max\_completion\_length 1024 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 4 \\ \--per\_device\_eval\_batch\_size 4 \\ \--learning\_rate 1e-5 \\ \--gradient\_accumulation\_steps 1 \\ \--eval\_steps 100 \\ \--save\_steps 100 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--dataset\_num\_proc 4 \\ \--num\_generations 4 \\ \--temperature 0.9 \\ \--system 'examples/train/grpo/prompt.txt' \\ \--log\_completions true Multi-GPU Training[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#multi-gpu-training "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- **LLM Example with DeepSpeed:** \# 60G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen2.5-7B \\ \--reward\_funcs accuracy format \\ \--use\_vllm true \\ \--vllm\_device auto \\ \--vllm\_gpu\_memory\_utilization 0.7 \\ \--vllm\_max\_model\_len 8192 \\ \--num\_infer\_workers 8 \\ \--train\_type lora \\ \--torch\_dtype bfloat16 \\ \--dataset 'AI-MO/NuminaMath-TIR' \\ \--max\_completion\_length 2048 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-6 \\ \--gradient\_accumulation\_steps 2 \\ \--eval\_steps 200 \\ \--save\_steps 200 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--max\_length 4096 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--dataset\_num\_proc 4 \\ \--num\_generations 8 \\ \--temperature 0.9 \\ \--system 'examples/train/grpo/prompt.txt' \\ \--deepspeed zero2 \\ \--log\_completions true \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--log\_completions true **MLLM Example with DeepSpeed:** \# 60G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ nohup swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen2.5-VL-7B-Instruct \\ \--external\_plugins examples/train/grpo/plugin/plugin.py \\ \--reward\_funcs external\_r1v\_acc format \\ \--use\_vllm true \\ \--vllm\_device auto \\ \--vllm\_gpu\_memory\_utilization 0.7 \\ \--vllm\_max\_model\_len 8192 \\ \--num\_infer\_workers 8 \\ \--train\_type lora \\ \--torch\_dtype bfloat16 \\ \--dataset 'lmms-lab/multimodal-open-r1-8k-verified' \\ \--max\_completion\_length 2048 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-6 \\ \--gradient\_accumulation\_steps 2 \\ \--eval\_steps 200 \\ \--save\_steps 200 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--vllm\_max\_model\_len 4196 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--dataset\_num\_proc 4 \\ \--num\_generations 8 \\ \--temperature 0.9 \\ \--system 'examples/train/grpo/prompt.txt' \\ \--deepspeed zero2 \\ \--log\_completions true \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--log\_completions true Model Export[¶](https://qwen.readthedocs.io/en/v2.5/training/RL/ms_swift.html#model-export "Link to this heading") ------------------------------------------------------------------------------------------------------------------- **Merge LoRA Adapters:** swift export \\ \--adapters output/checkpoint-xxx \\ \--merge\_lora true **Push to ModelScope Hub:** swift export \\ \--adapters output/checkpoint-xxx \\ \--push\_to\_hub true \\ \--hub\_model\_id '/' \\ \--hub\_token '' --- # HF Transformers Inference - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/benchmark/hf_infer.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/benchmark/hf_infer.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/benchmark/hf_infer.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/benchmark/hf_infer.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar HF Transformers Inference[¶](https://qwen.readthedocs.io/en/v1.5/benchmark/hf_infer.html#hf-transformers-inference "Permalink to this heading") ================================================================================================================================================ This section reports the performance of bf16 models and Int4 quantized models (including GPTQ and AWQ) of the Qwen1.5 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under the conditions of different context lengths. In terms of inference speed, we report those with or without Flash Attention 2. The environment of the performance evaluation is: * NVIDIA A100 80GB * CUDA 12.3 * Pytorch 2.1.2+cu118 * Flash Attention 2.5.6 Note that we use the batch size of 1 and the least number of GPUs as possible for the evalution. We test the speed and memory of generating 2048 tokens with the input lengths of 1, 6144, 14336, and 30720 tokens * 0.5B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-0.5B-Chat | 1 | 1 | 58.54 / 61.34 | 1.46 | | Qwen1.5-0.5B-Chat | 1 | 6144 | 57.93 / 63.57 | 6.87 | | Qwen1.5-0.5B-Chat | 1 | 14336 | 59.48 / 60.18 | 14.59 | | Qwen1.5-0.5B-Chat | 1 | 30720 | 47.65 / 35.44 | 30.04 | | Qwen1.5-0.5B-Chat-GPTQ-Int4 | 1 | 1 | 57.18 / 63.67 | 1.03 | | Qwen1.5-0.5B-Chat-GPTQ-Int4 | 1 | 6144 | 57.47 / 63.31 | 6.44 | | Qwen1.5-0.5B-Chat-GPTQ-Int4 | 1 | 14336 | 57.57 / 52.19 | 14.16 | | Qwen1.5-0.5B-Chat-GPTQ-Int4 | 1 | 30720 | 41.84 / 32.58 | 29.60 | | Qwen1.5-0.5B-Chat-AWQ | 1 | 1 | 45.04 / 48.54 | 1.02 | | Qwen1.5-0.5B-Chat-AWQ | 1 | 6144 | 43.30 / 47.62 | 6.43 | | Qwen1.5-0.5B-Chat-AWQ | 1 | 14336 | 42.98 / 48.05 | 14.15 | | Qwen1.5-0.5B-Chat-AWQ | 1 | 30720 | 42.18 / 33.58 | 29.59 | * 1.8B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-1.8B-Chat | 1 | 1 | 51.82 / 54.01 | 4.60 | | Qwen1.5-1.8B-Chat | 1 | 6144 | 51.56 / 51.45 | 10.21 | | Qwen1.5-1.8B-Chat | 1 | 14336 | 45.17 / 30.53 | 18.69 | | Qwen1.5-1.8B-Chat | 1 | 30720 | 29.21 / 16.70 | 35.67 | | Qwen1.5-1.8B-Chat-GPTQ-Int4 | 1 | 1 | 58.83 / 65.21 | 2.91 | | Qwen1.5-1.8B-Chat-GPTQ-Int4 | 1 | 6144 | 54.82 / 46.31 | 8.52 | | Qwen1.5-1.8B-Chat-GPTQ-Int4 | 1 | 14336 | 41.56 / 28.64 | 17.01 | | Qwen1.5-1.8B-Chat-GPTQ-Int4 | 1 | 30720 | 26.88 / 16.13 | 33.98 | | Qwen1.5-1.8B-Chat-AWQ | 1 | 1 | 45.78 / 48.02 | 2.89 | | Qwen1.5-1.8B-Chat-AWQ | 1 | 6144 | 44.95 / 47.64 | 8.50 | | Qwen1.5-1.8B-Chat-AWQ | 1 | 14336 | 42.44 / 29.48 | 16.98 | | Qwen1.5-1.8B-Chat-AWQ | 1 | 30720 | 28.34 / 16.38 | 33.96 | * 4B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-4B-Chat | 1 | 1 | 30.32 / 32.59 | 9.59 | | Qwen1.5-4B-Chat | 1 | 6144 | 30.72 / 28.61 | 16.19 | | Qwen1.5-4B-Chat | 1 | 14336 | 23.46 / 16.96 | 27.08 | | Qwen1.5-4B-Chat | 1 | 30720 | 14.76 / 9.19 | 48.85 | | Qwen1.5-4B-Chat-GPTQ-Int4 | 1 | 1 | 33.63 / 36.67 | 5.65 | | Qwen1.5-4B-Chat-GPTQ-Int4 | 1 | 6144 | 33.93 / 30.66 | 12.25 | | Qwen1.5-4B-Chat-GPTQ-Int4 | 1 | 14336 | 25.01 / 17.48 | 23.14 | | Qwen1.5-4B-Chat-GPTQ-Int4 | 1 | 30720 | 15.28 / 9.35 | 44.91 | | Qwen1.5-4B-Chat-AWQ | 1 | 1 | 28.09 / 28.64 | 5.19 | | Qwen1.5-4B-Chat-AWQ | 1 | 6144 | 28.00 / 27.83 | 11.79 | | Qwen1.5-4B-Chat-AWQ | 1 | 14336 | 22.95 / 16.49 | 22.67 | | Qwen1.5-4B-Chat-AWQ | 1 | 30720 | 14.50 / 9.06 | 44.45 | * MoE-A2.7B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-MoE-A2.7B-Chat | 1 | 1 | 8.49 / 8.52 | 27.82 | | Qwen1.5-MoE-A2.7B-Chat | 1 | 6144 | 8.73 / 8.41 | 33.43 | | Qwen1.5-MoE-A2.7B-Chat | 1 | 14336 | 8.30 / 7.43 | 41.91 | | Qwen1.5-MoE-A2.7B-Chat | 1 | 30720 | 7.40 / 6.34 | 58.89 | | Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 | 1 | 1 | 8.17 / 8.67 | 9.23 | | Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 | 1 | 6144 | 8.64 / 8.30 | 14.84 | | Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 | 1 | 14336 | 8.16 / 7.39 | 23.32 | | Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 | 1 | 30720 | 7.11 / 6.16 | 40.30 | * 7B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-7B-Chat | 1 | 1 | 37.07 / 40.05 | 16.90 | | Qwen1.5-7B-Chat | 1 | 6144 | 29.29 / 26.95 | 24.37 | | Qwen1.5-7B-Chat | 1 | 14336 | 19.93 / 16.18 | 37.01 | | Qwen1.5-7B-Chat | 1 | 30720 | 12.04 / 8.89 | 62.29 | | Qwen1.5-7B-Chat-GPTQ-Int4 | 1 | 1 | 38.73 / 46.46 | 8.78 | | Qwen1.5-7B-Chat-GPTQ-Int4 | 1 | 6144 | 34.33 / 30.76 | 16.26 | | Qwen1.5-7B-Chat-GPTQ-Int4 | 1 | 14336 | 22.04 / 17.46 | 28.90 | | Qwen1.5-7B-Chat-GPTQ-Int4 | 1 | 30720 | 12.82 / 9.26 | 54.17 | | Qwen1.5-7B-Chat-AWQ | 1 | 1 | 32.59 / 36.74 | 8.02 | | Qwen1.5-7B-Chat-AWQ | 1 | 6144 | 29.13 / 26.91 | 15.49 | | Qwen1.5-7B-Chat-AWQ | 1 | 14336 | 19.98 / 16.14 | 28.13 | | Qwen1.5-7B-Chat-AWQ | 1 | 30720 | 12.10 / 8.86 | 53.40 | * 14B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-14B-Chat | 1 | 1 | 26.89 / 31.36 | 30.18 | | Qwen1.5-14B-Chat | 1 | 6144 | 19.17 / 18.03 | 39.91 | | Qwen1.5-14B-Chat | 1 | 14336 | 12.91 / 11.01 | 57.05 | | Qwen1.5-14B-Chat | 2 | 30720 | 7.68 / 6.09 | 101.65 | | Qwen1.5-14B-Chat-GPTQ-Int4 | 1 | 1 | 32.79 / 36.88 | 13.87 | | Qwen1.5-14B-Chat-GPTQ-Int4 | 1 | 6144 | 23.30 / 21.49 | 23.59 | | Qwen1.5-14B-Chat-GPTQ-Int4 | 1 | 14336 | 14.69 / 12.21 | 40.74 | | Qwen1.5-14B-Chat-GPTQ-Int4 | 2 | 30720 | 8.14 / 7.68 | | | Qwen1.5-14B-Chat-AWQ | 1 | 1 | 27.51 / 29.50 | 12.88 | | Qwen1.5-14B-Chat-AWQ | 1 | 6144 | 20.37 / 19.03 | 22.61 | | Qwen1.5-14B-Chat-AWQ | 1 | 14336 | 13.50 / 11.35 | 39.76 | | Qwen1.5-14B-Chat-AWQ | 2 | 30720 | 7.74 / 6.03 | | * 72B: | Model | Num. GPU | Input Length | Speed (w/wo FA2) | Memory | | --- | --- | --- | --- | --- | | Qwen1.5-72B-Chat | 2 | 1 | 7.24 / 8.13 | 142.39 | | Qwen1.5-72B-Chat | 3 | 6144 | 4.89 / 4.82 | 174.66 | | Qwen1.5-72B-Chat | 4 | 14336 | 3.37 / 3.13 | 233.00 | | Qwen1.5-72B-Chat | 5 | 30720 | 2.17 / 2.00 | 344.17 | | Qwen1.5-72B-Chat-GPTQ-Int4 | 1 | 1 | 9.32 / 10.25 | 50.09 | | Qwen1.5-72B-Chat-GPTQ-Int4 | 2 | 6144 | 5.87 / 5.84 | 97.38 | | Qwen1.5-72B-Chat-GPTQ-Int4 | 3 | 14336 | 3.86 / 3.60 | 146.17 | | Qwen1.5-72B-Chat-GPTQ-Int4 | 4 | 30720 | 2.31 / 2.06 | 238.17 | | Qwen1.5-72B-Chat-AWQ | 1 | 1 | 10.59 / 12.06 | 49.68 | | Qwen1.5-72B-Chat-AWQ | 2 | 6144 | 6.47 / 6.41 | | | Qwen1.5-72B-Chat-AWQ | 3 | 14336 | 4.09 / 3.78 | | | Qwen1.5-72B-Chat-AWQ | 4 | 30720 | 2.35 / 2.10 | | (Note: we had problems with the statistics of memory footprint of AWQ models on multiple devices and thus we do not report the results. Also, the memory footprint of Qwen1.5-14B in the context of 32768 tokens is also inconsistent with our expectation and we don’t report here. Additionally, due to the implementation in our HF code, the MoE model runs much slower than expectation. Intead, we advise users to deploy the MoE model with vLLM.) --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/framework/qwen_agent.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/framework/qwen_agent.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen-Agent[¶](https://qwen.readthedocs.io/en/v2.0/framework/qwen_agent.html#qwen-agent "Link to this heading") =============================================================================================================== [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant. Installation[¶](https://qwen.readthedocs.io/en/v2.0/framework/qwen_agent.html#installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------- git clone https://github.com/QwenLM/Qwen-Agent.git cd Qwen-Agent pip install \-e ./ Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/v2.0/framework/qwen_agent.html#developing-your-own-agent "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Qwen-Agent provides atomic components such as LLMs and prompts, as well as high-level components such as Agents. The example below uses the Assistant component as an illustration, demonstrating how to add custom tools and quickly develop an agent that uses tools. import json import os import json5 import urllib.parse from qwen\_agent.agents import Assistant from qwen\_agent.tools.base import BaseTool, register\_tool llm\_cfg \= { \# Use the model service provided by DashScope: 'model': 'qwen-max', 'model\_server': 'dashscope', \# 'api\_key': 'YOUR\_DASHSCOPE\_API\_KEY', \# It will use the \`DASHSCOPE\_API\_KEY' environment variable if 'api\_key' is not set here. \# Use your own model service compatible with OpenAI API: \# 'model': 'Qwen/Qwen2-72B-Instruct', \# 'model\_server': 'http://localhost:8000/v1', # api\_base \# 'api\_key': 'EMPTY', \# (Optional) LLM hyperparameters for generation: 'generate\_cfg': { 'top\_p': 0.8 } } system \= 'According to the user\\'s request, you first draw a picture and then automatically run code to download the picture ' + \\ 'and select an image operation from the given document to process the image' \# Add a custom tool named my\_image\_gen： @register\_tool('my\_image\_gen') class MyImageGen(BaseTool): description \= 'AI painting (image generation) service, input text description, and return the image URL drawn based on text information.' parameters \= \[{\ 'name': 'prompt',\ 'type': 'string',\ 'description': 'Detailed description of the desired image content, in English',\ 'required': True\ }\] def call(self, params: str, \*\*kwargs) \-> str: prompt \= json5.loads(params)\['prompt'\] prompt \= urllib.parse.quote(prompt) return json.dumps( {'image\_url': f'https://image.pollinations.ai/prompt/{prompt}'}, ensure\_ascii\=False) tools \= \['my\_image\_gen', 'code\_interpreter'\] \# code\_interpreter is a built-in tool in Qwen-Agent bot \= Assistant(llm\=llm\_cfg, system\_message\=system, function\_list\=tools, files\=\[os.path.abspath('doc.pdf')\]) messages \= \[\] while True: query \= input('user question: ') messages.append({'role': 'user', 'content': query}) response \= \[\] for response in bot.run(messages\=messages): print('bot response:', response) messages.extend(response) The framework also provides more atomic components for developers to combine. For additional showcases, please refer to [examples](https://github.com/QwenLM/Qwen-Agent/tree/main/examples) . --- # Performance of Quantized Models - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/benchmark/quantization_benchmark.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/benchmark/quantization_benchmark.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/benchmark/quantization_benchmark.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/benchmark/quantization_benchmark.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Performance of Quantized Models[¶](https://qwen.readthedocs.io/en/v2.0/benchmark/quantization_benchmark.html#performance-of-quantized-models "Link to this heading") ===================================================================================================================================================================== This section reports the generation performance of quantized models (including GPTQ and AWQ) of the Qwen2 series. Specifically, we report: * MMLU (Accuracy) * C-Eval (Accuracy) * IFEval (Strict Prompt-Level Accuracy) We use greedy decoding in evaluating all models. | | Quantization | Average | MMLU | C-Eval | IFEval | | --- | --- | --- | --- | --- | --- | | Qwen2-72B-Instruct | BF16 | 81.3 | 82.3 | 83.8 | 77.6 | | GPTQ-Int8 | 80.7 | 81.3 | 83.4 | 77.5 | | GPTQ-Int4 | 81.2 | 80.8 | 83.9 | 78.9 | | AWQ | 80.4 | 80.5 | 83.9 | 76.9 | | Qwen2-7B-Instruct | BF16 | 66.9 | 70.5 | 77.2 | 53.1 | | GPTQ-Int8 | 66.2 | 69.1 | 76.7 | 52.9 | | GPTQ-Int4 | 64.1 | 67.8 | 75.2 | 49.4 | | AWQ | 64.1 | 67.4 | 73.6 | 51.4 | | Qwen2-1.5B-Instruct | BF16 | 48.4 | 52.4 | 63.8 | 29.0 | | GPTQ-Int8 | 48.1 | 53.0 | 62.5 | 28.8 | | GPTQ-Int4 | 45.0 | 50.7 | 57.4 | 27.0 | | AWQ | 46.5 | 51.6 | 58.1 | 29.9 | | Qwen2-0.5B-Instruct | BF16 | 34.4 | 37.9 | 45.2 | 20.0 | | GPTQ-Int8 | 32.6 | 35.6 | 43.9 | 18.1 | | GPTQ-Int4 | 29.7 | 33.0 | 39.2 | 16.8 | | AWQ | 31.1 | 34.4 | 42.1 | 16.7 | --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/v1.5/_sources/framework/Langchain.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v1.5/docs/source/framework/Langchain.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Langchain[¶](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html#langchain "Permalink to this heading") ================================================================================================================= This guide helps you build a question-answering application based on a local knowledge base using `Qwen1.5-7B-Chat` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html#basic-usage "Permalink to this heading") --------------------------------------------------------------------------------------------------------------------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen1.5-7B-Chat to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun device \= "cuda" \# the device to load the model onto model \= AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat", torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat") class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(device) generated\_ids \= model.generate( model\_inputs.input\_ids, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen1.5-7B-Chat model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language model." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/v1.5/framework/Langchain.html#next-step "Permalink to this heading") ----------------------------------------------------------------------------------------------------------------- Now you can chat with Qwen1.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/framework/LlamaIndex.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/framework/LlamaIndex.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LlamaIndex[¶](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#llamaindex "Link to this heading") =============================================================================================================== To connect Qwen2 with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2. Preparation[¶](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#preparation "Link to this heading") ----------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#set-parameters "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2-Instruct supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2 model families support a maximum of 32K context window size (up to 128K for Qwen2-7B-Instruct and Qwen2-72B-Instrut, requiring extra configuration) import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen2 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen2-7B-Instruct", tokenizer\_name\="Qwen/Qwen2-7B-Instruct", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#build-index "Link to this heading") ----------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/v2.0/framework/LlamaIndex.html#rag "Link to this heading") ------------------------------------------------------------------------------------------------- Now you can perform queries, and Qwen2 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/framework/Langchain.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/framework/Langchain.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Langchain[¶](https://qwen.readthedocs.io/en/v2.0/framework/Langchain.html#langchain "Link to this heading") ============================================================================================================ This guide helps you build a question-answering application based on a local knowledge base using `Qwen2-7B-Instruct` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/v2.0/framework/Langchain.html#basic-usage "Link to this heading") ---------------------------------------------------------------------------------------------------------------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2-7B-Instruct to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun device \= "cuda" \# the device to load the model onto model \= AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen2-7B-Instruct", torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained("Qwen/Qwen2-7B-Instruct") class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(device) generated\_ids \= model.generate( model\_inputs.input\_ids, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2-7B-Instruct model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language model." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/v2.0/framework/Langchain.html#next-step "Link to this heading") ------------------------------------------------------------------------------------------------------------ Now you can chat with Qwen2 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Speed Benchmark - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.0/benchmark/speed_benchmark.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.0/benchmark/speed_benchmark.html#) [View this page](https://qwen.readthedocs.io/en/v2.0/_sources/benchmark/speed_benchmark.rst.txt "View this page") [Edit this page](https://github.com/QwenLM/Qwen1.5/edit/v2.0/docs/source/benchmark/speed_benchmark.rst "Edit this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Speed Benchmark[¶](https://qwen.readthedocs.io/en/v2.0/benchmark/speed_benchmark.html#speed-benchmark "Link to this heading") ============================================================================================================================== This section reports the speed performance of bf16 models, quantized models (including GPTQ-Int4, GPTQ-Int8 and AWQ) of the Qwen2 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under the conditions of different context lengths. The environment of the evaluation with huggingface transformers is: * NVIDIA A100 80GB * CUDA 11.8 * Pytorch 2.1.2+cu118 * Flash Attention 2.3.3 * Transformers 4.38.2 * AutoGPTQ 0.7.1 * AutoAWQ 0.2.4 The environment of the evaluation with vLLM is: * NVIDIA A100 80GB * CUDA 11.8 * Pytorch 2.3.0+cu118 * Flash Attention 2.5.6 * Transformers 4.40.1 * vLLM 0.4.2 Note: * We use the batch size of 1 and the least number of GPUs as possible for the evalution. * We test the speed and memory of generating 2048 tokens with the input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens (>32k is only avaliable for Qwen2-72B-Instuct and Qwen2-7B-Instuct). * For vLLM, the memory usage is not reported because it pre-allocates all GPU memory. We use `gpu_memory_utilization=0.9 max_model_len=32768 enforce_eager=False` by default. * 0.5B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | | --- | --- | --- | --- | --- | --- | | Qwen2-0.5B-Instruct | 1 | BF16 | 1 | 49.94 | 1.17 | | GPTQ-Int8 | 1 | 36.35 | 0.85 | | GPTQ-Int4 | 1 | 49.56 | 0.68 | | AWQ | 1 | 38.78 | 0.68 | | 6144 | BF16 | 1 | 50.83 | 6.42 | | GPTQ-Int8 | 1 | 36.56 | 6.09 | | GPTQ-Int4 | 1 | 49.63 | 5.93 | | AWQ | 1 | 38.73 | 5.92 | | 14336 | BF16 | 1 | 49.56 | 13.48 | | GPTQ-Int8 | 1 | 36.23 | 13.15 | | GPTQ-Int4 | 1 | 48.68 | 12.97 | | AWQ | 1 | 38.94 | 12.99 | | 30720 | BF16 | 1 | 49.25 | 27.61 | | GPTQ-Int8 | 1 | 34.61 | 27.28 | | GPTQ-Int4 | 1 | 48.18 | 27.12 | | AWQ | 1 | 38.19 | 27.11 | * 0.5B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2-0.5B-Instruct | 1 | BF16 | 1 | 270.49 | | GPTQ-Int8 | 1 | 235.95 | | GPTQ-Int4 | 1 | 240.07 | | AWQ | 1 | 233.31 | | 6144 | BF16 | 1 | 256.16 | | GPTQ-Int8 | 1 | 224.30 | | GPTQ-Int4 | 1 | 226.41 | | AWQ | 1 | 222.83 | | 14336 | BF16 | 1 | 108.89 | | GPTQ-Int8 | 1 | 108.10 | | GPTQ-Int4 | 1 | 106.51 | | AWQ | 1 | 104.16 | | 30720 | BF16 | 1 | 97.20 | | GPTQ-Int8 | 1 | 94.49 | | GPTQ-Int4 | 1 | 93.94 | | AWQ | 1 | 92.23 | * 1.5B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | | --- | --- | --- | --- | --- | --- | | Qwen2-1.5B-Instruct | 1 | BF16 | 1 | 40.89 | 3.44 | | GPTQ-Int8 | 1 | 31.51 | 2.31 | | GPTQ-Int4 | 1 | 42.47 | 1.67 | | AWQ | 1 | 33.62 | 1.64 | | 6144 | BF16 | 1 | 40.86 | 8.74 | | GPTQ-Int8 | 1 | 31.31 | 7.59 | | GPTQ-Int4 | 1 | 42.78 | 6.95 | | AWQ | 1 | 32.90 | 6.92 | | 14336 | BF16 | 1 | 40.08 | 15.92 | | GPTQ-Int8 | 1 | 31.19 | 14.79 | | GPTQ-Int4 | 1 | 42.25 | 14.14 | | AWQ | 1 | 33.24 | 14.12 | | 30720 | BF16 | 1 | 34.09 | 30.31 | | GPTQ-Int8 | 1 | 28.52 | 29.18 | | GPTQ-Int4 | 1 | 31.30 | 28.54 | | AWQ | 1 | 32.16 | 28.51 | * 1.5B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2-1.5B-Instruct | 1 | BF16 | 1 | 175.55 | | GPTQ-Int8 | 1 | 172.28 | | GPTQ-Int4 | 1 | 184.58 | | AWQ | 1 | 170.87 | | 6144 | BF16 | 1 | 166.23 | | GPTQ-Int8 | 1 | 164.32 | | GPTQ-Int4 | 1 | 174.04 | | AWQ | 1 | 162.81 | | 14336 | BF16 | 1 | 83.67 | | GPTQ-Int8 | 1 | 98.63 | | GPTQ-Int4 | 1 | 97.65 | | AWQ | 1 | 92.48 | | 30720 | BF16 | 1 | 77.69 | | GPTQ-Int8 | 1 | 86.42 | | GPTQ-Int4 | 1 | 87.49 | | AWQ | 1 | 82.88 | * 7B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | | --- | --- | --- | --- | --- | --- | | Qwen2-7B-Instruct | 1 | BF16 | 1 | 37.97 | 14.92 | | GPTQ-Int8 | 1 | 30.85 | 8.97 | | GPTQ-Int4 | 1 | 36.17 | 6.06 | | AWQ | 1 | 33.08 | 5.93 | | 6144 | BF16 | 1 | 34.74 | 20.26 | | GPTQ-Int8 | 1 | 31.13 | 14.31 | | GPTQ-Int4 | 1 | 33.34 | 11.40 | | AWQ | 1 | 30.86 | 11.27 | | 14336 | BF16 | 1 | 26.63 | 27.71 | | GPTQ-Int8 | 1 | 24.58 | 21.76 | | GPTQ-Int4 | 1 | 25.81 | 18.86 | | AWQ | 1 | 27.61 | 18.72 | | 30720 | BF16 | 1 | 17.49 | 42.62 | | GPTQ-Int8 | 1 | 16.69 | 36.67 | | GPTQ-Int4 | 1 | 17.17 | 33.76 | | AWQ | 1 | 17.87 | 33.63 | * 7B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2-7B-Instruct | 1 | BF16 | 1 | 80.45 | | GPTQ-Int8 | 1 | 114.32 | | GPTQ-Int4 | 1 | 143.40 | | AWQ | 1 | 96.65 | | 6144 | BF16 | 1 | 76.41 | | GPTQ-Int8 | 1 | 107.02 | | GPTQ-Int4 | 1 | 131.55 | | AWQ | 1 | 91.38 | | 14336 | BF16 | 1 | 66.54 | | GPTQ-Int8 | 1 | 89.72 | | GPTQ-Int4 | 1 | 97.93 | | AWQ | 1 | 76.87 | | 30720 | BF16 | 1 | 55.83 | | GPTQ-Int8 | 1 | 71.58 | | GPTQ-Int4 | 1 | 81.48 | | AWQ | 1 | 63.62 | | 63488 | BF16 | 1 | 41.20 | | GPTQ-Int8 | 1 | 49.37 | | GPTQ-Int4 | 1 | 54.12 | | AWQ | 1 | 45.89 | | 129024 | BF16 | 1 | 25.01 | | GPTQ-Int8 | 1 | 27.73 | | GPTQ-Int4 | 1 | 29.39 | | AWQ | 1 | 27.13 | * 57B-A14B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | | --- | --- | --- | --- | --- | --- | | Qwen2-57B-A14B-Instruct | 1 | BF16 | 2 | 4.76 | 110.29 | | GPTQ-Int4 | 1 | 5.55 | 30.38 | | 6144 | BF16 | 2 | 4.90 | 117.80 | | GPTQ-Int4 | 1 | 5.44 | 35.67 | | 14336 | BF16 | 2 | 4.58 | 128.17 | | GPTQ-Int4 | 1 | 5.31 | 43.11 | | 30720 | BF16 | 2 | 4.12 | 163.77 | | GPTQ-Int4 | 1 | 4.72 | 58.01 | * 57B-A14B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2-57B-A14B-Instruct | 1 | BF16 | 2 | 31.44 | | | 6144 | BF16 | 2 | 31.77 | | | 14336 | BF16 | 2 | 21.25 | | | 30720 | BF16 | 2 | 20.24 | Note: Compared with dense models, MOE models have larger throughput when batch size is large, which is shown as follows: | Model | Quantization | \# Prompts | QPS | Tokens/s | | --- | --- | --- | --- | --- | | Qwen1.5-32B-Chat | BF16 | 100 | 6.68 | 7343.56 | | Qwen2-57B-A14B-Instruct | BF16 | 100 | 4.81 | 5291.15 | | Qwen1.5-32B-Chat | BF16 | 1000 | 7.99 | 8791.35 | | Qwen2-57B-A14B-Instruct | BF16 | 1000 | 5.18 | 5698.37 | The results are obtained from vLLM throughput benchmarking scripts, which can be reproduced by: `python vllm/benchmarks/benchmark_throughput.py --input-len 1000 --output-len 100 --model --num-prompts --enforce-eager -tp 2` * 72B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | | --- | --- | --- | --- | --- | --- | | Qwen2-72B-Instruct | 1 | BF16 | 2 | 7.45 | 134.74 | | GPTQ-Int8 | 2 | 7.30 | 71.00 | | GPTQ-Int4 | 1 | 9.05 | 41.80 | | AWQ | 1 | 9.96 | 41.31 | | 6144 | BF16 | 2 | 5.99 | 144.38 | | GPTQ-Int8 | 2 | 5.93 | 80.60 | | GPTQ-Int4 | 1 | 6.79 | 47.90 | | AWQ | 1 | 7.49 | 47.42 | | 14336 | BF16 | 3 | 4.12 | 169.93 | | GPTQ-Int8 | 2 | 4.43 | 95.14 | | GPTQ-Int4 | 1 | 4.87 | 57.79 | | AWQ | 1 | 5.23 | 57.30 | | 30720 | BF16 | 3 | 2.86 | 209.03 | | GPTQ-Int8 | 2 | 2.83 | 124.20 | | GPTQ-Int4 | 2 | 3.02 | 107.94 | | AWQ | 2 | 1.85 | 88.60 | * 72B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Setting | | --- | --- | --- | --- | --- | --- | | Qwen2-72B-Instruct | 1 | BF16 | 2 | 17.68 | \[Setting 1\] | | BF16 | 4 | 30.01 | | | GPTQ-Int8 | 2 | 27.56 | | | GPTQ-Int4 | 1 | 29.60 | \[Setting 2\] | | GPTQ-Int4 | 2 | 42.82 | | | AWQ | 2 | 27.73 | | | 6144 | BF16 | 4 | 27.98 | | | GPTQ-Int8 | 2 | 25.46 | | | GPTQ-Int4 | 1 | 25.16 | \[Setting 3\] | | GPTQ-Int4 | 2 | 38.23 | | | AWQ | 2 | 25.77 | | | 14336 | BF16 | 4 | 21.81 | | | GPTQ-Int8 | 2 | 22.71 | | | GPTQ-Int4 | 2 | 26.54 | | | AWQ | 2 | 21.50 | | | 30720 | BF16 | 4 | 19.43 | | | GPTQ-Int8 | 2 | 18.69 | | | GPTQ-Int4 | 2 | 23.12 | | | AWQ | 2 | 18.09 | | | 30720 | BF16 | 4 | 19.43 | | | GPTQ-Int8 | 2 | 18.69 | | | GPTQ-Int4 | 2 | 23.12 | | | AWQ | 2 | 18.09 | | | 63488 | BF16 | 4 | 17.46 | | | GPTQ-Int8 | 2 | 15.30 | | | GPTQ-Int4 | 2 | 13.23 | | | AWQ | 2 | 13.14 | | | 129024 | BF16 | 4 | 11.70 | | | GPTQ-Int8 | 4 | 12.94 | | | GPTQ-Int4 | 2 | 8.33 | | | AWQ | 2 | 7.78 | | > * \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) > > * \[Setting 1\]=(gpu\_memory\_utilization=0.98 max\_model\_len=4096 enforce\_eager=True) > > * \[Setting 2\]=(gpu\_memory\_utilization=1.0 max\_model\_len=4096 enforce\_eager=True) > > * \[Setting 3\]=(gpu\_memory\_utilization=1.0 max\_model\_len=8192 enforce\_eager=True) > --- # llama.cpp - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/quantization/llama.cpp.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar llama.cpp[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#llama-cpp "Link to this heading") ================================================================================================================= Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program `llama-quantize` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using `llama-imatrix`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don’t, check our guide [here](https://qwen.readthedocs.io/en/stable/run_locally/llama.cpp.html#getting-the-program) . Getting the GGUF[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#getting-the-gguf "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- Now, suppose you would like to quantize `Qwen3-8B`. You need to first make a GGUF file as shown below: python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outfile Qwen3-8B-F16.gguf Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outtype bf16 \--outfile Qwen3-8B-BF16.gguf Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outtype f32 \--outfile Qwen3-8B-F32.gguf Quantizing the GGUF without Calibration[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#quantizing-the-gguf-without-calibration "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8\_0.gguf Q8\_0 `Q8_0` is a code for a quantization preset. You can find all the presets in [the source code of `llama-quantize`](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp) . Look for the variable `QUANT_OPTIONS`. Common ones used for 8B models include `Q8_0`, `Q5_K_M`, and `Q4_K_M`. The letter case doesn’t matter, so `q8_0` or `q4_K_m` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. Quantizing the GGUF with AWQ Scale[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#quantizing-the-gguf-with-awq-scale "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Attention To be updated for Qwen3. To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following [this script](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export) . First, when you run `model.quantize()` with `autoawq`, remember to add `export_compatible=True` as shown below: ... model.quantize( tokenizer, quant\_config\=quant\_config, export\_compatible\=True ) model.save\_pretrained(quant\_path) ... The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are “easier” to quantize.[\[1\]](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#awq) Then, when you run `convert-hf-to-gguf.py`, remember to replace the model path with the path to the new model: python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf Finally, you can quantize the model as in the last example: ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 In this way, it should be possible to achieve similar quality with lower bit-per-weight. Quantizing the GGUF with Importance Matrix[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#quantizing-the-gguf-with-importance-matrix "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Another possible solution is to use the “important matrix”[\[2\]](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#imatrix) , following [this](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix) . First, you need to compute the importance matrix data of the weights of a model (`-m`) using a calibration dataset (`-f`): ./llama-imatrix \-m Qwen3-8B-F16.gguf \-f calibration-text.txt \--chunk 512 \-o Qwen3-8B-imatrix.dat \-ngl 80 The text is cut in chunks of length `--chunk` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named `Qwen3-8B-imatrix.dat` (`-o`), which can then be used: ./llama-quantize \--imatrix Qwen3-8B-imatrix.dat \\ Qwen3-8B-F16.gguf Qwen3-8B-Q4\_K\_M.gguf Q4\_K\_M For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide `--imatrix`, a helpful warning will be printed by `llama-quantize`. Perplexity Evaluation[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#perplexity-evaluation "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------- `llama.cpp` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say “wiki test”[\[3\]](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#wiki) . You can download the dataset with: wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref\=salesforce-research \-O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip Then you can run the test with the following command: ./llama-perplexity \-m Qwen3-8B-Q8\_0.gguf \-f wiki.test.raw \-ngl 80 Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture [here](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md) . It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. Finally[¶](https://qwen.readthedocs.io/en/stable/quantization/llama.cpp.html#finally "Link to this heading") ------------------------------------------------------------------------------------------------------------- In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the [llama.cpp GitHub repo](https://github.com/ggml-org/llama.cpp) . We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don’t provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven’t found what you need in our repos, look around. Enjoy your freshly quantized models! * * * --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/training/llama_factory.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LLaMA-Factory[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#llama-factory "Link to this heading") ======================================================================================================================= Attention To be updated for Qwen3. Here we provide a script for supervised finetuning Qwen2.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#installation "Link to this heading") --------------------------------------------------------------------------------------------------------------------- Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#data-preparation "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#training "Link to this heading") ------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#merge-lora "Link to this heading") ----------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/v3.0/training/llama_factory.html#conclusion "Link to this heading") ----------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/training/llama_factory.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LLaMA-Factory[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#llama-factory "Link to this heading") ========================================================================================================================= Attention To be updated for Qwen3. Here we provide a script for supervised finetuning Qwen2.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#installation "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#data-preparation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#training "Link to this heading") --------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#merge-lora "Link to this heading") ------------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/stable/training/llama_factory.html#conclusion "Link to this heading") ------------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # verl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/training/verl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/training/verl.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/training/verl.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar verl[¶](https://qwen.readthedocs.io/en/stable/training/verl.html#verl "Link to this heading") ============================================================================================== verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: [verl](https://github.com/volcengine/verl) verl is flexible and easy to use with: * **Easy extension of diverse RL algorithms**: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. * **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc * **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. * **Ready integration with popular HuggingFace models**: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: * **State-of-the-art throughput**: SOTA LLM training and inference engine integrations and SOTA RL throughput. * **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/stable/training/verl.html#reinforcement-learning-rl "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------ Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation[¶](https://qwen.readthedocs.io/en/stable/training/verl.html#step1-environment-and-training-preparation "Link to this heading") You can follow verl’s [installation guide](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py \--local\_dir ~/data/gsm8k Model download can be done using the following command: python3 \-c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" ### Step2: Start Training[¶](https://qwen.readthedocs.io/en/stable/training/verl.html#step2-start-training "Link to this heading") In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. python3 \-m verl.trainer.main\_ppo \\ algorithm.adv\_estimator\=grpo \\ data.train\_files\=$HOME/data/gsm8k/train.parquet \\ data.val\_files\=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size\=1024 \\ data.max\_prompt\_length\=512 \\ data.max\_response\_length\=1024 \\ data.filter\_overlong\_prompts\=True \\ data.truncation\='error' \\ actor\_rollout\_ref.model.path\=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr\=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding\=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size\=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss\=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef\=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type\=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff\=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing\=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload\=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload\=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size\=1 \\ actor\_rollout\_ref.rollout.name\=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization\=0.6 \\ actor\_rollout\_ref.rollout.n\=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload\=True \\ algorithm.use\_kl\_in\_reward\=False \\ trainer.critic\_warmup\=0 \\ trainer.logger\=\['console'\] \\ trainer.project\_name\='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name\='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node\=1 \\ trainer.nnodes\=1 \\ trainer.save\_freq\=\-1 \\ trainer.test\_freq\=5 \\ trainer.total\_epochs\=15 $@ Finally[¶](https://qwen.readthedocs.io/en/stable/training/verl.html#finally "Link to this heading") ---------------------------------------------------------------------------------------------------- If you encounter any difficulties during use, please join the discussion at [GitHub](https://github.com/volcengine/verl/discussions) . --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/framework/qwen_agent.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen-Agent[¶](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html#qwen-agent "Link to this heading") =============================================================================================================== [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant. Installation[¶](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html#installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------- git clone https://github.com/QwenLM/Qwen-Agent.git cd Qwen-Agent pip install \-e ./ Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html#developing-your-own-agent "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Qwen-Agent provides atomic components such as LLMs and prompts, as well as high-level components such as Agents. The example below uses the Assistant component as an illustration, demonstrating how to add custom tools and quickly develop an agent that uses tools. import json import os import json5 import urllib.parse from qwen\_agent.agents import Assistant from qwen\_agent.tools.base import BaseTool, register\_tool llm\_cfg \= { \# Use the model service provided by DashScope: 'model': 'qwen-max', 'model\_server': 'dashscope', \# 'api\_key': 'YOUR\_DASHSCOPE\_API\_KEY', \# It will use the \`DASHSCOPE\_API\_KEY' environment variable if 'api\_key' is not set here. \# Use your own model service compatible with OpenAI API: \# 'model': 'Qwen/Qwen2.5-7B-Instruct', \# 'model\_server': 'http://localhost:8000/v1', # api\_base \# 'api\_key': 'EMPTY', \# (Optional) LLM hyperparameters for generation: 'generate\_cfg': { 'top\_p': 0.8 } } system \= 'According to the user\\'s request, you first draw a picture and then automatically run code to download the picture ' + \\ 'and select an image operation from the given document to process the image' \# Add a custom tool named my\_image\_gen： @register\_tool('my\_image\_gen') class MyImageGen(BaseTool): description \= 'AI painting (image generation) service, input text description, and return the image URL drawn based on text information.' parameters \= \[{\ 'name': 'prompt',\ 'type': 'string',\ 'description': 'Detailed description of the desired image content, in English',\ 'required': True\ }\] def call(self, params: str, \*\*kwargs) \-> str: prompt \= json5.loads(params)\['prompt'\] prompt \= urllib.parse.quote(prompt) return json.dumps( {'image\_url': f'https://image.pollinations.ai/prompt/{prompt}'}, ensure\_ascii\=False) tools \= \['my\_image\_gen', 'code\_interpreter'\] \# code\_interpreter is a built-in tool in Qwen-Agent bot \= Assistant(llm\=llm\_cfg, system\_message\=system, function\_list\=tools, files\=\[os.path.abspath('doc.pdf')\]) messages \= \[\] while True: query \= input('user question: ') messages.append({'role': 'user', 'content': query}) response \= \[\] for response in bot.run(messages\=messages): print('bot response:', response) messages.extend(response) The framework also provides more atomic components for developers to combine. For additional showcases, please refer to [examples](https://github.com/QwenLM/Qwen-Agent/tree/main/examples) . --- # llama.cpp - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/quantization/llama.cpp.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar llama.cpp[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#llama-cpp "Link to this heading") =============================================================================================================== Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program `llama-quantize` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using `llama-imatrix`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don’t, check our guide [here](https://qwen.readthedocs.io/en/v3.0/run_locally/llama.cpp.html#getting-the-program) . Getting the GGUF[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#getting-the-gguf "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------- Now, suppose you would like to quantize `Qwen3-8B`. You need to first make a GGUF file as shown below: python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outfile Qwen3-8B-F16.gguf Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outtype bf16 \--outfile Qwen3-8B-BF16.gguf Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use python convert-hf-to-gguf.py Qwen/Qwen3-8B \--outtype f32 \--outfile Qwen3-8B-F32.gguf Quantizing the GGUF without Calibration[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#quantizing-the-gguf-without-calibration "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8\_0.gguf Q8\_0 `Q8_0` is a code for a quantization preset. You can find all the presets in [the source code of `llama-quantize`](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp) . Look for the variable `QUANT_OPTIONS`. Common ones used for 8B models include `Q8_0`, `Q5_K_M`, and `Q4_K_M`. The letter case doesn’t matter, so `q8_0` or `q4_K_m` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. Quantizing the GGUF with AWQ Scale[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#quantizing-the-gguf-with-awq-scale "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Attention To be updated for Qwen3. To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following [this script](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export) . First, when you run `model.quantize()` with `autoawq`, remember to add `export_compatible=True` as shown below: ... model.quantize( tokenizer, quant\_config\=quant\_config, export\_compatible\=True ) model.save\_pretrained(quant\_path) ... The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are “easier” to quantize.[\[1\]](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#awq) Then, when you run `convert-hf-to-gguf.py`, remember to replace the model path with the path to the new model: python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf Finally, you can quantize the model as in the last example: ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 In this way, it should be possible to achieve similar quality with lower bit-per-weight. Quantizing the GGUF with Importance Matrix[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#quantizing-the-gguf-with-importance-matrix "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Another possible solution is to use the “important matrix”[\[2\]](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#imatrix) , following [this](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix) . First, you need to compute the importance matrix data of the weights of a model (`-m`) using a calibration dataset (`-f`): ./llama-imatrix \-m Qwen3-8B-F16.gguf \-f calibration-text.txt \--chunk 512 \-o Qwen3-8B-imatrix.dat \-ngl 80 The text is cut in chunks of length `--chunk` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named `Qwen3-8B-imatrix.dat` (`-o`), which can then be used: ./llama-quantize \--imatrix Qwen3-8B-imatrix.dat \\ Qwen3-8B-F16.gguf Qwen3-8B-Q4\_K\_M.gguf Q4\_K\_M For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide `--imatrix`, a helpful warning will be printed by `llama-quantize`. Perplexity Evaluation[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#perplexity-evaluation "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------- `llama.cpp` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say “wiki test”[\[3\]](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#wiki) . You can download the dataset with: wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref\=salesforce-research \-O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip Then you can run the test with the following command: ./llama-perplexity \-m Qwen3-8B-Q8\_0.gguf \-f wiki.test.raw \-ngl 80 Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture [here](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md) . It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. Finally[¶](https://qwen.readthedocs.io/en/v3.0/quantization/llama.cpp.html#finally "Link to this heading") ----------------------------------------------------------------------------------------------------------- In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the [llama.cpp GitHub repo](https://github.com/ggml-org/llama.cpp) . We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don’t provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven’t found what you need in our repos, look around. Enjoy your freshly quantized models! * * * --- # verl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/training/verl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/training/verl.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/training/verl.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar verl[¶](https://qwen.readthedocs.io/en/v3.0/training/verl.html#verl "Link to this heading") ============================================================================================ verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: [verl](https://github.com/volcengine/verl) verl is flexible and easy to use with: * **Easy extension of diverse RL algorithms**: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. * **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc * **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. * **Ready integration with popular HuggingFace models**: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: * **State-of-the-art throughput**: SOTA LLM training and inference engine integrations and SOTA RL throughput. * **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/v3.0/training/verl.html#reinforcement-learning-rl "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------- Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation[¶](https://qwen.readthedocs.io/en/v3.0/training/verl.html#step1-environment-and-training-preparation "Link to this heading") You can follow verl’s [installation guide](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py \--local\_dir ~/data/gsm8k Model download can be done using the following command: python3 \-c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" ### Step2: Start Training[¶](https://qwen.readthedocs.io/en/v3.0/training/verl.html#step2-start-training "Link to this heading") In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. python3 \-m verl.trainer.main\_ppo \\ algorithm.adv\_estimator\=grpo \\ data.train\_files\=$HOME/data/gsm8k/train.parquet \\ data.val\_files\=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size\=1024 \\ data.max\_prompt\_length\=512 \\ data.max\_response\_length\=1024 \\ data.filter\_overlong\_prompts\=True \\ data.truncation\='error' \\ actor\_rollout\_ref.model.path\=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr\=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding\=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size\=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss\=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef\=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type\=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff\=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing\=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload\=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload\=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size\=1 \\ actor\_rollout\_ref.rollout.name\=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization\=0.6 \\ actor\_rollout\_ref.rollout.n\=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload\=True \\ algorithm.use\_kl\_in\_reward\=False \\ trainer.critic\_warmup\=0 \\ trainer.logger\=\['console'\] \\ trainer.project\_name\='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name\='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node\=1 \\ trainer.nnodes\=1 \\ trainer.save\_freq\=\-1 \\ trainer.test\_freq\=5 \\ trainer.total\_epochs\=15 $@ Finally[¶](https://qwen.readthedocs.io/en/v3.0/training/verl.html#finally "Link to this heading") -------------------------------------------------------------------------------------------------- If you encounter any difficulties during use, please join the discussion at [GitHub](https://github.com/volcengine/verl/discussions) . --- # Axolotl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/training/axolotl.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Axolotl[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#axolotl "Link to this heading") ===================================================================================================== This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. Requirements[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#requirements "Link to this heading") --------------------------------------------------------------------------------------------------------------- * **GPU:** NVIDIA Ampere (or newer) for `bf16` and `Flash Attention`, or AMD GPU * **Python:** ≥3.11 * **CUDA:** ≥12.4 (for NVIDIA GPUs) Installation[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#installation "Link to this heading") --------------------------------------------------------------------------------------------------------------- You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. Important Install PyTorch _before_ installing Axolotl to ensure CUDA compatibility. For the latest instructions, see the official [Axolotl Installation Guide](https://docs.axolotl.ai/docs/installation.html) . Quickstart[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#quickstart "Link to this heading") ----------------------------------------------------------------------------------------------------------- ### SFT[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#sft "Link to this heading") We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: [SFT 32B QLoRA config](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml) . \# Train the model axolotl train path/to/32b-qlora.yaml \# Merge LoRA weights with the base model \# This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml Tip To train a smaller model, edit the `base_model` in your config: base\_model: Qwen/Qwen3-8B Qwen3 works with all Axolotl features including `Flash Attention`, `bf16`, `LoRA`, `torch_compile`, and `QLoRA`. To run on more than single GPU, please take a look at the [Multi-GPU Training Guide](https://docs.axolotl.ai/docs/multi-gpu.html) or [Multi-node Training Guide](https://docs.axolotl.ai/docs/multi-node.html) . ### RLHF[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#rlhf "Link to this heading") See the [RLHF Guide](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#rm-prm "Link to this heading") Please refer to the [Reward Modelling Guide](https://docs.axolotl.ai/docs/reward_modelling.html) for required dataset formats and config examples. Dataset[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#dataset "Link to this heading") ----------------------------------------------------------------------------------------------------- By default, the example config uses the `mlabonne/FineTome-100k` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#sft-dataset-format "Link to this heading") Axolotl handles various SFT dataset formats, but the current **recommended** format (for use with `chat_template`) is the OpenAI Messages format: \[\ {\ "messages": \[\ {\ "role": "user",\ "content": "What is Qwen3?"\ },\ {\ "role": "assistant",\ "content": "Qwen3 is a language model..."\ }\ \]\ }\ \] Use this in your config: datasets: \- path: path/to/your/dataset.json type: chat\_template You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the [Dataset Loading Guide](https://docs.axolotl.ai/docs/dataset_loading.html) for more details. To load different dataset formats, refer to the [SFT Dataset Formats Guide](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft) . Optimizations[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#optimizations "Link to this heading") ----------------------------------------------------------------------------------------------------------------- With Qwen3/Qwen3\_MOE, you can leverage Axolotl’s custom optimizations for improved speed and reduced memory usage: * [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy) * [Liger Kernels](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels) * (LoRA/QLoRA only): [LoRA Kernels Optimization](https://docs.axolotl.ai/docs/lora_optims.html) Additional Suggestions[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#additional-suggestions "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------- ### Troubleshooting[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#troubleshooting "Link to this heading") * Ensure your CUDA version matches your GPU and PyTorch version. * If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. * Qwen3 MoE may have slower training due to the upstream transformer’s handling of MoE layers. * For help, check the help channel on [Axolotl Discord](https://discord.gg/7m9sfhzaf3) or create a Discussion on [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) . ### Links[¶](https://qwen.readthedocs.io/en/v3.0/training/axolotl.html#links "Link to this heading") * [Axolotl Documentation](https://docs.axolotl.ai/) * [Axolotl Discord](https://discord.gg/7m9sfhzaf3) * [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) * [Axolotl Website](https://axolotl.ai/) --- # Axolotl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/training/axolotl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/training/axolotl.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/training/axolotl.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Axolotl[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#axolotl "Link to this heading") ======================================================================================================= This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. Requirements[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#requirements "Link to this heading") ----------------------------------------------------------------------------------------------------------------- * **GPU:** NVIDIA Ampere (or newer) for `bf16` and `Flash Attention`, or AMD GPU * **Python:** ≥3.11 * **CUDA:** ≥12.4 (for NVIDIA GPUs) Installation[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#installation "Link to this heading") ----------------------------------------------------------------------------------------------------------------- You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. Important Install PyTorch _before_ installing Axolotl to ensure CUDA compatibility. For the latest instructions, see the official [Axolotl Installation Guide](https://docs.axolotl.ai/docs/installation.html) . Quickstart[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#quickstart "Link to this heading") ------------------------------------------------------------------------------------------------------------- ### SFT[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#sft "Link to this heading") We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: [SFT 32B QLoRA config](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml) . \# Train the model axolotl train path/to/32b-qlora.yaml \# Merge LoRA weights with the base model \# This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml Tip To train a smaller model, edit the `base_model` in your config: base\_model: Qwen/Qwen3-8B Qwen3 works with all Axolotl features including `Flash Attention`, `bf16`, `LoRA`, `torch_compile`, and `QLoRA`. To run on more than single GPU, please take a look at the [Multi-GPU Training Guide](https://docs.axolotl.ai/docs/multi-gpu.html) or [Multi-node Training Guide](https://docs.axolotl.ai/docs/multi-node.html) . ### RLHF[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#rlhf "Link to this heading") See the [RLHF Guide](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#rm-prm "Link to this heading") Please refer to the [Reward Modelling Guide](https://docs.axolotl.ai/docs/reward_modelling.html) for required dataset formats and config examples. Dataset[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#dataset "Link to this heading") ------------------------------------------------------------------------------------------------------- By default, the example config uses the `mlabonne/FineTome-100k` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#sft-dataset-format "Link to this heading") Axolotl handles various SFT dataset formats, but the current **recommended** format (for use with `chat_template`) is the OpenAI Messages format: \[\ {\ "messages": \[\ {\ "role": "user",\ "content": "What is Qwen3?"\ },\ {\ "role": "assistant",\ "content": "Qwen3 is a language model..."\ }\ \]\ }\ \] Use this in your config: datasets: \- path: path/to/your/dataset.json type: chat\_template You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the [Dataset Loading Guide](https://docs.axolotl.ai/docs/dataset_loading.html) for more details. To load different dataset formats, refer to the [SFT Dataset Formats Guide](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft) . Optimizations[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#optimizations "Link to this heading") ------------------------------------------------------------------------------------------------------------------- With Qwen3/Qwen3\_MOE, you can leverage Axolotl’s custom optimizations for improved speed and reduced memory usage: * [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy) * [Liger Kernels](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels) * (LoRA/QLoRA only): [LoRA Kernels Optimization](https://docs.axolotl.ai/docs/lora_optims.html) Additional Suggestions[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#additional-suggestions "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------- ### Troubleshooting[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#troubleshooting "Link to this heading") * Ensure your CUDA version matches your GPU and PyTorch version. * If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. * Qwen3 MoE may have slower training due to the upstream transformer’s handling of MoE layers. * For help, check the help channel on [Axolotl Discord](https://discord.gg/7m9sfhzaf3) or create a Discussion on [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) . ### Links[¶](https://qwen.readthedocs.io/en/stable/training/axolotl.html#links "Link to this heading") * [Axolotl Documentation](https://docs.axolotl.ai/) * [Axolotl Discord](https://discord.gg/7m9sfhzaf3) * [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) * [Axolotl Website](https://axolotl.ai/) --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/framework/qwen_agent.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen-Agent[¶](https://qwen.readthedocs.io/en/stable/framework/qwen_agent.html#qwen-agent "Link to this heading") ================================================================================================================= [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. Installation[¶](https://qwen.readthedocs.io/en/stable/framework/qwen_agent.html#installation "Link to this heading") --------------------------------------------------------------------------------------------------------------------- * Install the stable version from PyPI: pip install \-U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" \# Or use \`pip install -U qwen-agent\` for the minimal requirements. \# The optional requirements, specified in double brackets, are: \# \[gui\] for Gradio-based GUI support; \# \[rag\] for RAG support; \# \[code\_interpreter\] for Code Interpreter support; \# \[mcp\] for MCP support. Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/stable/framework/qwen_agent.html#developing-your-own-agent "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. import os from qwen\_agent.agents import Assistant \# Define LLM llm\_cfg \= { \# Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', \# api\_base 'api\_key': 'EMPTY', \# 'generate\_cfg': { \# # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way \# 'extra\_body': { \# 'chat\_template\_kwargs': {'enable\_thinking': False} \# }, # \# # Add: When the content is \`this is the thoughtthis is the answer\` \# # Do not add: When the response has been separated by reasoning\_content and content \# # This parameter will affect the parsing strategy of tool call \# # 'thought\_in\_content': True, \# }, } \# llm\_cfg = { \# # Use the model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_type': 'qwen\_dashscope', # \# # 'generate\_cfg': { \# # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way \# # 'enable\_thinking': False, \# # }, \# } \# llm\_cfg = { \# # Use the OpenAI-compatible model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', \# 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # \# # 'generate\_cfg': { \# # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way \# # 'extra\_body': { \# # 'enable\_thinking': False \# # }, \# # }, \# } \# Define Tools tools \= \[\ {'mcpServers': { \# You can specify the MCP configuration file\ 'time': {\ 'command': 'uvx',\ 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\]\ },\ "fetch": {\ "command": "uvx",\ "args": \["mcp-server-fetch"\]\ }\ }\ },\ 'code\_interpreter', \# Built-in tools\ \] \# Define Agent bot \= Assistant(llm\=llm\_cfg, function\_list\=tools) \# Streaming generation messages \= \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages\=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. --- # Performance of Quantized Models - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/benchmark/quantization_benchmark.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/benchmark/quantization_benchmark.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/benchmark/quantization_benchmark.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Performance of Quantized Models[¶](https://qwen.readthedocs.io/en/v2.5/benchmark/quantization_benchmark.html#performance-of-quantized-models "Link to this heading") ===================================================================================================================================================================== Attention To be updated for Qwen2.5. This section reports the generation performance of quantized models (including GPTQ and AWQ) of the Qwen2 series. Specifically, we report: * MMLU (Accuracy) * C-Eval (Accuracy) * IFEval (Strict Prompt-Level Accuracy) We use greedy decoding in evaluating all models. | | Quantization | Average | MMLU | C-Eval | IFEval | | --- | --- | --- | --- | --- | --- | | Qwen2-72B-Instruct | BF16 | 81.3 | 82.3 | 83.8 | 77.6 | | GPTQ-Int8 | 80.7 | 81.3 | 83.4 | 77.5 | | GPTQ-Int4 | 81.2 | 80.8 | 83.9 | 78.9 | | AWQ | 80.4 | 80.5 | 83.9 | 76.9 | | Qwen2-7B-Instruct | BF16 | 66.9 | 70.5 | 77.2 | 53.1 | | GPTQ-Int8 | 66.2 | 69.1 | 76.7 | 52.9 | | GPTQ-Int4 | 64.1 | 67.8 | 75.2 | 49.4 | | AWQ | 64.1 | 67.4 | 73.6 | 51.4 | | Qwen2-1.5B-Instruct | BF16 | 48.4 | 52.4 | 63.8 | 29.0 | | GPTQ-Int8 | 48.1 | 53.0 | 62.5 | 28.8 | | GPTQ-Int4 | 45.0 | 50.7 | 57.4 | 27.0 | | AWQ | 46.5 | 51.6 | 58.1 | 29.9 | | Qwen2-0.5B-Instruct | BF16 | 34.4 | 37.9 | 45.2 | 20.0 | | GPTQ-Int8 | 32.6 | 35.6 | 43.9 | 18.1 | | GPTQ-Int4 | 29.7 | 33.0 | 39.2 | 16.8 | | AWQ | 31.1 | 34.4 | 42.1 | 16.7 | --- # Axolotl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/training/axolotl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/training/axolotl.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/training/axolotl.md.txt "View this page") Axolotl[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#axolotl "Link to this heading") ======================================================================================================= This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. Requirements[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#requirements "Link to this heading") ----------------------------------------------------------------------------------------------------------------- * **GPU:** NVIDIA Ampere (or newer) for `bf16` and `Flash Attention`, or AMD GPU * **Python:** ≥3.11 * **CUDA:** ≥12.4 (for NVIDIA GPUs) Installation[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#installation "Link to this heading") ----------------------------------------------------------------------------------------------------------------- You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. Important Install PyTorch _before_ installing Axolotl to ensure CUDA compatibility. For the latest instructions, see the official [Axolotl Installation Guide](https://docs.axolotl.ai/docs/installation.html) . Quickstart[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#quickstart "Link to this heading") ------------------------------------------------------------------------------------------------------------- ### SFT[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#sft "Link to this heading") We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: [SFT 32B QLoRA config](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml) . \# Train the model axolotl train path/to/32b-qlora.yaml \# Merge LoRA weights with the base model \# This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml Tip To train a smaller model, edit the `base_model` in your config: base\_model: Qwen/Qwen3-8B Qwen3 works with all Axolotl features including `Flash Attention`, `bf16`, `LoRA`, `torch_compile`, and `QLoRA`. To run on more than single GPU, please take a look at the [Multi-GPU Training Guide](https://docs.axolotl.ai/docs/multi-gpu.html) or [Multi-node Training Guide](https://docs.axolotl.ai/docs/multi-node.html) . ### RLHF[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#rlhf "Link to this heading") See the [RLHF Guide](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#rm-prm "Link to this heading") Please refer to the [Reward Modelling Guide](https://docs.axolotl.ai/docs/reward_modelling.html) for required dataset formats and config examples. Dataset[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#dataset "Link to this heading") ------------------------------------------------------------------------------------------------------- By default, the example config uses the `mlabonne/FineTome-100k` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#sft-dataset-format "Link to this heading") Axolotl handles various SFT dataset formats, but the current **recommended** format (for use with `chat_template`) is the OpenAI Messages format: \[\ {\ "messages": \[\ {\ "role": "user",\ "content": "What is Qwen3?"\ },\ {\ "role": "assistant",\ "content": "Qwen3 is a language model..."\ }\ \]\ }\ \] Use this in your config: datasets: \- path: path/to/your/dataset.json type: chat\_template You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the [Dataset Loading Guide](https://docs.axolotl.ai/docs/dataset_loading.html) for more details. To load different dataset formats, refer to the [SFT Dataset Formats Guide](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft) . Optimizations[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#optimizations "Link to this heading") ------------------------------------------------------------------------------------------------------------------- With Qwen3/Qwen3\_MOE, you can leverage Axolotl’s custom optimizations for improved speed and reduced memory usage: * [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy) * [Liger Kernels](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels) * (LoRA/QLoRA only): [LoRA Kernels Optimization](https://docs.axolotl.ai/docs/lora_optims.html) Additional Suggestions[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#additional-suggestions "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------- ### Troubleshooting[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#troubleshooting "Link to this heading") * Ensure your CUDA version matches your GPU and PyTorch version. * If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. * Qwen3 MoE may have slower training due to the upstream transformer’s handling of MoE layers. * For help, check the help channel on [Axolotl Discord](https://discord.gg/7m9sfhzaf3) or create a Discussion on [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) . ### Links[¶](https://qwen.readthedocs.io/en/latest/training/axolotl.html#links "Link to this heading") * [Axolotl Documentation](https://docs.axolotl.ai/) * [Axolotl Discord](https://discord.gg/7m9sfhzaf3) * [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) * [Axolotl Website](https://axolotl.ai/) --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/framework/qwen_agent.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen-Agent[¶](https://qwen.readthedocs.io/en/v3.0/framework/qwen_agent.html#qwen-agent "Link to this heading") =============================================================================================================== [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. Installation[¶](https://qwen.readthedocs.io/en/v3.0/framework/qwen_agent.html#installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------- * Install the stable version from PyPI: pip install \-U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" \# Or use \`pip install -U qwen-agent\` for the minimal requirements. \# The optional requirements, specified in double brackets, are: \# \[gui\] for Gradio-based GUI support; \# \[rag\] for RAG support; \# \[code\_interpreter\] for Code Interpreter support; \# \[mcp\] for MCP support. Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/v3.0/framework/qwen_agent.html#developing-your-own-agent "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. import os from qwen\_agent.agents import Assistant \# Define LLM llm\_cfg \= { \# Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', \# api\_base 'api\_key': 'EMPTY', \# 'generate\_cfg': { \# # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way \# 'extra\_body': { \# 'chat\_template\_kwargs': {'enable\_thinking': False} \# }, # \# # Add: When the content is \`this is the thoughtthis is the answer\` \# # Do not add: When the response has been separated by reasoning\_content and content \# # This parameter will affect the parsing strategy of tool call \# # 'thought\_in\_content': True, \# }, } \# llm\_cfg = { \# # Use the model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_type': 'qwen\_dashscope', # \# # 'generate\_cfg': { \# # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way \# # 'enable\_thinking': False, \# # }, \# } \# llm\_cfg = { \# # Use the OpenAI-compatible model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', \# 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # \# # 'generate\_cfg': { \# # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way \# # 'extra\_body': { \# # 'enable\_thinking': False \# # }, \# # }, \# } \# Define Tools tools \= \[\ {'mcpServers': { \# You can specify the MCP configuration file\ 'time': {\ 'command': 'uvx',\ 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\]\ },\ "fetch": {\ "command": "uvx",\ "args": \["mcp-server-fetch"\]\ }\ }\ },\ 'code\_interpreter', \# Built-in tools\ \] \# Define Agent bot \= Assistant(llm\=llm\_cfg, function\_list\=tools) \# Streaming generation messages \= \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages\=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/framework/LlamaIndex.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LlamaIndex[¶](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#llamaindex "Link to this heading") =============================================================================================================== To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation[¶](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#preparation "Link to this heading") ----------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#set-parameters "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen2.5 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name\="Qwen/Qwen2.5-7B-Instruct", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#build-index "Link to this heading") ----------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/v2.5/framework/LlamaIndex.html#rag "Link to this heading") ------------------------------------------------------------------------------------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # LLaMA-Factory - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/training/llama_factory.md.txt "View this page") LLaMA-Factory[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#llama-factory "Link to this heading") ========================================================================================================================= Attention To be updated for Qwen3. Here we provide a script for supervised finetuning Qwen2.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) . This script for supervised finetuning (SFT) has the following features: * Support single-GPU and multi-GPU training; * Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. Installation[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#installation "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) , and build the environment. 2. Install these packages (Optional): pip install deepspeed pip install flash\-attn \--no\-build\-isolation 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) , make sure your CUDA is 11.6 and above. Data Preparation[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#data-preparation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. * The dataset in `alpaca` format should follow the below format: \[\ {\ "instruction": "user instruction (required)",\ "input": "user input (optional)",\ "output": "model response (required)",\ "system": "system prompt (optional)",\ "history": \[\ \["user instruction in the first round (optional)", "model response in the first round (optional)"\],\ \["user instruction in the second round (optional)", "model response in the second round (optional)"\]\ \]\ }\ \] * The dataset in `sharegpt` format should follow the below format: \[\ {\ "conversations": \[\ {\ "from": "human",\ "value": "user instruction"\ },\ {\ "from": "gpt",\ "value": "model response"\ }\ \],\ "system": "system prompt (optional)",\ "tools": "tool description (optional)"\ }\ \] 2. Provide your dataset definition in `data/dataset_info.json` in the following format . * For `alpaca` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } * For `sharegpt` format dataset, the columns in `dataset_info.json` should be: "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } Training[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#training "Link to this heading") --------------------------------------------------------------------------------------------------------------- Execute the following training command: DISTRIBUTED\_ARGS\=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ \--deepspeed $DS\_CONFIG\_PATH \\ \--stage sft \\ \--do\_train \\ \--use\_fast\_tokenizer \\ \--flash\_attn \\ \--model\_name\_or\_path $MODEL\_PATH \\ \--dataset your\_dataset \\ \--template qwen \\ \--finetuning\_type lora \\ \--lora\_target q\_proj,v\_proj\\ \--output\_dir $OUTPUT\_PATH \\ \--overwrite\_cache \\ \--overwrite\_output\_dir \\ \--warmup\_steps 100 \\ \--weight\_decay 0.1 \\ \--per\_device\_train\_batch\_size 4 \\ \--gradient\_accumulation\_steps 4 \\ \--ddp\_timeout 9000 \\ \--learning\_rate 5e-6 \\ \--lr\_scheduler\_type cosine \\ \--logging\_steps 1 \\ \--cutoff\_len 4096 \\ \--save\_steps 1000 \\ \--plot\_loss \\ \--num\_train\_epochs 3 \\ \--bf16 and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. Merge LoRA[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#merge-lora "Link to this heading") ------------------------------------------------------------------------------------------------------------------- If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. CUDA\_VISIBLE\_DEVICES\=0 llamafactory-cli export \\ \--model\_name\_or\_path path\_to\_base\_model \\ \--adapter\_name\_or\_path path\_to\_adapter \\ \--template qwen \\ \--finetuning\_type lora \\ \--export\_dir path\_to\_export \\ \--export\_size 2 \\ \--export\_legacy\_format False Conclusion[¶](https://qwen.readthedocs.io/en/latest/training/llama_factory.html#conclusion "Link to this heading") ------------------------------------------------------------------------------------------------------------------- The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # Unsloth - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/training/unsloth.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/training/unsloth.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/training/unsloth.md.txt "View this page") Unsloth[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#unsloth "Link to this heading") ======================================================================================================= This guide will teach you how to easily train Qwen3 models with Unsloth. Unsloth simplifies local model training, handling everything from loading and quantization to training, evaluation, running, and deployment with inference engines (Ollama, llama.cpp, vLLM). **Train Qwen** models 2× faster using 70% less VRAM. **GitHub repo:** [Unsloth](https://github.com/unslothai/unsloth) ⭐ Key Features[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#key-features "Link to this heading") ------------------------------------------------------------------------------------------------------------------- * Supports full fine-tuning, pretraining, LoRA, QLoRA, 8-bit training & more * Single and multi-GPU support (Linux, Windows, Colab, Kaggle; NVIDIA GPUs, soon AMD & Intel) * Compatible with all transformer models: TTS, multimodal, STT, BERT, RL * RLHF support: GRPO, DPO, DAPO, RM, PPO, KTO, etc. * Hand-written Triton kernels and a manual backprop engine ensure no accuracy degradation (0% approximation). Quickstart[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#quickstart "Link to this heading") ------------------------------------------------------------------------------------------------------------- **Local Installation (Linux recommended):** pip install unsloth You can view Unsloth’s full [installation instructions here.](https://docs.unsloth.ai/get-started/installing-+-updating) Fine-tuning Qwen3 with Unsloth[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#fine-tuning-qwen3-with-unsloth "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------- Unsloth makes Qwen3 fine-tuning 2× faster, uses 70% less VRAM, with 8× longer contexts. Qwen3 (14B) fits in a free 16 GB Colab Tesla T4 GPU. To retain Qwen3’s reasoning capabilities, use a 75% reasoning to 25% non-reasoning dataset ratio (e.g., NVIDIA’s math‑reasoning dataset + Maxime’s FineTome). For more details, see Unsloth’s full [Qwen3 fine-tuning guide](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) . ### Colab Notebooks[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#colab-notebooks "Link to this heading") * [Qwen3 (14B) Reasoning + Conversational](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) * [Qwen3 (4B) Advanced GRPO LoRA](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) * [Qwen3 (14B) Alpaca (Base model)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Alpaca.ipynb) **Update Unsloth locally:** pip install \--upgrade \--force-reinstall \--no-cache-dir unsloth unsloth\_zoo ### Fine-tuning Qwen3 MoE Models[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#fine-tuning-qwen3-moe-models "Link to this heading") Supported MoE models include 30B‑A3B and 235B‑A22B. Unsloth fine-tunes the 30B‑A3B model with just 17.5 GB VRAM. Router-layer fine-tuning is disabled by default. Use `FastModel` for MoE fine-tuning: from unsloth import FastModel model, tokenizer \= FastModel.from\_pretrained( model\_name\="unsloth/Qwen3-30B-A3B", max\_seq\_length\=2048, load\_in\_4bit\=True, load\_in\_8bit\=False, full\_finetuning\=False, ) ### Notebook Guide[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#notebook-guide "Link to this heading") For an end-to-end walkthrough, see Unsloth’s [full end-to-end fine-tuning guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) . * Open the notebook → click **Runtime ▸ Run all**. * Adjust settings (e.g., model name, context length) directly in the notebook: * `max_seq_length`: Recommended 2048 (Qwen3 supports up to 40960). * `load_in_4bit=True`: reduces memory usage by 4×. * Enable full fine-tuning (`full_finetuning=True`) or 8-bit training (`load_in_8bit=True`). If you want to use models directly from [ModelScope](https://modelscope.cn/organization/unsloth) , use: pip install modelscope \-qqq import os os.environ\["UNSLOTH\_USE\_MODELSCOPE"\] \= "1" from unsloth import FastLanguageModel model, tokenizer \= FastLanguageModel.from\_pretrained( model\_name\="unsloth/Qwen3-4B-Base", max\_seq\_length\=2048, ) RL & GRPO with Qwen3[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#rl-grpo-with-qwen3 "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- You can also train Qwen models with reinforcement learning (RL) using Unsloth. Explore Unsloth’s advanced GRPO notebook, featuring proximity-based reward scoring and Hugging Face’s Open‑R1 math dataset: [Qwen3 (4B) Advanced GRPO LoRA notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) . * Proximity-based rewards for closer answers * Custom GRPO formatting and templates * Enhanced evaluation accuracy with regex matching Resources & Links[¶](https://qwen.readthedocs.io/en/latest/training/unsloth.html#resources-links "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- That’s how you can easily train Qwen models with Unsloth. If you need any help, join the discussion on Unsloth’s [Discord](https://discord.com/invite/unsloth) or [GitHub](https://github.com/unslothai/unsloth) pages. **Links:** * [Unsloth Documentation](https://docs.unsloth.ai/) * [Unsloth Discord](https://discord.com/invite/unsloth) * [Unsloth Website](https://unsloth.ai/) * [Unsloth Reddit](https://www.reddit.com/r/unsloth/) --- # verl - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/training/verl.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/training/verl.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/training/verl.md.txt "View this page") verl[¶](https://qwen.readthedocs.io/en/latest/training/verl.html#verl "Link to this heading") ============================================================================================== verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: [verl](https://github.com/volcengine/verl) verl is flexible and easy to use with: * **Easy extension of diverse RL algorithms**: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. * **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc * **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. * **Ready integration with popular HuggingFace models**: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: * **State-of-the-art throughput**: SOTA LLM training and inference engine integrations and SOTA RL throughput. * **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/latest/training/verl.html#reinforcement-learning-rl "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------ Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation[¶](https://qwen.readthedocs.io/en/latest/training/verl.html#step1-environment-and-training-preparation "Link to this heading") You can follow verl’s [installation guide](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py \--local\_dir ~/data/gsm8k Model download can be done using the following command: python3 \-c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" ### Step2: Start Training[¶](https://qwen.readthedocs.io/en/latest/training/verl.html#step2-start-training "Link to this heading") In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. python3 \-m verl.trainer.main\_ppo \\ algorithm.adv\_estimator\=grpo \\ data.train\_files\=$HOME/data/gsm8k/train.parquet \\ data.val\_files\=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size\=1024 \\ data.max\_prompt\_length\=512 \\ data.max\_response\_length\=1024 \\ data.filter\_overlong\_prompts\=True \\ data.truncation\='error' \\ actor\_rollout\_ref.model.path\=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr\=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding\=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size\=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss\=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef\=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type\=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff\=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing\=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload\=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload\=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size\=1 \\ actor\_rollout\_ref.rollout.name\=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization\=0.6 \\ actor\_rollout\_ref.rollout.n\=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu\=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload\=True \\ algorithm.use\_kl\_in\_reward\=False \\ trainer.critic\_warmup\=0 \\ trainer.logger\=\['console'\] \\ trainer.project\_name\='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name\='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node\=1 \\ trainer.nnodes\=1 \\ trainer.save\_freq\=\-1 \\ trainer.test\_freq\=5 \\ trainer.total\_epochs\=15 $@ Finally[¶](https://qwen.readthedocs.io/en/latest/training/verl.html#finally "Link to this heading") ---------------------------------------------------------------------------------------------------- If you encounter any difficulties during use, please join the discussion at [GitHub](https://github.com/volcengine/verl/discussions) . --- # Qwen-Agent - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/framework/qwen_agent.rst.txt "View this page") Qwen-Agent[¶](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html#qwen-agent "Link to this heading") ================================================================================================================= [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. Installation[¶](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html#installation "Link to this heading") --------------------------------------------------------------------------------------------------------------------- * Install the stable version from PyPI: pip install \-U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" \# Or use \`pip install -U qwen-agent\` for the minimal requirements. \# The optional requirements, specified in double brackets, are: \# \[gui\] for Gradio-based GUI support; \# \[rag\] for RAG support; \# \[code\_interpreter\] for Code Interpreter support; \# \[mcp\] for MCP support. Developing Your Own Agent[¶](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html#developing-your-own-agent "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. import os from qwen\_agent.agents import Assistant \# Define LLM llm\_cfg \= { \# Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', \# api\_base 'api\_key': 'EMPTY', \# 'generate\_cfg': { \# # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way \# 'extra\_body': { \# 'chat\_template\_kwargs': {'enable\_thinking': False} \# }, # \# # Add: When the content is \`this is the thoughtthis is the answer\` \# # Do not add: When the response has been separated by reasoning\_content and content \# # This parameter will affect the parsing strategy of tool call \# # 'thought\_in\_content': True, \# }, } \# llm\_cfg = { \# # Use the model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_type': 'qwen\_dashscope', # \# # 'generate\_cfg': { \# # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way \# # 'enable\_thinking': False, \# # }, \# } \# llm\_cfg = { \# # Use the OpenAI-compatible model service provided by DashScope: \# 'model': 'qwen3-235b-a22b', \# 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', \# 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # \# # 'generate\_cfg': { \# # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way \# # 'extra\_body': { \# # 'enable\_thinking': False \# # }, \# # }, \# } \# Define Tools tools \= \[\ {'mcpServers': { \# You can specify the MCP configuration file\ 'time': {\ 'command': 'uvx',\ 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\]\ },\ "fetch": {\ "command": "uvx",\ "args": \["mcp-server-fetch"\]\ }\ }\ },\ 'code\_interpreter', \# Built-in tools\ \] \# Define Agent bot \= Assistant(llm\=llm\_cfg, function\_list\=tools) \# Streaming generation messages \= \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages\=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) repository. --- # MS-SWIFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/training/ms_swift.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar MS-SWIFT[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#ms-swift "Link to this heading") ========================================================================================================== ModelScope SWIFT (**ms-swift**) is the large model and multimodal large model training and deployment framework provided by the [ModelScope community](https://modelscope.cn/) . GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: * **Model Types**: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. * **Hardware Support**: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. * **Training Methods**: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. * **Distributed Training**: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. * **RLHF Training**: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift’s integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. pip install ms-swift \-U \# Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers \-U \# Optional packages pip install deepspeed \# multi-GPU training pip install liger-kernel \# save GPU memory resources pip install flash-attn \--no-build-isolation Supervised Fine-Tuning (SFT)[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#supervised-fine-tuning-sft "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ ### Data Preparation[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#data-preparation "Link to this heading") The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify `--dataset ` in the training script. For complete dataset formatting guidelines, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) * General format {"messages": \[\ {"role": "system", "content": ""},\ {"role": "user", "content": ""},\ {"role": "assistant", "content": ""}\ \]} * Format with think {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} If you want to train using data without a chain of thought but retain the model’s reasoning ability, there are two approaches to minimize disruption during fine-tuning: **Option 1**: During training, specify `--loss_scale ignore_empty_think` to ignore the loss calculation for `\n\n\n\n`, preventing the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo1.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} **Option 2**: Add `/no_think` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo2.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} ### 30-Minute Self-Cognition Fine-Tuning[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#minute-self-cognition-fine-tuning "Link to this heading") This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by [ModelScope’s free compute resources](https://modelscope.cn/my/mynotebook) . After training, the model will identify itself as “swift-robot,” trained by “swift,” instead of its original self-cognition as “Qwen,” trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify `--model ` and `--dataset `. The dataset can be found on [Modelscope Hub](https://modelscope.cn/datasets/swift/self-cognition) . For the meaning of each parameter in the training script, please refer to the [Command-line parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) . CUDA\_VISIBLE\_DEVICES\=0 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type lora \\ \--dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-4 \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--gradient\_accumulation\_steps 16 \\ \--eval\_steps 50 \\ \--save\_steps 50 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--max\_length 2048 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--model\_author swift \\ \--model\_name swift-robot After fine-tuning, you can use the following script to test the fine-tuning results. Note that the `--adapters` section needs to be modified to the directory path of the last saved checkpoint: CUDA\_VISIBLE\_DEVICES\=0 \\ swift infer \\ \--adapters output/vx-xxx/checkpoint-xxx \\ \--stream true \\ \--temperature 0 \\ \--max\_new\_tokens 2048 <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify `--use_hf true`. Merge LoRA weights: swift export \\ \--adapters output/checkpoint-xxx \\ \--merge\_lora true Push the model to ModelScope/HuggingFace: \# If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. \# The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ \--adapters output/checkpoint-xxx \\ \--push\_to\_hub true \\ \--hub\_model\_id '' \\ \--hub\_token '' \\ \--use\_hf false If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \# 4 \* 60GB \# You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. \# Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE\=4 \\ CUDA\_VISIBLE\_DEVICES\=0,1,2,3 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset '' \\ \--torch\_dtype bfloat16 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-5 \\ \--gradient\_accumulation\_steps 4 \\ \--packing true \\ \--eval\_steps 100 \\ \--save\_steps 100 \\ \--logging\_steps 5 \\ \--max\_length 8192 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--save\_total\_limit 2 \\ \--save\_only\_model true \\ \--output\_dir output \\ \--deepspeed zero3 \\ \--use\_liger\_kernel true \\ \--attn\_impl flash\_attn Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#reinforcement-learning-rl "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------- ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: [Supported Features](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html) . ### Environment Setup[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#environment-setup "Link to this heading") In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: pip install "math\_verify==0.5.2" pip install vllm ### Data Preparation[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#id1 "Link to this heading") The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a `solution` column is needed to calculate the accuracy. Example Dataset Formats: {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} For dataset preparation for other RLHF algorithms, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf) . Notes on Dataset Requirements: * **Reward Function Calculation**: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: * When using the built-in accuracy or cosine similarity reward, the dataset must include a `solution` column to calculate the accuracy of the responses. * Other columns in the dataset will be passed as `**kwargs` to the reward function for additional customization. * **Customizing the Reward Function**: To adapt the reward function to your specific needs, you can refer to the following resource: [External Reward Plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin) . This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting `num_infer_workers=8`, we deploy a vLLM engine for each device to speed up the sampling process. \# 70G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset 'AI-MO/NuminaMath-TIR#5000' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 2 \\ \--per\_device\_eval\_batch\_size 2 \\ \--learning\_rate 1e-6 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--gradient\_accumulation\_steps 1 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--max\_completion\_length 4096 \\ \--vllm\_max\_model\_len 8192 \\ \--reward\_funcs accuracy \\ \--num\_generations 16 \\ \--use\_vllm true \\ \--vllm\_gpu\_memory\_utilization 0.4 \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--deepspeed zero3 \\ \--num\_infer\_workers 8 \\ \--tensor\_parallel\_size 1 \\ \--temperature 1.0 \\ \--top\_p 0.85 \\ \--log\_completions true \\ \--overlong\_filter true Megatron-SWIFT[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#megatron-swift "Link to this heading") ---------------------------------------------------------------------------------------------------------------------- ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the [Supported Models Documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) . For environment preparation and the conversion between HF and MCore model weights, you can refer to the [Megatron-SWIFT Training Documentation](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html) . These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node) . \# https://help.aliyun.com/zh/pai/user-guide/general-environment-variables \# Ensure that the weight-saving paths on the two nodes are identical. NNODES\=$WORLD\_SIZE \\ NODE\_RANK\=$RANK \\ megatron sft \\ \--load Qwen3-30B-A3B-Base-mcore \\ \--dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ \--tensor\_model\_parallel\_size 2 \\ \--expert\_model\_parallel\_size 8 \\ \--moe\_grouped\_gemm true \\ \--moe\_shared\_expert\_overlap true \\ \--moe\_aux\_loss\_coeff 0.01 \\ \--micro\_batch\_size 1 \\ \--global\_batch\_size 16 \\ \--packing true \\ \--recompute\_granularity full \\ \--recompute\_method uniform \\ \--recompute\_num\_layers 1 \\ \--train\_iters 2000 \\ \--eval\_iters 50 \\ \--finetune true \\ \--cross\_entropy\_loss\_fusion true \\ \--lr 1e-5 \\ \--lr\_warmup\_iters 100 \\ \--min\_lr 1e-6 \\ \--save megatron\_output/Qwen3-30B-A3B-Base \\ \--eval\_interval 200 \\ \--save\_interval 200 \\ \--max\_length 8192 \\ \--num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--no\_save\_optim true \\ \--no\_save\_rng true \\ \--sequence\_parallel true \\ \--use\_flash\_attn true The custom dataset format is the same as `swift sft`, which can be found in the previous section. Simply specify `--dataset `. The following is a comparison of training speed and GPU memory usage between `megatron sft` and `swift sft` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | --- | --- | --- | --- | | Training Speed | 9.6s/it | \- | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | Conclusion[¶](https://qwen.readthedocs.io/en/stable/training/ms_swift.html#conclusion "Link to this heading") -------------------------------------------------------------------------------------------------------------- The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in [this issue](https://github.com/modelscope/ms-swift/issues/4030) . --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/framework/LlamaIndex.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LlamaIndex[¶](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#llamaindex "Link to this heading") =============================================================================================================== Attention To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation[¶](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#preparation "Link to this heading") ----------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#set-parameters "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen2.5 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name\="Qwen/Qwen2.5-7B-Instruct", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#build-index "Link to this heading") ----------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/v3.0/framework/LlamaIndex.html#rag "Link to this heading") ------------------------------------------------------------------------------------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # MS-SWIFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/training/ms_swift.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar MS-SWIFT[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#ms-swift "Link to this heading") ======================================================================================================== ModelScope SWIFT (**ms-swift**) is the large model and multimodal large model training and deployment framework provided by the [ModelScope community](https://modelscope.cn/) . GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: * **Model Types**: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. * **Hardware Support**: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. * **Training Methods**: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. * **Distributed Training**: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. * **RLHF Training**: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift’s integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. pip install ms-swift \-U \# Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers \-U \# Optional packages pip install deepspeed \# multi-GPU training pip install liger-kernel \# save GPU memory resources pip install flash-attn \--no-build-isolation Supervised Fine-Tuning (SFT)[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#supervised-fine-tuning-sft "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------- ### Data Preparation[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#data-preparation "Link to this heading") The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify `--dataset ` in the training script. For complete dataset formatting guidelines, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) * General format {"messages": \[\ {"role": "system", "content": ""},\ {"role": "user", "content": ""},\ {"role": "assistant", "content": ""}\ \]} * Format with think {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} If you want to train using data without a chain of thought but retain the model’s reasoning ability, there are two approaches to minimize disruption during fine-tuning: **Option 1**: During training, specify `--loss_scale ignore_empty_think` to ignore the loss calculation for `\n\n\n\n`, preventing the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo1.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} **Option 2**: Add `/no_think` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo2.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} ### 30-Minute Self-Cognition Fine-Tuning[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#minute-self-cognition-fine-tuning "Link to this heading") This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by [ModelScope’s free compute resources](https://modelscope.cn/my/mynotebook) . After training, the model will identify itself as “swift-robot,” trained by “swift,” instead of its original self-cognition as “Qwen,” trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify `--model ` and `--dataset `. The dataset can be found on [Modelscope Hub](https://modelscope.cn/datasets/swift/self-cognition) . For the meaning of each parameter in the training script, please refer to the [Command-line parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) . CUDA\_VISIBLE\_DEVICES\=0 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type lora \\ \--dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-4 \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--gradient\_accumulation\_steps 16 \\ \--eval\_steps 50 \\ \--save\_steps 50 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--max\_length 2048 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--model\_author swift \\ \--model\_name swift-robot After fine-tuning, you can use the following script to test the fine-tuning results. Note that the `--adapters` section needs to be modified to the directory path of the last saved checkpoint: CUDA\_VISIBLE\_DEVICES\=0 \\ swift infer \\ \--adapters output/vx-xxx/checkpoint-xxx \\ \--stream true \\ \--temperature 0 \\ \--max\_new\_tokens 2048 <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify `--use_hf true`. Merge LoRA weights: swift export \\ \--adapters output/checkpoint-xxx \\ \--merge\_lora true Push the model to ModelScope/HuggingFace: \# If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. \# The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ \--adapters output/checkpoint-xxx \\ \--push\_to\_hub true \\ \--hub\_model\_id '' \\ \--hub\_token '' \\ \--use\_hf false If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \# 4 \* 60GB \# You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. \# Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE\=4 \\ CUDA\_VISIBLE\_DEVICES\=0,1,2,3 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset '' \\ \--torch\_dtype bfloat16 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-5 \\ \--gradient\_accumulation\_steps 4 \\ \--packing true \\ \--eval\_steps 100 \\ \--save\_steps 100 \\ \--logging\_steps 5 \\ \--max\_length 8192 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--save\_total\_limit 2 \\ \--save\_only\_model true \\ \--output\_dir output \\ \--deepspeed zero3 \\ \--use\_liger\_kernel true \\ \--attn\_impl flash\_attn Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#reinforcement-learning-rl "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------- ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: [Supported Features](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html) . ### Environment Setup[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#environment-setup "Link to this heading") In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: pip install "math\_verify==0.5.2" pip install vllm ### Data Preparation[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#id1 "Link to this heading") The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a `solution` column is needed to calculate the accuracy. Example Dataset Formats: {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} For dataset preparation for other RLHF algorithms, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf) . Notes on Dataset Requirements: * **Reward Function Calculation**: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: * When using the built-in accuracy or cosine similarity reward, the dataset must include a `solution` column to calculate the accuracy of the responses. * Other columns in the dataset will be passed as `**kwargs` to the reward function for additional customization. * **Customizing the Reward Function**: To adapt the reward function to your specific needs, you can refer to the following resource: [External Reward Plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin) . This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting `num_infer_workers=8`, we deploy a vLLM engine for each device to speed up the sampling process. \# 70G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset 'AI-MO/NuminaMath-TIR#5000' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 2 \\ \--per\_device\_eval\_batch\_size 2 \\ \--learning\_rate 1e-6 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--gradient\_accumulation\_steps 1 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--max\_completion\_length 4096 \\ \--vllm\_max\_model\_len 8192 \\ \--reward\_funcs accuracy \\ \--num\_generations 16 \\ \--use\_vllm true \\ \--vllm\_gpu\_memory\_utilization 0.4 \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--deepspeed zero3 \\ \--num\_infer\_workers 8 \\ \--tensor\_parallel\_size 1 \\ \--temperature 1.0 \\ \--top\_p 0.85 \\ \--log\_completions true \\ \--overlong\_filter true Megatron-SWIFT[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#megatron-swift "Link to this heading") -------------------------------------------------------------------------------------------------------------------- ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the [Supported Models Documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) . For environment preparation and the conversion between HF and MCore model weights, you can refer to the [Megatron-SWIFT Training Documentation](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html) . These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node) . \# https://help.aliyun.com/zh/pai/user-guide/general-environment-variables \# Ensure that the weight-saving paths on the two nodes are identical. NNODES\=$WORLD\_SIZE \\ NODE\_RANK\=$RANK \\ megatron sft \\ \--load Qwen3-30B-A3B-Base-mcore \\ \--dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ \--tensor\_model\_parallel\_size 2 \\ \--expert\_model\_parallel\_size 8 \\ \--moe\_grouped\_gemm true \\ \--moe\_shared\_expert\_overlap true \\ \--moe\_aux\_loss\_coeff 0.01 \\ \--micro\_batch\_size 1 \\ \--global\_batch\_size 16 \\ \--packing true \\ \--recompute\_granularity full \\ \--recompute\_method uniform \\ \--recompute\_num\_layers 1 \\ \--train\_iters 2000 \\ \--eval\_iters 50 \\ \--finetune true \\ \--cross\_entropy\_loss\_fusion true \\ \--lr 1e-5 \\ \--lr\_warmup\_iters 100 \\ \--min\_lr 1e-6 \\ \--save megatron\_output/Qwen3-30B-A3B-Base \\ \--eval\_interval 200 \\ \--save\_interval 200 \\ \--max\_length 8192 \\ \--num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--no\_save\_optim true \\ \--no\_save\_rng true \\ \--sequence\_parallel true \\ \--use\_flash\_attn true The custom dataset format is the same as `swift sft`, which can be found in the previous section. Simply specify `--dataset `. The following is a comparison of training speed and GPU memory usage between `megatron sft` and `swift sft` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | --- | --- | --- | --- | | Training Speed | 9.6s/it | \- | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | Conclusion[¶](https://qwen.readthedocs.io/en/v3.0/training/ms_swift.html#conclusion "Link to this heading") ------------------------------------------------------------------------------------------------------------ The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in [this issue](https://github.com/modelscope/ms-swift/issues/4030) . --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/framework/LlamaIndex.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar LlamaIndex[¶](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#llamaindex "Link to this heading") ================================================================================================================= Attention To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation[¶](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#preparation "Link to this heading") ------------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#set-parameters "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen2.5 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name\="Qwen/Qwen2.5-7B-Instruct", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#build-index "Link to this heading") ------------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/stable/framework/LlamaIndex.html#rag "Link to this heading") --------------------------------------------------------------------------------------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/framework/Langchain.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Langchain[¶](https://qwen.readthedocs.io/en/v2.5/framework/Langchain.html#langchain "Link to this heading") ============================================================================================================ This guide helps you build a question-answering application based on a local knowledge base using `Qwen2.5-7B-Instruct` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/v2.5/framework/Langchain.html#basic-usage "Link to this heading") ---------------------------------------------------------------------------------------------------------------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name \= "Qwen/Qwen2.5-7B-Instruct" model \= AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(model.device) generated\_ids \= model.generate( \*\*model\_inputs, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language model." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/v2.5/framework/Langchain.html#next-step "Link to this heading") ------------------------------------------------------------------------------------------------------------ Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Qwen2.5 Speed Benchmark - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/benchmark/speed_benchmark.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/benchmark/speed_benchmark.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/benchmark/speed_benchmark.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Qwen2.5 Speed Benchmark[¶](https://qwen.readthedocs.io/en/v2.5/benchmark/speed_benchmark.html#qwen2-5-speed-benchmark "Link to this heading") ============================================================================================================================================== This section reports the speed performance of bf16 models, quantized models (including GPTQ-Int4, GPTQ-Int8 and AWQ) of the Qwen2.5 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under the conditions of different context lengths. The environment of the evaluation with huggingface transformers is: * NVIDIA A100 80GB * CUDA 12.1 * Pytorch 2.3.1 * Flash Attention 2.5.8 * Transformers 4.46.0 * AutoGPTQ 0.7.1+cu121 (Compiled from source code) * AutoAWQ 0.2.6 The environment of the evaluation with vLLM is: * NVIDIA A100 80GB * CUDA 12.1 * vLLM 0.6.3 * Pytorch 2.4.0 * Flash Attention 2.6.3 * Transformers 4.46.0 Notes: * We use the batch size of 1 and the least number of GPUs as possible for the evaluation. * We test the speed and memory of generating 2048 tokens with the input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens. * For vLLM, the memory usage is not reported because it pre-allocates all GPU memory. We use `gpu_memory_utilization=0.9 max_model_len=32768 enforce_eager=False` by default. * 0.5B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-0.5B-Instruct | 1 | BF16 | 1 | 47.40 | 0.97 | | | GPTQ-Int8 | 1 | 35.17 | 0.64 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 50.60 | 0.48 | | | AWQ | 1 | 37.09 | 0.68 | | | 6144 | BF16 | 1 | 47.45 | 1.23 | | | GPTQ-Int8 | 1 | 36.47 | 0.90 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 48.89 | 0.73 | | | AWQ | 1 | 37.04 | 0.72 | | | 14336 | BF16 | 1 | 47.11 | 1.60 | | | GPTQ-Int8 | 1 | 35.44 | 1.26 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 48.26 | 1.10 | | | AWQ | 1 | 37.14 | 1.10 | | | 30720 | BF16 | 1 | 47.16 | 2.34 | | | GPTQ-Int8 | 1 | 36.25 | 2.01 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 49.22 | 1.85 | | | AWQ | 1 | 36.90 | 1.84 | | * 0.5B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2.5-0.5B-Instruct | 1 | BF16 | 1 | 311.55 | | GPTQ-Int8 | 1 | 257.07 | | GPTQ-Int4 | 1 | 260.93 | | AWQ | 1 | 261.95 | | 6144 | BF16 | 1 | 304.79 | | GPTQ-Int8 | 1 | 254.10 | | GPTQ-Int4 | 1 | 257.33 | | AWQ | 1 | 259.80 | | 14336 | BF16 | 1 | 290.28 | | GPTQ-Int8 | 1 | 243.69 | | GPTQ-Int4 | 1 | 247.01 | | AWQ | 1 | 249.58 | | 30720 | BF16 | 1 | 264.51 | | GPTQ-Int8 | 1 | 223.86 | | GPTQ-Int4 | 1 | 226.50 | | AWQ | 1 | 229.84 | * 1.5B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-1.5B-Instruct | 1 | BF16 | 1 | 39.68 | 2.95 | | | GPTQ-Int8 | 1 | 32.62 | 1.82 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 43.33 | 1.18 | | | AWQ | 1 | 31.70 | 1.51 | | | 6144 | BF16 | 1 | 40.88 | 3.43 | | | GPTQ-Int8 | 1 | 31.46 | 2.30 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 43.96 | 1.66 | | | AWQ | 1 | 32.30 | 1.63 | | | 14336 | BF16 | 1 | 40.43 | 4.16 | | | GPTQ-Int8 | 1 | 31.06 | 3.03 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 43.66 | 2.39 | | | AWQ | 1 | 32.39 | 2.36 | | | 30720 | BF16 | 1 | 38.59 | 5.62 | | | GPTQ-Int8 | 1 | 31.04 | 4.49 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 35.68 | 3.85 | | | AWQ | 1 | 31.95 | 3.82 | | * 1.5B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2.5-1.5B-Instruct | 1 | BF16 | 1 | 183.33 | | GPTQ-Int8 | 1 | 201.67 | | GPTQ-Int4 | 1 | 217.03 | | AWQ | 1 | 213.74 | | 6144 | BF16 | 1 | 176.68 | | GPTQ-Int8 | 1 | 192.83 | | GPTQ-Int4 | 1 | 206.63 | | AWQ | 1 | 203.64 | | 14336 | BF16 | 1 | 168.69 | | GPTQ-Int8 | 1 | 183.69 | | GPTQ-Int4 | 1 | 195.88 | | AWQ | 1 | 192.64 | | 30720 | BF16 | 1 | 152.04 | | GPTQ-Int8 | 1 | 162.82 | | GPTQ-Int4 | 1 | 173.57 | | AWQ | 1 | 170.20 | * 3B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-3B-Instruct | 1 | BF16 | 1 | 30.80 | 5.95 | | | GPTQ-Int8 | 1 | 25.69 | 3.38 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 35.21 | 2.06 | | | AWQ | 1 | 25.29 | 2.50 | | | 6144 | BF16 | 1 | 32.20 | 6.59 | | | GPTQ-Int8 | 1 | 24.69 | 3.98 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 34.47 | 2.67 | | | AWQ | 1 | 24.86 | 2.62 | | | 14336 | BF16 | 1 | 31.72 | 7.47 | | | GPTQ-Int8 | 1 | 24.70 | 4.89 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 34.36 | 3.58 | | | AWQ | 1 | 25.19 | 3.54 | | | 30720 | BF16 | 1 | 25.37 | 9.30 | | | GPTQ-Int8 | 1 | 21.67 | 6.72 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 23.60 | 5.41 | | | AWQ | 1 | 24.56 | 5.37 | | * 3B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | | --- | --- | --- | --- | --- | | Qwen2.5-3B-Instruct | 1 | BF16 | 1 | 127.61 | | GPTQ-Int8 | 1 | 150.02 | | GPTQ-Int4 | 1 | 168.20 | | AWQ | 1 | 165.50 | | 6144 | BF16 | 1 | 123.15 | | GPTQ-Int8 | 1 | 143.09 | | GPTQ-Int4 | 1 | 159.85 | | AWQ | 1 | 156.38 | | 14336 | BF16 | 1 | 117.35 | | GPTQ-Int8 | 1 | 135.50 | | GPTQ-Int4 | 1 | 149.35 | | AWQ | 1 | 147.75 | | 30720 | BF16 | 1 | 105.88 | | GPTQ-Int8 | 1 | 118.38 | | GPTQ-Int4 | 1 | 129.28 | | AWQ | 1 | 127.19 | * 7B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-7B-Instruct | 1 | BF16 | 1 | 40.38 | 14.38 | | | GPTQ-Int8 | 1 | 31.55 | 8.42 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 43.10 | 5.52 | | | AWQ | 1 | 32.03 | 5.39 | | | 6144 | BF16 | 1 | 38.76 | 15.38 | | | GPTQ-Int8 | 1 | 31.26 | 9.43 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 38.27 | 6.52 | | | AWQ | 1 | 32.37 | 6.39 | | | 14336 | BF16 | 1 | 29.78 | 16.91 | | | GPTQ-Int8 | 1 | 26.86 | 10.96 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 28.70 | 8.05 | | | AWQ | 1 | 30.23 | 7.92 | | | 30720 | BF16 | 1 | 18.83 | 19.97 | | | GPTQ-Int8 | 1 | 17.59 | 14.01 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 18.45 | 11.11 | | | AWQ | 1 | 19.11 | 10.98 | | * 7B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | | --- | --- | --- | --- | --- | --- | | Qwen2.5-7B-Instruct | 1 | BF16 | 1 | 84.28 | | | GPTQ-Int8 | 1 | 122.01 | | | GPTQ-Int4 | 1 | 154.05 | | | AWQ | 1 | 148.10 | | | 6144 | BF16 | 1 | 80.70 | | | GPTQ-Int8 | 1 | 112.38 | | | GPTQ-Int4 | 1 | 141.98 | | | AWQ | 1 | 137.64 | | | 14336 | BF16 | 1 | 77.69 | | | GPTQ-Int8 | 1 | 105.25 | | | GPTQ-Int4 | 1 | 129.35 | | | AWQ | 1 | 124.91 | | | 30720 | BF16 | 1 | 70.33 | | | GPTQ-Int8 | 1 | 90.71 | | | GPTQ-Int4 | 1 | 108.30 | | | AWQ | 1 | 104.66 | | | 63488 | BF16 | 1 | 50.86 | setting-64k | | GPTQ-Int8 | 1 | 60.52 | setting-64k | | GPTQ-Int4 | 1 | 67.97 | setting-64k | | AWQ | 1 | 66.42 | setting-64k | | 129024 | BF16 | 1 | 28.94 | vllm==0.6.2, new sample config | | GPTQ-Int8 | 1 | 25.97 | vllm==0.6.2, new sample config | | GPTQ-Int4 | 1 | 26.37 | vllm==0.6.2, new sample config | | AWQ | 1 | 26.57 | vllm==0.6.2, new sample config | * \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) * \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) * 14B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-14B-Instruct | 1 | BF16 | 1 | 24.74 | 28.08 | | | GPTQ-Int8 | 1 | 18.84 | 16.11 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 25.89 | 9.94 | | | AWQ | 1 | 19.23 | 9.79 | | | 6144 | BF16 | 1 | 20.51 | 29.50 | | | GPTQ-Int8 | 1 | 17.80 | 17.61 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 20.06 | 11.36 | | | AWQ | 1 | 19.21 | 11.22 | | | 14336 | BF16 | 1 | 13.92 | 31.95 | | | GPTQ-Int8 | 1 | 12.66 | 19.98 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 13.79 | 13.81 | | | AWQ | 1 | 14.17 | 13.67 | | | 30720 | BF16 | 1 | 8.20 | 36.85 | | | GPTQ-Int8 | 1 | 7.77 | 24.88 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 8.14 | 18.71 | | | AWQ | 1 | 8.31 | 18.57 | | * 14B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | | --- | --- | --- | --- | --- | --- | | Qwen2.5-14B-Instruct | 1 | BF16 | 1 | 46.30 | | | GPTQ-Int8 | 1 | 70.40 | | | GPTQ-Int4 | 1 | 98.02 | | | AWQ | 1 | 92.66 | | | 6144 | BF16 | 1 | 43.83 | | | GPTQ-Int8 | 1 | 64.33 | | | GPTQ-Int4 | 1 | 86.10 | | | AWQ | 1 | 83.11 | | | 14336 | BF16 | 1 | 41.91 | | | GPTQ-Int8 | 1 | 59.21 | | | GPTQ-Int4 | 1 | 76.85 | | | AWQ | 1 | 74.03 | | | 30720 | BF16 | 1 | 37.18 | | | GPTQ-Int8 | 1 | 49.23 | | | GPTQ-Int4 | 1 | 60.91 | | | AWQ | 1 | 59.01 | | | 63488 | BF16 | 1 | 26.85 | setting-64k | | GPTQ-Int8 | 1 | 32.83 | setting-64k | | GPTQ-Int4 | 1 | 37.67 | setting-64k | | AWQ | 1 | 36.71 | setting-64k | | 129024 | BF16 | 1 | 14.53 | vllm==0.6.2, new sample config | | GPTQ-Int8 | 1 | 15.10 | vllm==0.6.2, new sample config | | GPTQ-Int4 | 1 | 15.13 | vllm==0.6.2, new sample config | | AWQ | 1 | 15.25 | vllm==0.6.2, new sample config | * \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) * \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) * 32B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-32B-Instruct | 1 | BF16 | 1 | 17.54 | 61.58 | | | GPTQ-Int8 | 1 | 14.52 | 33.56 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 19.20 | 18.94 | | | AWQ | 1 | 14.60 | 18.67 | | | 6144 | BF16 | 1 | 12.49 | 63.72 | | | GPTQ-Int8 | 1 | 11.61 | 35.86 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 13.42 | 21.09 | | | AWQ | 1 | 13.81 | 20.81 | | | 14336 | BF16 | 1 | 8.95 | 67.31 | | | GPTQ-Int8 | 1 | 8.53 | 39.28 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 9.48 | 24.67 | | | AWQ | 1 | 9.71 | 24.39 | | | 30720 | BF16 | 1 | 5.59 | 74.47 | | | GPTQ-Int8 | 1 | 5.42 | 46.45 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 5.79 | 31.84 | | | AWQ | 1 | 5.85 | 31.56 | | * 32B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | | --- | --- | --- | --- | --- | --- | | Qwen2.5-32B-Instruct | 1 | BF16 | 1 | 22.13 | setting1 | | GPTQ-Int8 | 1 | 37.57 | | | GPTQ-Int4 | 1 | 55.83 | | | AWQ | 1 | 51.92 | | | 6144 | BF16 | 1 | 21.05 | setting1 | | GPTQ-Int8 | 1 | 34.67 | | | GPTQ-Int4 | 1 | 49.96 | | | AWQ | 1 | 46.68 | | | 14336 | BF16 | 1 | 19.91 | setting1 | | GPTQ-Int8 | 1 | 31.89 | | | GPTQ-Int4 | 1 | 44.79 | | | AWQ | 1 | 41.83 | | | 30720 | BF16 | 2 | 31.82 | | | GPTQ-Int8 | 1 | 26.88 | | | GPTQ-Int4 | 1 | 35.66 | | | AWQ | 1 | 33.75 | | | 63488 | BF16 | 2 | 24.45 | setting-64k | | GPTQ-Int8 | 1 | 18.60 | setting-64k | | GPTQ-Int4 | 1 | 22.72 | setting-64k | | AWQ | 1 | 21.79 | setting-64k | | 129024 | BF16 | 2 | 14.31 | vllm==0.6.2, new sample config | | GPTQ-Int8 | 1 | 9.77 | vllm==0.6.2, new sample config | | GPTQ-Int4 | 1 | 10.39 | vllm==0.6.2, new sample config | | AWQ | 1 | 10.34 | vllm==0.6.2, new sample config | > * For context length 129024, the model needs to be predicted with the following config: “model\_max\_length”=131072 > > * \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) > > * \[Setting 1\]=(gpu\_memory\_utilization=1.0 max\_model\_len=32768 enforce\_eager=True) > > * \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) > > * \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) > * 72B (Transformer) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-72B-Instruct | 1 | BF16 | 2 | 8.73 | 136.20 | | | GPTQ-Int8 | 2 | 8.66 | 72.61 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 11.07 | 39.91 | | | AWQ | 1 | 11.50 | 39.44 | | | 6144 | BF16 | 2 | 6.39 | 140.00 | | | GPTQ-Int8 | 2 | 6.39 | 77.81 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 7.56 | 42.50 | | | AWQ | 1 | 8.17 | 42.13 | | | 14336 | BF16 | 3 | 4.25 | 149.14 | | | GPTQ-Int8 | 2 | 4.66 | 82.55 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 1 | 5.27 | 46.86 | | | AWQ | 1 | 5.57 | 46.38 | | | 30720 | BF16 | 3 | 2.94 | 164.79 | | | GPTQ-Int8 | 2 | 2.94 | 94.75 | auto\_gptq==0.6.0+cu1210 | | GPTQ-Int4 | 2 | 3.14 | 62.57 | | | AWQ | 2 | 3.23 | 61.64 | | * 72B (vLLM) | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | | --- | --- | --- | --- | --- | --- | | Qwen2.5-72B-Instruct | 1 | BF16 | 2 | 18.19 | Setting 1 | | | BF16 | 4 | 31.37 | Default | | | GPTQ-Int8 | 2 | 31.40 | Default | | | GPTQ-Int4 | 1 | 16.47 | Default | | | GPTQ-Int4 | 2 | 46.30 | Setting 2 | | | AWQ | 2 | 44.30 | Default | | 6144 | BF16 | 4 | 29.90 | Default | | | GPTQ-Int8 | 2 | 29.37 | Default | | | GPTQ-Int4 | 1 | 13.88 | Default | | | GPTQ-Int4 | 2 | 42.50 | Setting 3 | | | AWQ | 2 | 40.67 | Default | | 14336 | BF16 | 4 | 30.10 | Default | | | GPTQ-Int8 | 2 | 27.20 | Default | | | GPTQ-Int4 | 2 | 38.10 | Default | | | AWQ | 2 | 36.63 | Default | | 30720 | BF16 | 4 | 27.53 | Default | | | GPTQ-Int8 | 2 | 23.32 | Default | | | GPTQ-Int4 | 2 | 30.98 | Default | | | AWQ | 2 | 30.02 | Default | | 63488 | BF16 | 4 | 20.74 | Setting 4 | | | GPTQ-Int8 | 2 | 16.27 | Setting 4 | | | GPTQ-Int4 | 2 | 19.84 | Setting 4 | | | AWQ | 2 | 19.32 | Setting 4 | | 129024 | BF16 | 4 | 12.68 | Setting 5 | | | GPTQ-Int8 | 4 | 14.11 | Setting 5 | | | GPTQ-Int4 | 2 | 10.11 | Setting 5 | | | AWQ | 2 | 9.88 | Setting 5 | > * \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) > > * \[Setting 1\]=(gpu\_memory\_utilization=0.98 max\_model\_len=4096 enforce\_eager=True) > > * \[Setting 2\]=(gpu\_memory\_utilization=1.0 max\_model\_len=4096 enforce\_eager=True) > > * \[Setting 3\]=(gpu\_memory\_utilization=1.0 max\_model\_len=8192 enforce\_eager=True) > > * \[Setting 4\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) > > * \[Setting 5\]=(gpu\_memory\_utilization=0.9 max\_model\_len=131072 enforce\_eager=False) > --- # LlamaIndex - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/framework/LlamaIndex.rst.txt "View this page") LlamaIndex[¶](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#llamaindex "Link to this heading") ================================================================================================================= Attention To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on [LlamaIndex](https://www.llamaindex.ai/) . This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation[¶](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#preparation "Link to this heading") ------------------------------------------------------------------------------------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters[¶](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#set-parameters "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the `bge-base-en-v1.5` model to retrieve from English documents, and you can download the `bge-base-zh-v1.5` model to retrieve from Chinese documents. You can also choose `bge-large` or `bge-small` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding \# Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt \= "" for message in messages: if message.role \== "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role \== "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role \== "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt \= "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt \= prompt + "<|im\_start|>assistant\\n" return prompt \# Set Qwen2.5 as the language model and set generation config Settings.llm \= HuggingFaceLLM( model\_name\="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name\="Qwen/Qwen2.5-7B-Instruct", context\_window\=30000, max\_new\_tokens\=2000, generate\_kwargs\={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt\=messages\_to\_prompt, completion\_to\_prompt\=completion\_to\_prompt, device\_map\="auto", ) \# Set embedding model Settings.embed\_model \= HuggingFaceEmbedding( model\_name \= "BAAI/bge-base-en-v1.5" ) \# Set the size of the text chunk for retrieval Settings.transformations \= \[SentenceSplitter(chunk\_size\=1024)\] Build Index[¶](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#build-index "Link to this heading") ------------------------------------------------------------------------------------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named ‘document’. from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleDirectoryReader("./document").load\_data() index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents \= SimpleWebPageReader(html\_to\_text\=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index \= VectorStoreIndex.from\_documents( documents, embed\_model\=Settings.embed\_model, transformations\=Settings.transformations ) To save and load the index, you can use the following code snippet. from llama\_index.core import StorageContext, load\_index\_from\_storage \# save index storage\_context \= StorageContext.from\_defaults(persist\_dir\="save") \# load index index \= load\_index\_from\_storage(storage\_context) RAG[¶](https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html#rag "Link to this heading") --------------------------------------------------------------------------------------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. query\_engine \= index.as\_query\_engine() your\_query \= "" print(query\_engine.query(your\_query).response) --- # MS-SWIFT - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/training/ms_swift.md.txt "View this page") MS-SWIFT[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#ms-swift "Link to this heading") ========================================================================================================== ModelScope SWIFT (**ms-swift**) is the large model and multimodal large model training and deployment framework provided by the [ModelScope community](https://modelscope.cn/) . GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: * **Model Types**: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. * **Hardware Support**: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. * **Training Methods**: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. * **Distributed Training**: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. * **RLHF Training**: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift’s integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. pip install ms-swift \-U \# Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers \-U \# Optional packages pip install deepspeed \# multi-GPU training pip install liger-kernel \# save GPU memory resources pip install flash-attn \--no-build-isolation Supervised Fine-Tuning (SFT)[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#supervised-fine-tuning-sft "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ ### Data Preparation[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#data-preparation "Link to this heading") The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify `--dataset ` in the training script. For complete dataset formatting guidelines, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) * General format {"messages": \[\ {"role": "system", "content": ""},\ {"role": "user", "content": ""},\ {"role": "assistant", "content": ""}\ \]} * Format with think {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} If you want to train using data without a chain of thought but retain the model’s reasoning ability, there are two approaches to minimize disruption during fine-tuning: **Option 1**: During training, specify `--loss_scale ignore_empty_think` to ignore the loss calculation for `\n\n\n\n`, preventing the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo1.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang?"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} **Option 2**: Add `/no_think` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo2.sh) . The custom dataset format is as follows: {"messages": \[\ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"},\ {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."}\ \]} ### 30-Minute Self-Cognition Fine-Tuning[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#minute-self-cognition-fine-tuning "Link to this heading") This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by [ModelScope’s free compute resources](https://modelscope.cn/my/mynotebook) . After training, the model will identify itself as “swift-robot,” trained by “swift,” instead of its original self-cognition as “Qwen,” trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify `--model ` and `--dataset `. The dataset can be found on [Modelscope Hub](https://modelscope.cn/datasets/swift/self-cognition) . For the meaning of each parameter in the training script, please refer to the [Command-line parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) . CUDA\_VISIBLE\_DEVICES\=0 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type lora \\ \--dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-4 \\ \--lora\_rank 8 \\ \--lora\_alpha 32 \\ \--target\_modules all-linear \\ \--gradient\_accumulation\_steps 16 \\ \--eval\_steps 50 \\ \--save\_steps 50 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--max\_length 2048 \\ \--output\_dir output \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--model\_author swift \\ \--model\_name swift-robot After fine-tuning, you can use the following script to test the fine-tuning results. Note that the `--adapters` section needs to be modified to the directory path of the last saved checkpoint: CUDA\_VISIBLE\_DEVICES\=0 \\ swift infer \\ \--adapters output/vx-xxx/checkpoint-xxx \\ \--stream true \\ \--temperature 0 \\ \--max\_new\_tokens 2048 <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify `--use_hf true`. Merge LoRA weights: swift export \\ \--adapters output/checkpoint-xxx \\ \--merge\_lora true Push the model to ModelScope/HuggingFace: \# If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. \# The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ \--adapters output/checkpoint-xxx \\ \--push\_to\_hub true \\ \--hub\_model\_id '' \\ \--hub\_token '' \\ \--use\_hf false If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \# 4 \* 60GB \# You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. \# Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE\=4 \\ CUDA\_VISIBLE\_DEVICES\=0,1,2,3 \\ swift sft \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset '' \\ \--torch\_dtype bfloat16 \\ \--per\_device\_train\_batch\_size 1 \\ \--per\_device\_eval\_batch\_size 1 \\ \--learning\_rate 1e-5 \\ \--gradient\_accumulation\_steps 4 \\ \--packing true \\ \--eval\_steps 100 \\ \--save\_steps 100 \\ \--logging\_steps 5 \\ \--max\_length 8192 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--save\_total\_limit 2 \\ \--save\_only\_model true \\ \--output\_dir output \\ \--deepspeed zero3 \\ \--use\_liger\_kernel true \\ \--attn\_impl flash\_attn Reinforcement Learning (RL)[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#reinforcement-learning-rl "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------- ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: [Supported Features](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html) . ### Environment Setup[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#environment-setup "Link to this heading") In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: pip install "math\_verify==0.5.2" pip install vllm ### Data Preparation[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#id1 "Link to this heading") The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a `solution` column is needed to calculate the accuracy. Example Dataset Formats: {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} For dataset preparation for other RLHF algorithms, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf) . Notes on Dataset Requirements: * **Reward Function Calculation**: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: * When using the built-in accuracy or cosine similarity reward, the dataset must include a `solution` column to calculate the accuracy of the responses. * Other columns in the dataset will be passed as `**kwargs` to the reward function for additional customization. * **Customizing the Reward Function**: To adapt the reward function to your specific needs, you can refer to the following resource: [External Reward Plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin) . This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting `num_infer_workers=8`, we deploy a vLLM engine for each device to speed up the sampling process. \# 70G\*8 CUDA\_VISIBLE\_DEVICES\=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE\=8 \\ swift rlhf \\ \--rlhf\_type grpo \\ \--model Qwen/Qwen3-8B \\ \--train\_type full \\ \--dataset 'AI-MO/NuminaMath-TIR#5000' \\ \--torch\_dtype bfloat16 \\ \--num\_train\_epochs 1 \\ \--per\_device\_train\_batch\_size 2 \\ \--per\_device\_eval\_batch\_size 2 \\ \--learning\_rate 1e-6 \\ \--save\_total\_limit 2 \\ \--logging\_steps 5 \\ \--output\_dir output \\ \--gradient\_accumulation\_steps 1 \\ \--warmup\_ratio 0.05 \\ \--dataloader\_num\_workers 4 \\ \--max\_completion\_length 4096 \\ \--vllm\_max\_model\_len 8192 \\ \--reward\_funcs accuracy \\ \--num\_generations 16 \\ \--use\_vllm true \\ \--vllm\_gpu\_memory\_utilization 0.4 \\ \--sleep\_level 1 \\ \--offload\_model true \\ \--offload\_optimizer true \\ \--gc\_collect\_after\_offload true \\ \--deepspeed zero3 \\ \--num\_infer\_workers 8 \\ \--tensor\_parallel\_size 1 \\ \--temperature 1.0 \\ \--top\_p 0.85 \\ \--log\_completions true \\ \--overlong\_filter true Megatron-SWIFT[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#megatron-swift "Link to this heading") ---------------------------------------------------------------------------------------------------------------------- ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the [Supported Models Documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) . For environment preparation and the conversion between HF and MCore model weights, you can refer to the [Megatron-SWIFT Training Documentation](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html) . These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node) . \# https://help.aliyun.com/zh/pai/user-guide/general-environment-variables \# Ensure that the weight-saving paths on the two nodes are identical. NNODES\=$WORLD\_SIZE \\ NODE\_RANK\=$RANK \\ megatron sft \\ \--load Qwen3-30B-A3B-Base-mcore \\ \--dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ \--tensor\_model\_parallel\_size 2 \\ \--expert\_model\_parallel\_size 8 \\ \--moe\_grouped\_gemm true \\ \--moe\_shared\_expert\_overlap true \\ \--moe\_aux\_loss\_coeff 0.01 \\ \--micro\_batch\_size 1 \\ \--global\_batch\_size 16 \\ \--packing true \\ \--recompute\_granularity full \\ \--recompute\_method uniform \\ \--recompute\_num\_layers 1 \\ \--train\_iters 2000 \\ \--eval\_iters 50 \\ \--finetune true \\ \--cross\_entropy\_loss\_fusion true \\ \--lr 1e-5 \\ \--lr\_warmup\_iters 100 \\ \--min\_lr 1e-6 \\ \--save megatron\_output/Qwen3-30B-A3B-Base \\ \--eval\_interval 200 \\ \--save\_interval 200 \\ \--max\_length 8192 \\ \--num\_workers 8 \\ \--dataset\_num\_proc 8 \\ \--no\_save\_optim true \\ \--no\_save\_rng true \\ \--sequence\_parallel true \\ \--use\_flash\_attn true The custom dataset format is the same as `swift sft`, which can be found in the previous section. Simply specify `--dataset `. The following is a comparison of training speed and GPU memory usage between `megatron sft` and `swift sft` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | --- | --- | --- | --- | | Training Speed | 9.6s/it | \- | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | Conclusion[¶](https://qwen.readthedocs.io/en/latest/training/ms_swift.html#conclusion "Link to this heading") -------------------------------------------------------------------------------------------------------------- The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in [this issue](https://github.com/modelscope/ms-swift/issues/4030) . --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/framework/Langchain.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Langchain[¶](https://qwen.readthedocs.io/en/v3.0/framework/Langchain.html#langchain "Link to this heading") ============================================================================================================ Attention To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using `Qwen2.5-7B-Instruct` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/v3.0/framework/Langchain.html#basic-usage "Link to this heading") ---------------------------------------------------------------------------------------------------------------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name \= "Qwen/Qwen2.5-7B-Instruct" model \= AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(model.device) generated\_ids \= model.generate( \*\*model\_inputs, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language models." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/v3.0/framework/Langchain.html#next-step "Link to this heading") ------------------------------------------------------------------------------------------------------------ Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/framework/Langchain.rst.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Langchain[¶](https://qwen.readthedocs.io/en/stable/framework/Langchain.html#langchain "Link to this heading") ============================================================================================================== Attention To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using `Qwen2.5-7B-Instruct` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/stable/framework/Langchain.html#basic-usage "Link to this heading") ------------------------------------------------------------------------------------------------------------------ The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name \= "Qwen/Qwen2.5-7B-Instruct" model \= AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(model.device) generated\_ids \= model.generate( \*\*model\_inputs, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language models." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/stable/framework/Langchain.html#next-step "Link to this heading") -------------------------------------------------------------------------------------------------------------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Langchain - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/framework/Langchain.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/framework/Langchain.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/framework/Langchain.rst.txt "View this page") Langchain[¶](https://qwen.readthedocs.io/en/latest/framework/Langchain.html#langchain "Link to this heading") ============================================================================================================== Attention To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using `Qwen2.5-7B-Instruct` with `langchain`. The goal is to establish a knowledge base Q&A solution. Basic Usage[¶](https://qwen.readthedocs.io/en/latest/framework/Langchain.html#basic-usage "Link to this heading") ------------------------------------------------------------------------------------------------------------------ The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: pip install langchain\==0.0.174 pip install faiss-gpu from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name \= "Qwen/Qwen2.5-7B-Instruct" model \= AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype\="auto", device\_map\="auto" ) tokenizer \= AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int \= 10000 temperature: float \= 0.01 top\_p \= 0.9 history\_len: int \= 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) \-> str: return "Qwen" @property def \_history\_len(self) \-> int: return self.history\_len def set\_history\_len(self, history\_len: int \= 10) \-> None: self.history\_len \= history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] \= None, run\_manager: Optional\[CallbackManagerForLLMRun\] \= None, ) \-> str: messages \= \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},\ {"role": "user", "content": prompt}\ \] text \= tokenizer.apply\_chat\_template( messages, tokenize\=False, add\_generation\_prompt\=True ) model\_inputs \= tokenizer(\[text\], return\_tensors\="pt").to(model.device) generated\_ids \= model.generate( \*\*model\_inputs, max\_new\_tokens\=512 ) generated\_ids \= \[\ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)\ \] response \= tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens\=True)\[0\] return response @property def \_identifying\_params(self) \-> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool \= False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf \= pdf def split\_text(self, text: str) \-> List\[str\]: if self.pdf: text \= re.sub(r"\\n{3,}", "\\n", text) text \= re.sub('\\s', ' ', text) text \= text.replace("\\n\\n", "") sent\_sep\_pattern \= re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list \= \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[\-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader \= TextLoader(filepath, autodetect\_encoding\=True) textsplitter \= ChineseTextSplitter(pdf\=False) docs \= loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path \= os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp \= os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding\='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) \-> List\[List\[int\]\]: lists \= \[\] ls1 \= \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i \- 1\] + 1 \== ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 \= \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size \= 250 chunk\_conent \= True score\_threshold \= 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int \= 4 ) \-> List\[Tuple\[Document, float\]\]: scores, indices \= self.index.search(np.array(\[embedding\], dtype\=np.float32), k) docs \= \[\] id\_set \= set() store\_len \= len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i \== \-1 or 0 < self.score\_threshold < scores\[0\]\[j\]: \# This happens when not enough docs are returned. continue \_id \= self.index\_to\_docstore\_id\[i\] doc \= self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] \= int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len \= len(doc.page\_content) for k in range(1, max(i, store\_len \- i)): break\_flag \= False for l in \[i + k, i \- k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 \= self.index\_to\_docstore\_id\[l\] doc0 \= self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) \> self.chunk\_size: break\_flag \= True break elif doc0.metadata\["source"\] \== doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) \== 0 and self.score\_threshold \> 0: return \[\] id\_list \= sorted(list(id\_set)) id\_lists \= separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id \== id\_seq\[0\]: \_id \= self.index\_to\_docstore\_id\[id\] doc \= self.docstore.search(\_id) else: \_id0 \= self.index\_to\_docstore\_id\[id\] doc0 \= self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score \= min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] \= int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ \== '\_\_main\_\_': \# load docs (pdf file or txt file) filepath \= 'your file path' \# Embedding model name EMBEDDING\_MODEL \= 'text2vec' PROMPT\_TEMPLATE \= """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" \# Embedding running device EMBEDDING\_DEVICE \= "cuda" \# return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K \= 3 CHAIN\_TYPE \= 'stuff' embedding\_model\_dict \= { "text2vec": "your text2vec model path", } llm \= Qwen() embeddings \= HuggingFaceEmbeddings(model\_name\=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs\={'device': EMBEDDING\_DEVICE}) docs \= load\_file(filepath) docsearch \= FAISSWrapper.from\_documents(docs, embeddings) prompt \= PromptTemplate( template\=PROMPT\_TEMPLATE, input\_variables\=\["context\_str", "question"\] ) chain\_type\_kwargs \= {"prompt": prompt, "document\_variable\_name": "context\_str"} qa \= RetrievalQA.from\_chain\_type( llm\=llm, chain\_type\=CHAIN\_TYPE, retriever\=docsearch.as\_retriever(search\_kwargs\={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs\=chain\_type\_kwargs) query \= "Give me a short introduction to large language models." print(qa.run(query)) Next Step[¶](https://qwen.readthedocs.io/en/latest/framework/Langchain.html#next-step "Link to this heading") -------------------------------------------------------------------------------------------------------------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Function Calling - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#) [View this page](https://qwen.readthedocs.io/en/v3.0/_sources/framework/function_call.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Function Calling[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#function-calling "Link to this heading") ============================================================================================================================== Preface[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#preface "Link to this heading") ------------------------------------------------------------------------------------------------------------ Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: * either for AI-native applications that strive to work around the shortcomings of current AI technology, * or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, * We will first demonstrate how to use function calling with Qwen3. * Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is … What is function calling?[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#what-is-function-calling "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------- Note There is another term “tool use” that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. * On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. * On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. Inference with Function Calling[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#inference-with-function-calling "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------ As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with * **Qwen-Agent**, * **vLLM**. ### The Example Case[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#the-example-case "Link to this heading") Let’s also use an example to demonstrate the inference usage. We assume **Python 3.11** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: Preparation Code import json def get\_current\_temperature(location: str, unit: str \= "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str \= "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name \== "get\_current\_temperature": return get\_current\_temperature if name \== "get\_temperature\_date": return get\_temperature\_date TOOLS \= \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location"\],\ },\ },\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "date": {\ "type": "string",\ "description": 'The date to get the temperature for, in the format "Year-Month-Day".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location", "date"\],\ },\ },\ },\ \] MESSAGES \= \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ \] In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: Example Tools The tools should be described using the following JSON: \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location"\ \]\ }\ }\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "date": {\ "type": "string",\ "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location",\ "date"\ \]\ }\ }\ }\ \] For each **tool**, it is a JSON object with two fields: * `type`: a string specifying the type of the tool, currently only `"function"` is valid * `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: * `name`: a string indicating the name of the function * `description`: a string describing what the function is used for * `parameters`: [a JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.`. \[ \ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}\ \] ### Qwen-Agent[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#qwen-agent "Link to this heading") [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is _not recommended_ to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let’s make sure the latest library is installed: pip install \-U qwen-agent #### Preparing[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#preparing "Link to this heading") Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: from qwen\_agent.llm import get\_chat\_model llm \= get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True } } }) In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by `generate_cfg`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: messages \= MESSAGES\[:\] At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: functions \= \[tool\["function"\] for tool in TOOLS\] #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#tool-calls-and-tool-results "Link to this heading") To interact with the model, the `chat` method should be used: for responses in llm.chat( messages\=messages, functions\=functions, ): pass messages.extend(responses) The `chat` method returns a generator of list, each of which may contain multiple messages. * The results of `no_think` mode: \[\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] * The results of `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: * `name`: a string representing the function to call * `arguments`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part – checking and applying the function call: 1for message in responses: 2 if fn\_call := message.get("function\_call", None): 3 fn\_name: str \= fn\_call\['name'\] 4 fn\_args: dict \= json.loads(fn\_call\["arguments"\]) 5 6 fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) 7 8 messages.append({ 9 "role": "function", 10 "name": fn\_name, 11 "content": fn\_res, 12 }) To get tool results: * line 1: We should iterate the function calls in the order the model generates them. * line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. * line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. * line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#prepcode) to help us get the related function by its name. * line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are: * `no_think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] * `think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#final-response "Link to this heading") Finally, run the model again to get the final model results: for responses in llm.chat(messages\=messages, functions\=functions): pass messages.extend(responses) The final response should be like * `no_think` mode: \[\ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] * `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, \ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] ### vLLM[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#vllm "Link to this heading") vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. * `vllm` >= v0.8.5. For more information, check the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api) . We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#id1 "Link to this heading") For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: vllm serve Qwen/Qwen3-8B \--enable-auto-tool-choice \--tool-call-parser hermes \--reasoning-parser deepseek\_r1 The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES Let’s also initialize the client: from openai import OpenAI openai\_api\_key \= "EMPTY" openai\_api\_base \= "http://localhost:8000/v1" client \= OpenAI( api\_key\=openai\_api\_key, base\_url\=openai\_api\_base, ) model\_name \= "Qwen/Qwen3-8B" #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#id2 "Link to this heading") We can use the create chat completions endpoint to query the model. Here is an example of the `no_think` mode: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True }, ) vLLM should be able to parse the tool calls for us, and the main fields in the response (`response.choices[0]`) should be like Choice( finish\_reason\='tool\_calls', index\=0, logprobs\=None, message\=ChatCompletionMessage( content\=None, role\='assistant', function\_call\=None, tool\_calls\=\[\ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA"}', name\='get\_current\_temperature'), \ type\='function',\ ), \ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name\='get\_temperature\_date'), \ type\='function',\ ),\ \], ), stop\_reason\=None, ) Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str \= tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= json.loads(fn\_call\["arguments"\]) fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like \[\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[\ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'},\ {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'},\ \]},\ {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'},\ {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#id3 "Link to this heading") Let’s call the endpoint again to seed the tool results and get response: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) The final response (`response.choices[0].message.content`) should be like The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. Finally[¶](https://qwen.readthedocs.io/en/v3.0/framework/function_call.html#finally "Link to this heading") ------------------------------------------------------------------------------------------------------------ In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: * It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. * If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Function Calling - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/stable/framework/function_call.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/stable/framework/function_call.html#) [View this page](https://qwen.readthedocs.io/en/stable/_sources/framework/function_call.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Function Calling[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#function-calling "Link to this heading") ================================================================================================================================ Preface[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#preface "Link to this heading") -------------------------------------------------------------------------------------------------------------- Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: * either for AI-native applications that strive to work around the shortcomings of current AI technology, * or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, * We will first demonstrate how to use function calling with Qwen3. * Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is … What is function calling?[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#what-is-function-calling "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------- Note There is another term “tool use” that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. * On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. * On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. Inference with Function Calling[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#inference-with-function-calling "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------- As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with * **Qwen-Agent**, * **vLLM**. ### The Example Case[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#the-example-case "Link to this heading") Let’s also use an example to demonstrate the inference usage. We assume **Python 3.11** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: Preparation Code import json def get\_current\_temperature(location: str, unit: str \= "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str \= "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name \== "get\_current\_temperature": return get\_current\_temperature if name \== "get\_temperature\_date": return get\_temperature\_date TOOLS \= \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location"\],\ },\ },\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "date": {\ "type": "string",\ "description": 'The date to get the temperature for, in the format "Year-Month-Day".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location", "date"\],\ },\ },\ },\ \] MESSAGES \= \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ \] In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: Example Tools The tools should be described using the following JSON: \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location"\ \]\ }\ }\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "date": {\ "type": "string",\ "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location",\ "date"\ \]\ }\ }\ }\ \] For each **tool**, it is a JSON object with two fields: * `type`: a string specifying the type of the tool, currently only `"function"` is valid * `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: * `name`: a string indicating the name of the function * `description`: a string describing what the function is used for * `parameters`: [a JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.`. \[ \ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}\ \] ### Qwen-Agent[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#qwen-agent "Link to this heading") [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is _not recommended_ to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let’s make sure the latest library is installed: pip install \-U qwen-agent #### Preparing[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#preparing "Link to this heading") Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: from qwen\_agent.llm import get\_chat\_model llm \= get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True } } }) In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by `generate_cfg`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: messages \= MESSAGES\[:\] At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: functions \= \[tool\["function"\] for tool in TOOLS\] #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#tool-calls-and-tool-results "Link to this heading") To interact with the model, the `chat` method should be used: for responses in llm.chat( messages\=messages, functions\=functions, ): pass messages.extend(responses) The `chat` method returns a generator of list, each of which may contain multiple messages. * The results of `no_think` mode: \[\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] * The results of `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: * `name`: a string representing the function to call * `arguments`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part – checking and applying the function call: 1for message in responses: 2 if fn\_call := message.get("function\_call", None): 3 fn\_name: str \= fn\_call\['name'\] 4 fn\_args: dict \= json.loads(fn\_call\["arguments"\]) 5 6 fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) 7 8 messages.append({ 9 "role": "function", 10 "name": fn\_name, 11 "content": fn\_res, 12 }) To get tool results: * line 1: We should iterate the function calls in the order the model generates them. * line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. * line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. * line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](https://qwen.readthedocs.io/en/stable/framework/function_call.html#prepcode) to help us get the related function by its name. * line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are: * `no_think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] * `think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#final-response "Link to this heading") Finally, run the model again to get the final model results: for responses in llm.chat(messages\=messages, functions\=functions): pass messages.extend(responses) The final response should be like * `no_think` mode: \[\ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] * `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, \ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] ### vLLM[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#vllm "Link to this heading") vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. * `vllm` >= v0.8.5. For more information, check the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api) . We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#id1 "Link to this heading") For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: vllm serve Qwen/Qwen3-8B \--enable-auto-tool-choice \--tool-call-parser hermes \--reasoning-parser deepseek\_r1 The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/stable/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES Let’s also initialize the client: from openai import OpenAI openai\_api\_key \= "EMPTY" openai\_api\_base \= "http://localhost:8000/v1" client \= OpenAI( api\_key\=openai\_api\_key, base\_url\=openai\_api\_base, ) model\_name \= "Qwen/Qwen3-8B" #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#id2 "Link to this heading") We can use the create chat completions endpoint to query the model. Here is an example of the `no_think` mode: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True }, ) vLLM should be able to parse the tool calls for us, and the main fields in the response (`response.choices[0]`) should be like Choice( finish\_reason\='tool\_calls', index\=0, logprobs\=None, message\=ChatCompletionMessage( content\=None, role\='assistant', function\_call\=None, tool\_calls\=\[\ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA"}', name\='get\_current\_temperature'), \ type\='function',\ ), \ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name\='get\_temperature\_date'), \ type\='function',\ ),\ \], ), stop\_reason\=None, ) Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str \= tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= json.loads(fn\_call\["arguments"\]) fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like \[\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[\ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'},\ {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'},\ \]},\ {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'},\ {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#id3 "Link to this heading") Let’s call the endpoint again to seed the tool results and get response: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) The final response (`response.choices[0].message.content`) should be like The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. Finally[¶](https://qwen.readthedocs.io/en/stable/framework/function_call.html#finally "Link to this heading") -------------------------------------------------------------------------------------------------------------- In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: * It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. * If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Function Calling - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/latest/framework/function_call.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/latest/framework/function_call.html#) [View this page](https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt "View this page") Function Calling[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#function-calling "Link to this heading") ================================================================================================================================ Preface[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#preface "Link to this heading") -------------------------------------------------------------------------------------------------------------- Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: * either for AI-native applications that strive to work around the shortcomings of current AI technology, * or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, * We will first demonstrate how to use function calling with Qwen3. * Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is … What is function calling?[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#what-is-function-calling "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------- Note There is another term “tool use” that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. * On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. * On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. Inference with Function Calling[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#inference-with-function-calling "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------- As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with * **Qwen-Agent**, * **vLLM**. ### The Example Case[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#the-example-case "Link to this heading") Let’s also use an example to demonstrate the inference usage. We assume **Python 3.11** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: Preparation Code import json def get\_current\_temperature(location: str, unit: str \= "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str \= "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name \== "get\_current\_temperature": return get\_current\_temperature if name \== "get\_temperature\_date": return get\_temperature\_date TOOLS \= \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location"\],\ },\ },\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "date": {\ "type": "string",\ "description": 'The date to get the temperature for, in the format "Year-Month-Day".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location", "date"\],\ },\ },\ },\ \] MESSAGES \= \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ \] In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: Example Tools The tools should be described using the following JSON: \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location"\ \]\ }\ }\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "date": {\ "type": "string",\ "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location",\ "date"\ \]\ }\ }\ }\ \] For each **tool**, it is a JSON object with two fields: * `type`: a string specifying the type of the tool, currently only `"function"` is valid * `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: * `name`: a string indicating the name of the function * `description`: a string describing what the function is used for * `parameters`: [a JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.`. \[ \ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}\ \] ### Qwen-Agent[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#qwen-agent "Link to this heading") [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is _not recommended_ to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let’s make sure the latest library is installed: pip install \-U qwen-agent #### Preparing[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#preparing "Link to this heading") Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: from qwen\_agent.llm import get\_chat\_model llm \= get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True } } }) In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by `generate_cfg`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: messages \= MESSAGES\[:\] At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: functions \= \[tool\["function"\] for tool in TOOLS\] #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#tool-calls-and-tool-results "Link to this heading") To interact with the model, the `chat` method should be used: for responses in llm.chat( messages\=messages, functions\=functions, ): pass messages.extend(responses) The `chat` method returns a generator of list, each of which may contain multiple messages. * The results of `no_think` mode: \[\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] * The results of `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ \] As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: * `name`: a string representing the function to call * `arguments`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part – checking and applying the function call: 1for message in responses: 2 if fn\_call := message.get("function\_call", None): 3 fn\_name: str \= fn\_call\['name'\] 4 fn\_args: dict \= json.loads(fn\_call\["arguments"\]) 5 6 fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) 7 8 messages.append({ 9 "role": "function", 10 "name": fn\_name, 11 "content": fn\_res, 12 }) To get tool results: * line 1: We should iterate the function calls in the order the model generates them. * line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. * line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. * line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](https://qwen.readthedocs.io/en/latest/framework/function_call.html#prepcode) to help us get the related function by its name. * line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are: * `no_think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] * `think` mode: \[\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}},\ {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},\ {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#final-response "Link to this heading") Finally, run the model again to get the final model results: for responses in llm.chat(messages\=messages, functions\=functions): pass messages.extend(responses) The final response should be like * `no_think` mode: \[\ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] * `think` mode: \[\ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, \ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."}\ \] ### vLLM[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#vllm "Link to this heading") vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. * `vllm` >= v0.8.5. For more information, check the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api) . We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#id1 "Link to this heading") For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: vllm serve Qwen/Qwen3-8B \--enable-auto-tool-choice \--tool-call-parser hermes \--reasoning-parser deepseek\_r1 The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/latest/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES Let’s also initialize the client: from openai import OpenAI openai\_api\_key \= "EMPTY" openai\_api\_base \= "http://localhost:8000/v1" client \= OpenAI( api\_key\=openai\_api\_key, base\_url\=openai\_api\_base, ) model\_name \= "Qwen/Qwen3-8B" #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#id2 "Link to this heading") We can use the create chat completions endpoint to query the model. Here is an example of the `no_think` mode: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} \# default to True }, ) vLLM should be able to parse the tool calls for us, and the main fields in the response (`response.choices[0]`) should be like Choice( finish\_reason\='tool\_calls', index\=0, logprobs\=None, message\=ChatCompletionMessage( content\=None, role\='assistant', function\_call\=None, tool\_calls\=\[\ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA"}', name\='get\_current\_temperature'), \ type\='function',\ ), \ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name\='get\_temperature\_date'), \ type\='function',\ ),\ \], ), stop\_reason\=None, ) Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str \= tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= json.loads(fn\_call\["arguments"\]) fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like \[\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},\ {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[\ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'},\ {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'},\ \]},\ {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'},\ {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#id3 "Link to this heading") Let’s call the endpoint again to seed the tool results and get response: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) The final response (`response.choices[0].message.content`) should be like The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. Finally[¶](https://qwen.readthedocs.io/en/latest/framework/function_call.html#finally "Link to this heading") -------------------------------------------------------------------------------------------------------------- In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: * It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. * If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Function Calling - Qwen Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode [Skip to content](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#furo-main-content) [Back to top](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#) [View this page](https://qwen.readthedocs.io/en/v2.5/_sources/framework/function_call.md.txt "View this page") Toggle Light / Dark / Auto color theme Toggle table of contents sidebar Function Calling[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#function-calling "Link to this heading") ============================================================================================================================== Preface[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#preface "Link to this heading") ------------------------------------------------------------------------------------------------------------ Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: * either for AI-native applications that strive to work around the shortcomings of current AI technology, * or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. This guide will not delve into those discussions or which role an LLM should play in an application and the related best practice. Those views are reflected in the design of AI application frameworks: from LangChain to LlamaIndex to QwenAgent. Instead, we will talk about how Qwen2.5 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, * We will first demonstrate how to use function calling with Qwen2.5. * Then, we will introduce the technical details on functional calling with Qwen2.5, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is … What is function calling?[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#what-is-function-calling "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------- Note There is another term “tool use” that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. * On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. * On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. They are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. Qwen2.5 were pre-trained with various types of templates that could support function calling, so that users can directly make use of this procedure. Inference with Function Calling[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#inference-with-function-calling "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------ Note Please be aware that the inference usage is subject to change as the frameworks and the Qwen models evolve. As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen2 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with * **Qwen-Agent**, * **Hugging Face transformers**, * **Ollama**, and * **vLLM**. If you are familiar with the usage of OpenAI API, you could also directly use the OpenAI-compatible API services for Qwen2.5. However, not all of them support function calling for Qwen2.5. Currently, supported solutions include the self-hosted service by [Ollama](https://github.com/ollama/ollama/blob/main/docs/openai.md) or [vLLM](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api) and the cloud service of [ModelStudio \[zh\]](https://help.aliyun.com/zh/model-studio/developer-reference/compatibility-of-openai-with-dashscope#97e2b45391x08) . If you are familiar with application frameworks, e.g., LangChain, you can also use function calling abilities in Qwen2.5 via ReAct Prompting. ### The Example Case[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#the-example-case "Link to this heading") Let’s also use an example to demonstrate the inference usage. We assume **Python 3.11** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: Preparation Code import json def get\_current\_temperature(location: str, unit: str \= "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str \= "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name \== "get\_current\_temperature": return get\_current\_temperature if name \== "get\_temperature\_date": return get\_temperature\_date TOOLS \= \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location"\],\ },\ },\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": 'The location to get the temperature for, in the format "City, State, Country".',\ },\ "date": {\ "type": "string",\ "description": 'The date to get the temperature for, in the format "Year-Month-Day".',\ },\ "unit": {\ "type": "string",\ "enum": \["celsius", "fahrenheit"\],\ "description": 'The unit to return the temperature in. Defaults to "celsius".',\ },\ },\ "required": \["location", "date"\],\ },\ },\ },\ \] MESSAGES \= \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"},\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"},\ \] In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: Example Tools The tools should be described using the following JSON: \[\ {\ "type": "function",\ "function": {\ "name": "get\_current\_temperature",\ "description": "Get current temperature at a location.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location"\ \]\ }\ }\ },\ {\ "type": "function",\ "function": {\ "name": "get\_temperature\_date",\ "description": "Get temperature at a location and date.",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."\ },\ "date": {\ "type": "string",\ "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."\ },\ "unit": {\ "type": "string",\ "enum": \[\ "celsius",\ "fahrenheit"\ \],\ "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."\ }\ },\ "required": \[\ "location",\ "date"\ \]\ }\ }\ }\ \] For each **tool**, it is a JSON object with two fields: * `type`: a string specifying the type of the tool, currently only `"function"` is valid * `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: * `name`: a string indicating the name of the function * `description`: a string describing what the function is used for * `parameters`: [a JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow?`. Since the model does not know what the current date is, let alone tomorrow, we should provide the date in the inputs. Here, we decide to supply that information in the system message after the default system message `You are Qwen, created by Alibaba Cloud. You are a helpful assistant.`. You could append the date to user message in your application code. \[\ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"},\ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}\ \] ### Qwen-Agent[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#qwen-agent "Link to this heading") [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen2.5. It provides the function calling ability for Qwen2.5 to an OpenAI-compatible API through templates that is transparent to users. It’s worth noting that since a lot of stuff can be done under the scene with application frameworks, currently the official function calling implementation for Qwen2.5 is very flexible and beyond simple templating, making it hard to adapt it other frameworks that use less capable templating engines. Before starting, let’s make sure the latest library is installed: pip install \-U qwen-agent For this guide, we are at version v0.0.10. #### Preparing[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#preparing "Link to this heading") Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: from qwen\_agent.llm import get\_chat\_model llm \= get\_chat\_model({ "model": "Qwen/Qwen2.5-7B-Instruct", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", }) In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. For model inputs, the common message structure for system, user, and assistant history should be used: messages \= MESSAGES\[:\] \# \[\ \# {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"},\ \# {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}\ \# \] We add the current date to the system message so that the “tomorrow” in the user message is anchored. It can also be added to the user message if one desires. At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: functions \= \[tool\["function"\] for tool in TOOLS\] #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#tool-calls-and-tool-results "Link to this heading") To interact with the model, the `chat` method should be used: for responses in llm.chat( messages\=messages, functions\=functions, extra\_generate\_cfg\=dict(parallel\_function\_calls\=True), ): pass messages.extend(responses) In the above code, the `chat` method receives the `messages`, the `functions`, and an `extra_generate_cfg` parameter. You can put sampling parameters, such as `temperature`, and `top_p`, in the `extra_generate_cfg`. Here, we add to it a special control `parallel_function_calls` provided by Qwen-Agent. As its name suggests, it will enable parallel function calls, which means that the model may generate multiple function calls for a single turn as it deems fit. The `chat` method returns a generator of list, each of which may contain multiple messages. Since we enable `parallel_function_calls`, we should get two messages in the responses: \[\ {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_current\_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}},\ {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_temperature\_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}},\ \] As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: * `name`: a string representing the function to call * `arguments`: a JSON-formatted string representing the arguments the function should be called with Note that Qwen2.5-7B-Instruct is quite capable: * It has followed the function instructions to add the state and the country to the location. * It has correctly induced the date of tomorrow and given in the format required by the function. Then comes the critical part – checking and applying the function call: 1for message in responses: 2 if fn\_call := message.get("function\_call", None): 3 fn\_name: str \= fn\_call\['name'\] 4 fn\_args: dict \= json.loads(fn\_call\["arguments"\]) 5 6 fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) 7 8 messages.append({ 9 "role": "function", 10 "name": fn\_name, 11 "content": fn\_res, 12 }) To get tool results: * line 1: We should iterate the function calls in the order the model generates them. * line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. * line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. * line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#prepcode) to help us get the related function by its name. * line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are \[\ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'},\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},\ {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_current\_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}},\ {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_temperature\_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}},\ {'role': 'function', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'},\ {'role': 'function', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#final-response "Link to this heading") Finally, run the model again to get the final model results: for responses in llm.chat(messages\=messages, functions\=functions): pass messages.extend(responses) The final response should be like {'role': 'assistant', 'content': 'Currently, the temperature in San Francisco is approximately 26.1°C. Tomorrow, on 2024-10-01, the temperature is forecasted to be around 25.9°C.'} ### Hugging Face transformers[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#hugging-face-transformers "Link to this heading") Since function calling is based on prompt engineering and templates, `transformers` supports it with its tokenizer utilities, in particular, the `tokenizer.apply_chat_template` method, which hides the sophistication of constructing the model inputs, using the Jinja templating engine. However, it means that users should handle the model output part on their own, which includes parsing the generated function call message. The blog piece [_Tool Use, Unified_](https://huggingface.co/blog/unified-tool-use) is very helpful in understanding its design. Be sure to take a look. Tool use API is available in transformers since v4.42.0. Before starting, let’s check that: pip install "transformers>4.42.0" For this guide, we are at version v4.44.2. #### Preparing[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id1 "Link to this heading") For Qwen2.5, the chat template in `tokenizer_config.json` has already included support for the Hermes-style tool use. We simply need to load the model and the tokenizer: from transformers import AutoModelForCausalLM, AutoTokenizer model\_name\_or\_path \= "Qwen/Qwen2.5-7B-Instruct" tokenizer \= AutoTokenizer.from\_pretrained(model\_name\_or\_path) model \= AutoModelForCausalLM.from\_pretrained( model\_name\_or\_path, torch\_dtype\="auto", device\_map\="auto", ) The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES\[:\] In `transformers`, you can also directly use Python functions as tools with certain constraints[\[1\]](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#get-json-schema-note) : tools \= \[get\_current\_temperature, get\_temperature\_date\] #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id3 "Link to this heading") To construct the input sequence, we should use the `apply_chat_template` method and then let the model continue the texts: text \= tokenizer.apply\_chat\_template(messages, tools\=tools, add\_generation\_prompt\=True, tokenize\=False) inputs \= tokenizer(text, return\_tensors\="pt").to(model.device) outputs \= model.generate(\*\*inputs, max\_new\_tokens\=512) output\_text \= tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] The output texts should be like {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA"}} {"name": "get\_temperature\_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} <|im\_end|> Now we need to do two things: 1. Parse the generated tool calls to a message and add them to the messages, so that the model knows which tools are used. 2. Obtain the results of the tools and add them to the messages, so that the model knows the results of the tool calls. In `transformers`, the tool calls should be a field of assistant messages. Let’s use a simple function called `try_parse_tool_calls` to parse the tool calls: import re def try\_parse\_tool\_calls(content: str): """Try parse the tool calls.""" tool\_calls \= \[\] offset \= 0 for i, m in enumerate(re.finditer(r"\\n(.+)?\\n", content)): if i \== 0: offset \= m.start() try: func \= json.loads(m.group(1)) tool\_calls.append({"type": "function", "function": func}) if isinstance(func\["arguments"\], str): func\["arguments"\] \= json.loads(func\["arguments"\]) except json.JSONDecodeError as e: print(f"Failed to parse tool calls: the content is {m.group(1)} and {e}") pass if tool\_calls: if offset \> 0 and content\[:offset\].strip(): c \= content\[:offset\] else: c \= "" return {"role": "assistant", "content": c, "tool\_calls": tool\_calls} return {"role": "assistant", "content": re.sub(r"<\\|im\_end\\|>$", "", content)} This function does not cover all possible scenarios and thus is prone to errors. But it should suffice for the purpose of this guide. Note The template in the `tokenizer_config.json` assumes that the generated content alongside tool calls is in the same message instead of separate assistant messages, e.g., { "role": "assistant", "content": "To obtain the current temperature, I should call the functions \`get\_current\_temperate\`.", "tool\_calls": \[\ {"type": "function", "function": {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}}\ \] } instead of \[\ {\ "role": "assistant", \ "content": "To obtain the current temperature, I should call the functions \`get\_current\_temperate\`.", \ },\ {\ "role": "assistant", \ "content": "", \ "tool\_calls": \[\ {"type": "function", "function": {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}}\ \]\ }\ \] This is implemented roughly in `try_parse_tool_calls` but keep that in mind if you are writing your own tool call parser. messages.append(try\_parse\_tool\_calls(output\_text)) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= fn\_call\["arguments"\] fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) The messages now should be like \[\ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'},\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},\ {'role': 'assistant', 'content': '', 'tool\_calls': \[\ {'type': 'function', 'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, \ {'type': 'function', 'function': {'name': 'get\_temperature\_date', 'arguments': {'location': 'San Francisco, CA, USA', 'date': '2024-10-01'}}},\ \]},\ {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'},\ {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'},\ \] The messages are similar to those of Qwen-Agent, but there are some major differences: * Tools instead of functions * Parallel calls are by default * Multiple tool calls as a list in a single assistant message, instead of multiple messages. * The function arguments are parsed into a dict if it is a valid JSON-formatted string. #### Final Response[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id4 "Link to this heading") Then it’s time for the model to generate the actual response for us based on the tool results. Let’s query the model again: text \= tokenizer.apply\_chat\_template(messages, tools\=tools, add\_generation\_prompt\=True, tokenize\=False) inputs \= tokenizer(text, return\_tensors\="pt").to(model.device) outputs \= model.generate(\*\*inputs, max\_new\_tokens\=512) output\_text \= tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] The output\_text should be like The current temperature in San Francisco is approximately 26.1°C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9°C.<|im\_end|> Add the result text as an assistant message and the final messages should be ready for further interaction: messages.append(try\_parse\_tool\_calls(output\_text)) ### Ollama[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#ollama "Link to this heading") Ollama is a set of tools for serving LLMs locally. It also relies on its template implementation to support function calling. Different from transformers, which is written in Python and uses the Jinja template whose syntax is heavily inspired by Django and Python, Ollama, which is mostly written in Go, uses Go’s [text/template](https://pkg.go.dev/text/template) packages. In addition, Ollama implements internally a helper function so that it can automatically parse the generated tool calls in texts to structured messages if the format supported. You could check the [Tool support](https://ollama.com/blog/tool-support) blog post first. Tool support has been available in Ollama since v0.3.0. You can run the following to check the Ollama version: ollama \-v If lower than expected, follow [the official instructions](https://ollama.com/download) to install the latest version. In this guide, we will aslo use [ollama-python](https://github.com/ollama/ollama-python) , before starting, make sure it is available in your environment: pip install ollama For this guide, the `ollama` binary is at v0.3.9 and the `ollama` Python library is at v0.3.2. #### Preparing[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id5 "Link to this heading") The messages structure used in Ollama is the same with that in `transformers` and the template in [Qwen2.5 Ollama models](https://ollama.com/library/qwen2.5) has supported tool use. The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES\[:\] model\_name \= "qwen2.5:7b" Note that you cannot pass Python functions as tools directly and `tools` has to be a `dict`. #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id6 "Link to this heading") We can use the `ollama.chat` method to directly query the underlying API: import ollama response \= ollama.chat( model\=model\_name, messages\=messages, tools\=tools, ) The main fields in the response could be: { 'model': 'qwen2.5:7b', 'message': { 'role': 'assistant', 'content': '', 'tool\_calls': \[\ {'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}},\ {'function': {'name': 'get\_temperature\_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}},\ \], }, } Ollama’s tool call parser has succeeded in parsing the tool results. If not, you may refine [the `try_parse_tool_calls` function above](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#parse-function) . Then, we can obtain the tool results and add them to the messages. The following is basically the same with `transformers`: messages.append(response\["message"\]) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= fn\_call\["arguments"\] fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) The messages are now like \[\ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'},\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},\ {'role': 'assistant', 'content': '', 'tool\_calls': \[\ {'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}},\ {'function': {'name': 'get\_temperature\_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}},\ \]},\ {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'},\ {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id7 "Link to this heading") The rest are easy: response \= ollama.chat( model\=model\_name, messages\=messages, tools\=tools, ) messages.append(response\["message"\]) The final message should be like the following: {'role': 'assistant', 'content': 'The current temperature in San Francisco is approximately 26.1°C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9°C.'} ### vLLM[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#vllm "Link to this heading") vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. Tool support has been available in `vllm` since v0.6.0. Be sure to install a version that supports tool use. For more information, check the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api) . For this guide, we are at version v0.6.1.post2. We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id8 "Link to this heading") For Qwen2.5, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: vllm serve Qwen/Qwen2.5-7B-Instruct \--enable-auto-tool-choice \--tool-call-parser hermes The inputs are the same with those in [the preparation code](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#prepcode) : tools \= TOOLS messages \= MESSAGES\[:\] Let’s also initialize the client: from openai import OpenAI openai\_api\_key \= "EMPTY" openai\_api\_base \= "http://localhost:8000/v1" client \= OpenAI( api\_key\=openai\_api\_key, base\_url\=openai\_api\_base, ) model\_name \= "Qwen/Qwen2.5-7B-Instruct" #### Tool Calls and Tool Results[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id9 "Link to this heading") We can use the create chat completions endpoint to query the model: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, }, ) vLLM should be able to parse the tool calls for us, and the main fields in the response (`response.choices[0]`) should be like Choice( finish\_reason\='tool\_calls', index\=0, logprobs\=None, message\=ChatCompletionMessage( content\=None, role\='assistant', function\_call\=None, tool\_calls\=\[\ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA"}', name\='get\_current\_temperature'), \ type\='function',\ ), \ ChatCompletionMessageToolCall(\ id\='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', \ function\=Function(arguments\='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name\='get\_temperature\_date'), \ type\='function',\ ),\ \], ), stop\_reason\=None, ) Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows but `transformers` and Ollama differs. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[\-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str \= tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str \= fn\_call\["name"\] fn\_args: dict \= json.loads(fn\_call\["arguments"\]) fn\_res: str \= json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like \[\ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'},\ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},\ {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[\ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'},\ {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'},\ \]},\ {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'},\ {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'},\ \] #### Final Response[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#id10 "Link to this heading") Let’s call the endpoint again to seed the tool results and get response: response \= client.chat.completions.create( model\=model\_name, messages\=messages, tools\=tools, temperature\=0.7, top\_p\=0.8, max\_tokens\=512, extra\_body\={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) The final response (`response.choices[0].message.content`) should be like The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. ### Discussions[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#discussions "Link to this heading") Now, we have introduced how to conduct inference with function calling using Qwen2 in three different frameworks! Let’s make a brief comparison. | Item | OpenAI API | Hugging Face transformers | Ollama | vLLM | Qwen-Agent | | --- | --- | --- | --- | --- | --- | | Type | HTTP API | Python Library | HTTP API | HTTP API | Python Library | | Inference Backend | \- | PyTorch | llama.cpp | PyTorch | HTTP API | | Templating Backend | \- | Jinja | Go `text/template` | Jinja | Python | | Tools/Functions | Tools | Tools | Tools | Tools | Functions | | Parallel Calls | Default Yes (Configurable) | Yes | Yes | Yes | Default No (Configurable) | | Call Format | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Multiple assistant messages with `function_call` | | Call Argument Format | string | object | object | string | string | | Call Result Format | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple function messages with `content` | There are some details not shown in the above table: * OpenAI API comes with Python, Node.js, Go, and .NET SDKs. It also follows the OpenAPI standard. * Ollama comes with Python and Node.js SDKs. It has OpenAI-compatible API at a different base url that can be accessed using OpenAI API SDK. * Qwen-Agent as an application framework can call the tools automatically for you, which is introduced in [the Qwen-Agent guide](https://qwen.readthedocs.io/en/v2.5/framework/qwen_agent.html) . In addition, there are more on the model side of function calling, which means you may need to consider more things in production code: * **Accuracy of function calling**: When it comes to evaluate the accuracy of function calling, there are two aspects: (a) whether the correct functions (including no functions) are selected and (b) whether the correct function arguments are generated. It is not always the case that Qwen2.5 will be accurate. Function calling can involve knowledge that is deep and domain-specific. Sometimes, it doesn’t fully understand the function and select the wrong one by mistake. Sometimes, it can fall into a loop and require calling the same function again and again. Sometimes, it will fabricate required function arguments instead of asking the user for input. To improve the function calling accuracy, it is advised to first try prompt engineering: does a more detailed function description help? can we provide instructions and examples to the model in the system message? If not, finetuning on your own data could also improve performance. * **Protocol consistency**: Even with the proper function calling template, the protocol may break. The model may generate extra texts to tool calls, e.g., explanations. The generated tool call may be invalid JSON-formatted string but a representation of a Python dict The generated tool call may be valid JSON but not conforms to the provided JSON Schema. For those kinds of issues, while some of them could be addressed with prompt engineering, some are caused by the nature of LLMs and can be hard to resolve in a general manner by LLMs themselves. While we strive to improve Qwen2.5 in this regard, edge cases are unlikely to be eliminated completely. Function Calling Templates[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#function-calling-templates "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- The template design for function calling often includes the following aspects: * How to describe the functions to the model, so that the model understands what they are and how to use them. * How to prompt the model, so that it knows that functions can be used and in what format to generate the function calls. * How to tell a function call generation from others in generated text, so that we can extract the calls from the generated texts and actually make the calls. * How to incorporate the function results to the text, so that the model can tell them from its own generation and make connection among the calls and the results. For experienced prompt engineers, it should be possible to make any LLM support function calling, using in-context learning techniques and with representative examples, though with varied accuracy and stability depending on how “zero-shot” the task at hand is. ### Starting from ReAct Prompting[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#starting-from-react-prompting "Link to this heading") For example, ReAct Prompting can be used to implement function calling with an extra element of planning: * **Thought**: the overt reasoning path, analyzing the functions and the user query and saying it out “loud” * **Action**: the function to use and the arguments with which the function should be called * **Observation**: the results of the function In fact, Qwen2 is verse in the following variant of ReAct Prompting (similar to LangChain ReAct) to make the intermediate texts more structured: Answer the following questions as best you can. You have access to the following tools: {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of \[{function\_name},{function\_name}\] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can be repeated zero or more times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: {query} Thought: {some\_text} Action: {function\_name} Action Input: {function\_arguments} Observation: {function\_results} Final Answer: {response} As you can see, there is no apparent user/assistant conversation structure in the template. The model will simply continue the texts. One should write the code to actively detect which step the model is at and in particular to add the observations in the process, until the Final Answer is generated. However, as most programming interfaces accept the message structure, there should be some kind of adapter between the two. [The ReAct Chat Agent](https://github.com/QwenLM/Qwen-Agent/blob/v0.0.10/qwen_agent/agents/react_chat.py) in Qwen-Agent facilitates this kind of conversion. ### Qwen2 Function Calling Template[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#qwen2-function-calling-template "Link to this heading") As a step forward, the official Qwen2 function calling template is in the vein of the ReAct Prompting format but focuses more on * differentiating the keywords like `Question`, `Thought`, `Action`, etc., from generation, * simplifying the process, * supporting better multi-turn conversation, and * adding controls for specialized usage. An equivalent example would be <|im\_start|>system {system message} ## Tools You have access to the following tools: ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of \[{function\_name},{function\_name}\] ✿ARGS✿: The input of the tool ✿RESULT✿: Tool results ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user {query}<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: {function\_name} ✿ARGS✿: {function\_arguments} ✿RESULT✿: {function\_result} ✿RETURN✿:{response}<|im\_end|> Let’s first list the obvious differences: * Keywords (`✿FUNCTION✿`, `✿ARGS✿`, etc.) seem rare in ordinary text and more semantically related to function calling, but not special tokens yet. * Thought is omitted. This could affect accuracy for some use cases. * Use the system-user-assistant format for multi-turn conversations. Function calling prompting is moved to the system message. How about adding controls for specialized usage? The template actually has the following variants: * Language: the above is for non-Chinese language; there is another template in Chinese. * Parallel Calls: the above is for non-parallel calls; there is another template for parallel calls. In the canonical implementation in Qwen-Agent, those switches are implemented in Python, according to the configuration and current input. The actual text with _parallel calls_ should be like the following: <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 ## Tools You have access to the following tools: ### get\_current\_temperature get\_current\_temperature: Get current temperature at a location. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]} Format the arguments as a JSON object. ### get\_temperature\_date get\_temperature\_date: Get temperature at a location and date. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]} Format the arguments as a JSON object. ## Insert the following command in your reply when you need to call N tools in parallel: ✿FUNCTION✿: The name of tool 1, should be one of \[get\_current\_temperature,get\_temperature\_date\] ✿ARGS✿: The input of tool 1 ✿FUNCTION✿: The name of tool 2 ✿ARGS✿: The input of tool 2 ... ✿FUNCTION✿: The name of tool N ✿ARGS✿: The input of tool N ✿RESULT✿: The result of tool 1 ✿RESULT✿: The result of tool 2 ... ✿RESULT✿: The result of tool N ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: get\_current\_temperature ✿ARGS✿: {"location": "San Francisco, CA, USA"} ✿FUNCTION✿: get\_temperature\_date ✿ARGS✿: {"location": "San Francisco, CA, USA", "date": "2024-10-01"} ✿RESULT✿: {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} ✿RESULT✿: {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} ✿RETURN✿: The current temperature in San Francisco is approximately 26.1°C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9°C.<|im\_end|> This template is hard to adapt it for other frameworks that use less capable templating engines. But it is doable at least partially for Jinja, which is Python-oriented after all. We didn’t use it because using the template in `transformers` leads to more changes to the inference usage, which are not very common for beginners. For the interested, you can find the Jinja template and key points on usage below: Qwen2 Function Calling Jinja Template {%\- if messages\[0\]\["role"\] \== "system" %} {%\- set system\_message \= messages\[0\]\["content"\] %} {%\- set loop\_messages \= messages\[1:\] %} {%\- else %} {%\- set system\_message \= "You are a helpful assistant." %} {%\- set loop\_messages \= messages %} {%\- endif %} {%\- if parallel\_tool\_calls is undefined %} {%\- set parallel\_tool\_calls \= false %} {%\- endif %} {%\- if language is undefined or language != "zh" %} {%\- set language \= "en" %} {%\- endif %} {{\- "<|im\_start|>system\\n" + system\_message|trim }} {%\- if tools is defined %} {{\- "\\n\\n# 工具\\n\\n## 你拥有如下工具：\\n\\n" if language \== "zh" else "\\n\\n## Tools\\n\\nYou have access to the following tools:\\n\\n" }} {%\- set functions \= tools|map(attribute\="function")|list %} {%\- set function\_names \= functions|map(attribute\="name")|join(",") %} {%\- for function in functions %} {{\- "### " + function.name + "\\n\\n" + function.name + ": " + function.description + (" 输入参数：" if language \== "zh" else " Parameters: ") + function.parameters|tojson + (" 此工具的输入应为JSON对象。\\n\\n" if language \== "zh" else " Format the arguments as a JSON object.\\n\\n") }} {%\- endfor %} {%\- if parallel\_tool\_calls and language \== "zh" %} {{\- "## 你可以在回复中插入以下命令以并行调用N个工具：\\n\\n✿FUNCTION✿: 工具1的名称，必须是\[" + function\_names + "\]之一\\n✿ARGS✿: 工具1的输入\\n✿FUNCTION✿: 工具2的名称\\n✿ARGS✿: 工具2的输入\\n...\\n✿FUNCTION✿: 工具N的名称\\n✿ARGS✿: 工具N的输入\\n✿RESULT✿: 工具1的结果\\n✿RESULT✿: 工具2的结果\\n...\\n✿RESULT✿: 工具N的结果\\n✿RETURN✿: 根据工具结 果进行回复，需将图片用!\[\](url)渲染出来" }} {%\- elif parallel\_tool\_calls %} {{\- "## Insert the following command in your reply when you need to call N tools in parallel:\\n\\n✿FUNCTION✿: The name of tool 1, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of tool 1\\n✿FUNCTION✿: The name of tool 2\\n✿ARGS✿: The input of tool 2\\n...\\n✿FUNCTION✿: The name of tool N\\n✿ARGS✿: The input of tool N\\n✿RESULT✿: The result of tool 1\\n✿RESULT✿: The result of tool 2\\n...\\n✿RESULT✿: The result of tool N\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%\- elif language \== "zh" %} {{\- "## 你可以在回复中插入零次、一次或多次以下命令以调用工具：\\n\\n✿FUNCTION✿: 工具名称，必须是\[" + function\_names + "\]之一。\\n✿ARGS✿: 工具输入\\n✿RESULT✿: 工具结果\\n✿RETURN✿: 根据工具结果进行回复，需将图片用!\[\](url)渲染出来" }} {%\- else %} {{\- "## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs:\\n\\n✿FUNCTION✿: The tool to use, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of the tool\\n✿RESULT✿: Tool results\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%\- endif %} {%\- endif %} {{\- "<|im\_end|>" }} {%\- for message in loop\_messages %} {%\- if message.role \== "user" %} {{\- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%\- if loop.last and add\_generation\_prompt %} {{\- "\\n<|im\_start|>assistant\\n" }} {%\- endif %} {%\- elif message.role \== "tool" %} {{\- "✿RESULT✿: " + message.content + "\\n" }} {%\- if loop.last and add\_generation\_prompt %} {{\- "✿RETURN✿:" }} {%\- endif %} {%\- elif message.role \== "assistant" and message.tool\_calls is defined %} {%\- if loop.previtem.role \== "user" %} {{\- "\\n<|im\_start|>assistant\\n" }} {%\- endif %} {%\- for function in message.tool\_calls|map(attribute\="function") %} {{\- "✿FUNCTION✿: " + function.name + "\\n✿ARGS✿: " + function.arguments|tojson + "\\n" }} {%\- endfor %} {%\- elif message.role \== "assistant" %} {%\- if loop.previtem.role \== "user" %} {{\- "\\n<|im\_start|>assistant\\n" }} {%\- elif loop.previtem.role \== "tool" %} {{\- "✿RETURN✿:" }} {%\- endif %} {{\- message.content }} {%\- if loop.nextitem is undefined or loop.nextitem.role \== "user" %} {{\- "<|im\_end|>" }} {%\- endif %} {%\- else %} {{\- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%\- endif %} {%\- endfor %} To use this template in `transformers`: * Switches can be enabled by passing them to the `apply_chat_template` method, e.g., `tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, parallel_tool_call=True, language="zh", tokenize=False)`. By default, it is for English non-parallel function calling. * The tool arguments should be a Python `dict` instead of a JSON-formatted object `str`. * Since the generation needs to be stopped at `✿RESULT✿` or else the model will generate fabricated tool results, we should add it to `stop_strings` in `generation_config`: model.generation\_config.stop\_strings \= \["✿RESULT✿:", "✿RETURN✿:"\] * As a result of using `stop_strings`, you need to pass the tokenizer to `model.generate` as `model.generate(**inputs, tokenizer=tokenizer, max_new_tokens=512)`. * `response`, i.e., the model generation based on the tool calls and tool results, may contain a leading space. You should not strip it for the model. It is resulted from the tokenization and the template design. * The `try_parse_tool_calls` function should also be modified accordingly. ### Qwen2.5 Function Calling Templates[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#qwen2-5-function-calling-templates "Link to this heading") For `transformers` and Ollama, we have also used templates that are easier to implement with Jinja or Go. They are variants of [the Nous Research’s Hermes function calling template](https://github.com/NousResearch/Hermes-Function-Calling#prompt-format-for-function-calling) . The Jinja template and the Go template should produce basically the same results. They final text should look like the following: <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {"type": "function", "function": {"name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]}}} {"type": "function", "function": {"name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]}}} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA"}} {"name": "get\_temperature\_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} <|im\_end|> <|im\_start|>user {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} <|im\_end|> <|im\_start|>assistant The current temperature in San Francisco is approximately 26.1°C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9°C.<|im\_end|> While the text may seem different from the previous one, the basic prompting structure is still the same. There are just more structural tags and more JSON-formatted strings. * * * There is one thing we haven’t talked about: how should functions be described to the LLMs. In short, you could describe them as you would normally describe them in an API documentation, as long as you can effectively parse, validate, and execute the tool calls generated by the models. The format with JSON Schema appears a valid and common choice. Finally[¶](https://qwen.readthedocs.io/en/v2.5/framework/function_call.html#finally "Link to this heading") ------------------------------------------------------------------------------------------------------------ In whichever way you choose to use function calling with Qwen2.5, keep in mind that the limitation and the perks of prompt engineering applies: * It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. * If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! * * * --- # Unknown Installation ============ To get a quick start with Qwen1.5, you can install the \`\`transformers\`\` library from Hugging Face and use models in the \`Qwen1.5 collection \`\_\_. We advise you to install the latest \`\`transformers\`\` or at least version 4.37.0. Install with Pip ~~~~~~~~~~~~~~~~ .. code:: bash pip install transformers -U Install with Conda ~~~~~~~~~~~~~~~~~~ .. code:: bash conda install conda-forge::transformers Install from Source ~~~~~~~~~~~~~~~~~~~ .. code:: bash pip install git+https://github.com/huggingface/transformers We advise you to use Python 3.8 or higher, and Pytorch 2.0 or higher. --- # Unknown \# Quickstart This guide helps you quickly start using Qwen2. We provide examples of \[Hugging Face Transformers\](https://github.com/huggingface/transformers) as well as \[ModelScope\](https://github.com/modelscope/modelscope), and \[vLLM\](https://github.com/vllm-project/vllm) for deployment. You can find Qwen2 models in the \[Qwen2 collection\](https://huggingface.co/collections/Qwen/qwen2-6659360b33528ced941e557f). ## Hugging Face Transformers & ModelScope To get a quick start with Qwen2, we advise you to try with the inference with \`transformers\` first. Make sure that you have installed \`transformers>=4.40.0\`. We advise you to use Python 3.8 or higher, and PyTorch 2.2 or higher. :::{dropdown} Install \`\`transformers\`\` \* Install with \`\`pip\`\`: \`\`\`bash pip install transformers -U \`\`\` \* Install with \`\`conda\`\`: \`\`\`bash conda install conda-forge::transformers \`\`\` \* Install from source: \`\`\`bash pip install git+https://github.com/huggingface/transformers \`\`\` ::: The following is a very simple code snippet showing how to run Qwen2-Instruct, with an example of Qwen2-7B-Instruct: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer # Now you do not need to add "trust\_remote\_code=True" model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen2-7B-Instruct") # Instead of using model.chat(), we directly use model.generate() # But you need to use tokenizer.apply\_chat\_template() to format your inputs as shown below prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # Directly use generate() and tokenizer.decode() to get the output. # Use \`max\_new\_tokens\` to control the maximum output length. generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` Previously, we use \`model.chat()\` (see \`modeling\_qwen.py\` in previous Qwen models for more information). Now, we follow the practice of \`transformers\` and directly use \`model.generate()\` with \`apply\_chat\_template()\` in tokenizer. ### Streaming Generation Streaming mode for model chat is simple with the help of \`TextStreamer\`. Below we show you an example of how to use it: \`\`\`python ... # Reuse the code before \`model.generate()\` in the last code snippet from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, streamer=streamer, ) \`\`\` It will print the text to the console or the terminal as being generated. ### ModelScope To tackle with downloading issues, we advise you to try \[ModelScope\](https://github.com/modelscope/modelscope). Before starting, you need to install \`modelscope\` with \`pip\`. \`modelscope\` adopts a programmatic interface similar (but not identical) to \`transformers\`. For basic usage, you can simply change the first line of code above to the following: \`\`\`python from modelscope import AutoModelForCausalLM, AutoTokenizer \`\`\` For more information, please refer to \[the documentation of \`modelscope\`\](https://www.modelscope.cn/docs). ## vLLM for Deployment To deploy Qwen2, we advise you to use vLLM. vLLM is a fast and easy-to-use framework for LLM inference and serving. In the following, we demonstrate how to build a OpenAI-API compatible API service with vLLM. First, make sure you have installed \`vllm>=0.4.0\`: \`\`\`bash pip install vllm \`\`\` Run the following code to build up a vLLM service. Here we take Qwen2-7B-Instruct as an example: \`\`\`bash python -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen2-7B-Instruct \`\`\` with \`vllm>=0.5.3\`, you can also use \`\`\`bash vllm serve Qwen/Qwen2-7B-Instruct \`\`\` Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2-7B-Instruct", "messages": \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use Python client with \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2-7B-Instruct", messages=\[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` For more information, please refer to \[the documentation of \`vllm\`\](https://docs.vllm.ai/en/stable/). ## Next Step Now, you can have fun with Qwen2 models. Would love to know more about its usages? Feel free to check other documents in this documentation. --- # Unknown Quickstart ========== This guide helps you quickly start using Qwen1.5. We provide examples of \`Hugging Face Transformers \`\_\_ as well as \`ModelScope \`\_\_, and \`vLLM \`\_\_ for deployment. Hugging Face Transformers & ModelScope -------------------------------------- To get a quick start with Qwen1.5, we advise you to try with the inference with \`\`transformers\`\` first. Make sure that you have installed \`\`transformers>=4.37.0\`\`. The following is a very simple code snippet showing how to run Qwen1.5-Chat, with an example of Qwen1.5-7B-Chat: .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" # the device to load the model onto # Now you do not need to add "trust\_remote\_code=True" model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat", torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat") # Instead of using model.chat(), we directly use model.generate() # But you need to use tokenizer.apply\_chat\_template() to format your inputs as shown below prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device) # Directly use generate() and tokenizer.decode() to get the output. # Use \`max\_new\_tokens\` to control the maximum output length. generated\_ids = model.generate( model\_inputs.input\_ids, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] Previously, we use \`\`model.chat()\`\` (see \`\`modeling\_qwen.py\`\` in previous Qwen models for more information). Now, we follow the practice of \`\`transformers\`\` and directly use \`\`model.generate()\`\` with \`\`apply\_chat\_template()\`\` in tokenizer. If you would like to apply Flash Attention 2, you can load the model as shown below: .. code:: python model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat", torch\_dtype="auto", device\_map="auto", attn\_implementation="flash\_attention\_2", ) To tackle with downloading issues, we advise you to try with from ModelScope, just changing the first line of code above to the following: .. code:: python from modelscope import AutoModelForCausalLM, AutoTokenizer Streaming mode for model chat is simple with the help of \`\`TextStreamer\`\`. Below we show you an example of how to use it: .. code:: python ... # Reuse the code before \`model.generate()\` in the last code snippet from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) generated\_ids = model.generate( model\_inputs.input\_ids, max\_new\_tokens=512, streamer=streamer, ) vLLM for Deployment ------------------- To deploy Qwen1.5, we advise you to use vLLM. vLLM is a fast and easy-to-use framework for LLM inference and serving. In the following, we demonstrate how to build a OpenAI-API compatible API service with vLLM. First, make sure you have installed \`\`vLLM>=0.3.0\`\`: .. code:: bash pip install vllm Run the following code to build up a vllm service. Here we take Qwen1.5-7B-Chat as an example: .. code:: bash python -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen1.5-7B-Chat Then, you can use the \`create chat interface \`\_\_ to communicate with Qwen: .. code:: bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen1.5-7B-Chat", "messages": \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], }' or you can use python client with \`\`openai\`\` python package as shown below: .. code:: python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen1.5-7B-Chat", messages=\[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \] ) print("Chat response:", chat\_response) Next Step --------- Now, you can have fun with Qwen models. Would love to know more about its usages? Feel free to check other documents in this documentation. --- # Unknown \# Quickstart This guide helps you quickly start using Qwen3. We provide examples of \[Hugging Face Transformers\](https://github.com/huggingface/transformers) as well as \[ModelScope\](https://github.com/modelscope/modelscope), and \[vLLM\](https://github.com/vllm-project/vllm) and \[SGLang\](https://github.com/sgl-project/sglang) for deployment. You can find Qwen3 models in \[the Qwen3 collection\](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) at Hugging Face Hub and \[the Qwen3 collection\](https://www.modelscope.cn/collections/Qwen3-9743180bdc6b48) at ModelScope. ## Transformers To get a quick start with Qwen3, you can try the inference with \`transformers\` first. Make sure that you have installed \`transformers>=4.51.0\`. We advise you to use Python 3.10 or higher, and PyTorch 2.6 or higher. :::::{tab-set} :sync-group: model ::::{tab-item} Qwen3-Instruct-2507 :sync: instruct :::{important} Qwen3-Instruct-2507 supports \*\*only non-thinking mode\*\* and \*\*does not generate \`\`\`\` blocks\*\* in its output. Different from Qwen3-2504, \*\*specifying \`enable\_thinking=False\` is no longer required or supported\*\*. ::: The following contains a code snippet illustrating how to use Qwen3-235B-A22B-Instruct-2507 to generate content based on given inputs. \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-235B-A22B-Instruct-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from\_pretrained(model\_name) model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = \[ {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # conduct text completion generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=16384 ) output\_ids = generated\_ids\[0\]\[len(model\_inputs.input\_ids\[0\]):\].tolist() content = tokenizer.decode(output\_ids, skip\_special\_tokens=True) print("content:", content) \`\`\` :::{Note} We recommend \`temperature=0.7\`, \`top\_p=0.8\`, \`top\_k=20\`, and \`min\_p=0\` for Qwen3-Instruct-2507 models. For supported frameworks, adjust \`presence\_penalty\` between 0 and 2 to reduce repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance. ::: :::{Note} Qwen3-Instruct-2507 may use CoT (chain-of-thoughts) automatically for complex tasks. We recommend using an output length of 16,384 tokens for most queries. ::: :::: ::::{tab-item} Qwen3-Thinking-2507 :sync: thinking :::{important} Qwen3-Thinking-2507 supports \*\*only thinking mode\*\*. Additionally, to enforce model thinking, the default chat template automatically includes \`\`. Therefore, it is normal for the model's output to contain only \`\` without an explicit opening \`\` tag. ::: The following contains a code snippet illustrating how to use Qwen3-235B-A22B-Thinking-2507 to generate content based on given inputs. \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-235B-A22B-Thinking-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from\_pretrained(model\_name) model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = \[ {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # conduct text completion generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=32768 ) output\_ids = generated\_ids\[0\]\[len(model\_inputs.input\_ids\[0\]):\].tolist() # parsing thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) # no opening tag print("content:", content) \`\`\` :::{note} We recommend \`temperature=0.6\`, \`top\_p=0.95\`, \`top\_k=20\`, and \`min\_p=0\` for Qwen3-Thinking-2507 models. For supported frameworks, adjust \`presence\_penalty\` between 0 and 2 to reduce repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance. ::: :::{note} Qwen3-Thinking-2507 features increased thinking depth. We strongly recommend its use in highly complex reasoning tasks with adequate maximum generation length. ::: :::: ::::{tab-item} Qwen3 :sync: hybrid The following is a very simple code snippet showing how to run Qwen3-8B: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # conduct text completion generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=32768 ) output\_ids = generated\_ids\[0\]\[len(model\_inputs.input\_ids\[0\]):\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` Qwen3 will think before respond, similar to QwQ models. This means the model will use its reasoning abilities to enhance the quality of generated responses. The model will first generate thinking content wrapped in a \`...\` block, followed by the final response. - Hard Switch: To strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5-Instruct models, you can set \`enable\_thinking=False\` when formatting the text. \`\`\`python text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=False, # Setting enable\_thinking=False disables thinking mode ) \`\`\` It can be particularly useful in scenarios where disabling thinking is essential for enhancing efficiency. - Soft Switch: Qwen3 also understands the user's instruction on its thinking behavior, in particular, the soft switch \`/think\` and \`/no\_think\`. You can add them to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations. :::{note} For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in \`generation\_config.json\`). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. ::: :::: ::::: ## ModelScope To tackle with downloading issues, we advise you to try \[ModelScope\](https://github.com/modelscope/modelscope). Before starting, you need to install \`modelscope\` with \`pip\`. \`modelscope\` adopts a programmatic interface similar (but not identical) to \`transformers\`. For basic usage, you can simply change the first line of code above to the following: \`\`\`python from modelscope import AutoModelForCausalLM, AutoTokenizer \`\`\` For more information, please refer to \[the documentation of \`modelscope\`\](https://www.modelscope.cn/docs). ## OpenAI API Compatibility You can serve Qwen3 via OpenAI-compatible APIs using frameworks such as vLLM, SGLang, and interact with the API using common HTTP clients or the OpenAI SDKs. :::::{tab-set} :sync-group: model ::::{tab-item} Qwen3-Instruct-2507 :sync: instruct Here we take Qwen3-235B-A22B-Instruct-2507 as an example to start the API: - SGLang (\`sglang>=0.4.6.post1\` is required): \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tp 8 --context-length 262144 \`\`\` - vLLM (\`vllm>=0.9.0\` is recommended): \`\`\`shell vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 \`\`\` :::{note} Consider adjusting the context length according to the available GPU memory. ::: :::: ::::{tab-item} Qwen3-Thinking-2507 :sync: thinking Here we take Qwen3-235B-A22B-Thinking-2507 as an example to start the API: - SGLang (\`sglang>=0.4.6.post1\` is required): \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-235B-A22B-Thinking-2507 --port 8000 --tp 8 --context-length 262144 --reasoning-parser deepseek-r1 \`\`\` - vLLM (\`vllm>=0.9.0\` is recommended): \`\`\`shell vllm serve Qwen/Qwen3-235B-A22B-Thinking-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` :::{note} Consider adjusting the context length according to the available GPU memory. ::: :::{important} We are currently working on adapting the \`qwen3\` reasoning parsers to the new behavior. Please follow the command above at the moment. ::: :::: ::::{tab-item} Qwen3 :sync: hybrid Here we take Qwen3-8B as an example to start the API: - SGLang (\`sglang>=0.4.6.post1\` is required): \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --port 8000 --reasoning-parser qwen3 \`\`\` - vLLM (\`vllm>=0.9.0\` is recommended): \`\`\`shell vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser qwen3 \`\`\` :::: ::::: Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::::{tab-set} :sync-group: model :::::{tab-item} Qwen3-Instruct-2507 :sync: instruct Here we show the basic command to interact with the chat completion API using Qwen3-235B-A22B-Instruct-2507. ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-235B-A22B-Instruct-2507", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 16384 }' \`\`\` ::: :::{tab-item} Python :sync: python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Instruct-2507", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=16384, temperature=0.7, top\_p=0.8, extra\_body={ "top\_k": 20, } ) print("Chat response:", chat\_response) \`\`\` :::: ::::: :::::{tab-item} Qwen3-Thinking-2507 :sync: thinking Here we show the basic command to interact with the chat completion API using Qwen3-235B-A22B-Thinking-2507. ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-235B-A22B-Thinking-2507", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python :sync: python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, } ) print("Chat response:", chat\_response) \`\`\` :::: ::::: :::::{tab-item} Qwen3 :sync: hybrid Here we show the basic command to interact with the chat completion API using Qwen3-8B. The default is with thinking enabled: ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python :sync: python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, } ) print("Chat response:", chat\_response) \`\`\` ::: :::: To disable thinking, one could use the soft switch (e.g., appending \`/nothink\` to the user query). The hard switch can also be used as follows: ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python :sync: python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": False}, } ) print("Chat response:", chat\_response) \`\`\` ::: ::::: :::::: For more usage, please refer to our document on \[SGLang\](../deployment/sglang) and \[vLLM\](../deployment/vllm). ## Thinking Budget Qwen3 supports the configuration of thinking budget. It is achieved by ending the thinking process once the budget is reached and guiding the model to generate the "summary" with an early-stopping prompt. Since this feature involves customization specific to each model, it is currently not available in the open-source frameworks and only implemented by \[the Alibaba Cloud Model Studio API\](https://www.alibabacloud.com/help/en/model-studio/deep-thinking#6f0633b9cdts1). However, with existing open-source frameworks, one can generate twice to implement this feature as follows: 1. For the first time, generate tokens up to the thinking budget and check if the thinking process is finished. If the thinking process is not finished, append the early-stopping prompt. 2. For the second time, continue generation until the end of the content or the upper length limit is fulfilled. The following snippet shows the implementation with Hugging Face Transformers: \`\`\`python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" thinking\_budget = 16 max\_new\_tokens = 32768 # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) input\_length = model\_inputs.input\_ids.size(-1) # first generation until thinking budget generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=thinking\_budget ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # check if the generation has already finished (151645 is <|im\_end|>) if 151645 not in output\_ids: # check if the thinking process has finished (151668 is ) # and prepare the second model input if 151668 not in output\_ids: print("thinking budget is reached") early\_stopping\_text = "\\n\\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\\n\\n\\n" early\_stopping\_ids = tokenizer(\[early\_stopping\_text\], return\_tensors="pt", return\_attention\_mask=False).input\_ids.to(model.device) input\_ids = torch.cat(\[generated\_ids, early\_stopping\_ids\], dim=-1) else: input\_ids = generated\_ids attention\_mask = torch.ones\_like(input\_ids, dtype=torch.int64) # second generation generated\_ids = model.generate( input\_ids=input\_ids, attention\_mask=attention\_mask, max\_new\_tokens=input\_length + max\_new\_tokens - input\_ids.size(-1) # could be negative if max\_new\_tokens is not large enough (early stopping text is 24 tokens) ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` You should see the output in the console like the following \`\`\`text thinking budget is reached thinking content: Okay, the user is asking for a short introduction to large language models Considering the limited time by the user, I have to give the solution based on the thinking directly now. content: Large language models (LLMs) are advanced artificial intelligence systems trained on vast amounts of text data to understand and generate human-like language. They can perform tasks such as answering questions, writing stories, coding, and translating languages. LLMs are powered by deep learning techniques and have revolutionized natural language processing by enabling more context-aware and versatile interactions with text. Examples include models like GPT, BERT, and others developed by companies like OpenAI and Alibaba. \`\`\` :::{note} For purpose of demonstration only, \`thinking\_budget\` is set to 16. However, \`thinking\_budget\` should not be set to that low in practice. We recommend tuning \`thinking\_budget\` based on the latency users can accept and setting it higher than 1024 for meaningful improvements across tasks. If thinking is not desired at all, developers should make use of the hard switch instead. ::: ## Next Step Now, you can have fun with Qwen3 models. Would love to know more about its usage? Feel free to check other documents in this documentation. --- # Unknown Using Transformers to Chat ========================== The most significant but also the simplest usage of Qwen1.5 is to chat with it using the \`\`transformers\`\` library. In this document, we show how to chat with \`\`Qwen1.5-7B-Chat\`\`, in either streaming mode or not. Basic Usage ----------- You can just write several lines of code with \`\`transformers\`\` to chat with Qwen1.5-Chat. Essentially, we build the tokenizer and the model with \`\`from\_pretrained\`\` method, and we use \`\`generate\`\` method to perform chatting with the help of chat template provided by the tokenizer. Below is an example of how to chat with Qwen1.5-7B-Chat: .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" # the device to load the model onto # Now you do not need to add "trust\_remote\_code=True" model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat", torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat") # Instead of using model.chat(), we directly use model.generate() # But you need to use tokenizer.apply\_chat\_template() to format your inputs as shown below prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device) # Directly use generate() and tokenizer.decode() to get the output. # Use \`max\_new\_tokens\` to control the maximum output length. generated\_ids = model.generate( model\_inputs.input\_ids, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] If you would like to apply Flash Attention 2, you can load the model as shown below: .. code:: python model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat", torch\_dtype="auto", device\_map="auto", attn\_implementation="flash\_attention\_2", ) Note that the previous method in the original Qwen repo \`\`chat()\`\` is now replaced by \`\`generate()\`\`. The \`\`apply\_chat\_template()\`\` function is used to convert the messages into a format that the model can understand. The \`\`add\_generation\_prompt\`\` argument is used to add a generation prompt, which refers to \`\`<|im\_start|>assistant\\n\`\` to the input. Notably, we apply ChatML template for chat models following our previous practice. The \`\`max\_new\_tokens\`\` argument is used to set the maximum length of the response. The \`\`tokenizer.batch\_decode()\`\` function is used to decode the response. In terms of the input, the above \`\`messages\`\` is an example to show how to format your dialog history and system prompt. By default, if you do not specify system prompt, we directly use \`\`You are a helpful assistant.\`\`. Streaming Mode -------------- With the help of \`\`TextStreamer\`\`, you can modify your chatting with Qwen to streaming mode. Below we show you an example of how to use it: .. code:: python # Repeat the code above before model.generate() # Starting here, we add streamer for text generation. from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) # This will print the output in the streaming mode. generated\_ids = model.generate( model\_inputs, max\_new\_tokens=512, streamer=streamer, ) Besides using \`\`TextStreamer\`\`, we can also use \`\`TextIteratorStreamer\`\` which stores print-ready text in a queue, to be used by a downstream application as an iterator: .. code:: python # Repeat the code above before model.generate() # Starting here, we add streamer for text generation. from transformers import TextIteratorStreamer streamer = TextIteratorStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) from threading import Thread generation\_kwargs = dict(model\_inputs, streamer=streamer, max\_new\_tokens=512) thread = Thread(target=model.generate, kwargs=generation\_kwargs) thread.start() generated\_text = "" for new\_text in streamer: generated\_text += new\_text print(generated\_text) Next Step --------- Now you can chat with Qwen1.5 in either streaming mode or not. Continue to read the documentation and try to figure out more advanced usages of model inference! --- # Unknown \# Quickstart This guide helps you quickly start using Qwen2.5. We provide examples of \[Hugging Face Transformers\](https://github.com/huggingface/transformers) as well as \[ModelScope\](https://github.com/modelscope/modelscope), and \[vLLM\](https://github.com/vllm-project/vllm) for deployment. You can find Qwen2.5 models in the \[Qwen2.5 collection\](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e) at Hugging Face Hub. ## Hugging Face Transformers & ModelScope To get a quick start with Qwen2.5, we advise you to try with the inference with \`transformers\` first. Make sure that you have installed \`transformers>=4.37.0\`. We advise you to use Python 3.10 or higher, and PyTorch 2.3 or higher. :::{dropdown} Install \`transformers\` \* Install with \`pip\`: \`\`\`bash pip install transformers -U \`\`\` \* Install with \`conda\`: \`\`\`bash conda install conda-forge::transformers \`\`\` \* Install from source: \`\`\`bash pip install git+https://github.com/huggingface/transformers \`\`\` ::: The following is a very simple code snippet showing how to run Qwen2.5-7B-Instruct: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` As you can see, it's just standard usage for casual LMs in \`transformers\`! ### Streaming Generation Streaming mode for model chat is simple with the help of \`TextStreamer\`. Below we show you an example of how to use it: \`\`\`python ... # Reuse the code before \`model.generate()\` in the last code snippet from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, streamer=streamer, ) \`\`\` It will print the text to the console or the terminal as being generated. ### ModelScope To tackle with downloading issues, we advise you to try \[ModelScope\](https://github.com/modelscope/modelscope). Before starting, you need to install \`modelscope\` with \`pip\`. \`modelscope\` adopts a programmatic interface similar (but not identical) to \`transformers\`. For basic usage, you can simply change the first line of code above to the following: \`\`\`python from modelscope import AutoModelForCausalLM, AutoTokenizer \`\`\` For more information, please refer to \[the documentation of \`modelscope\`\](https://www.modelscope.cn/docs). ## vLLM for Deployment To deploy Qwen2.5, we advise you to use vLLM. vLLM is a fast and easy-to-use framework for LLM inference and serving. In the following, we demonstrate how to build a OpenAI-API compatible API service with vLLM. First, make sure you have installed \`vllm>=0.4.0\`: \`\`\`bash pip install vllm \`\`\` Run the following code to build up a vLLM service. Here we take Qwen2.5-7B-Instruct as an example: \`\`\`bash python -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen2.5-7B-Instruct \`\`\` with \`vllm>=0.5.3\`, you can also use \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct \`\`\` Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use Python client with \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` For more information, please refer to \[the documentation of \`vllm\`\](https://docs.vllm.ai/en/stable/). ## Next Step Now, you can have fun with Qwen2.5 models. Would love to know more about its usages? Feel free to check other documents in this documentation. --- # Unknown llama.cpp =========================== \`llama.cpp \`\_\_ is a C++ library for LLM inference with mimimal setup. It enables running Qwen on your local machine. It is a plain C/C++ implementation without dependencies, and it has AVX, AVX2 and AVX512 support for x86 architectures. It provides 2, 3, 4, 5, 6, and 8-bit quantization for faster inference and reduced memory footprint. CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity is also supported. Essentially, the usage of llama.cpp is to run the GGUF (GPT-Generated Unified Format ) models. For more information, please refer to the official GitHub repo. Here we demonstrate how to run Qwen with llama.cpp. Prerequisites ------------- This example is for the usage on Linux or MacOS. For the first step, clone the repo and enter the directory: .. code:: bash git clone https://github.com/ggerganov/llama.cpp cd llama.cpp Then use \`\`make\`\`: .. code:: bash make Then you can run GGUF files with \`\`llama.cpp\`\`. Running Qwen GGUF Files ----------------------- We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with \`\`-GGUF\`\`. Download the GGUF model that you want with \`\`huggingface-cli\`\` (you need to install it first with \`\`pip install huggingface\_hub\`\`): .. code:: bash huggingface-cli download \--local-dir \--local-dir-use-symlinks False for example: .. code:: bash huggingface-cli download Qwen/Qwen1.5-7B-Chat-GGUF qwen1\_5-7b-chat-q5\_k\_m.gguf --local-dir . --local-dir-use-symlinks False Then you can run the model with the following command: .. code:: bash ./main -m qwen1\_5-7b-chat-q5\_k\_m.gguf -n 512 --color -i -cml -f prompts/chat-with-qwen.txt where \`\`-n\`\` refers to the maximum number of tokens to generate. There are other hyperparameters for you to choose and you can run .. code:: bash ./main -h to figure them out. Make Your GGUF Files -------------------- We introduce the method of creating and quantizing GGUF files in \`quantization/llama.cpp <../quantization/gguf.html>\`\_\_. You can refer to that document for more information. Perplexity Evaluation --------------------- \`\`llama.cpp\`\` provides methods for us to evaluate the perplexity performance of the GGUF models. To do this, you need to prepare the dataset, say "wiki test". Here we demonstrate an example to run the test. For the first step, download the dataset: .. code:: bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip Then you can run the test with the following command: .. code:: bash ./perplexity -m models/7B/ggml-model-q4\_0.gguf -f wiki.test.raw where the output is like .. code:: text perplexity : calculating perplexity over 655 chunks 24.43 seconds per pass - ETA 4.45 hours \[1\]4.5970,\[2\]5.1807,\[3\]6.0382,... Wait for some time and you will get the perplexity of the model. Use GGUF with LM Studio ----------------------- If you still find it difficult to use \`\`llama.cpp\`\`, I advise you to play with \`LM Studio \`\_\_, which is a platform for your to search and run local LLMs. Qwen1.5 has already been officially part of LM Studio. Have fun! --- # Unknown \# Key Concepts ## Qwen Qwen (Chinese: 通义千问; pinyin: \_Tongyi Qianwen\_) is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Both language models and multimodal models are pre-trained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. There is the proprietary version hosted exclusively at \[Alibaba Cloud \\\[zh\\\]\](https://help.aliyun.com/zh/model-studio/developer-reference/tongyi-qianwen-llm/) and the open-weight version. The spectrum for the open-weight models spans over - Qwen: the language models - \[Qwen\](https://github.com/QwenLM/Qwen): 1.8B, 7B, 14B, and 72B models - \[Qwen1.5\](https://github.com/QwenLM/Qwen1.5/tree/v1.5): 0.5B, 1.8B, 4B, 14BA2.7B, 7B, 14B, 32B, 72B, and 110B models - \[Qwen2\](https://github.com/QwenLM/Qwen2): 0.5B, 1.5B, 7B, 57A14B, and 72B models - Qwen-VL: the vision-language models - \[Qwen-VL\](https://github.com/QwenLM/Qwen-VL): 7B-based models - Qwen-Audio: the audio-language models - \[Qwen-Audio\](https://github.com/QwenLM/Qwen-Audio): 7B-based models - \[Qwen2-Audio\](https://github.com/QwenLM/Qwen2-Audio): 7B-based models - CodeQwen: the language models for coding - \[CodeQwen1.5\](https://github.com/QwenLM/CodeQwen1.5): 7B models \*\*In this document, our focus is Qwen, the language models.\*\* ## Causal Language Models Causal language models, also known as autoregressive language models or decoder-only language models, are a type of machine learning model designed to predict the next token in a sequence based on the preceding tokens. In other words, they generate text one token at a time, using the previously generated tokens as context. The "causal" aspect refers to the fact that the model only considers the past context (the already generated tokens) when predicting the next token, not any future tokens. Causal language models are widely used for various natural language processing tasks involving text completion and generation. They have been particularly successful in generating coherent and contextually relevant text, making them a cornerstone of modern natural language understanding and generation systems. \*\*Takeaway: Qwen models are causal language models suitable for text completion.\*\* :::{dropdown} Learn more about language models They are three main kinds of models that are commonly referred to as language models in deep learning: - Sequence-to-sequence models: T5 and the likes Sequence-to-sequence models use both an encoder to capture the entire input sequence and a decoder to generate an output sequence. They are widely used for tasks like machine translation, text summarization, etc. - Bidirectional models or encoder-only models: BERT and the likes Bidirectional models can access both past and future context in a sequence during training. They cannot generate sequential outputs in real-time due to the need for future context. They are widely used as embedding models and subsequently used for text classification. - Casual language models or decoder-only models: GPT and the likes Causal language models operate unidirectionally in a strictly forward direction, predicting each subsequent word based only on the previous words in the sequence. This unidirectional nature ensures that the model's predictions do not rely on future context, making them suitable for tasks like text completion and generation. ::: ### Pre-training & Base models Base language models are foundational models trained on extensive corpora of text to predict the next word in a sequence. Their main goal is to capture the statistical patterns and structures of language, enabling them to generate coherent and contextually relevant text. These models are versatile and can be adapted to various natural language processing tasks through fine-tuning. While adept at producing fluent text, they may require in-context learning or additional training to follow specific instructions or perform complex reasoning tasks effectively. For Qwen models, the base models are those without "-Instruct" indicators, such as Qwen2-7B and Qwen2-72B. \*\*Takeaway: Use base models for in-context learning, downstream fine-tuning, etc.\*\* ### Post-training & Instruction-tuned models Instruction-tuned language models are specialized models designed to understand and execute specific instructions in conversational styles. These models are fine-tuned to interpret user commands accurately and can perform tasks such as summarization, translation, and question answering with improved accuracy and consistency. Unlike base models, which are trained on large corpora of text, instruction-tuned models undergo additional training using datasets that contain examples of instructions and their desired outcomes, often in multiple turns. This kind of training makes them ideal for applications requiring targeted functionalities while maintaining the ability to generate fluent and coherent text. For Qwen models, the instruction-tuned models are those with the "-Instruct" suffix, such as Qwen2-7B-Instruct and Qwen2-72B-Instruct. \[^instruct-chat\] \*\*Takeaway: Use instruction-tuned models for conducting tasks in conversations, downstream fine-tuning, etc.\*\* \[^instruct-chat\]: Previously, they are known as the chat models and with the "-Chat" suffix. Starting from Qwen2, the name is changed to follow the common practice. For Qwen, "-Instruct" and "-Chat" should be regarded as synonymous. ## Tokens & Tokenization Tokens represent the fundamental units that models process and generate. They can represent texts in human languages (regular tokens) or represent specific functionality like keywords in programming languages (control tokens \[^special\]). Typically, a tokenizer is used to split text into regular tokens, which can be words, subwords, or characters depending on the specific tokenization scheme employed, and furnish the token sequence with control tokens as needed. The vocabulary size, or the total number of unique tokens a model recognizes, significantly impacts its performance and versatility. Larger language models often use sophisticated tokenization methods to handle the vast diversity of human language while keeping the vocabulary size manageable. Qwen use a relatively large vocabulary of 151,646 tokens in total. \[^special\]: Control tokens can be called special tokens. However, the meaning of special tokens need to be interpreted based on the contexts: special tokens may contain extra regular tokens. \*\*Takeaway: Tokenization method and vocabulary size is important.\*\* ### Byte-level Byte Pair Encoding Qwen adopts a subword tokenization method called Byte Pair Encoding (BPE), which attempts to learn the composition of tokens that can represent the text with the fewest tokens. For example, the string " tokenization" is decomposed as " token" and "ization" (note that the space is part of the token). Especially, the tokenization of Qwen ensures that there is no unknown words and all texts can be transformed to token sequences. There are 151,643 tokens as a result of BPE in the vocabulary of Qwen, which is a large vocabulary efficient for diverse languages. As a rule of thumb, 1 token is 3~4 characters for English texts and 1.5~1.8 characters for Chinese texts. \*\*Takeaway: Qwen processes texts in subwords and there are no unknown words.\*\* :::{dropdown} Learn more about tokenization in Qwen Qwen uses byte-level BPE (BBPE) on UTF-8 encoded texts. It starts by treating each byte as a token and then iteratively merges the most frequent pairs of tokens occurring the texts into larger tokens until the desired vocabulary size is met. In byte-level BPE, minimum 256 tokens are needed to tokenize every piece of text and avoid the out of vocabulary (OOV) problem. In comparison, character-level BPE needs every Unicode character in its vocabulary to avoid OOV and the Unicode Standard contains 154,998 characters as of Unicode Version 16.0. One limitation to keep in mind for byte-level BPE is that the individual tokens in the vocabulary may not be seemingly semantically meaningful or even valid UTF-8 byte sequences, and in certain aspects, they should be viewed as a text compression scheme. ::: ### Control Tokens & Chat Template Control tokens and chat templates both serve as mechanisms to guide the model's behavior and outputs. Control tokens are special tokens inserted into the sequence that signifies meta information. For example, in pre-training, multiple documents may be packed into a single sequence. For Qwen, the control token "<|endoftext|>" is inserted after each document to signify that the document has ended and a new document will proceed. Chat templates provide a structured format for conversational interactions, where predefined placeholders or prompts are used to elicit responses from the model that adhere to a desired dialogue flow or context. Different models may use different kinds of chat template to format the conversations. It is crucial to use the designated one to ensure the precise control over the LLM's generation process. Qwen uses the following format (ChatML\[^chatml\]), making use of control tokens to format each turn in the conversations \`\`\`text <|im\_start|>{{role}} {{content}}<|im\_end|> \`\`\` The user input take the role of \`user\` and the model generation takes the role of \`assistant\`. Qwen also supports the meta message that instruct the model to perform specific actions or generate text with certain characteristics, such as altering tone, style, or content, which takes the role of \`system\` and the content defaults to "You are a helpful assistant." The following is a full example: \`\`\`text <|im\_start|>system You are a helpful assistant.<|im\_end|> <|im\_start|>user hello<|im\_end|> <|im\_start|>assistant Hello! How can I assist you today? Let me know if you have any questions or need help with something. I'm here to help!<|im\_end|> <|im\_start|>user Explain large language models like I'm 5.<|im\_end|> <|im\_start|>assistant Sure, I'd be happy to explain large language models in a simple way! Imagine that you have a friend who is really good at playing with words. This friend has read so many books and talked to so many people that they know how to use words in all sorts of different ways. They can tell stories, answer questions, and even make up new words if they need to. Now imagine that this friend is actually a computer program, called a "large language model". It's been trained on lots and lots of text, like books, articles, and conversations, so it knows how to use words just like your word-loving friend does. When you ask the model a question or give it a task, it uses all the knowledge it's learned to come up with a response that makes sense. Just like your friend might sometimes make mistakes or say things in a funny way, the large language model isn't perfect either. But it's still really impressive, because it can understand and generate human language in a way that was once thought impossible for machines!<|im\_end|><|endoftext|> \`\`\` There are 3 control tokens in the vocabulary of Qwen, making the vocabulary size totaling 151,646. \*\*Takeaway: Qwen uses ChatML with 3 control tokens for chat template.\*\* \[^chatml\]: For historical reference only, ChatML is first described by the OpenAI Python SDK. The last available version is \[this\](https://github.com/openai/openai-python/blob/v0.28.1/chatml.md). Please also be aware that that document lists use cases intended for OpenAI models. For Qwen2 models, please only use as in our guide. ## Length Limit As Qwen models are causal language models, in theory there is only one length limit of the entire sequence. However, since there is often packing in training and each sequence may contain multiple individual pieces of texts. \*\*How long the model can generate or complete ultimately depends on the use case and in that case how long each document (for pre-training) or each turn (for post-training) is in training.\*\* For Qwen2, the packed sequence length in training is 32,768 tokens.\[^yarn\] The maximum document length in pre-training is this length. The maximum message length for user and assistant is different in post-training. In general, the assistant message could be up to 2048 tokens and for tasks with less variation like tables to HTML, it could be 6-8K tokens. \[^yarn\]: The sequence length can be extended to 131,072 tokens for Qwen2-7B and Qwen2-72B models with YaRN. Please refer to the model card on how to enable YaRN in vLLM. \*\*Takeaway: Qwen2 models can process texts of 32K or 128K tokens but not all of them can be output.\*\* --- # Unknown \# Quickstart This guide helps you quickly start using Qwen3. We provide examples of \[Hugging Face Transformers\](https://github.com/huggingface/transformers) as well as \[ModelScope\](https://github.com/modelscope/modelscope), and \[vLLM\](https://github.com/vllm-project/vllm) for deployment. You can find Qwen3 models in \[the Qwen3 collection\](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) at Hugging Face Hub and \[the Qwen3 collection\](https://www.modelscope.cn/collections/Qwen3-9743180bdc6b48) at ModelScope. ## Transformers To get a quick start with Qwen3, you can try the inference with \`transformers\` first. Make sure that you have installed \`transformers>=4.51.0\`. We advise you to use Python 3.10 or higher, and PyTorch 2.6 or higher. The following is a very simple code snippet showing how to run Qwen3-8B: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # conduct text completion generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=32768 ) output\_ids = generated\_ids\[0\]\[len(model\_inputs.input\_ids\[0\]):\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` Qwen3 will think before respond, similar to QwQ models. This means the model will use its reasoning abilities to enhance the quality of generated responses. The model will first generate thinking content wrapped in a \`...\` block, followed by the final response. - Hard Switch: To strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5-Instruct models, you can set \`enable\_thinking=False\` when formatting the text. \`\`\`python text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=False, # Setting enable\_thinking=False disables thinking mode ) \`\`\` It can be particularly useful in scenarios where disabling thinking is essential for enhancing efficiency. - Soft Switch: Qwen3 also understands the user's instruction on its thinking behavior, in particular, the soft switch \`/think\` and \`/no\_think\`. You can add them to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations. :::{note} For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in \`generation\_config.json\`). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section. For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. ::: ## ModelScope To tackle with downloading issues, we advise you to try \[ModelScope\](https://github.com/modelscope/modelscope). Before starting, you need to install \`modelscope\` with \`pip\`. \`modelscope\` adopts a programmatic interface similar (but not identical) to \`transformers\`. For basic usage, you can simply change the first line of code above to the following: \`\`\`python from modelscope import AutoModelForCausalLM, AutoTokenizer \`\`\` For more information, please refer to \[the documentation of \`modelscope\`\](https://www.modelscope.cn/docs). ## OpenAI API Compatibility You can serve Qwen3 via OpenAI-compatible APIs using frameworks such as vLLM, SGLang, and interact with the API using common HTTP clients or the OpenAI SDKs. Here we take Qwen3-8B as an example to start the API: - SGLang (\`sglang>=0.4.6.post1\` is required): \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --port 8000 --reasoning-parser qwen3 \`\`\` - vLLM (\`vllm>=0.8.5\` is recommended): \`\`\`shell vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, } ) print("Chat response:", chat\_response) \`\`\` :::: While the soft switch is always available, the hard switch is also available in the API through the following configuration to the API call. For more usage, please refer to our document on \[SGLang\](../deployment/sglang) and \[vLLM\](../deployment/vllm). ## Thinking Budget Qwen3 supports the configuration of thinking budget. It is achieved by ending the thinking process once the budget is reached and guiding the model to generate the "summary" with an early-stopping prompt. Since this feature involves customization specific to each model, it is currently not available in the open-source frameworks and only implemented by \[the Alibaba Cloud Model Studio API\](https://www.alibabacloud.com/help/en/model-studio/deep-thinking#6f0633b9cdts1). However, with existing open-source frameworks, one can generate twice to implement this feature as follows: 1. For the first time, generate tokens up to the thinking budget and check if the thinking process is finished. If the thinking process is not finished, append the early-stopping prompt. 2. For the second time, continue generation until the end of the content or the upper length limit is fulfilled. The following snippet shows the implementation with Hugging Face Transformers: \`\`\`python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" thinking\_budget = 16 max\_new\_tokens = 32768 # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) input\_length = model\_inputs.input\_ids.size(-1) # first generation until thinking budget generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=thinking\_budget ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # check if the generation has already finished (151645 is <|im\_end|>) if 151645 not in output\_ids: # check if the thinking process has finished (151668 is ) # and prepare the second model input if 151668 not in output\_ids: print("thinking budget is reached") early\_stopping\_text = "\\n\\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\\n\\n\\n" early\_stopping\_ids = tokenizer(\[early\_stopping\_text\], return\_tensors="pt", return\_attention\_mask=False).input\_ids.to(model.device) input\_ids = torch.cat(\[generated\_ids, early\_stopping\_ids\], dim=-1) else: input\_ids = generated\_ids attention\_mask = torch.ones\_like(input\_ids, dtype=torch.int64) # second generation generated\_ids = model.generate( input\_ids=input\_ids, attention\_mask=attention\_mask, max\_new\_tokens=input\_length + max\_new\_tokens - input\_ids.size(-1) # could be negative if max\_new\_tokens is not large enough (early stopping text is 24 tokens) ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` You should see the output in the console like the following \`\`\`text thinking budget is reached thinking content: Okay, the user is asking for a short introduction to large language models Considering the limited time by the user, I have to give the solution based on the thinking directly now. content: Large language models (LLMs) are advanced artificial intelligence systems trained on vast amounts of text data to understand and generate human-like language. They can perform tasks such as answering questions, writing stories, coding, and translating languages. LLMs are powered by deep learning techniques and have revolutionized natural language processing by enabling more context-aware and versatile interactions with text. Examples include models like GPT, BERT, and others developed by companies like OpenAI and Alibaba. \`\`\` :::{note} For purpose of demonstration only, \`thinking\_budget\` is set to 16. However, \`thinking\_budget\` should not be set to that low in practice. We recommend tuning \`thinking\_budget\` based on the latency users can accept and setting it higher than 1024 for meaningful improvements across tasks. If thinking is not desired at all, developers should make use of the hard switch instead. ::: ## Next Step Now, you can have fun with Qwen3 models. Would love to know more about its usage? Feel free to check other documents in this documentation. --- # Unknown Ollama =========================== \`Ollama \`\_\_ helps you run LLMs locally with only a few commands. It is available at MacOS, Linux, and Windows. Now, Qwen1.5 is officially on Ollama, and you can run it with one command: .. code:: bash ollama run qwen Next, we introduce more detailed usages of Ollama for running Qwen models. Quickstart ---------- Visit the official website \`Ollama \`\_\_ and click download to install Ollama on your device. You can also search models in the website, where you can find the Qwen1.5 models. Except for the default one, you can choose to run Qwen1.5-Chat models of different sizes by: - \`\`ollama run qwen:0.5b\`\` - \`\`ollama run qwen:1.8b\`\` - \`\`ollama run qwen:4b\`\` - \`\`ollama run qwen:7b\`\` - \`\`ollama run qwen:14b\`\` - \`\`ollama run qwen:72b\`\` Run Ollama with Your GGUF Files ------------------------------- Sometimes you don't want to pull models and you just want to use Ollama with your own GGUF files. Suppose you have a GGUF file of Qwen, \`\`qwen1\_5-7b-chat-q4\_0.gguf\`\`. For the first step, you need to create a file called \`\`Modelfile\`\`. The content of the file is shown below: .. code:: text FROM qwen1\_5-7b-chat-q4\_0.gguf # set the temperature to 1 \[higher is more creative, lower is more coherent\] PARAMETER temperature 0.7 PARAMETER top\_p 0.8 PARAMETER repeat\_penalty 1.05 PARAMETER top\_k 20 TEMPLATE """{{ if and .First .System }}<|im\_start|>system {{ .System }}<|im\_end|> {{ end }}<|im\_start|>user {{ .Prompt }}<|im\_end|> <|im\_start|>assistant {{ .Response }}""" # set the system message SYSTEM """ You are a helpful assistant. """ Then create the ollama model by running: .. code:: bash ollama create qwen7b -f Modelfile Once it is finished, you can run your ollama model by: .. code:: bash ollama run qwen7b --- # Unknown Ollama =========================== \`Ollama \`\_\_ helps you run LLMs locally with only a few commands. It is available at MacOS, Linux, and Windows. Now, Qwen2 is officially on Ollama, and you can run it with one command: .. code:: bash ollama run qwen2 Next, we introduce more detailed usages of Ollama for running Qwen2 models. Quickstart ---------- Visit the official website \`Ollama \`\_\_ and click download to install Ollama on your device. You can also search models in the website, where you can find the Qwen2 models. Except for the default one, you can choose to run Qwen2-Instruct models of different sizes by: - \`\`ollama run qwen2:0.5b\`\` - \`\`ollama run qwen2:1.5b\`\` - \`\`ollama run qwen2:7b\`\` - \`\`ollama run qwen2:72b\`\` .. note:: \`\`ollama\`\` does not host base models. Even though the tag may not have the instruct suffix, they are all instruct models. Run Ollama with Your GGUF Files ------------------------------- Sometimes you don't want to pull models and you just want to use Ollama with your own GGUF files. Suppose you have a GGUF file of Qwen2, \`\`qwen2-7b-instruct-q5\_0.gguf\`\`. For the first step, you need to create a file called \`\`Modelfile\`\`. The content of the file is shown below: .. code:: text FROM qwen2-7b-instruct-q5\_0.gguf # set the temperature to 1 \[higher is more creative, lower is more coherent\] PARAMETER temperature 0.7 PARAMETER top\_p 0.8 PARAMETER repeat\_penalty 1.05 PARAMETER top\_k 20 TEMPLATE """{{ if and .First .System }}<|im\_start|>system {{ .System }}<|im\_end|> {{ end }}<|im\_start|>user {{ .Prompt }}<|im\_end|> <|im\_start|>assistant {{ .Response }}""" # set the system message SYSTEM """ You are a helpful assistant. """ Then create the ollama model by running: .. code:: bash ollama create qwen2\_7b -f Modelfile Once it is finished, you can run your ollama model by: .. code:: bash ollama run qwen2\_7b --- # Unknown Speed Benchmark ========================= This section reports the speed performance of bf16 models, quantized models (including GPTQ-Int4, GPTQ-Int8 and AWQ) of the Qwen2 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under the conditions of different context lengths. The environment of the evaluation with huggingface transformers is: - NVIDIA A100 80GB - CUDA 11.8 - Pytorch 2.1.2+cu118 - Flash Attention 2.3.3 - Transformers 4.38.2 - AutoGPTQ 0.7.1 - AutoAWQ 0.2.4 The environment of the evaluation with vLLM is: - NVIDIA A100 80GB - CUDA 11.8 - Pytorch 2.3.0+cu118 - Flash Attention 2.5.6 - Transformers 4.40.1 - vLLM 0.4.2 Note: - We use the batch size of 1 and the least number of GPUs as possible for the evalution. - We test the speed and memory of generating 2048 tokens with the input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens (\\>32k is only avaliable for Qwen2-72B-Instuct and Qwen2-7B-Instuct). - For vLLM, the memory usage is not reported because it pre-allocates all GPU memory. We use \`\`gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False\`\` by default. - 0.5B (Transformer) +---------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | +=====================+==============+==============+=========+=================+================+ | Qwen2-0.5B-Instruct | 1 | BF16 | 1 | 49.94 | 1.17 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 36.35 | 0.85 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 49.56 | 0.68 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 38.78 | 0.68 | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 1 | 50.83 | 6.42 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 36.56 | 6.09 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 49.63 | 5.93 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 38.73 | 5.92 | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 1 | 49.56 | 13.48 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 36.23 | 13.15 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 48.68 | 12.97 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 38.94 | 12.99 | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 1 | 49.25 | 27.61 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 34.61 | 27.28 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 48.18 | 27.12 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 38.19 | 27.11 | +---------------------+--------------+--------------+---------+-----------------+----------------+ - 0.5B (vLLM) +---------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +=====================+==============+==============+=========+=================+ | Qwen2-0.5B-Instruct | 1 | BF16 | 1 | 270.49 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 235.95 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 240.07 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 233.31 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 256.16 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 224.30 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 226.41 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 222.83 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 108.89 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 108.10 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 106.51 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 104.16 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 97.20 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 94.49 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 93.94 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 92.23 | +---------------------+--------------+--------------+---------+-----------------+ - 1.5B (Transformer) +---------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | +=====================+==============+==============+=========+=================+================+ | Qwen2-1.5B-Instruct | 1 | BF16 | 1 | 40.89 | 3.44 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 31.51 | 2.31 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 42.47 | 1.67 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 33.62 | 1.64 | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 1 | 40.86 | 8.74 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 31.31 | 7.59 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 42.78 | 6.95 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 32.90 | 6.92 | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 1 | 40.08 | 15.92 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 31.19 | 14.79 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 42.25 | 14.14 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 33.24 | 14.12 | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 1 | 34.09 | 30.31 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 28.52 | 29.18 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 31.30 | 28.54 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 32.16 | 28.51 | +---------------------+--------------+--------------+---------+-----------------+----------------+ - 1.5B (vLLM) +---------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +=====================+==============+==============+=========+=================+ | Qwen2-1.5B-Instruct | 1 | BF16 | 1 | 175.55 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 172.28 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 184.58 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 170.87 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 166.23 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 164.32 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 174.04 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 162.81 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 83.67 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 98.63 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 97.65 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 92.48 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 77.69 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 86.42 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 87.49 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 82.88 | +---------------------+--------------+--------------+---------+-----------------+ - 7B (Transformer) +-------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | +===================+==============+==============+=========+=================+================+ | Qwen2-7B-Instruct | 1 | BF16 | 1 | 37.97 | 14.92 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 30.85 | 8.97 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 36.17 | 6.06 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 33.08 | 5.93 | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 1 | 34.74 | 20.26 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 31.13 | 14.31 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 33.34 | 11.40 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 30.86 | 11.27 | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 1 | 26.63 | 27.71 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 24.58 | 21.76 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 25.81 | 18.86 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 27.61 | 18.72 | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 1 | 17.49 | 42.62 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 1 | 16.69 | 36.67 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 17.17 | 33.76 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 17.87 | 33.63 | +-------------------+--------------+--------------+---------+-----------------+----------------+ - 7B (vLLM) +-------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +===================+==============+==============+=========+=================+ | Qwen2-7B-Instruct | 1 | BF16 | 1 | 80.45 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 114.32 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 143.40 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 96.65 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 76.41 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 107.02 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 131.55 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 91.38 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 66.54 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 89.72 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 97.93 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 76.87 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 55.83 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 71.58 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 81.48 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 63.62 | + +--------------+--------------+---------+-----------------+ | | 63488 | BF16 | 1 | 41.20 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 49.37 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 54.12 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 45.89 | + +--------------+--------------+---------+-----------------+ | | 129024 | BF16 | 1 | 25.01 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 27.73 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 29.39 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 27.13 | +-------------------+--------------+--------------+---------+-----------------+ - 57B-A14B (Transformer) +--------------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | +==========================+==============+==============+=========+=================+================+ | Qwen2-57B-A14B-Instruct | 1 | BF16 | 2 | 4.76 | 110.29 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 5.55 | 30.38 | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 2 | 4.90 | 117.80 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 5.44 | 35.67 | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 2 | 4.58 | 128.17 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 5.31 | 43.11 | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 2 | 4.12 | 163.77 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 4.72 | 58.01 | +--------------------------+--------------+--------------+---------+-----------------+----------------+ - 57B-A14B (vLLM) +--------------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +==========================+==============+==============+=========+=================+ | Qwen2-57B-A14B-Instruct | 1 | BF16 | 2 | 31.44 | +--------------------------+--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 2 | 31.77 | +--------------------------+--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 2 | 21.25 | +--------------------------+--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 2 | 20.24 | +--------------------------+--------------+--------------+---------+-----------------+ Note: Compared with dense models, MOE models have larger throughput when batch size is large, which is shown as follows: +--------------------------+--------------+-------------+------+----------+ | Model | Quantization | # Prompts | QPS | Tokens/s | +==========================+==============+=============+======+==========+ | Qwen1.5-32B-Chat | BF16 | 100 | 6.68 | 7343.56 | +--------------------------+--------------+-------------+------+----------+ | Qwen2-57B-A14B-Instruct | BF16 | 100 | 4.81 | 5291.15 | +--------------------------+--------------+-------------+------+----------+ | Qwen1.5-32B-Chat | BF16 | 1000 | 7.99 | 8791.35 | +--------------------------+--------------+-------------+------+----------+ | Qwen2-57B-A14B-Instruct | BF16 | 1000 | 5.18 | 5698.37 | +--------------------------+--------------+-------------+------+----------+ The results are obtained from vLLM throughput benchmarking scripts, which can be reproduced by: \`\`python vllm/benchmarks/benchmark\_throughput.py --input-len 1000 --output-len 100 --model \--num-prompts \--enforce-eager -tp 2\`\` - 72B (Transformer) +--------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | +====================+==============+==============+=========+=================+================+ | Qwen2-72B-Instruct | 1 | BF16 | 2 | 7.45 | 134.74 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 7.30 | 71.00 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 9.05 | 41.80 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 9.96 | 41.31 | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 2 | 5.99 | 144.38 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 5.93 | 80.60 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 6.79 | 47.90 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 7.49 | 47.42 | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 3 | 4.12 | 169.93 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 4.43 | 95.14 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 4.87 | 57.79 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 1 | 5.23 | 57.30 | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 3 | 2.86 | 209.03 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 2.83 | 124.20 | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 3.02 | 107.94 | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 1.85 | 88.60 | +--------------------+--------------+--------------+---------+-----------------+----------------+ - 72B (vLLM) +--------------------+--------------+--------------+---------+-----------------+----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Setting | +====================+==============+==============+=========+=================+================+ | Qwen2-72B-Instruct | 1 | BF16 | 2 | 17.68 | \[Setting 1\] | + + +--------------+---------+-----------------+----------------+ | | | BF16 | 4 | 30.01 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 27.56 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 29.60 | \[Setting 2\] | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 42.82 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 27.73 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 6144 | BF16 | 4 | 27.98 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 25.46 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 1 | 25.16 | \[Setting 3\] | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 38.23 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 25.77 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 14336 | BF16 | 4 | 21.81 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 22.71 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 26.54 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 21.50 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 4 | 19.43 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 18.69 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 23.12 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 18.09 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 30720 | BF16 | 4 | 19.43 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 18.69 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 23.12 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 18.09 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 63488 | BF16 | 4 | 17.46 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 2 | 15.30 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 13.23 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 13.14 | - | + +--------------+--------------+---------+-----------------+----------------+ | | 129024 | BF16 | 4 | 11.70 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int8 | 4 | 12.94 | - | + + +--------------+---------+-----------------+----------------+ | | | GPTQ-Int4 | 2 | 8.33 | - | + + +--------------+---------+-----------------+----------------+ | | | AWQ | 2 | 7.78 | - | +--------------------+--------------+--------------+---------+-----------------+----------------+ \* \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) \* \[Setting 1\]=(gpu\_memory\_utilization=0.98 max\_model\_len=4096 enforce\_eager=True) \* \[Setting 2\]=(gpu\_memory\_utilization=1.0 max\_model\_len=4096 enforce\_eager=True) \* \[Setting 3\]=(gpu\_memory\_utilization=1.0 max\_model\_len=8192 enforce\_eager=True) --- # Unknown \# MLX LM :::{attention} To be updated for Qwen3. ::: \[mlx-lm\](https://github.com/ml-explore/mlx-examples/tree/main/llms) helps you run LLMs locally on Apple Silicon. It is available at macOS. It has already supported Qwen models and this time, we have also provided checkpoints that you can directly use with it. ## Prerequisites The easiest way to get started is to install the \`mlx-lm\` package: - with \`pip\`: \`\`\`bash pip install mlx-lm \`\`\` - with \`conda\`: \`\`\`bash conda install -c conda-forge mlx-lm \`\`\` ## Running with Qwen MLX Files We provide model checkpoints with \`mlx-lm\` in our Hugging Face organization, and to search for what you need you can search the repo names with \`-MLX\`. Here provides a code snippet with \`apply\_chat\_template\` to show you how to load the tokenizer and model and how to generate contents. \`\`\`python from mlx\_lm import load, generate model, tokenizer = load('Qwen/Qwen2.5-7B-Instruct-MLX', tokenizer\_config={"eos\_token": "<|im\_end|>"}) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) response = generate(model, tokenizer, prompt=text, verbose=True, top\_p=0.8, temp=0.7, repetition\_penalty=1.05, max\_tokens=512) \`\`\` ## Make Your MLX files You can make MLX files with just one command: \`\`\`bash mlx\_lm.convert --hf-path Qwen/Qwen2.5-7B-Instruct --mlx-path mlx/Qwen2.5-7B-Instruct/ -q \`\`\` where - \`--hf-path\`: the model name on Hugging Face Hub or the local path - \`--mlx-path\`: the path for output files - \`-q\`: enable quantization --- # Unknown LlamaIndex ========== .. attention:: To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on \`LlamaIndex \`\_\_. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: .. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the \`\`bge-base-en-v1.5\`\` model to retrieve from English documents, and you can download the \`\`bge-base-zh-v1.5\`\` model to retrieve from Chinese documents. You can also choose \`\`bge-large\`\` or \`\`bge-small\`\` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) .. code:: python import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt = "" for message in messages: if message.role == "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role == "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role == "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt = "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt = prompt + "<|im\_start|>assistant\\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings.llm = HuggingFaceLLM( model\_name="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name="Qwen/Qwen2.5-7B-Instruct", context\_window=30000, max\_new\_tokens=2000, generate\_kwargs={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt=messages\_to\_prompt, completion\_to\_prompt=completion\_to\_prompt, device\_map="auto", ) # Set embedding model Settings.embed\_model = HuggingFaceEmbedding( model\_name = "BAAI/bge-base-en-v1.5" ) # Set the size of the text chunk for retrieval Settings.transformations = \[SentenceSplitter(chunk\_size=1024)\] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'. .. code:: python from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load\_data() index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. .. code:: python from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html\_to\_text=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) To save and load the index, you can use the following code snippet. .. code:: python from llama\_index.core import StorageContext, load\_index\_from\_storage # save index storage\_context = StorageContext.from\_defaults(persist\_dir="save") # load index index = load\_index\_from\_storage(storage\_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. .. code:: python query\_engine = index.as\_query\_engine() your\_query = "" print(query\_engine.query(your\_query).response) --- # Unknown Langchain ========================== This guide helps you build a question-answering application based on a local knowledge base using \`\`Qwen2.5-7B-Instruct\`\` with \`\`langchain\`\`. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: .. code:: bash pip install langchain==0.0.174 pip install faiss-gpu .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int = 10000 temperature: float = 0.01 top\_p = 0.9 history\_len: int = 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) -> str: return "Qwen" @property def \_history\_len(self) -> int: return self.history\_len def set\_history\_len(self, history\_len: int = 10) -> None: self.history\_len = history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] = None, run\_manager: Optional\[CallbackManagerForLLMRun\] = None, ) -> str: messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] return response @property def \_identifying\_params(self) -> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. .. code:: python import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool = False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf = pdf def split\_text(self, text: str) -> List\[str\]: if self.pdf: text = re.sub(r"\\n{3,}", "\\n", text) text = re.sub('\\s', ' ', text) text = text.replace("\\n\\n", "") sent\_sep\_pattern = re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list = \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader = TextLoader(filepath, autodetect\_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path = os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp = os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) -> List\[List\[int\]\]: lists = \[\] ls1 = \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i - 1\] + 1 == ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 = \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size = 250 chunk\_conent = True score\_threshold = 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int = 4 ) -> List\[Tuple\[Document, float\]\]: scores, indices = self.index.search(np.array(\[embedding\], dtype=np.float32), k) docs = \[\] id\_set = set() store\_len = len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i == -1 or 0 < self.score\_threshold < scores\[0\]\[j\]: # This happens when not enough docs are returned. continue \_id = self.index\_to\_docstore\_id\[i\] doc = self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] = int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len = len(doc.page\_content) for k in range(1, max(i, store\_len - i)): break\_flag = False for l in \[i + k, i - k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 = self.index\_to\_docstore\_id\[l\] doc0 = self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) > self.chunk\_size: break\_flag = True break elif doc0.metadata\["source"\] == doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) == 0 and self.score\_threshold > 0: return \[\] id\_list = sorted(list(id\_set)) id\_lists = separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id == id\_seq\[0\]: \_id = self.index\_to\_docstore\_id\[id\] doc = self.docstore.search(\_id) else: \_id0 = self.index\_to\_docstore\_id\[id\] doc0 = self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score = min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] = int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ == '\_\_main\_\_': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING\_MODEL = 'text2vec' PROMPT\_TEMPLATE = """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING\_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K = 3 CHAIN\_TYPE = 'stuff' embedding\_model\_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model\_name=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs={'device': EMBEDDING\_DEVICE}) docs = load\_file(filepath) docsearch = FAISSWrapper.from\_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT\_TEMPLATE, input\_variables=\["context\_str", "question"\] ) chain\_type\_kwargs = {"prompt": prompt, "document\_variable\_name": "context\_str"} qa = RetrievalQA.from\_chain\_type( llm=llm, chain\_type=CHAIN\_TYPE, retriever=docsearch.as\_retriever(search\_kwargs={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs=chain\_type\_kwargs) query = "Give me a short introduction to large language model." print(qa.run(query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Unknown Langchain ========================== .. attention:: To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using \`\`Qwen2.5-7B-Instruct\`\` with \`\`langchain\`\`. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: .. code:: bash pip install langchain==0.0.174 pip install faiss-gpu .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int = 10000 temperature: float = 0.01 top\_p = 0.9 history\_len: int = 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) -> str: return "Qwen" @property def \_history\_len(self) -> int: return self.history\_len def set\_history\_len(self, history\_len: int = 10) -> None: self.history\_len = history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] = None, run\_manager: Optional\[CallbackManagerForLLMRun\] = None, ) -> str: messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] return response @property def \_identifying\_params(self) -> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. .. code:: python import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool = False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf = pdf def split\_text(self, text: str) -> List\[str\]: if self.pdf: text = re.sub(r"\\n{3,}", "\\n", text) text = re.sub('\\s', ' ', text) text = text.replace("\\n\\n", "") sent\_sep\_pattern = re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list = \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader = TextLoader(filepath, autodetect\_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path = os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp = os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) -> List\[List\[int\]\]: lists = \[\] ls1 = \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i - 1\] + 1 == ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 = \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size = 250 chunk\_conent = True score\_threshold = 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int = 4 ) -> List\[Tuple\[Document, float\]\]: scores, indices = self.index.search(np.array(\[embedding\], dtype=np.float32), k) docs = \[\] id\_set = set() store\_len = len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i == -1 or 0 < self.score\_threshold < scores\[0\]\[j\]: # This happens when not enough docs are returned. continue \_id = self.index\_to\_docstore\_id\[i\] doc = self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] = int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len = len(doc.page\_content) for k in range(1, max(i, store\_len - i)): break\_flag = False for l in \[i + k, i - k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 = self.index\_to\_docstore\_id\[l\] doc0 = self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) > self.chunk\_size: break\_flag = True break elif doc0.metadata\["source"\] == doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) == 0 and self.score\_threshold > 0: return \[\] id\_list = sorted(list(id\_set)) id\_lists = separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id == id\_seq\[0\]: \_id = self.index\_to\_docstore\_id\[id\] doc = self.docstore.search(\_id) else: \_id0 = self.index\_to\_docstore\_id\[id\] doc0 = self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score = min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] = int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ == '\_\_main\_\_': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING\_MODEL = 'text2vec' PROMPT\_TEMPLATE = """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING\_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K = 3 CHAIN\_TYPE = 'stuff' embedding\_model\_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model\_name=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs={'device': EMBEDDING\_DEVICE}) docs = load\_file(filepath) docsearch = FAISSWrapper.from\_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT\_TEMPLATE, input\_variables=\["context\_str", "question"\] ) chain\_type\_kwargs = {"prompt": prompt, "document\_variable\_name": "context\_str"} qa = RetrievalQA.from\_chain\_type( llm=llm, chain\_type=CHAIN\_TYPE, retriever=docsearch.as\_retriever(search\_kwargs={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs=chain\_type\_kwargs) query = "Give me a short introduction to large language models." print(qa.run(query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Unknown Langchain ========================== .. attention:: To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using \`\`Qwen2.5-7B-Instruct\`\` with \`\`langchain\`\`. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: .. code:: bash pip install langchain==0.0.174 pip install faiss-gpu .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int = 10000 temperature: float = 0.01 top\_p = 0.9 history\_len: int = 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) -> str: return "Qwen" @property def \_history\_len(self) -> int: return self.history\_len def set\_history\_len(self, history\_len: int = 10) -> None: self.history\_len = history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] = None, run\_manager: Optional\[CallbackManagerForLLMRun\] = None, ) -> str: messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] return response @property def \_identifying\_params(self) -> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. .. code:: python import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool = False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf = pdf def split\_text(self, text: str) -> List\[str\]: if self.pdf: text = re.sub(r"\\n{3,}", "\\n", text) text = re.sub('\\s', ' ', text) text = text.replace("\\n\\n", "") sent\_sep\_pattern = re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list = \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader = TextLoader(filepath, autodetect\_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path = os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp = os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) -> List\[List\[int\]\]: lists = \[\] ls1 = \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i - 1\] + 1 == ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 = \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size = 250 chunk\_conent = True score\_threshold = 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int = 4 ) -> List\[Tuple\[Document, float\]\]: scores, indices = self.index.search(np.array(\[embedding\], dtype=np.float32), k) docs = \[\] id\_set = set() store\_len = len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i == -1 or 0 < self.score\_threshold < scores\[0\]\[j\]: # This happens when not enough docs are returned. continue \_id = self.index\_to\_docstore\_id\[i\] doc = self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] = int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len = len(doc.page\_content) for k in range(1, max(i, store\_len - i)): break\_flag = False for l in \[i + k, i - k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 = self.index\_to\_docstore\_id\[l\] doc0 = self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) > self.chunk\_size: break\_flag = True break elif doc0.metadata\["source"\] == doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) == 0 and self.score\_threshold > 0: return \[\] id\_list = sorted(list(id\_set)) id\_lists = separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id == id\_seq\[0\]: \_id = self.index\_to\_docstore\_id\[id\] doc = self.docstore.search(\_id) else: \_id0 = self.index\_to\_docstore\_id\[id\] doc0 = self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score = min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] = int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ == '\_\_main\_\_': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING\_MODEL = 'text2vec' PROMPT\_TEMPLATE = """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING\_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K = 3 CHAIN\_TYPE = 'stuff' embedding\_model\_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model\_name=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs={'device': EMBEDDING\_DEVICE}) docs = load\_file(filepath) docsearch = FAISSWrapper.from\_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT\_TEMPLATE, input\_variables=\["context\_str", "question"\] ) chain\_type\_kwargs = {"prompt": prompt, "document\_variable\_name": "context\_str"} qa = RetrievalQA.from\_chain\_type( llm=llm, chain\_type=CHAIN\_TYPE, retriever=docsearch.as\_retriever(search\_kwargs={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs=chain\_type\_kwargs) query = "Give me a short introduction to large language models." print(qa.run(query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Unknown {%- if tools %} {{- '<|im\_start|>system\\n' }} {%- if messages\[0\].role == 'system' %} {{- messages\[0\].content + '\\n\\n' }} {%- endif %} {{- "# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n" }} {%- for tool in tools %} {{- "\\n" }} {{- tool | tojson }} {%- endfor %} {{- "\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\"name\\": , \\"arguments\\": }\\n<|im\_end|>\\n" }} {%- else %} {%- if messages\[0\].role == 'system' %} {{- '<|im\_start|>system\\n' + messages\[0\].content + '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- set ns = namespace(multi\_step\_tool=true, last\_query\_index=messages|length - 1) %} {%- for message in messages\[::-1\] %} {%- set index = (messages|length - 1) - loop.index0 %} {%- if ns.multi\_step\_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} {%- set ns.multi\_step\_tool = false %} {%- set ns.last\_query\_index = index %} {%- endif %} {%- endfor %} {%- for message in messages %} {%- if message.content is string %} {%- set content = message.content %} {%- else %} {%- set content = '' %} {%- endif %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im\_start|>' + message.role + '\\n' + content + '<|im\_end|>' + '\\n' }} {%- elif message.role == "assistant" %} {%- set reasoning\_content = '' %} {%- if message.reasoning\_content is string %} {%- set reasoning\_content = message.reasoning\_content %} {%- else %} {%- if '' in content %} {%- set reasoning\_content = content.split('')\[0\].rstrip('\\n').split('')\[-1\].lstrip('\\n') %} {%- set content = content.split('')\[-1\].lstrip('\\n') %} {%- endif %} {%- endif %} {%- if loop.index0 > ns.last\_query\_index %} {%- if loop.last or (not loop.last and reasoning\_content) %} {{- '<|im\_start|>' + message.role + '\\n\\n' + reasoning\_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- if message.tool\_calls %} {%- for tool\_call in message.tool\_calls %} {%- if (loop.first and content) or (not loop.first) %} {{- '\\n' }} {%- endif %} {%- if tool\_call.function %} {%- set tool\_call = tool\_call.function %} {%- endif %} {{- '\\n{"name": "' }} {{- tool\_call.name }} {{- '", "arguments": ' }} {%- if tool\_call.arguments is string %} {{- tool\_call.arguments }} {%- else %} {{- tool\_call.arguments | tojson }} {%- endif %} {{- '}\\n' }} {%- endfor %} {%- endif %} {{- '<|im\_end|>\\n' }} {%- elif message.role == "tool" %} {%- if loop.first or (messages\[loop.index0 - 1\].role != "tool") %} {{- '<|im\_start|>user' }} {%- endif %} {{- '\\n\\n' }} {{- content }} {{- '\\n' }} {%- if loop.last or (messages\[loop.index0 + 1\].role != "tool") %} {{- '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add\_generation\_prompt %} {{- '<|im\_start|>assistant\\n\\n\\n\\n\\n' }} {%- endif %} --- # Unknown \--- myst: number\_code\_blocks: \["python3"\] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. This guide will not delve into those discussions or which role an LLM should play in an application and the related best practice. Those views are reflected in the design of AI application frameworks: from LangChain to LlamaIndex to QwenAgent. Instead, we will talk about how Qwen2.5 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen2.5. - Then, we will introduce the technical details on functional calling with Qwen2.5, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. They are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. Qwen2.5 were pre-trained with various types of templates that could support function calling, so that users can directly make use of this procedure. ## Inference with Function Calling :::{note} Please be aware that the inference usage is subject to change as the frameworks and the Qwen models evolve. ::: As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen2 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - \*\*Qwen-Agent\*\*, - \*\*Hugging Face transformers\*\*, - \*\*Ollama\*\*, and - \*\*vLLM\*\*. If you are familiar with the usage of OpenAI API, you could also directly use the OpenAI-compatible API services for Qwen2.5. However, not all of them support function calling for Qwen2.5. Currently, supported solutions include the self-hosted service by \[Ollama\](https://github.com/ollama/ollama/blob/main/docs/openai.md) or \[vLLM\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#tool-calling-in-the-chat-completion-api) and the cloud service of \[ModelStudio \\\[zh\\\]\](https://help.aliyun.com/zh/model-studio/developer-reference/compatibility-of-openai-with-dashscope#97e2b45391x08). If you are familiar with application frameworks, e.g., LangChain, you can also use function calling abilities in Qwen2.5 via ReAct Prompting. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume \*\*Python 3.11\*\* is used as the programming language. \*\*Scenario\*\*: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode \`\`\`python import json def get\_current\_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name == "get\_current\_temperature": return get\_current\_temperature if name == "get\_temperature\_date": return get\_temperature\_date TOOLS = \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location"\], }, }, }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location", "date"\], }, }, }, \] MESSAGES = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, \] \`\`\` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: \`\`\`json \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location" \] } } }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location", "date" \] } } } \] \`\`\` For each \*\*tool\*\*, it is a JSON object with two fields: - \`type\`: a string specifying the type of the tool, currently only \`"function"\` is valid - \`function\`: an object detailing the instructions to use the function For each \*\*function\*\*, it is a JSON object with three fields: - \`name\`: a string indicating the name of the function - \`description\`: a string describing what the function is used for - \`parameters\`: \[a JSON Schema\](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include \`type\`, \`required\`, and \`enum\`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is \`What's the temperature in San Francisco now? How about tomorrow?\`. Since the model does not know what the current date is, let alone tomorrow, we should provide the date in the inputs. Here, we decide to supply that information in the system message after the default system message \`You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\`. You could append the date to user message in your application code. \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"} \] \`\`\` ::: ### Qwen-Agent \[Qwen-Agent\](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the \*\*canonical implementation\*\* of function calling for Qwen2.5. It provides the function calling ability for Qwen2.5 to an OpenAI-compatible API through templates that is transparent to users. {#note-official-template} It's worth noting that since a lot of stuff can be done under the scene with application frameworks, currently the official function calling implementation for Qwen2.5 is very flexible and beyond simple templating, making it hard to adapt it other frameworks that use less capable templating engines. Before starting, let's make sure the latest library is installed: \`\`\`bash pip install -U qwen-agent \`\`\` For this guide, we are at version v0.0.10. #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at \`http://localhost:8000/v1\`, Qwen-Agent provides a shortcut function \`get\_chat\_model\` to obtain a model inference class with function calling support: \`\`\`python from qwen\_agent.llm import get\_chat\_model llm = get\_chat\_model({ "model": "Qwen/Qwen2.5-7B-Instruct", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", }) \`\`\` In the above, \`model\_server\` is the \`api\_base\` common used in other OpenAI-compatible API clients. It is advised to provide the \`api\_key\` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. For model inputs, the common message structure for system, user, and assistant history should be used: \`\`\`python messages = MESSAGES\[:\] # \[ # {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30"}, # {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"} # \] \`\`\` We add the current date to the system message so that the "tomorrow" in the user message is anchored. It can also be added to the user message if one desires. At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: \`\`\`python functions = \[tool\["function"\] for tool in TOOLS\] \`\`\` #### Tool Calls and Tool Results To interact with the model, the \`chat\` method should be used: \`\`\`python for responses in llm.chat( messages=messages, functions=functions, extra\_generate\_cfg=dict(parallel\_function\_calls=True), ): pass messages.extend(responses) \`\`\` In the above code, the \`chat\` method receives the \`messages\`, the \`functions\`, and an \`extra\_generate\_cfg\` parameter. You can put sampling parameters, such as \`temperature\`, and \`top\_p\`, in the \`extra\_generate\_cfg\`. Here, we add to it a special control \`parallel\_function\_calls\` provided by Qwen-Agent. As its name suggests, it will enable parallel function calls, which means that the model may generate multiple function calls for a single turn as it deems fit. The \`chat\` method returns a generator of list, each of which may contain multiple messages. Since we enable \`parallel\_function\_calls\`, we should get two messages in the responses: \`\`\`python \[ {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_current\_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}}, {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_temperature\_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}}, \] \`\`\` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the \`function\_call\` field of the messages: - \`name\`: a string representing the function to call - \`arguments\`: a JSON-formatted string representing the arguments the function should be called with Note that Qwen2.5-7B-Instruct is quite capable: - It has followed the function instructions to add the state and the country to the location. - It has correctly induced the date of tomorrow and given in the format required by the function. Then comes the critical part -- checking and applying the function call: \`\`\`python3 for message in responses: if fn\_call := message.get("function\_call", None): fn\_name: str = fn\_call\['name'\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "function", "name": fn\_name, "content": fn\_res, }) \`\`\` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the \`function\_call\` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are \`name\` and \`arguments\` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named \[\`get\_function\_by\_name\`\](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as \`content\` and with \`role\` as \`"function"\`. Now the messages are \`\`\`python \[ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_current\_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}}, {'role': 'assistant', 'content': '', 'function\_call': {'name': 'get\_temperature\_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}}, {'role': 'function', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'function', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response Finally, run the model again to get the final model results: \`\`\`python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) \`\`\` The final response should be like \`\`\`python {'role': 'assistant', 'content': 'Currently, the temperature in San Francisco is approximately 26.1°C. Tomorrow, on 2024-10-01, the temperature is forecasted to be around 25.9°C.'} \`\`\` ### Hugging Face transformers Since function calling is based on prompt engineering and templates, \`transformers\` supports it with its tokenizer utilities, in particular, the \`tokenizer.apply\_chat\_template\` method, which hides the sophistication of constructing the model inputs, using the Jinja templating engine. However, it means that users should handle the model output part on their own, which includes parsing the generated function call message. The blog piece \[\_Tool Use, Unified\_\](https://huggingface.co/blog/unified-tool-use) is very helpful in understanding its design. Be sure to take a look. Tool use API is available in transformers since v4.42.0. Before starting, let's check that: \`\`\`bash pip install "transformers>4.42.0" \`\`\` For this guide, we are at version v4.44.2. #### Preparing For Qwen2.5, the chat template in \`tokenizer\_config.json\` has already included support for the Hermes-style tool use. We simply need to load the model and the tokenizer: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name\_or\_path = "Qwen/Qwen2.5-7B-Instruct" tokenizer = AutoTokenizer.from\_pretrained(model\_name\_or\_path) model = AutoModelForCausalLM.from\_pretrained( model\_name\_or\_path, torch\_dtype="auto", device\_map="auto", ) \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES\[:\] \`\`\` In \`transformers\`, you can also directly use Python functions as tools with certain constraints\[^get\_json\_schema\_note\]: \`\`\`python tools = \[get\_current\_temperature, get\_temperature\_date\] \`\`\` \[^get\_json\_schema\_note\]: \`transformers\` will use \`transformers.utils.get\_json\_schema\` to generate the tool descriptions from Python functions. There are some gotchas with \`get\_json\_schema\`, and it is advised to check \[its doc \\\[v4.44.2\\\]\](https://github.com/huggingface/transformers/blob/v4.44.2/src/transformers/utils/chat\_template\_utils.py#L183-L288) before relying on it. - The function should use Python type hints for parameter types and has a Google-style docstring for function description and parameter descriptions. - Supported types are limited, since the types needs to be mapped to JSON Schema. In particular, \`typing.Literal\` is not supported. You can instead add \`(choices: ...)\` at the end of a parameter description, which will be mapped to a \`enum\` type in JSON Schema. Please be aware that all the returned results in the examples in the linked docstring are actually the content of the \`function\` field in the actual returned results. #### Tool Calls and Tool Results To construct the input sequence, we should use the \`apply\_chat\_template\` method and then let the model continue the texts: \`\`\`python text = tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, tokenize=False) inputs = tokenizer(text, return\_tensors="pt").to(model.device) outputs = model.generate(\*\*inputs, max\_new\_tokens=512) output\_text = tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] \`\`\` The output texts should be like \`\`\`text {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA"}} {"name": "get\_temperature\_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} <|im\_end|> \`\`\` Now we need to do two things: 1. Parse the generated tool calls to a message and add them to the messages, so that the model knows which tools are used. 2. Obtain the results of the tools and add them to the messages, so that the model knows the results of the tool calls. In \`transformers\`, the tool calls should be a field of assistant messages. Let's use a simple function called \`try\_parse\_tool\_calls\` to parse the tool calls: {#parse-function} \`\`\`python import re def try\_parse\_tool\_calls(content: str): """Try parse the tool calls.""" tool\_calls = \[\] offset = 0 for i, m in enumerate(re.finditer(r"\\n(.+)?\\n", content)): if i == 0: offset = m.start() try: func = json.loads(m.group(1)) tool\_calls.append({"type": "function", "function": func}) if isinstance(func\["arguments"\], str): func\["arguments"\] = json.loads(func\["arguments"\]) except json.JSONDecodeError as e: print(f"Failed to parse tool calls: the content is {m.group(1)} and {e}") pass if tool\_calls: if offset > 0 and content\[:offset\].strip(): c = content\[:offset\] else: c = "" return {"role": "assistant", "content": c, "tool\_calls": tool\_calls} return {"role": "assistant", "content": re.sub(r"<\\|im\_end\\|>$", "", content)} \`\`\` This function does not cover all possible scenarios and thus is prone to errors. But it should suffice for the purpose of this guide. :::{note} The template in the \`tokenizer\_config.json\` assumes that the generated content alongside tool calls is in the same message instead of separate assistant messages, e.g., \`\`\`json { "role": "assistant", "content": "To obtain the current temperature, I should call the functions \`get\_current\_temperate\`.", "tool\_calls": \[ {"type": "function", "function": {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}} \] } \`\`\` instead of \`\`\`json \[ { "role": "assistant", "content": "To obtain the current temperature, I should call the functions \`get\_current\_temperate\`.", }, { "role": "assistant", "content": "", "tool\_calls": \[ {"type": "function", "function": {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}} \] } \] \`\`\` This is implemented roughly in \`try\_parse\_tool\_calls\` but keep that in mind if you are writing your own tool call parser. ::: \`\`\`python messages.append(try\_parse\_tool\_calls(output\_text)) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = fn\_call\["arguments"\] fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) \`\`\` The messages now should be like \`\`\`python \[ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'tool\_calls': \[ {'type': 'function', 'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'type': 'function', 'function': {'name': 'get\_temperature\_date', 'arguments': {'location': 'San Francisco, CA, USA', 'date': '2024-10-01'}}}, \]}, {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` The messages are similar to those of Qwen-Agent, but there are some major differences: - Tools instead of functions - Parallel calls are by default - Multiple tool calls as a list in a single assistant message, instead of multiple messages. - The function arguments are parsed into a dict if it is a valid JSON-formatted string. #### Final Response Then it's time for the model to generate the actual response for us based on the tool results. Let's query the model again: \`\`\`python text = tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, tokenize=False) inputs = tokenizer(text, return\_tensors="pt").to(model.device) outputs = model.generate(\*\*inputs, max\_new\_tokens=512) output\_text = tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] \`\`\` The output\_text should be like \`\`\` The current temperature in San Francisco is approximately 26.1°C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9°C.<|im\_end|> \`\`\` Add the result text as an assistant message and the final messages should be ready for further interaction: \`\`\`python messages.append(try\_parse\_tool\_calls(output\_text)) \`\`\` ### Ollama Ollama is a set of tools for serving LLMs locally. It also relies on its template implementation to support function calling. Different from transformers, which is written in Python and uses the Jinja template whose syntax is heavily inspired by Django and Python, Ollama, which is mostly written in Go, uses Go's \[text/template\](https://pkg.go.dev/text/template) packages. In addition, Ollama implements internally a helper function so that it can automatically parse the generated tool calls in texts to structured messages if the format supported. You could check the \[Tool support\](https://ollama.com/blog/tool-support) blog post first. Tool support has been available in Ollama since v0.3.0. You can run the following to check the Ollama version: \`\`\`bash ollama -v \`\`\` If lower than expected, follow \[the official instructions\](https://ollama.com/download) to install the latest version. In this guide, we will aslo use \[ollama-python\](https://github.com/ollama/ollama-python), before starting, make sure it is available in your environment: \`\`\`bash pip install ollama \`\`\` For this guide, the \`ollama\` binary is at v0.3.9 and the \`ollama\` Python library is at v0.3.2. #### Preparing The messages structure used in Ollama is the same with that in \`transformers\` and the template in \[Qwen2.5 Ollama models\](https://ollama.com/library/qwen2.5) has supported tool use. The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES\[:\] model\_name = "qwen2.5:7b" \`\`\` Note that you cannot pass Python functions as tools directly and \`tools\` has to be a \`dict\`. #### Tool Calls and Tool Results We can use the \`ollama.chat\` method to directly query the underlying API: \`\`\`python import ollama response = ollama.chat( model=model\_name, messages=messages, tools=tools, ) \`\`\` The main fields in the response could be: \`\`\`python { 'model': 'qwen2.5:7b', 'message': { 'role': 'assistant', 'content': '', 'tool\_calls': \[ {'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'function': {'name': 'get\_temperature\_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}}, \], }, } \`\`\` Ollama's tool call parser has succeeded in parsing the tool results. If not, you may refine \[the \`try\_parse\_tool\_calls\` function above\](#parse-function). Then, we can obtain the tool results and add them to the messages. The following is basically the same with \`transformers\`: \`\`\`python messages.append(response\["message"\]) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = fn\_call\["arguments"\] fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) \`\`\` The messages are now like \`\`\`python \[ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'tool\_calls': \[ {'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'function': {'name': 'get\_temperature\_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}}, \]}, {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response The rest are easy: \`\`\`python response = ollama.chat( model=model\_name, messages=messages, tools=tools, ) messages.append(response\["message"\]) \`\`\` The final message should be like the following: \`\`\`python {'role': 'assistant', 'content': 'The current temperature in San Francisco is approximately 26.1°C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9°C.'} \`\`\` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from \`transformers\` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. Tool support has been available in \`vllm\` since v0.6.0. Be sure to install a version that supports tool use. For more information, check the \[vLLM documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#tool-calling-in-the-chat-completion-api). For this guide, we are at version v0.6.1.post2. We will use the OpenAI-Compatible API by \`vllm\` with the API client from the \`openai\` Python library. #### Preparing For Qwen2.5, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct --enable-auto-tool-choice --tool-call-parser hermes \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES\[:\] \`\`\` Let's also initialize the client: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) model\_name = "Qwen/Qwen2.5-7B-Instruct" \`\`\` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) \`\`\` vLLM should be able to parse the tool calls for us, and the main fields in the response (\`response.choices\[0\]\`) should be like \`\`\`python Choice( finish\_reason='tool\_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function\_call=None, tool\_calls=\[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get\_current\_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get\_temperature\_date'), type='function', ), \], ), stop\_reason=None, ) \`\`\` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows but \`transformers\` and Ollama differs. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: \`\`\`python messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str = tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) \`\`\` It should be noted that the OpenAI API uses \`tool\_call\_id\` to identify the relation between tool results and tool calls. The messages are now like \`\`\`python \[ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\\n\\nCurrent Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'}, \]}, {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, \] \`\`\` #### Final Response Let's call the endpoint again to seed the tool results and get response: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) \`\`\` The final response (\`response.choices\[0\].message.content\`) should be like \`\`\`text The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. \`\`\` ### Discussions Now, we have introduced how to conduct inference with function calling using Qwen2 in three different frameworks! Let's make a brief comparison. | Item | OpenAI API | Hugging Face transformers | Ollama | vLLM | Qwen-Agent | | :----- | :---: | :---: | :---: | :---: | :---: | | Type | HTTP API | Python Library | HTTP API | HTTP API | Python Library | | Inference Backend | - | PyTorch | llama.cpp | PyTorch | HTTP API | | Templating Backend | - | Jinja | Go \`text/template\` | Jinja | Python | | Tools/Functions | Tools | Tools | Tools | Tools | Functions | | Parallel Calls | Default Yes (Configurable) | Yes | Yes | Yes | Default No (Configurable) | | Call Format | Single assistant message with \`tool\_calls\` | Single assistant message with \`tool\_calls\` | Single assistant message with \`tool\_calls\` | Single assistant message with \`tool\_calls\` | Multiple assistant messages with \`function\_call\` | | Call Argument Format | string | object | object | string | string | | Call Result Format | Multiple tool messages with \`content\` | Multiple tool messages with \`content\` | Multiple tool messages with \`content\` | Multiple tool messages with \`content\` | Multiple function messages with \`content\` | There are some details not shown in the above table: - OpenAI API comes with Python, Node.js, Go, and .NET SDKs. It also follows the OpenAPI standard. - Ollama comes with Python and Node.js SDKs. It has OpenAI-compatible API at a different base url that can be accessed using OpenAI API SDK. - Qwen-Agent as an application framework can call the tools automatically for you, which is introduced in \[the Qwen-Agent guide\](./qwen\_agent). In addition, there are more on the model side of function calling, which means you may need to consider more things in production code: - \*\*Accuracy of function calling\*\*: When it comes to evaluate the accuracy of function calling, there are two aspects: (a) whether the correct functions (including no functions) are selected and (b) whether the correct function arguments are generated. It is not always the case that Qwen2.5 will be accurate. Function calling can involve knowledge that is deep and domain-specific. Sometimes, it doesn't fully understand the function and select the wrong one by mistake. Sometimes, it can fall into a loop and require calling the same function again and again. Sometimes, it will fabricate required function arguments instead of asking the user for input. To improve the function calling accuracy, it is advised to first try prompt engineering: does a more detailed function description help? can we provide instructions and examples to the model in the system message? If not, finetuning on your own data could also improve performance. - \*\*Protocol consistency\*\*: Even with the proper function calling template, the protocol may break. The model may generate extra texts to tool calls, e.g., explanations. The generated tool call may be invalid JSON-formatted string but a representation of a Python dict The generated tool call may be valid JSON but not conforms to the provided JSON Schema. For those kinds of issues, while some of them could be addressed with prompt engineering, some are caused by the nature of LLMs and can be hard to resolve in a general manner by LLMs themselves. While we strive to improve Qwen2.5 in this regard, edge cases are unlikely to be eliminated completely. ## Function Calling Templates The template design for function calling often includes the following aspects: - How to describe the functions to the model, so that the model understands what they are and how to use them. - How to prompt the model, so that it knows that functions can be used and in what format to generate the function calls. - How to tell a function call generation from others in generated text, so that we can extract the calls from the generated texts and actually make the calls. - How to incorporate the function results to the text, so that the model can tell them from its own generation and make connection among the calls and the results. For experienced prompt engineers, it should be possible to make any LLM support function calling, using in-context learning techniques and with representative examples, though with varied accuracy and stability depending on how "zero-shot" the task at hand is. ### Starting from ReAct Prompting For example, ReAct Prompting can be used to implement function calling with an extra element of planning: - \*\*Thought\*\*: the overt reasoning path, analyzing the functions and the user query and saying it out "loud" - \*\*Action\*\*: the function to use and the arguments with which the function should be called - \*\*Observation\*\*: the results of the function In fact, Qwen2 is verse in the following variant of ReAct Prompting (similar to LangChain ReAct) to make the intermediate texts more structured: \`\`\` Answer the following questions as best you can. You have access to the following tools: {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of \[{function\_name},{function\_name}\] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can be repeated zero or more times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: {query} Thought: {some\_text} Action: {function\_name} Action Input: {function\_arguments} Observation: {function\_results} Final Answer: {response} \`\`\` As you can see, there is no apparent user/assistant conversation structure in the template. The model will simply continue the texts. One should write the code to actively detect which step the model is at and in particular to add the observations in the process, until the Final Answer is generated. However, as most programming interfaces accept the message structure, there should be some kind of adapter between the two. \[The ReAct Chat Agent\](https://github.com/QwenLM/Qwen-Agent/blob/v0.0.10/qwen\_agent/agents/react\_chat.py) in Qwen-Agent facilitates this kind of conversion. ### Qwen2 Function Calling Template As a step forward, the official Qwen2 function calling template is in the vein of the ReAct Prompting format but focuses more on - differentiating the keywords like \`Question\`, \`Thought\`, \`Action\`, etc., from generation, - simplifying the process, - supporting better multi-turn conversation, and - adding controls for specialized usage. An equivalent example would be \`\`\` <|im\_start|>system {system message} ## Tools You have access to the following tools: ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of \[{function\_name},{function\_name}\] ✿ARGS✿: The input of the tool ✿RESULT✿: Tool results ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user {query}<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: {function\_name} ✿ARGS✿: {function\_arguments} ✿RESULT✿: {function\_result} ✿RETURN✿:{response}<|im\_end|> \`\`\` Let's first list the obvious differences: - Keywords (\`✿FUNCTION✿\`, \`✿ARGS✿\`, etc.) seem rare in ordinary text and more semantically related to function calling, but not special tokens yet. - Thought is omitted. This could affect accuracy for some use cases. - Use the system-user-assistant format for multi-turn conversations. Function calling prompting is moved to the system message. How about adding controls for specialized usage? The template actually has the following variants: - Language: the above is for non-Chinese language; there is another template in Chinese. - Parallel Calls: the above is for non-parallel calls; there is another template for parallel calls. In the canonical implementation in Qwen-Agent, those switches are implemented in Python, according to the configuration and current input. The actual text with \_parallel calls\_ should be like the following: \`\`\`text <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 ## Tools You have access to the following tools: ### get\_current\_temperature get\_current\_temperature: Get current temperature at a location. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]} Format the arguments as a JSON object. ### get\_temperature\_date get\_temperature\_date: Get temperature at a location and date. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]} Format the arguments as a JSON object. ## Insert the following command in your reply when you need to call N tools in parallel: ✿FUNCTION✿: The name of tool 1, should be one of \[get\_current\_temperature,get\_temperature\_date\] ✿ARGS✿: The input of tool 1 ✿FUNCTION✿: The name of tool 2 ✿ARGS✿: The input of tool 2 ... ✿FUNCTION✿: The name of tool N ✿ARGS✿: The input of tool N ✿RESULT✿: The result of tool 1 ✿RESULT✿: The result of tool 2 ... ✿RESULT✿: The result of tool N ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: get\_current\_temperature ✿ARGS✿: {"location": "San Francisco, CA, USA"} ✿FUNCTION✿: get\_temperature\_date ✿ARGS✿: {"location": "San Francisco, CA, USA", "date": "2024-10-01"} ✿RESULT✿: {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} ✿RESULT✿: {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} ✿RETURN✿: The current temperature in San Francisco is approximately 26.1°C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9°C.<|im\_end|> \`\`\` This template is hard to adapt it for other frameworks that use less capable templating engines. But it is doable at least partially for Jinja, which is Python-oriented after all. We didn't use it because using the template in \`transformers\` leads to more changes to the inference usage, which are not very common for beginners. For the interested, you can find the Jinja template and key points on usage below: :::{dropdown} Qwen2 Function Calling Jinja Template \`\`\`jinja {%- if messages\[0\]\["role"\] == "system" %} {%- set system\_message = messages\[0\]\["content"\] %} {%- set loop\_messages = messages\[1:\] %} {%- else %} {%- set system\_message = "You are a helpful assistant." %} {%- set loop\_messages = messages %} {%- endif %} {%- if parallel\_tool\_calls is undefined %} {%- set parallel\_tool\_calls = false %} {%- endif %} {%- if language is undefined or language != "zh" %} {%- set language = "en" %} {%- endif %} {{- "<|im\_start|>system\\n" + system\_message|trim }} {%- if tools is defined %} {{- "\\n\\n# 工具\\n\\n## 你拥有如下工具：\\n\\n" if language == "zh" else "\\n\\n## Tools\\n\\nYou have access to the following tools:\\n\\n" }} {%- set functions = tools|map(attribute="function")|list %} {%- set function\_names = functions|map(attribute="name")|join(",") %} {%- for function in functions %} {{- "### " + function.name + "\\n\\n" + function.name + ": " + function.description + (" 输入参数：" if language == "zh" else " Parameters: ") + function.parameters|tojson + (" 此工具的输入应为JSON对象。\\n\\n" if language == "zh" else " Format the arguments as a JSON object.\\n\\n") }} {%- endfor %} {%- if parallel\_tool\_calls and language == "zh" %} {{- "## 你可以在回复中插入以下命令以并行调用N个工具：\\n\\n✿FUNCTION✿: 工具1的名称，必须是\[" + function\_names + "\]之一\\n✿ARGS✿: 工具1的输入\\n✿FUNCTION✿: 工具2的名称\\n✿ARGS✿: 工具2的输入\\n...\\n✿FUNCTION✿: 工具N的名称\\n✿ARGS✿: 工具N的输入\\n✿RESULT✿: 工具1的结果\\n✿RESULT✿: 工具2的结果\\n...\\n✿RESULT✿: 工具N的结果\\n✿RETURN✿: 根据工具结 果进行回复，需将图片用!\[\](url)渲染出来" }} {%- elif parallel\_tool\_calls %} {{- "## Insert the following command in your reply when you need to call N tools in parallel:\\n\\n✿FUNCTION✿: The name of tool 1, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of tool 1\\n✿FUNCTION✿: The name of tool 2\\n✿ARGS✿: The input of tool 2\\n...\\n✿FUNCTION✿: The name of tool N\\n✿ARGS✿: The input of tool N\\n✿RESULT✿: The result of tool 1\\n✿RESULT✿: The result of tool 2\\n...\\n✿RESULT✿: The result of tool N\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%- elif language == "zh" %} {{- "## 你可以在回复中插入零次、一次或多次以下命令以调用工具：\\n\\n✿FUNCTION✿: 工具名称，必须是\[" + function\_names + "\]之一。\\n✿ARGS✿: 工具输入\\n✿RESULT✿: 工具结果\\n✿RETURN✿: 根据工具结果进行回复，需将图片用!\[\](url)渲染出来" }} {%- else %} {{- "## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs:\\n\\n✿FUNCTION✿: The tool to use, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of the tool\\n✿RESULT✿: Tool results\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%- endif %} {%- endif %} {{- "<|im\_end|>" }} {%- for message in loop\_messages %} {%- if message.role == "user" %} {{- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%- if loop.last and add\_generation\_prompt %} {{- "\\n<|im\_start|>assistant\\n" }} {%- endif %} {%- elif message.role == "tool" %} {{- "✿RESULT✿: " + message.content + "\\n" }} {%- if loop.last and add\_generation\_prompt %} {{- "✿RETURN✿:" }} {%- endif %} {%- elif message.role == "assistant" and message.tool\_calls is defined %} {%- if loop.previtem.role == "user" %} {{- "\\n<|im\_start|>assistant\\n" }} {%- endif %} {%- for function in message.tool\_calls|map(attribute="function") %} {{- "✿FUNCTION✿: " + function.name + "\\n✿ARGS✿: " + function.arguments|tojson + "\\n" }} {%- endfor %} {%- elif message.role == "assistant" %} {%- if loop.previtem.role == "user" %} {{- "\\n<|im\_start|>assistant\\n" }} {%- elif loop.previtem.role == "tool" %} {{- "✿RETURN✿:" }} {%- endif %} {{- message.content }} {%- if loop.nextitem is undefined or loop.nextitem.role == "user" %} {{- "<|im\_end|>" }} {%- endif %} {%- else %} {{- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%- endif %} {%- endfor %} \`\`\` To use this template in \`transformers\`: - Switches can be enabled by passing them to the \`apply\_chat\_template\` method, e.g., \`tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, parallel\_tool\_call=True, language="zh", tokenize=False)\`. By default, it is for English non-parallel function calling. - The tool arguments should be a Python \`dict\` instead of a JSON-formatted object \`str\`. - Since the generation needs to be stopped at \`✿RESULT✿\` or else the model will generate fabricated tool results, we should add it to \`stop\_strings\` in \`generation\_config\`: \`\`\`python model.generation\_config.stop\_strings = \["✿RESULT✿:", "✿RETURN✿:"\] \`\`\` - As a result of using \`stop\_strings\`, you need to pass the tokenizer to \`model.generate\` as \`model.generate(\*\*inputs, tokenizer=tokenizer, max\_new\_tokens=512)\`. - \`response\`, i.e., the model generation based on the tool calls and tool results, may contain a leading space. You should not strip it for the model. It is resulted from the tokenization and the template design. - The \`try\_parse\_tool\_calls\` function should also be modified accordingly. ::: ### Qwen2.5 Function Calling Templates For \`transformers\` and Ollama, we have also used templates that are easier to implement with Jinja or Go. They are variants of \[the Nous Research's Hermes function calling template\](https://github.com/NousResearch/Hermes-Function-Calling#prompt-format-for-function-calling). The Jinja template and the Go template should produce basically the same results. They final text should look like the following: \`\`\`text <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {"type": "function", "function": {"name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]}}} {"type": "function", "function": {"name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]}}} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant {"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA"}} {"name": "get\_temperature\_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} <|im\_end|> <|im\_start|>user {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} <|im\_end|> <|im\_start|>assistant The current temperature in San Francisco is approximately 26.1°C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9°C.<|im\_end|> \`\`\` While the text may seem different from the previous one, the basic prompting structure is still the same. There are just more structural tags and more JSON-formatted strings. --- There is one thing we haven't talked about: how should functions be described to the LLMs. In short, you could describe them as you would normally describe them in an API documentation, as long as you can effectively parse, validate, and execute the tool calls generated by the models. The format with JSON Schema appears a valid and common choice. ## Finally In whichever way you choose to use function calling with Qwen2.5, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Unknown \--- myst: number\_code\_blocks: \["python3"\] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen3. - Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. ## Inference with Function Calling As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - \*\*Qwen-Agent\*\*, - \*\*vLLM\*\*. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume \*\*Python 3.11\*\* is used as the programming language. \*\*Scenario\*\*: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode \`\`\`python import json def get\_current\_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name == "get\_current\_temperature": return get\_current\_temperature if name == "get\_temperature\_date": return get\_temperature\_date TOOLS = \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location"\], }, }, }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location", "date"\], }, }, }, \] MESSAGES = \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, \] \`\`\` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: \`\`\`json \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location" \] } } }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location", "date" \] } } } \] \`\`\` For each \*\*tool\*\*, it is a JSON object with two fields: - \`type\`: a string specifying the type of the tool, currently only \`"function"\` is valid - \`function\`: an object detailing the instructions to use the function For each \*\*function\*\*, it is a JSON object with three fields: - \`name\`: a string indicating the name of the function - \`description\`: a string describing what the function is used for - \`parameters\`: \[a JSON Schema\](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include \`type\`, \`required\`, and \`enum\`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is \`What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.\`. \`\`\`json \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."} \] \`\`\` ::: ### Qwen-Agent \[Qwen-Agent\](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the \*\*canonical implementation\*\* of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is \*not recommended\* to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let's make sure the latest library is installed: \`\`\`bash pip install -U qwen-agent \`\`\` #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at \`http://localhost:8000/v1\`, Qwen-Agent provides a shortcut function \`get\_chat\_model\` to obtain a model inference class with function calling support: \`\`\`python from qwen\_agent.llm import get\_chat\_model llm = get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} # default to True } } }) \`\`\` In the above, \`model\_server\` is the \`api\_base\` common used in other OpenAI-compatible API clients. It is advised to provide the \`api\_key\` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by \`generate\_cfg\`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: \`\`\`python messages = MESSAGES\[:\] \`\`\` At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: \`\`\`python functions = \[tool\["function"\] for tool in TOOLS\] \`\`\` #### Tool Calls and Tool Results To interact with the model, the \`chat\` method should be used: \`\`\`python for responses in llm.chat( messages=messages, functions=functions, ): pass messages.extend(responses) \`\`\` The \`chat\` method returns a generator of list, each of which may contain multiple messages. - The results of \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` - The results of \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the \`function\_call\` field of the messages: - \`name\`: a string representing the function to call - \`arguments\`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part -- checking and applying the function call: \`\`\`python3 for message in responses: if fn\_call := message.get("function\_call", None): fn\_name: str = fn\_call\['name'\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "function", "name": fn\_name, "content": fn\_res, }) \`\`\` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the \`function\_call\` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are \`name\` and \`arguments\` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named \[\`get\_function\_by\_name\`\](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as \`content\` and with \`role\` as \`"function"\`. Now the messages are: - \`no\_think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response Finally, run the model again to get the final model results: \`\`\`python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) \`\`\` The final response should be like - \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from \`transformers\` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. - \`vllm\` >= v0.8.5. For more information, check the \[vLLM documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#tool-calling-in-the-chat-completion-api). We will use the OpenAI-Compatible API by \`vllm\` with the API client from the \`openai\` Python library. #### Preparing For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: \`\`\`bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek\_r1 \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES \`\`\` Let's also initialize the client: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) model\_name = "Qwen/Qwen3-8B" \`\`\` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model. Here is an example of the \`no\_think\` mode: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} # default to True }, ) \`\`\` vLLM should be able to parse the tool calls for us, and the main fields in the response (\`response.choices\[0\]\`) should be like \`\`\`python Choice( finish\_reason='tool\_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function\_call=None, tool\_calls=\[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get\_current\_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get\_temperature\_date'), type='function', ), \], ), stop\_reason=None, ) \`\`\` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: \`\`\`python messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str = tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) \`\`\` It should be noted that the OpenAI API uses \`tool\_call\_id\` to identify the relation between tool results and tool calls. The messages are now like \`\`\`python \[ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'}, \]}, {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, \] \`\`\` #### Final Response Let's call the endpoint again to seed the tool results and get response: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) \`\`\` The final response (\`response.choices\[0\].message.content\`) should be like \`\`\`text The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. \`\`\` ## Finally In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Unknown {%- if tools %} {{- '<|im\_start|>system\\n' }} {%- if messages\[0\].role == 'system' %} {{- messages\[0\].content + '\\n\\n' }} {%- endif %} {{- "# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n" }} {%- for tool in tools %} {{- "\\n" }} {{- tool | tojson }} {%- endfor %} {{- "\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\"name\\": , \\"arguments\\": }\\n<|im\_end|>\\n" }} {%- else %} {%- if messages\[0\].role == 'system' %} {{- '<|im\_start|>system\\n' + messages\[0\].content + '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- set ns = namespace(multi\_step\_tool=true, last\_query\_index=messages|length - 1) %} {%- for message in messages\[::-1\] %} {%- set index = (messages|length - 1) - loop.index0 %} {%- if ns.multi\_step\_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} {%- set ns.multi\_step\_tool = false %} {%- set ns.last\_query\_index = index %} {%- endif %} {%- endfor %} {%- for message in messages %} {%- if message.content is string %} {%- set content = message.content %} {%- else %} {%- set content = '' %} {%- endif %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im\_start|>' + message.role + '\\n' + content + '<|im\_end|>' + '\\n' }} {%- elif message.role == "assistant" %} {%- set reasoning\_content = '' %} {%- if message.reasoning\_content is string %} {%- set reasoning\_content = message.reasoning\_content %} {%- else %} {%- if '' in content %} {%- set reasoning\_content = content.split('')\[0\].rstrip('\\n').split('')\[-1\].lstrip('\\n') %} {%- set content = content.split('')\[-1\].lstrip('\\n') %} {%- endif %} {%- endif %} {%- if loop.index0 > ns.last\_query\_index %} {%- if loop.last or (not loop.last and reasoning\_content) %} {{- '<|im\_start|>' + message.role + '\\n\\n' + reasoning\_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- if message.tool\_calls %} {%- for tool\_call in message.tool\_calls %} {%- if (loop.first and content) or (not loop.first) %} {{- '\\n' }} {%- endif %} {%- if tool\_call.function %} {%- set tool\_call = tool\_call.function %} {%- endif %} {{- '\\n{"name": "' }} {{- tool\_call.name }} {{- '", "arguments": ' }} {%- if tool\_call.arguments is string %} {{- tool\_call.arguments }} {%- else %} {{- tool\_call.arguments | tojson }} {%- endif %} {{- '}\\n' }} {%- endfor %} {%- endif %} {{- '<|im\_end|>\\n' }} {%- elif message.role == "tool" %} {%- if loop.first or (messages\[loop.index0 - 1\].role != "tool") %} {{- '<|im\_start|>user' }} {%- endif %} {{- '\\n\\n' }} {{- content }} {{- '\\n' }} {%- if loop.last or (messages\[loop.index0 + 1\].role != "tool") %} {{- '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add\_generation\_prompt %} {{- '<|im\_start|>assistant\\n\\n\\n\\n\\n' }} {%- endif %} --- # Unknown Langchain ========================== .. attention:: To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using \`\`Qwen2.5-7B-Instruct\`\` with \`\`langchain\`\`. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: .. code:: bash pip install langchain==0.0.174 pip install faiss-gpu .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model\_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) class Qwen(LLM, ABC): max\_token: int = 10000 temperature: float = 0.01 top\_p = 0.9 history\_len: int = 3 def \_\_init\_\_(self): super().\_\_init\_\_() @property def \_llm\_type(self) -> str: return "Qwen" @property def \_history\_len(self) -> int: return self.history\_len def set\_history\_len(self, history\_len: int = 10) -> None: self.history\_len = history\_len def \_call( self, prompt: str, stop: Optional\[List\[str\]\] = None, run\_manager: Optional\[CallbackManagerForLLMRun\] = None, ) -> str: messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] return response @property def \_identifying\_params(self) -> Mapping\[str, Any\]: """Get the identifying parameters.""" return {"max\_token": self.max\_token, "temperature": self.temperature, "top\_p": self.top\_p, "history\_len": self.history\_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. .. code:: python import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document\_loaders import TextLoader from langchain.text\_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def \_\_init\_\_(self, pdf: bool = False, \*\*kwargs): super().\_\_init\_\_(\*\*kwargs) self.pdf = pdf def split\_text(self, text: str) -> List\[str\]: if self.pdf: text = re.sub(r"\\n{3,}", "\\n", text) text = re.sub('\\s', ' ', text) text = text.replace("\\n\\n", "") sent\_sep\_pattern = re.compile( '(\[﹒﹔﹖﹗．。！？\]\["’”」』\]{0,2}|(?=\["‘“「『\]{1,2}|$))') sent\_list = \[\] for ele in sent\_sep\_pattern.split(text): if sent\_sep\_pattern.match(ele) and sent\_list: sent\_list\[-1\] += ele elif ele: sent\_list.append(ele) return sent\_list def load\_file(filepath): loader = TextLoader(filepath, autodetect\_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader.load\_and\_split(textsplitter) write\_check\_file(filepath, docs) return docs def write\_check\_file(filepath, docs): folder\_path = os.path.join(os.path.dirname(filepath), "tmp\_files") if not os.path.exists(folder\_path): os.makedirs(folder\_path) fp = os.path.join(folder\_path, 'load\_file.txt') with open(fp, 'a+', encoding='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\\n') for i in docs: fout.write(str(i)) fout.write('\\n') fout.close() def separate\_list(ls: List\[int\]) -> List\[List\[int\]\]: lists = \[\] ls1 = \[ls\[0\]\] for i in range(1, len(ls)): if ls\[i - 1\] + 1 == ls\[i\]: ls1.append(ls\[i\]) else: lists.append(ls1) ls1 = \[ls\[i\]\] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk\_size = 250 chunk\_conent = True score\_threshold = 0 def similarity\_search\_with\_score\_by\_vector( self, embedding: List\[float\], k: int = 4 ) -> List\[Tuple\[Document, float\]\]: scores, indices = self.index.search(np.array(\[embedding\], dtype=np.float32), k) docs = \[\] id\_set = set() store\_len = len(self.index\_to\_docstore\_id) for j, i in enumerate(indices\[0\]): if i == -1 or 0 < self.score\_threshold < scores\[0\]\[j\]: # This happens when not enough docs are returned. continue \_id = self.index\_to\_docstore\_id\[i\] doc = self.docstore.search(\_id) if not self.chunk\_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc.metadata\["score"\] = int(scores\[0\]\[j\]) docs.append(doc) continue id\_set.add(i) docs\_len = len(doc.page\_content) for k in range(1, max(i, store\_len - i)): break\_flag = False for l in \[i + k, i - k\]: if 0 <= l < len(self.index\_to\_docstore\_id): \_id0 = self.index\_to\_docstore\_id\[l\] doc0 = self.docstore.search(\_id0) if docs\_len + len(doc0.page\_content) > self.chunk\_size: break\_flag = True break elif doc0.metadata\["source"\] == doc.metadata\["source"\]: docs\_len += len(doc0.page\_content) id\_set.add(l) if break\_flag: break if not self.chunk\_conent: return docs if len(id\_set) == 0 and self.score\_threshold > 0: return \[\] id\_list = sorted(list(id\_set)) id\_lists = separate\_list(id\_list) for id\_seq in id\_lists: for id in id\_seq: if id == id\_seq\[0\]: \_id = self.index\_to\_docstore\_id\[id\] doc = self.docstore.search(\_id) else: \_id0 = self.index\_to\_docstore\_id\[id\] doc0 = self.docstore.search(\_id0) doc.page\_content += " " + doc0.page\_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {\_id}, got {doc}") doc\_score = min(\[scores\[0\]\[id\] for id in \[indices\[0\].tolist().index(i) for i in id\_seq if i in indices\[0\]\]\]) doc.metadata\["score"\] = int(doc\_score) docs.append((doc, doc\_score)) return docs if \_\_name\_\_ == '\_\_main\_\_': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING\_MODEL = 'text2vec' PROMPT\_TEMPLATE = """Known information: {context\_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING\_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR\_SEARCH\_TOP\_K = 3 CHAIN\_TYPE = 'stuff' embedding\_model\_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model\_name=embedding\_model\_dict\[EMBEDDING\_MODEL\],model\_kwargs={'device': EMBEDDING\_DEVICE}) docs = load\_file(filepath) docsearch = FAISSWrapper.from\_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT\_TEMPLATE, input\_variables=\["context\_str", "question"\] ) chain\_type\_kwargs = {"prompt": prompt, "document\_variable\_name": "context\_str"} qa = RetrievalQA.from\_chain\_type( llm=llm, chain\_type=CHAIN\_TYPE, retriever=docsearch.as\_retriever(search\_kwargs={"k": VECTOR\_SEARCH\_TOP\_K}), chain\_type\_kwargs=chain\_type\_kwargs) query = "Give me a short introduction to large language models." print(qa.run(query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- # Unknown \--- myst: number\_code\_blocks: \["python3"\] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. This guide will not delve into those discussions or which role an LLM should play in an application and the related best practice. Those views are reflected in the design of AI application frameworks: from LangChain to LlamaIndex to QwenAgent. Instead, we will talk about how Qwen2 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen2. - Then, we will introduce the technical details on functional calling with Qwen2, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. They are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. Qwen2 were pre-trained with two types of templates that could support function calling, one in-house designed and one based on ReAct Prompting, so that users can directly make use of this procedure. ## Inference with Function Calling :::{note} Please be aware that the inference usage is subject to change as the frameworks and the Qwen models evolve. ::: As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen2 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - \*\*Qwen-Agent\*\*, - \*\*Hugging Face transformers\*\*, and - \*\*Ollama\*\*. If you are familiar with the usage of OpenAI API, you could also directly use the OpenAI-compatible API services for Qwen2. However, not all of them support function calling for Qwen2. Currently, supported solutions include the self-hosted service by \[Ollama\](https://github.com/ollama/ollama/blob/main/docs/openai.md) (with a proper TEMPLATE in the Model File) and the cloud service of \[ModelStudio \\\[zh\\\]\](https://help.aliyun.com/zh/model-studio/developer-reference/compatibility-of-openai-with-dashscope#97e2b45391x08). Note that although vLLM has supported function calling in its OpenAI-compatible API, it cannot be used with Qwen2 due to the lack of required tool call parsers. If you are familiar with application frameworks, e.g., LangChain, you can also use function calling abilities in Qwen2 via ReAct Prompting. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume \*\*Python 3.11\*\* is used as the programming language. \*\*Scenario\*\*: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode \`\`\`python import json import re def get\_current\_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name == "get\_current\_temperature": return get\_current\_temperature if name == "get\_temperature\_date": return get\_temperature\_date def try\_parse\_tool\_calls(content: str): """Try parse the tool calls.""" tool\_calls = \[\] for m in re.finditer(r"<\\|tool\_call\_start\\|>(.+)?<\\|tool\_call\_end\\|>", content): try: func = json.loads(m.group(1)) tool\_calls.append({"type": "function", "function": func}) if isinstance(func\["arguments"\], str): func\["arguments"\] = json.loads(func\["arguments"\]) except json.JSONDecodeError as \_: print(m) pass if tool\_calls: return {"role": "assistant", "tool\_calls": tool\_calls} return {"role": "assistant", "content": re.sub(r"<\\|im\_end\\|>$", "", content)} TOOLS = \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location"\], }, }, }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location", "date"\], }, }, }, \] MESSAGES = \[ {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, \] \`\`\` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: \`\`\`json \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location" \] } } }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location", "date" \] } } } \] \`\`\` For each \*\*tool\*\*, it is a JSON object with two fields: - \`type\`: a string specifying the type of the tool, currently only \`"function"\` is valid - \`function\`: an object detailing the instructions to use the function For each \*\*function\*\*, it is a JSON object with three fields: - \`name\`: a string indicating the name of the function - \`description\`: a string describing what the function is used for - \`parameters\`: \[a JSON Schema\](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include \`type\`, \`required\`, and \`enum\`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is \`What's the temperature in San Francisco now? How about tomorrow?\`. Since the model does not know what the current date is, let alone tomorrow, we should provide the date in the inputs. Here, we decide to supply that information in the system message after the default system message \`You are a helpful assistant.\`. You could append the date to user message in your application code. \`\`\`json \[ {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"} \] \`\`\` ::: ### Qwen-Agent \[Qwen-Agent\](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the \*\*canonical implementation\*\* of function calling for Qwen2. It provides the function calling ability for Qwen2 to an OpenAI-compatible API through templates that is transparent to users. {#note-official-template} It's worth noting that since a lot of stuff can be done under the scene with application frameworks, currently the official function calling implementation for Qwen2 is very flexible and beyond simple templating, making it hard to adapt it other frameworks that use less capable templating engines. Before starting, let's make sure the latest library is installed: \`\`\`bash pip install -U qwen-agent \`\`\` For this guide, we are at version v0.0.9. #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at \`http://localhost:8000/v1\`, Qwen-Agent provides a shortcut function \`get\_chat\_model\` to obtain a model inference class with function calling support: \`\`\`python from qwen\_agent.llm import get\_chat\_model llm = get\_chat\_model({ "model": "Qwen/Qwen2-7B-Instruct", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", }) \`\`\` In the above, \`model\_server\` is the \`api\_base\` common used in other OpenAI-compatible API clients. It is advised to provide the \`api\_key\` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. For model inputs, the common message structure for system, user, and assistant history should be used: \`\`\`python messages = MESSAGES\[:\] # \[ # {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, # {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, # \] \`\`\` We add the current date to the system message so that the "tomorrow" in the user message is anchored. It can also be added to the user message if one desires. At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: \`\`\`python functions = \[tool\["function"\] for tool in TOOLS\] \`\`\` #### Tool Calls and Tool Results To interact with the model, the \`chat\` method should be used: \`\`\`python for responses in llm.chat( messages=messages, functions=functions, extra\_generate\_cfg=dict(parallel\_function\_calls=True), ): pass messages.extend(responses) \`\`\` In the above code, the \`chat\` method receives the \`messages\`, the \`functions\`, and an \`extra\_generate\_cfg\` parameter. You can put sampling parameters, such as \`temperature\`, and \`top\_p\`, in the \`extra\_generate\_cfg\`. Here, we add to it a special control \`parallel\_function\_calls\` provided by Qwen-Agent. As its name suggests, it will enable parallel function calls, which means that the model may generate multiple function calls for a single turn as it deems fit. The \`chat\` method returns a generator of list, each of which may contain multiple messages. Since we enable \`parallel\_function\_calls\`, we should get two messages in the responses: \`\`\`python \[ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": '{"location": "San Francisco, CA, USA"}'}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": '{"location": "San Francisco, CA, USA", "date": "2024-09-01"}'}}, \] \`\`\` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the \`function\_call\` field of the messages: - \`name\`: a string representing the function to call - \`arguments\`: a JSON-formatted string representing the arguments the function should be called with Note that Qwen2-7B-Instruct is quite capable: - It has followed the function instructions to add the state and the country to the location. - It has correctly induced the date of tomorrow and given in the format required by the function. Then comes the critical part -- checking and applying the function call: \`\`\`python3 for message in responses: if fn\_call := message.get("function\_call", None): fn\_name: str = fn\_call\['name'\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "function", "name": fn\_name, "content": fn\_res, }) \`\`\` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the \`function\_call\` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are \`name\` and \`arguments\` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named \[\`get\_function\_by\_name\`\](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as \`content\` and with \`role\` as \`"function"\`. Now the messages are \`\`\`python \[ {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": '{"location": "San Francisco, CA, USA"}'}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": '{"location": "San Francisco, CA, USA", "date": "2024-09-01"}'}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response Finally, run the model again to get the final model results: \`\`\`python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) \`\`\` The final response should be like \`\`\`python {"role": "assistant", "content": "The current temperature in San Francisco is 26.1 degrees Celsius. For tomorrow, the forecasted temperature is 25.9 degrees Celsius."} \`\`\` ### Hugging Face transformers :::{attention} We demonstrate the usage of function calling with Hugging Face transformers using an unofficial template for Qwen2. You may be expecting varied performance compared with Qwen-Agent. ::: Since function calling is based on prompt engineering and templates, \`transformers\` supports it with its tokenizer utilities, in particular, the \`tokenizer.apply\_chat\_template\` method, which hides the sophistication of constructing the model inputs, using the Jinja templating engine. However, it means that users should handle the model output part on their own, which includes parsing the generated function call message. The blog piece \[\_Tool Use, Unified\_\](https://huggingface.co/blog/unified-tool-use) is very helpful in understanding its design. Be sure to take a look. Tool use API is available in transformers since v4.42.0. Before starting, let's check that: \`\`\`bash pip install "transformers>4.42.0" \`\`\` For this guide, we are at version v4.44.2. #### Preparing As explained \[before\](#note-official-template), since the official Qwen2 function calling template can be complicated to implement with Jinja, here we will use a custom template: \`\`\`python chat\_template = r"""{%- if messages\[0\]\["role"\] == "system" %} {%- set system\_message = messages\[0\]\["content"\] %} {%- set loop\_messages = messages\[1:\] %} {%- else %} {%- set system\_message = "You are a helpful assistant." %} {%- set loop\_messages = messages %} {%- endif %} {{- "<|im\_start|>system\\n" + system\_message|trim }} {%- if tools %} {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <|tools\_start|><|tools\_end|> tags:\\n<|tools\_start|>" + tools|map(attribute="function")|list|tojson + "<|tools\_end|>\\n\\nFor each function call, return a JSON object with function name and arguments within <|tool\_call\_start|><|tool\_call\_end|> tags:\\n<|tool\_call\_start|>{\\"name\\": , \\"arguments\\": }<|tool\_call\_end|>" }} {%- endif %} {{- "<|im\_end|>" }} {%- for message in loop\_messages %} {%- if message.role == "assistant" and message.tool\_calls is defined %} {{- "\\n<|im\_start|>assistant" }} {%- for tool\_call in message.tool\_calls %} {{- "\\n<|tool\_call\_start|>" + tool\_call.function|tojson + "<|tool\_call\_end|>" }} {%- endfor %} {{- "<|im\_end|>" }} {%- else %} {{- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%- endif %} {%- if loop.last and add\_generation\_prompt and message.role != "assistant" %} {{- "\\n<|im\_start|>assistant\\n" }} {%- endif %} {%- endfor %}""" \`\`\` We need to load the model and the tokenizer and make sure the tokenizer use the tool use template: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name\_or\_path = "Qwen/Qwen2-7B-Instruct" tokenizer = AutoTokenizer.from\_pretrained( model\_name\_or\_path, chat\_template=chat\_template, ) model = AutoModelForCausalLM.from\_pretrained( model\_name\_or\_path, torch\_dtype="auto", device\_map="auto", ) \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES\[:\] \`\`\` In \`transformers\`, you can also directly use Python functions as tools with certain constraints\[^get\_json\_schema\_note\]: \`\`\`python tools = \[get\_current\_temperature, get\_temperature\_date\] \`\`\` \[^get\_json\_schema\_note\]: \`transformers\` will use \`transformers.utils.get\_json\_schema\` to generate the tool descriptions from Python functions. There are some gotchas with \`get\_json\_schema\`, and it is advised to check \[its doc \\\[v4.44.2\\\]\](https://github.com/huggingface/transformers/blob/v4.44.2/src/transformers/utils/chat\_template\_utils.py#L183-L288) before relying on it. - The function should use Python type hints for parameter types and has a Google-style docstring for function description and parameter descriptions. - Supported types are limited, since the types needs to be mapped to JSON Schema. In particular, \`typing.Literal\` is not supported. You can instead add \`(choices: ...)\` at the end of a parameter description, which will be mapped to a \`enum\` type in JSON Schema. Please be aware that all the returned results in the examples in the linked docstring are actually the content of the \`function\` field in the actual returned results. #### Tool Calls and Tool Results To construct the input sequence, we should use the \`apply\_chat\_template\` method and then let the model continue the texts: \`\`\`python text = tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, tokenize=False) inputs = tokenizer(text, return\_tensors="pt").to(model.device) outputs = model.generate(\*\*inputs, max\_new\_tokens=512) output\_text = tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] \`\`\` The output texts should be like \`\`\` <|tool\_call\_start|>{"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, CA, USA\\", \\"unit\\": \\"celsius\\"}"}<|tool\_call\_end|> <|tool\_call\_start|>{"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, CA, USA\\", \\"date\\": \\"2024-09-01\\", \\"unit\\": \\"celsius\\"}"}<|tool\_call\_end|><|im\_end|> \`\`\` Now we need to do two things: 1. Parse the generated tool calls to a message and add them to the messages, so that the model knows which tools are used. 2. Obtain the results of the tools and add them to the messages, so that the model knows the results of the tool calls. In \`transformers\`, the tool calls should be a field of assistant messages.\[^tool\_call\_arg\_format\] Let's use a simple function called \`try\_parse\_tool\_calls\` to parse the tool calls, which can be found in \[the preparation code\](#prepcode). This function does not cover all possible scenarios and thus is prone to errors. But it should suffice for the purpose of this guide. \`\`\`python messages.append(try\_parse\_tool\_calls(output\_text)) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = fn\_call\["arguments"\] fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) \`\`\` The messages now should be like \`\`\`python \[ {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, {"role": "assistant", "tool\_calls": \[ {'type': 'function', 'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA', 'unit': 'celsius'}}}, {'type': 'function', 'function': {'name': 'get\_temperature\_date', 'arguments': {'location': 'San Francisco, CA, USA', 'date': '2024-09-01', 'unit': 'celsius'}}} \]}, {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"}'}, \] \`\`\` The messages are similar to those of Qwen-Agent, but there are some major differences: - Tools instead of functions - Parallel calls are by default - Multiple tool calls as a list in a single assistant message, instead of multiple messages - The function arguments are parsed into a dict if it is a valid JSON-formatted string. #### Final Response Then it's time for the model to generate the actual response for us based on the tool results. Let's query the model again: \`\`\`python text = tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, tokenize=False) inputs = tokenizer(text, return\_tensors="pt").to(model.device) outputs = model.generate(\*\*inputs, max\_new\_tokens=512) output\_text = tokenizer.batch\_decode(outputs)\[0\]\[len(text):\] \`\`\` The output\_text should be like \`\`\` The current temperature in San Francisco is 26.1°C. The temperature for tomorrow in San Francisco is expected to be 25.9°C.<|im\_end|> \`\`\` Add the result text as an assistant message and the final messages should be ready for further interaction: \`\`\`python messages.append(try\_parse\_tool\_calls(output\_text)) \`\`\` \[^tool\_call\_arg\_format\]: However, note that the model generates arguments in tool calls not as a JSON object but a JSON-formatted string of the JSON object. For \`transformers\` and \`ollama\`, as the interfaces require the arguments to be JSON objects or Python dicts, there will be differences between the actual model generation and the template results for tool call arguments. ### Ollama :::{attention} We demonstrate the usage of function calling with Ollama using an unofficial template for Qwen2. You may be expecting varied performance compared with Qwen-Agent. ::: Ollama is a set of tools for serving LLMs locally. It also relies on its template implementation to support function calling. Different from transformers, which is written in Python and uses the Jinja template whose syntax is heavily inspired by Django and Python, Ollama, which is mostly written in Go, uses Go's \[text/template\](https://pkg.go.dev/text/template) packages. In addition, Ollama implements internally a helper function so that it can automatically parse the generated tool calls in texts to structured messages if the format supported. You could check the \[Tool support\](https://ollama.com/blog/tool-support) blog post first. Tool support has been available in Ollama since v0.3.0. You can run the following to check the Ollama version: \`\`\`bash ollama -v \`\`\` If lower than expected, follow \[the official instructions\](https://ollama.com/download) to install the latest version. In this guide, we will aslo use ollama-python, before starting, make sure it is available in your environment: \`\`\`bash pip install ollama \`\`\` For this guide, the ollama binary is at v0.3.9 and the ollama Python library is at v0.3.2. #### Preparing The messages structure used in Ollama is the same with that in \`transformers\`. However, as explained \[before\](#note-official-template), it's quite difficult to implement the Qwen2 official template using Go templates. The following can be used to create a new model with a function calling template: \`\`\`python import ollama model\_name = "qwen2fc:7b-instruct-q4\_K\_M" modelfile = r'''FROM qwen2:7b-instruct-q4\_K\_M TEMPLATE """ {{- $system\_message := "You are a helpful assistant." }} {{- $loop\_messages := .Messages }} {{- if .System }} {{- $system\_message = .System }} {{- end }} {{- if eq ((index .Messages 0).Role) "system" }} {{- $system\_message = ((index .Messages 0).Content) }} {{- $loop\_messages = slice .Messages 1 }} {{- end -}} <|im\_start|>system {{ $system\_message }} {{- if .Tools }} # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <|tools\_start|><|tools\_end|> tags: <|tools\_start|>\[{{ range $index, $\_ := .Tools }}{{ if ne $index 0 }},{{ end }}{{ json .Function }}{{ end }}\]<|tools\_end|> For each function call, return a JSON object with function name and arguments within <|tool\_call\_start|><|tool\_call\_end|> tags: <|tool\_call\_start|>{"name": , "arguments": }<|tool\_call\_end|> {{- end -}} <|im\_end|> {{- range $loop\_messages }} {{- if and (eq .Role "assistant") .ToolCalls }} <|im\_start|>assistant {{- range .ToolCalls }}<|tool\_call\_start|>{{ json .Function }}<|tool\_call\_end|>{{ end }}<|im\_end|> {{- else }} <|im\_start|>{{ .Role }} {{ .Content }}<|im\_end|> {{- end }} {{- end }} <|im\_start|>assistant """ ''' ollama.create(model\_name, modelfile=modelfile) \`\`\` You can change the model name and tag after \`FROM\` to use another as base. If that model is not available locally, it will download the model and may take some time. If successful, you will receive the following return values: \`\`\`python {"status": "success"} \`\`\` Then, the new model can be used later as \`qwen2fc:7b-instruct-q4\_K\_M\`. The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES\[:\] \`\`\` Note that you cannot pass Python functions as tools directly and \`tools\` has to be a \`dict\`. #### Tool Calls and Tool Results We can use the \`ollama.chat\` method to directly query the underlying API: \`\`\`python import ollama response = ollama.chat( model=model\_name, messages=messages, tools=tools, ) \`\`\` The main fields in the response could be: \`\`\`python { "model": "qwen2fc:7b-instruct-q4\_K\_M", "message": { "role": "assistant", "content": '<|tool\_call\_start|>{"name": "get\_current\_temperature", "arguments": "{\\\\"location\\\\": \\\\"San Francisco, CA, USA\\\\", \\\\"unit\\\\": \\\\"celsius\\\\"}"}<|tool\_call\_end|>\\n<|tool\_call\_start|>{"name": "get\_temperature\_date", "arguments": "{\\\\"date\\\\": \\\\"2024-09-01\\\\", \\\\"location\\\\": \\\\"San Francisco, CA, USA\\\\", \\\\"unit\\\\": \\\\"celsius\\\\"}"}<|tool\_call\_end|>' }, "done\_reason": 'stop', "done": True, } \`\`\` Well, it seems that the format we adopt does not work well with Ollama's tool call parser currently. Let's reuse \[the \`try\_parse\_tool\_calls\` function above\](#prepcode) since Ollama also needs the tool call arguments to be JSON objects.\[^tool\_call\_arg\_format\] Then, we can obtain the tool results and add them to the messages. The following is basically the same with \`transformers\`: \`\`\`python messages.append(try\_parse\_tool\_calls(response\["message"\]\["content"\])) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = fn\_call\["arguments"\] fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "name": fn\_name, "content": fn\_res, }) \`\`\` The messages are now like \`\`\`python \[ {"role": "system", "content": "You are a helpful assistant.\\n\\nCurrent Date: 2024-08-31"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, {"role": "assistant", "content": "", "tool\_calls": \[ {'function': {'name': 'get\_current\_temperature', 'arguments': {'location': 'San Francisco, CA, USA', "unit": "celsius"}}}, {'function': {'name': 'get\_temperature\_date', 'arguments': {'date': '2024-09-01', 'location': 'San Francisco, CA, USA', "unit": "celsius"}}} \]}, {'role': 'tool', 'name': 'get\_current\_temperature', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get\_temperature\_date', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response The rest are easy: \`\`\`python response = ollama.chat( model=model\_name, messages=messages, tools=tools, ) messages.append(response\["message"\]) \`\`\` The final message should be like the following: \`\`\`python {"role": "assistant", "content": "The current temperature in San Francisco is approximately 26.1 degrees Celsius. Tomorrow, the forecasted temperature in San Francisco will be around 25.9 degrees Celsius."} \`\`\` ### Discussions Now, we have introduced how to conduct inference with function calling using Qwen2 in three different frameworks! Let's make a brief comparison. | Item | OpenAI API | Hugging Face transformers | Ollama | Qwen-Agent | | :----- | :---: | :---: | :---: | :---: | | Type | HTTP API | Python Library | HTTP API | Python Library | | Inference Backend | - | PyTorch | llama.cpp | HTTP API | | Templating Backend | - | Jinja | Go \`text/template\` | Python | | Tools/Functions | Tools | Tools | Tools | Functions | | Parallel Calls | Default Yes (Configurable) | Yes | Yes | Default No (Configurable) | | Call Format | Single assistant message with \`tool\_calls\` | Single assistant message with \`tool\_calls\` | Single assistant message with \`tool\_calls\` | Multiple assistant messages with \`function\_call\` | | Call Argument Format | string | object | object | string | | Call Result Format | Multiple tool messages with \`content\` | Multiple tool messages with \`content\` | Multiple tool messages with \`content\` | Multiple function messages with \`content\` | There are some details not shown in the above table: - OpenAI API comes with Python, Node.js, Go, and .NET SDKs. It also follows the OpenAPI standard. - Ollama comes with Python and Node.js SDKs. It has OpenAI-compatible API at a different base url that can be accessed using OpenAI API SDK. - Qwen-Agent as an application framework can call the tools automatically for you, which is introduced in \[the Qwen-Agent guide\](./qwen\_agent). In addition, there are more on the model side of function calling, which means you may need to consider more things in production code: - \*\*Accuracy of function calling\*\*: When it comes to evaluate the accuracy of function calling, there are two aspects: (a) whether the correct functions (including no functions) are selected and (b) whether the correct function arguments are generated. It is not always the case that Qwen2 will be accurate. Function calling can involve knowledge that is deep and domain-specific. Sometimes, it doesn't fully understand the function and select the wrong one by mistake. Sometimes, it can fall into a loop and require calling the same function again and again. Sometimes, it will fabricate required function arguments instead of asking the user for input. To improve the function calling accuracy, it is advised to first try prompt engineering: does a more detailed function description help? can we provide instructions and examples to the model in the system message? If not, finetuning on your own data could also improve performance. - \*\*Protocol consistency\*\*: Even with the proper function calling template, the protocol may break. The model may generate extra texts to tool calls, e.g., explanations. The generated tool call may be invalid JSON-formatted string but a representation of a Python dict The generated tool call may be valid JSON but not conforms to the provided JSON Schema. For those kinds of issues, while some of them could be addressed with prompt engineering, some are caused by the nature of LLMs and can be hard to resolve in a general manner by LLMs themselves. While we strive to improve Qwen2 in this regard, edge cases are unlikely to be eliminated completely. ## Function Calling Templates The template design for function calling often includes the following aspects: - How to describe the functions to the model, so that the model understands what they are and how to use them. - How to prompt the model, so that it knows that functions can be used and in what format to generate the function calls. - How to tell a function call generation from others in generated text, so that we can extract the calls from the generated texts and actually make the calls. - How to incorporate the function results to the text, so that the model can tell them from its own generation and make connection among the calls and the results. For experienced prompt engineers, it should be possible to make any LLM support function calling, using in-context learning techniques and with representative examples, though with varied accuracy and stability depending on how "zero-shot" the task at hand is. ### Starting from ReAct Prompting For example, ReAct Prompting can be used to implement function calling with an extra element of planning: - \*\*Thought\*\*: the overt reasoning path, analyzing the functions and the user query and saying it out "loud" - \*\*Action\*\*: the function to use and the arguments with which the function should be called - \*\*Observation\*\*: the results of the function In fact, Qwen2 is verse in the following variant of ReAct Prompting (similar to LangChain ReAct) to make the intermediate texts more structured: \`\`\` Answer the following questions as best you can. You have access to the following tools: {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} {function\_name}: Call this tool to interact with the {function\_name\_human\_readable} API. What is the {function\_name\_human\_readable} API useful for? {function\_desciption} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of \[{function\_name},{function\_name}\] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can be repeated zero or more times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: {query} Thought: {some\_text} Action: {function\_name} Action Input: {function\_arguments} Observation: {function\_results} Final Answer: {response} \`\`\` As you can see, there is no apparent user/assistant conversation structure in the template. The model will simply continue the texts. One should write the code to actively detect which step the model is at and in particular to add the observations in the process, until the Final Answer is generated. However, as most programming interfaces accept the message structure, there should be some kind of adapter between the two. \[The ReAct Chat Agent\](https://github.com/QwenLM/Qwen-Agent/blob/v0.0.9/qwen\_agent/agents/react\_chat.py) in Qwen-Agent facilitates this kind of conversion. ### Qwen2 "Official" Function Calling Template As a step forward, the official Qwen2 function calling template is in the vein of the ReAct Prompting format but focuses more on - differentiating the keywords like \`Question\`, \`Thought\`, \`Action\`, etc., from generation, - simplifying the process, - supporting better multi-turn conversation, and - adding controls for specialized usage. An equivalent example would be \`\`\` <|im\_start|>system You are a helpful assistant. ## Tools You have access to the following tools: ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ### {function\_name\_human\_readable} {function\_name}: {function\_description} Parameters: {function\_parameter\_descriptions} {argument\_formatting\_instructions} ## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of \[{function\_name},{function\_name}\] ✿ARGS✿: The input of the tool ✿RESULT✿: Tool results ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user {query}<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: {function\_name} ✿ARGS✿: {function\_arguments} ✿RESULT✿: {function\_result} ✿RETURN✿:{response}<|im\_end|> \`\`\` Let's first list the obvious differences: - Keywords (\`✿FUNCTION✿\`, \`✿ARGS✿\`, etc.) seem rare in ordinary text and more semantically related to function calling, but not special tokens yet. - Thought is omitted. This could affect accuracy for some use cases. - Use the system-user-assistant format for multi-turn conversations. Function calling prompting is moved to the system message. How about adding controls for specialized usage? The template actually has the following variants: - Language: the above is for non-Chinese language; there is another template in Chinese. - Parallel Calls: the above is for non-parallel calls; there is another template for parallel calls. In the canonical implementation in Qwen-Agent, those switches are implemented in Python, according to the configuration and current input. The actual text with parallel calls should be like the following: \`\`\`text <|im\_start|>system You are a helpful assistant. Current Date: 2024-08-31 ## Tools You have access to the following tools: ### get\_current\_temperature get\_current\_temperature: Get current temperature at a location. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]} Format the arguments as a JSON object. ### get\_temperature\_date get\_temperature\_date: Get temperature at a location and date. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]} Format the arguments as a JSON object. ## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of \[get\_current\_temperature,get\_temperature\_date\] ✿ARGS✿: The input of the tool ✿RESULT✿: Tool results ✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)<|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant ✿FUNCTION✿: get\_current\_temperature ✿ARGS✿: {"location": "San Francisco, CA, USA", "unit": "celsius"} ✿FUNCTION✿: get\_temperature\_date ✿ARGS✿: {"date": "2024-09-01", "location": "San Francisco, CA, USA", "unit": "celsius"} ✿RESULT✿: {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"} ✿RESULT✿: {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"} ✿RETURN✿: The current temperature in San Francisco is 26.1°C. The temperature for tomorrow in San Francisco is expected to be 25.9°C.<|im\_end|> \`\`\` \[Previously\](#note-official-template), we have said that it is hard to adapt it for other frameworks that use less capable templating engines. But it is doable at least partially for Jinja, which is Python-oriented after all. We didn't use it because using the template in \`transformers\` leads to more changes to the inference usage, which are not very common for beginners. For the interested, you can find the Jinja template and key points on usage below: :::{dropdown} Qwen2 Function Calling Jinja Template \`\`\`jinja {%- if messages\[0\]\["role"\] == "system" %} {%- set system\_message = messages\[0\]\["content"\] %} {%- set loop\_messages = messages\[1:\] %} {%- else %} {%- set system\_message = "You are a helpful assistant." %} {%- set loop\_messages = messages %} {%- endif %} {%- if parallel\_tool\_calls is undefined %} {%- set parallel\_tool\_calls = false %} {%- endif %} {%- if language is undefined or language != "zh" %} {%- set language = "en" %} {%- endif %} {{- "<|im\_start|>system\\n" + system\_message|trim }} {%- if tools is defined %} {{- "\\n\\n# 工具\\n\\n## 你拥有如下工具：\\n\\n" if language == "zh" else "\\n\\n## Tools\\n\\nYou have access to the following tools:\\n\\n" }} {%- set functions = tools|map(attribute="function")|list %} {%- set function\_names = functions|map(attribute="name")|join(",") %} {%- for function in functions %} {{- "### " + function.name + "\\n\\n" + function.name + ": " + function.description + (" 输入参数：" if language == "zh" else " Parameters: ") + function.parameters|tojson + (" 此工具的输入应为JSON对象。\\n\\n" if language == "zh" else " Format the arguments as a JSON object.\\n\\n") }} {%- endfor %} {%- if parallel\_tool\_calls and language == "zh" %} {{- "## 你可以在回复中插入以下命令以并行调用N个工具：\\n\\n✿FUNCTION✿: 工具1的名称，必须是\[" + function\_names + "\]之一\\n✿ARGS✿: 工具1的输入\\n✿FUNCTION✿: 工具2的名称\\n✿ARGS✿: 工具2的输入\\n...\\n✿FUNCTION✿: 工具N的名称\\n✿ARGS✿: 工具N的输入\\n✿RESULT✿: 工具1的结果\\n✿RESULT✿: 工具2的结果\\n...\\n✿RESULT✿: 工具N的结果\\n✿RETURN✿: 根据工具结 果进行回复，需将图片用!\[\](url)渲染出来" }} {%- elif parallel\_tool\_calls %} {{- "## Insert the following command in your reply when you need to call N tools in parallel:\\n\\n✿FUNCTION✿: The name of tool 1, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of tool 1\\n✿FUNCTION✿: The name of tool 2\\n✿ARGS✿: The input of tool 2\\n...\\n✿FUNCTION✿: The name of tool N\\n✿ARGS✿: The input of tool N\\n✿RESULT✿: The result of tool 1\\n✿RESULT✿: The result of tool 2\\n...\\n✿RESULT✿: The result of tool N\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%- elif language == "zh" %} {{- "## 你可以在回复中插入零次、一次或多次以下命令以调用工具：\\n\\n✿FUNCTION✿: 工具名称，必须是\[" + function\_names + "\]之一。\\n✿ARGS✿: 工具输入\\n✿RESULT✿: 工具结果\\n✿RETURN✿: 根据工具结果进行回复，需将图片用!\[\](url)渲染出来" }} {%- else %} {{- "## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs:\\n\\n✿FUNCTION✿: The tool to use, should be one of \[" + function\_names + "\]\\n✿ARGS✿: The input of the tool\\n✿RESULT✿: Tool results\\n✿RETURN✿: Reply based on tool results. Images need to be rendered as !\[\](url)" }} {%- endif %} {%- endif %} {{- "<|im\_end|>" }} {%- for message in loop\_messages %} {%- if message.role == "user" %} {{- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%- if loop.last and add\_generation\_prompt %} {{- "\\n<|im\_start|>assistant\\n" }} {%- endif %} {%- elif message.role == "tool" %} {{- "✿RESULT✿: " + message.content + "\\n" }} {%- if loop.last and add\_generation\_prompt %} {{- "✿RETURN✿:" }} {%- endif %} {%- elif message.role == "assistant" and message.tool\_calls is defined %} {%- if loop.previtem.role == "user" %} {{- "\\n<|im\_start|>assistant\\n" }} {%- endif %} {%- for function in message.tool\_calls|map(attribute="function") %} {{- "✿FUNCTION✿: " + function.name + "\\n✿ARGS✿: " + function.arguments|tojson + "\\n" }} {%- endfor %} {%- elif message.role == "assistant" %} {%- if loop.previtem.role == "user" %} {{- "\\n<|im\_start|>assistant\\n" }} {%- elif loop.previtem.role == "tool" %} {{- "✿RETURN✿:" }} {%- endif %} {{- message.content }} {%- if loop.nextitem is undefined or loop.nextitem.role == "user" %} {{- "<|im\_end|>" }} {%- endif %} {%- else %} {{- "\\n<|im\_start|>" + message.role + "\\n" + message.content + "<|im\_end|>" }} {%- endif %} {%- endfor %} \`\`\` To use this template in \`transformers\`: - Switches can be enabled by passing them to the \`apply\_chat\_template\` method, e.g., \`tokenizer.apply\_chat\_template(messages, tools=tools, add\_generation\_prompt=True, parallel\_tool\_call=True, language="zh", tokenize=False)\`. By default, it is for English non-parallel function calling. - Since the generation needs to be stopped at \`✿RESULT✿\` or else the model will generate fabricated tool results, we should add it to \`stop\_strings\` in \`generation\_config\`: \`\`\`python model.generation\_config.stop\_strings = \["✿RESULT✿:", "✿RETURN✿:"\] \`\`\` - As a result of using \`stop\_strings\`, you need to pass the tokenizer to \`model.generate\` as \`model.generate(\*\*inputs, tokenizer=tokenizer, max\_new\_tokens=512)\`. - \`response\`, i.e., the model generation based on the tool calls and tool results, may contain a leading space. You should not strip it for the model. It is resulted from the tokenization and the template design. - The \`try\_parse\_tool\_calls\` function should also be modified accordingly. ::: ### The Unofficial Function Calling Templates For \`transformers\` and Ollama, we have also used unofficial templates and they also seem to work. What do they look like? The templates we have used for \`transformers\` and Ollama are variants of \[the Nous Research's Hermes function calling template\](https://github.com/NousResearch/Hermes-Function-Calling#prompt-format-for-function-calling). The Jinja template and the Go template should produce basically the same results. They final text should look like the following: \`\`\`text <|im\_start|>system You are a helpful assistant. Current Date: 2024-08-31 # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <|tools\_start|><|tools\_end|> tags: <|tools\_start|>\[{"name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location"\]}}, {"name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"."}, "unit": {"type": "string", "enum": \["celsius", "fahrenheit"\], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"."}}, "required": \["location", "date"\]}}\]<|tools\_end|> For each function call, return a JSON object with function name and arguments within <|tool\_call\_start|><|tool\_call\_end|> tags: <|tool\_call\_start|>{"name": , "arguments": }<|tool\_call\_end|><|im\_end|> <|im\_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im\_end|> <|im\_start|>assistant <|tool\_call\_start|>{"name": "get\_current\_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}<|tool\_call\_end|> <|tool\_call\_start|>{"name": "get\_temperature\_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"}}<|tool\_call\_end|><|im\_end|> <|im\_start|>tool {"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}<|im\_end|> <|im\_start|>tool {"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-09-01", "unit": "celsius"}<|im\_end|> <|im\_start|>assistant The current temperature in San Francisco is 26.1°C. The temperature for tomorrow in San Francisco is expected to be 25.9°C.<|im\_end|> \`\`\` While the text may seem different from the previous one, the basic prompting structure is still the same. There are just more structural tags and more JSON-formatted strings. We also demonstrate the usage with those templates to show that - LLMs are not necessarily fragile. Using a different template won't break it. - Prompt engineering can be helpful in steering LLM generations to one's need. --- There is one thing we haven't talked about: how should functions be described to the LLMs. In short, you could describe them as you would normally describe them in an API documentation, as long as you can effectively parse, validate, and execute the tool calls generated by the models. The format with JSON Schema appears a valid and common choice. ## Finally In whichever way you choose to use function calling with Qwen2, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Unknown {%- if tools %} {{- '<|im\_start|>system\\n' }} {%- if messages\[0\].role == 'system' %} {{- messages\[0\].content + '\\n\\n' }} {%- endif %} {{- "# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n" }} {%- for tool in tools %} {{- "\\n" }} {{- tool | tojson }} {%- endfor %} {{- "\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\"name\\": , \\"arguments\\": }\\n<|im\_end|>\\n" }} {%- else %} {%- if messages\[0\].role == 'system' %} {{- '<|im\_start|>system\\n' + messages\[0\].content + '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- set ns = namespace(multi\_step\_tool=true, last\_query\_index=messages|length - 1) %} {%- for message in messages\[::-1\] %} {%- set index = (messages|length - 1) - loop.index0 %} {%- if ns.multi\_step\_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} {%- set ns.multi\_step\_tool = false %} {%- set ns.last\_query\_index = index %} {%- endif %} {%- endfor %} {%- for message in messages %} {%- if message.content is string %} {%- set content = message.content %} {%- else %} {%- set content = '' %} {%- endif %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im\_start|>' + message.role + '\\n' + content + '<|im\_end|>' + '\\n' }} {%- elif message.role == "assistant" %} {%- set reasoning\_content = '' %} {%- if message.reasoning\_content is string %} {%- set reasoning\_content = message.reasoning\_content %} {%- else %} {%- if '' in content %} {%- set reasoning\_content = content.split('')\[0\].rstrip('\\n').split('')\[-1\].lstrip('\\n') %} {%- set content = content.split('')\[-1\].lstrip('\\n') %} {%- endif %} {%- endif %} {%- if loop.index0 > ns.last\_query\_index %} {%- if loop.last or (not loop.last and reasoning\_content) %} {{- '<|im\_start|>' + message.role + '\\n\\n' + reasoning\_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- else %} {{- '<|im\_start|>' + message.role + '\\n' + content }} {%- endif %} {%- if message.tool\_calls %} {%- for tool\_call in message.tool\_calls %} {%- if (loop.first and content) or (not loop.first) %} {{- '\\n' }} {%- endif %} {%- if tool\_call.function %} {%- set tool\_call = tool\_call.function %} {%- endif %} {{- '\\n{"name": "' }} {{- tool\_call.name }} {{- '", "arguments": ' }} {%- if tool\_call.arguments is string %} {{- tool\_call.arguments }} {%- else %} {{- tool\_call.arguments | tojson }} {%- endif %} {{- '}\\n' }} {%- endfor %} {%- endif %} {{- '<|im\_end|>\\n' }} {%- elif message.role == "tool" %} {%- if loop.first or (messages\[loop.index0 - 1\].role != "tool") %} {{- '<|im\_start|>user' }} {%- endif %} {{- '\\n\\n' }} {{- content }} {{- '\\n' }} {%- if loop.last or (messages\[loop.index0 + 1\].role != "tool") %} {{- '<|im\_end|>\\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add\_generation\_prompt %} {{- '<|im\_start|>assistant\\n\\n\\n\\n\\n' }} {%- endif %} --- # Unknown \--- myst: number\_code\_blocks: \["python3"\] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen3. - Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. ## Inference with Function Calling As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - \*\*Qwen-Agent\*\*, - \*\*vLLM\*\*. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume \*\*Python 3.11\*\* is used as the programming language. \*\*Scenario\*\*: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode \`\`\`python import json def get\_current\_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name == "get\_current\_temperature": return get\_current\_temperature if name == "get\_temperature\_date": return get\_temperature\_date TOOLS = \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location"\], }, }, }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location", "date"\], }, }, }, \] MESSAGES = \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, \] \`\`\` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: \`\`\`json \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location" \] } } }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location", "date" \] } } } \] \`\`\` For each \*\*tool\*\*, it is a JSON object with two fields: - \`type\`: a string specifying the type of the tool, currently only \`"function"\` is valid - \`function\`: an object detailing the instructions to use the function For each \*\*function\*\*, it is a JSON object with three fields: - \`name\`: a string indicating the name of the function - \`description\`: a string describing what the function is used for - \`parameters\`: \[a JSON Schema\](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include \`type\`, \`required\`, and \`enum\`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is \`What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.\`. \`\`\`json \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."} \] \`\`\` ::: ### Qwen-Agent \[Qwen-Agent\](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the \*\*canonical implementation\*\* of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is \*not recommended\* to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let's make sure the latest library is installed: \`\`\`bash pip install -U qwen-agent \`\`\` #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at \`http://localhost:8000/v1\`, Qwen-Agent provides a shortcut function \`get\_chat\_model\` to obtain a model inference class with function calling support: \`\`\`python from qwen\_agent.llm import get\_chat\_model llm = get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} # default to True } } }) \`\`\` In the above, \`model\_server\` is the \`api\_base\` common used in other OpenAI-compatible API clients. It is advised to provide the \`api\_key\` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by \`generate\_cfg\`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: \`\`\`python messages = MESSAGES\[:\] \`\`\` At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: \`\`\`python functions = \[tool\["function"\] for tool in TOOLS\] \`\`\` #### Tool Calls and Tool Results To interact with the model, the \`chat\` method should be used: \`\`\`python for responses in llm.chat( messages=messages, functions=functions, ): pass messages.extend(responses) \`\`\` The \`chat\` method returns a generator of list, each of which may contain multiple messages. - The results of \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` - The results of \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the \`function\_call\` field of the messages: - \`name\`: a string representing the function to call - \`arguments\`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part -- checking and applying the function call: \`\`\`python3 for message in responses: if fn\_call := message.get("function\_call", None): fn\_name: str = fn\_call\['name'\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "function", "name": fn\_name, "content": fn\_res, }) \`\`\` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the \`function\_call\` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are \`name\` and \`arguments\` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named \[\`get\_function\_by\_name\`\](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as \`content\` and with \`role\` as \`"function"\`. Now the messages are: - \`no\_think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response Finally, run the model again to get the final model results: \`\`\`python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) \`\`\` The final response should be like - \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from \`transformers\` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. - \`vllm\` >= v0.8.5. For more information, check the \[vLLM documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#tool-calling-in-the-chat-completion-api). We will use the OpenAI-Compatible API by \`vllm\` with the API client from the \`openai\` Python library. #### Preparing For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: \`\`\`bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek\_r1 \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES \`\`\` Let's also initialize the client: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) model\_name = "Qwen/Qwen3-8B" \`\`\` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model. Here is an example of the \`no\_think\` mode: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} # default to True }, ) \`\`\` vLLM should be able to parse the tool calls for us, and the main fields in the response (\`response.choices\[0\]\`) should be like \`\`\`python Choice( finish\_reason='tool\_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function\_call=None, tool\_calls=\[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get\_current\_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get\_temperature\_date'), type='function', ), \], ), stop\_reason=None, ) \`\`\` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: \`\`\`python messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str = tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) \`\`\` It should be noted that the OpenAI API uses \`tool\_call\_id\` to identify the relation between tool results and tool calls. The messages are now like \`\`\`python \[ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'}, \]}, {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, \] \`\`\` #### Final Response Let's call the endpoint again to seed the tool results and get response: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) \`\`\` The final response (\`response.choices\[0\].message.content\`) should be like \`\`\`text The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. \`\`\` ## Finally In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Unknown \--- myst: number\_code\_blocks: \["python3"\] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen3. - Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. ## Inference with Function Calling As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - \*\*Qwen-Agent\*\*, - \*\*vLLM\*\*. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume \*\*Python 3.11\*\* is used as the programming language. \*\*Scenario\*\*: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode \`\`\`python import json def get\_current\_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get\_temperature\_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: \["celsius", "fahrenheit"\]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get\_function\_by\_name(name): if name == "get\_current\_temperature": return get\_current\_temperature if name == "get\_temperature\_date": return get\_temperature\_date TOOLS = \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location"\], }, }, }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": \["celsius", "fahrenheit"\], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": \["location", "date"\], }, }, }, \] MESSAGES = \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, \] \`\`\` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: \`\`\`json \[ { "type": "function", "function": { "name": "get\_current\_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location" \] } } }, { "type": "function", "function": { "name": "get\_temperature\_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \\"City, State, Country\\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \\"Year-Month-Day\\"." }, "unit": { "type": "string", "enum": \[ "celsius", "fahrenheit" \], "description": "The unit to return the temperature in. Defaults to \\"celsius\\"." } }, "required": \[ "location", "date" \] } } } \] \`\`\` For each \*\*tool\*\*, it is a JSON object with two fields: - \`type\`: a string specifying the type of the tool, currently only \`"function"\` is valid - \`function\`: an object detailing the instructions to use the function For each \*\*function\*\*, it is a JSON object with three fields: - \`name\`: a string indicating the name of the function - \`description\`: a string describing what the function is used for - \`parameters\`: \[a JSON Schema\](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include \`type\`, \`required\`, and \`enum\`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is \`What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.\`. \`\`\`json \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."} \] \`\`\` ::: ### Qwen-Agent \[Qwen-Agent\](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the \*\*canonical implementation\*\* of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is \*not recommended\* to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let's make sure the latest library is installed: \`\`\`bash pip install -U qwen-agent \`\`\` #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at \`http://localhost:8000/v1\`, Qwen-Agent provides a shortcut function \`get\_chat\_model\` to obtain a model inference class with function calling support: \`\`\`python from qwen\_agent.llm import get\_chat\_model llm = get\_chat\_model({ "model": "Qwen/Qwen3-8B", "model\_server": "http://localhost:8000/v1", "api\_key": "EMPTY", "generate\_cfg": { "extra\_body": { "chat\_template\_kwargs": {"enable\_thinking": False} # default to True } } }) \`\`\` In the above, \`model\_server\` is the \`api\_base\` common used in other OpenAI-compatible API clients. It is advised to provide the \`api\_key\` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by \`generate\_cfg\`. Here we demonstrate how to control the think and no\_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: \`\`\`python messages = MESSAGES\[:\] \`\`\` At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: \`\`\`python functions = \[tool\["function"\] for tool in TOOLS\] \`\`\` #### Tool Calls and Tool Results To interact with the model, the \`chat\` method should be used: \`\`\`python for responses in llm.chat( messages=messages, functions=functions, ): pass messages.extend(responses) \`\`\` The \`chat\` method returns a generator of list, each of which may contain multiple messages. - The results of \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` - The results of \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, \] \`\`\` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the \`function\_call\` field of the messages: - \`name\`: a string representing the function to call - \`arguments\`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part -- checking and applying the function call: \`\`\`python3 for message in responses: if fn\_call := message.get("function\_call", None): fn\_name: str = fn\_call\['name'\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "function", "name": fn\_name, "content": fn\_res, }) \`\`\` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the \`function\_call\` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are \`name\` and \`arguments\` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named \[\`get\_function\_by\_name\`\](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as \`content\` and with \`role\` as \`"function"\`. Now the messages are: - \`no\_think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\\n\\nFirst, there's the get\_current\_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \\"San Francisco, State, Country\\". Wait, the example format is \\"City, State, Country\\", but San Francisco is a city in California, USA. So the location parameter would be \\"San Francisco, California, United States\\".\\n\\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get\_temperature\_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \\"Year-Month-Day\\", which is 2024-10-01.\\n\\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \\"2024-10-01\\".\\n\\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_current\_temperature", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "assistant", "content": "", "function\_call": {"name": "get\_temperature\_date", "arguments": "{\\"location\\": \\"San Francisco, California, United States\\", \\"date\\": \\"2024-10-01\\", \\"unit\\": \\"celsius\\"}"}}, {"role": "function", "name": "get\_current\_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get\_temperature\_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, \] \`\`\` #### Final Response Finally, run the model again to get the final model results: \`\`\`python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) \`\`\` The final response should be like - \`no\_think\` mode: \`\`\`python \[ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` - \`think\` mode: \`\`\`python \[ {"role": "assistant", "content": "", "reasoning\_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get\_current\_temperature function for now and get\_temperature\_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\\n\\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is \*\*26.1°C\*\*. \\n\\nFor tomorrow (2024-10-01), the temperature is projected to be \*\*25.9°C\*\*. \\n\\nThere is a slight decrease in temperature expected from today to tomorrow."} \] \`\`\` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from \`transformers\` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. - \`vllm\` >= v0.8.5. For more information, check the \[vLLM documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#tool-calling-in-the-chat-completion-api). We will use the OpenAI-Compatible API by \`vllm\` with the API client from the \`openai\` Python library. #### Preparing For Qwen3, the chat template in tokenizer\_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: \`\`\`bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek\_r1 \`\`\` The inputs are the same with those in \[the preparation code\](#prepcode): \`\`\`python tools = TOOLS messages = MESSAGES \`\`\` Let's also initialize the client: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) model\_name = "Qwen/Qwen3-8B" \`\`\` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model. Here is an example of the \`no\_think\` mode: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, "chat\_template\_kwargs": {"enable\_thinking": False} # default to True }, ) \`\`\` vLLM should be able to parse the tool calls for us, and the main fields in the response (\`response.choices\[0\]\`) should be like \`\`\`python Choice( finish\_reason='tool\_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function\_call=None, tool\_calls=\[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get\_current\_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get\_temperature\_date'), type='function', ), \], ), stop\_reason=None, ) \`\`\` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: \`\`\`python messages.append(response.choices\[0\].message.model\_dump()) if tool\_calls := messages\[-1\].get("tool\_calls", None): for tool\_call in tool\_calls: call\_id: str = tool\_call\["id"\] if fn\_call := tool\_call.get("function"): fn\_name: str = fn\_call\["name"\] fn\_args: dict = json.loads(fn\_call\["arguments"\]) fn\_res: str = json.dumps(get\_function\_by\_name(fn\_name)(\*\*fn\_args)) messages.append({ "role": "tool", "content": fn\_res, "tool\_call\_id": call\_id, }) \`\`\` It should be noted that the OpenAI API uses \`tool\_call\_id\` to identify the relation between tool results and tool calls. The messages are now like \`\`\`python \[ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {'content': None, 'role': 'assistant', 'function\_call': None, 'tool\_calls': \[ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get\_current\_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get\_temperature\_date'}, 'type': 'function'}, \]}, {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool\_call\_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, \] \`\`\` #### Final Response Let's call the endpoint again to seed the tool results and get response: \`\`\`python response = client.chat.completions.create( model=model\_name, messages=messages, tools=tools, temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) messages.append(response.choices\[0\].message.model\_dump()) \`\`\` The final response (\`response.choices\[0\].message.content\`) should be like \`\`\`text The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. \`\`\` ## Finally In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- # Unknown Qwen2.5 Speed Benchmark ========================= This section reports the speed performance of bf16 models, quantized models (including GPTQ-Int4, GPTQ-Int8 and AWQ) of the Qwen2.5 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under the conditions of different context lengths. The environment of the evaluation with huggingface transformers is: - NVIDIA A100 80GB - CUDA 12.1 - Pytorch 2.3.1 - Flash Attention 2.5.8 - Transformers 4.46.0 - AutoGPTQ 0.7.1+cu121 (Compiled from source code) - AutoAWQ 0.2.6 The environment of the evaluation with vLLM is: - NVIDIA A100 80GB - CUDA 12.1 - vLLM 0.6.3 - Pytorch 2.4.0 - Flash Attention 2.6.3 - Transformers 4.46.0 Notes: - We use the batch size of 1 and the least number of GPUs as possible for the evaluation. - We test the speed and memory of generating 2048 tokens with the input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens. - For vLLM, the memory usage is not reported because it pre-allocates all GPU memory. We use \`\`gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False\`\` by default. - 0.5B (Transformer) +-------------------------+--------------+--------------+---------+-----------------+----------------+---------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +=========================+==============+==============+=========+=================+================+===========================+ | Qwen2.5-0.5B-Instruct | 1 | BF16 | 1 | 47.40 | 0.97 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int8 | 1 | 35.17 | 0.64 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int4 | 1 | 50.60 | 0.48 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | AWQ | 1 | 37.09 | 0.68 | | + +--------------+--------------+---------+-----------------+----------------+---------------------------+ | | 6144 | BF16 | 1 | 47.45 | 1.23 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int8 | 1 | 36.47 | 0.90 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int4 | 1 | 48.89 | 0.73 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | AWQ | 1 | 37.04 | 0.72 | | + +--------------+--------------+---------+-----------------+----------------+---------------------------+ | | 14336 | BF16 | 1 | 47.11 | 1.60 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int8 | 1 | 35.44 | 1.26 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int4 | 1 | 48.26 | 1.10 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | AWQ | 1 | 37.14 | 1.10 | | + +--------------+--------------+---------+-----------------+----------------+---------------------------+ | | 30720 | BF16 | 1 | 47.16 | 2.34 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int8 | 1 | 36.25 | 2.01 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | GPTQ-Int4 | 1 | 49.22 | 1.85 | | + + +--------------+---------+-----------------+----------------+---------------------------+ | | | AWQ | 1 | 36.90 | 1.84 | | +-------------------------+--------------+--------------+---------+-----------------+----------------+---------------------------+ - 0.5B (vLLM) +-------------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +=========================+==============+==============+=========+=================+ | Qwen2.5-0.5B-Instruct | 1 | BF16 | 1 | 311.55 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 257.07 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 260.93 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 261.95 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 304.79 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 254.10 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 257.33 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 259.80 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 290.28 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 243.69 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 247.01 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 249.58 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 264.51 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 223.86 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 226.50 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 229.84 | +-------------------------+--------------+--------------+---------+-----------------+ - 1.5B (Transformer) +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +==========================+==============+==============+=========+=================+================+=========================+ | Qwen2.5-1.5B-Instruct | 1 | BF16 | 1 | 39.68 | 2.95 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 32.62 | 1.82 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 43.33 | 1.18 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 31.70 | 1.51 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 6144 | BF16 | 1 | 40.88 | 3.43 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 31.46 | 2.30 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 43.96 | 1.66 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 32.30 | 1.63 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 14336 | BF16 | 1 | 40.43 | 4.16 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 31.06 | 3.03 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 43.66 | 2.39 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 32.39 | 2.36 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 30720 | BF16 | 1 | 38.59 | 5.62 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 31.04 | 4.49 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 35.68 | 3.85 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 31.95 | 3.82 | | +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ - 1.5B (vLLM) +--------------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +==========================+==============+==============+=========+=================+ | Qwen2.5-1.5B-Instruct | 1 | BF16 | 1 | 183.33 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 201.67 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 217.03 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 213.74 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 176.68 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 192.83 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 206.63 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 203.64 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 168.69 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 183.69 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 195.88 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 192.64 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 152.04 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 162.82 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 173.57 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 170.20 | +--------------------------+--------------+--------------+---------+-----------------+ - 3B (Transformer) +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +==========================+==============+==============+=========+=================+================+=========================+ | Qwen2.5-3B-Instruct | 1 | BF16 | 1 | 30.80 | 5.95 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 25.69 | 3.38 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 35.21 | 2.06 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 25.29 | 2.50 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 6144 | BF16 | 1 | 32.20 | 6.59 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 24.69 | 3.98 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 34.47 | 2.67 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 24.86 | 2.62 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 14336 | BF16 | 1 | 31.72 | 7.47 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 24.70 | 4.89 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 34.36 | 3.58 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 25.19 | 3.54 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 30720 | BF16 | 1 | 25.37 | 9.30 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 21.67 | 6.72 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 23.60 | 5.41 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 24.56 | 5.37 | | +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ - 3B (vLLM) +--------------------------+--------------+--------------+---------+-----------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | +==========================+==============+==============+=========+=================+ | Qwen2.5-3B-Instruct | 1 | BF16 | 1 | 127.61 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 150.02 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 168.20 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 165.50 | + +--------------+--------------+---------+-----------------+ | | 6144 | BF16 | 1 | 123.15 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 143.09 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 159.85 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 156.38 | + +--------------+--------------+---------+-----------------+ | | 14336 | BF16 | 1 | 117.35 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 135.50 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 149.35 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 147.75 | + +--------------+--------------+---------+-----------------+ | | 30720 | BF16 | 1 | 105.88 | + + +--------------+---------+-----------------+ | | | GPTQ-Int8 | 1 | 118.38 | + + +--------------+---------+-----------------+ | | | GPTQ-Int4 | 1 | 129.28 | + + +--------------+---------+-----------------+ | | | AWQ | 1 | 127.19 | +--------------------------+--------------+--------------+---------+-----------------+ - 7B (Transformer) +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +=============================+==============+==============+=========+=================+================+=========================+ | Qwen2.5-7B-Instruct | 1 | BF16 | 1 | 40.38 | 14.38 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 31.55 | 8.42 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 43.10 | 5.52 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 32.03 | 5.39 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 6144 | BF16 | 1 | 38.76 | 15.38 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 31.26 | 9.43 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 38.27 | 6.52 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 32.37 | 6.39 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 14336 | BF16 | 1 | 29.78 | 16.91 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 26.86 | 10.96 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 28.70 | 8.05 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 30.23 | 7.92 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 30720 | BF16 | 1 | 18.83 | 19.97 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 17.59 | 14.01 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 18.45 | 11.11 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 19.11 | 10.98 | | +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ - 7B (vLLM) +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | +=============================+==============+==============+=========+=================+===========================================+ | Qwen2.5-7B-Instruct | 1 | BF16 | 1 | 84.28 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 122.01 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 154.05 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 148.10 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 6144 | BF16 | 1 | 80.70 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 112.38 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 141.98 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 137.64 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 14336 | BF16 | 1 | 77.69 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 105.25 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 129.35 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 124.91 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 30720 | BF16 | 1 | 70.33 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 90.71 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 108.30 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 104.66 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 63488 | BF16 | 1 | 50.86 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 60.52 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 67.97 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 66.42 | setting-64k | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 129024 | BF16 | 1 | 28.94 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 25.97 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 26.37 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 26.57 | vllm==0.6.2, new sample config | +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ \* \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) \* \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) - 14B (Transformer) +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +==========================+==============+==============+=========+=================+================+=========================+ | Qwen2.5-14B-Instruct | 1 | BF16 | 1 | 24.74 | 28.08 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 18.84 | 16.11 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 25.89 | 9.94 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 19.23 | 9.79 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 6144 | BF16 | 1 | 20.51 | 29.50 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 17.80 | 17.61 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 20.06 | 11.36 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 19.21 | 11.22 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 14336 | BF16 | 1 | 13.92 | 31.95 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 12.66 | 19.98 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 13.79 | 13.81 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 14.17 | 13.67 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------+ | | 30720 | BF16 | 1 | 8.20 | 36.85 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int8 | 1 | 7.77 | 24.88 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | GPTQ-Int4 | 1 | 8.14 | 18.71 | | + + +--------------+---------+-----------------+----------------+-------------------------+ | | | AWQ | 1 | 8.31 | 18.57 | | +--------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------+ - 14B (vLLM) +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | +=============================+==============+==============+=========+=================+===========================================+ | Qwen2.5-14B-Instruct | 1 | BF16 | 1 | 46.30 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 70.40 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 98.02 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 92.66 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 6144 | BF16 | 1 | 43.83 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 64.33 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 86.10 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 83.11 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 14336 | BF16 | 1 | 41.91 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 59.21 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 76.85 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 74.03 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 30720 | BF16 | 1 | 37.18 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 49.23 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 60.91 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 59.01 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 63488 | BF16 | 1 | 26.85 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 32.83 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 37.67 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 36.71 | setting-64k | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 129024 | BF16 | 1 | 14.53 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 15.10 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 15.13 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 15.25 | vllm==0.6.2, new sample config | +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ \* \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) \* \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) - 32B (Transformer) +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +=============================+==============+==============+=========+=================+================+===========================================+ | Qwen2.5-32B-Instruct | 1 | BF16 | 1 | 17.54 | 61.58 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 14.52 | 33.56 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 19.20 | 18.94 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 14.60 | 18.67 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 6144 | BF16 | 1 | 12.49 | 63.72 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 11.61 | 35.86 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 13.42 | 21.09 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 13.81 | 20.81 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 14336 | BF16 | 1 | 8.95 | 67.31 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 8.53 | 39.28 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 9.48 | 24.67 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 9.71 | 24.39 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 30720 | BF16 | 1 | 5.59 | 74.47 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 5.42 | 46.45 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 5.79 | 31.84 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 5.85 | 31.56 | | +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ - 32B (vLLM) +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | +=============================+==============+==============+=========+=================+===========================================+ | Qwen2.5-32B-Instruct | 1 | BF16 | 1 | 22.13 | setting1 | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 37.57 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 55.83 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 51.92 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 6144 | BF16 | 1 | 21.05 | setting1 | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 34.67 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 49.96 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 46.68 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 14336 | BF16 | 1 | 19.91 | setting1 | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 31.89 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 44.79 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 41.83 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 30720 | BF16 | 2 | 31.82 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 26.88 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 35.66 | | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 33.75 | | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 63488 | BF16 | 2 | 24.45 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 18.60 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 22.72 | setting-64k | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 21.79 | setting-64k | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 129024 | BF16 | 2 | 14.31 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 1 | 9.77 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 10.39 | vllm==0.6.2, new sample config | + + +--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 1 | 10.34 | vllm==0.6.2, new sample config | +-----------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ \* For context length 129024, the model needs to be predicted with the following config: "model\_max\_length"=131072 \* \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) \* \[Setting 1\]=(gpu\_memory\_utilization=1.0 max\_model\_len=32768 enforce\_eager=True) \* \[Setting-64k\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) \* \[new sample config\]: for vLLM, set the following sampling parameters: SamplingParams(temperature=0.7,top\_p=0.8,top\_k=20,repetition\_penalty=1,presence\_penalty=0,frequency\_penalty=0,max\_tokens=out\_length) - 72B (Transformer) +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | GPU Memory(GB) | Note | +=============================+==============+==============+=========+=================+================+===========================================+ | Qwen2.5-72B-Instruct | 1 | BF16 | 2 | 8.73 | 136.20 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 8.66 | 72.61 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 11.07 | 39.91 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 11.50 | 39.44 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 6144 | BF16 | 2 | 6.39 | 140.00 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 6.39 | 77.81 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 7.56 | 42.50 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 8.17 | 42.13 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 14336 | BF16 | 3 | 4.25 | 149.14 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 4.66 | 82.55 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 5.27 | 46.86 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 1 | 5.57 | 46.38 | | + +--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ | | 30720 | BF16 | 3 | 2.94 | 164.79 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 2.94 | 94.75 | auto\_gptq==0.6.0+cu1210 | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 3.14 | 62.57 | | + + +--------------+---------+-----------------+----------------+-------------------------------------------+ | | | AWQ | 2 | 3.23 | 61.64 | | +-----------------------------+--------------+--------------+---------+-----------------+----------------+-------------------------------------------+ - 72B (vLLM) +------------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ | Model | Input Length | Quantization | GPU Num | Speed(tokens/s) | Note | +==============================+==============+==============+=========+=================+===========================================+ | Qwen2.5-72B-Instruct | 1 | BF16 | 2 | 18.19 | Setting 1 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | BF16 | 4 | 31.37 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 31.40 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 16.47 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 46.30 | Setting 2 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 44.30 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 6144 | BF16 | 4 | 29.90 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 29.37 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 1 | 13.88 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 42.50 | Setting 3 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 40.67 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 14336 | BF16 | 4 | 30.10 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 27.20 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 38.10 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 36.63 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 30720 | BF16 | 4 | 27.53 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 23.32 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 30.98 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 30.02 | Default | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 63488 | BF16 | 4 | 20.74 | Setting 4 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 2 | 16.27 | Setting 4 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 19.84 | Setting 4 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 19.32 | Setting 4 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | 129024 | BF16 | 4 | 12.68 | Setting 5 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int8 | 4 | 14.11 | Setting 5 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | GPTQ-Int4 | 2 | 10.11 | Setting 5 | + +--------------+--------------+---------+-----------------+-------------------------------------------+ | | | AWQ | 2 | 9.88 | Setting 5 | +------------------------------+--------------+--------------+---------+-----------------+-------------------------------------------+ \* \[Default Setting\]=(gpu\_memory\_utilization=0.9 max\_model\_len=32768 enforce\_eager=False) \* \[Setting 1\]=(gpu\_memory\_utilization=0.98 max\_model\_len=4096 enforce\_eager=True) \* \[Setting 2\]=(gpu\_memory\_utilization=1.0 max\_model\_len=4096 enforce\_eager=True) \* \[Setting 3\]=(gpu\_memory\_utilization=1.0 max\_model\_len=8192 enforce\_eager=True) \* \[Setting 4\]=(gpu\_memory\_utilization=0.9 max\_model\_len=65536 enforce\_eager=False) \* \[Setting 5\]=(gpu\_memory\_utilization=0.9 max\_model\_len=131072 enforce\_eager=False) --- # Unknown LlamaIndex ========== .. attention:: To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on \`LlamaIndex \`\_\_. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: .. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the \`\`bge-base-en-v1.5\`\` model to retrieve from English documents, and you can download the \`\`bge-base-zh-v1.5\`\` model to retrieve from Chinese documents. You can also choose \`\`bge-large\`\` or \`\`bge-small\`\` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) .. code:: python import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt = "" for message in messages: if message.role == "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role == "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role == "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt = "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt = prompt + "<|im\_start|>assistant\\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings.llm = HuggingFaceLLM( model\_name="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name="Qwen/Qwen2.5-7B-Instruct", context\_window=30000, max\_new\_tokens=2000, generate\_kwargs={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt=messages\_to\_prompt, completion\_to\_prompt=completion\_to\_prompt, device\_map="auto", ) # Set embedding model Settings.embed\_model = HuggingFaceEmbedding( model\_name = "BAAI/bge-base-en-v1.5" ) # Set the size of the text chunk for retrieval Settings.transformations = \[SentenceSplitter(chunk\_size=1024)\] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'. .. code:: python from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load\_data() index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. .. code:: python from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html\_to\_text=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) To save and load the index, you can use the following code snippet. .. code:: python from llama\_index.core import StorageContext, load\_index\_from\_storage # save index storage\_context = StorageContext.from\_defaults(persist\_dir="save") # load index index = load\_index\_from\_storage(storage\_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. .. code:: python query\_engine = index.as\_query\_engine() your\_query = "" print(query\_engine.query(your\_query).response) --- # Unknown \# MS-SWIFT ModelScope SWIFT (\*\*ms-swift\*\*) is the large model and multimodal large model training and deployment framework provided by the \[ModelScope community\](https://modelscope.cn/). GitHub repository: \[ms-swift\](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: - \*\*Model Types\*\*: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. - \*\*Hardware Support\*\*: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. - \*\*Training Methods\*\*: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. - \*\*Distributed Training\*\*: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. - \*\*RLHF Training\*\*: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift's integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. \`\`\`shell pip install ms-swift -U # Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers -U # Optional packages pip install deepspeed # multi-GPU training pip install liger-kernel # save GPU memory resources pip install flash-attn --no-build-isolation \`\`\` ## Supervised Fine-Tuning (SFT) ### Data Preparation The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify \`--dataset \` in the training script. For complete dataset formatting guidelines, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) - General format \`\`\`json {"messages": \[ {"role": "system", "content": ""}, {"role": "user", "content": ""}, {"role": "assistant", "content": ""} \]} \`\`\` - Format with think \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` If you want to train using data without a chain of thought but retain the model's reasoning ability, there are two approaches to minimize disruption during fine-tuning: \*\*Option 1\*\*: During training, specify \`--loss\_scale ignore\_empty\_think\` to ignore the loss calculation for \`\\n\\n\\n\\n\`, preventing the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo1.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` \*\*Option 2\*\*: Add \`/no\_think\` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo2.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` ### 30-Minute Self-Cognition Fine-Tuning This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by \[ModelScope's free compute resources\](https://modelscope.cn/my/mynotebook). After training, the model will identify itself as "swift-robot," trained by "swift," instead of its original self-cognition as "Qwen," trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify \`--model \` and \`--dataset \`. The dataset can be found on \[Modelscope Hub\](https://modelscope.cn/datasets/swift/self-cognition). For the meaning of each parameter in the training script, please refer to the \[Command-line parameters documentation\](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type lora \\ --dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-4 \\ --lora\_rank 8 \\ --lora\_alpha 32 \\ --target\_modules all-linear \\ --gradient\_accumulation\_steps 16 \\ --eval\_steps 50 \\ --save\_steps 50 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --max\_length 2048 \\ --output\_dir output \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --model\_author swift \\ --model\_name swift-robot \`\`\` After fine-tuning, you can use the following script to test the fine-tuning results. Note that the \`--adapters\` section needs to be modified to the directory path of the last saved checkpoint: \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift infer \\ --adapters output/vx-xxx/checkpoint-xxx \\ --stream true \\ --temperature 0 \\ --max\_new\_tokens 2048 \`\`\` \`\`\`text <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! \`\`\` By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify \`--use\_hf true\`. Merge LoRA weights: \`\`\`shell swift export \\ --adapters output/checkpoint-xxx \\ --merge\_lora true \`\`\` Push the model to ModelScope/HuggingFace: \`\`\`shell # If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. # The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ --adapters output/checkpoint-xxx \\ --push\_to\_hub true \\ --hub\_model\_id '' \\ --hub\_token '' \\ --use\_hf false \`\`\` If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \`\`\`shell # 4 \* 60GB # You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. # Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE=4 \\ CUDA\_VISIBLE\_DEVICES=0,1,2,3 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset '' \\ --torch\_dtype bfloat16 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-5 \\ --gradient\_accumulation\_steps 4 \\ --packing true \\ --eval\_steps 100 \\ --save\_steps 100 \\ --logging\_steps 5 \\ --max\_length 8192 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 8 \\ --dataset\_num\_proc 8 \\ --save\_total\_limit 2 \\ --save\_only\_model true \\ --output\_dir output \\ --deepspeed zero3 \\ --use\_liger\_kernel true \\ --attn\_impl flash\_attn \`\`\` ## Reinforcement Learning (RL) ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: \[Supported Features\](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). ### Environment Setup In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: \`\`\`shell pip install "math\_verify==0.5.2" pip install vllm \`\`\` ### Data Preparation The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a \`solution\` column is needed to calculate the accuracy. Example Dataset Formats: \`\`\`json {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} \`\`\` For dataset preparation for other RLHF algorithms, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf). Notes on Dataset Requirements: - \*\*Reward Function Calculation\*\*: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: - When using the built-in accuracy or cosine similarity reward, the dataset must include a \`solution\` column to calculate the accuracy of the responses. - Other columns in the dataset will be passed as \`\*\*kwargs\` to the reward function for additional customization. - \*\*Customizing the Reward Function\*\*: To adapt the reward function to your specific needs, you can refer to the following resource: \[External Reward Plugin\](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin). This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting \`num\_infer\_workers=8\`, we deploy a vLLM engine for each device to speed up the sampling process. \`\`\`shell # 70G\*8 CUDA\_VISIBLE\_DEVICES=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE=8 \\ swift rlhf \\ --rlhf\_type grpo \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset 'AI-MO/NuminaMath-TIR#5000' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 2 \\ --per\_device\_eval\_batch\_size 2 \\ --learning\_rate 1e-6 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --output\_dir output \\ --gradient\_accumulation\_steps 1 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --max\_completion\_length 4096 \\ --vllm\_max\_model\_len 8192 \\ --reward\_funcs accuracy \\ --num\_generations 16 \\ --use\_vllm true \\ --vllm\_gpu\_memory\_utilization 0.4 \\ --sleep\_level 1 \\ --offload\_model true \\ --offload\_optimizer true \\ --gc\_collect\_after\_offload true \\ --deepspeed zero3 \\ --num\_infer\_workers 8 \\ --tensor\_parallel\_size 1 \\ --temperature 1.0 \\ --top\_p 0.85 \\ --log\_completions true \\ --overlong\_filter true \`\`\` ## Megatron-SWIFT ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the \[Supported Models Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For environment preparation and the conversion between HF and MCore model weights, you can refer to the \[Megatron-SWIFT Training Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html). These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to \[here\](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). \`\`\`shell # https://help.aliyun.com/zh/pai/user-guide/general-environment-variables # Ensure that the weight-saving paths on the two nodes are identical. NNODES=$WORLD\_SIZE \\ NODE\_RANK=$RANK \\ megatron sft \\ --load Qwen3-30B-A3B-Base-mcore \\ --dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ --tensor\_model\_parallel\_size 2 \\ --expert\_model\_parallel\_size 8 \\ --moe\_grouped\_gemm true \\ --moe\_shared\_expert\_overlap true \\ --moe\_aux\_loss\_coeff 0.01 \\ --micro\_batch\_size 1 \\ --global\_batch\_size 16 \\ --packing true \\ --recompute\_granularity full \\ --recompute\_method uniform \\ --recompute\_num\_layers 1 \\ --train\_iters 2000 \\ --eval\_iters 50 \\ --finetune true \\ --cross\_entropy\_loss\_fusion true \\ --lr 1e-5 \\ --lr\_warmup\_iters 100 \\ --min\_lr 1e-6 \\ --save megatron\_output/Qwen3-30B-A3B-Base \\ --eval\_interval 200 \\ --save\_interval 200 \\ --max\_length 8192 \\ --num\_workers 8 \\ --dataset\_num\_proc 8 \\ --no\_save\_optim true \\ --no\_save\_rng true \\ --sequence\_parallel true \\ --use\_flash\_attn true \`\`\` The custom dataset format is the same as \`swift sft\`, which can be found in the previous section. Simply specify \`--dataset \`. The following is a comparison of training speed and GPU memory usage between \`megatron sft\` and \`swift sft\` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | ---------------- | ----------- | --------------- | --------------- | | Training Speed | 9.6s/it | - | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | ## Conclusion The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in \[this issue\](https://github.com/modelscope/ms-swift/issues/4030). --- # Unknown \# SGLang \[SGLang\](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. To learn more about SGLang, please refer to the \[documentation\](https://docs.sglang.ai/). ## Environment Setup By default, you can install \`sglang\` with pip in a clean environment: \`\`\`shell pip install "sglang\[all\]>=0.4.6.post1" \`\`\` If you have encountered issues in installation, please feel free to check the official document for installation (\[link\](https://docs.sglang.ai/start/install.html)). ## API Service It is easy to build an OpenAI-compatible API service with SGLang, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:30000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B \`\`\` By default, if the \`--model-path\` does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export SGLANG\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in SGLang through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": True}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content SGLang supports parsing the thinking content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} \`enable\_thinking=False\` may not be compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, please consider disabling parsing thinking content. ::: ### Parsing Tool Calls SGLang supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25 \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md). ### Structured/JSON Output SGLang supports structured/JSON output. Please refer to \[SGLang's documentation\](https://docs.sglang.ai/backend/structured\_outputs.html#OpenAI-Compatible-API). Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-FP8 # For AWQ quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-AWQ \`\`\` ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. SGLang supports YaRN, which can be configured as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --json-model-override-args '{"rope\_scaling":{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}}' --context-length 131072 \`\`\` :::{note} SGLang implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which is used by SGLang. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: --- # Unknown dstack ======== \`dstack \`\_\_ is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML teams across top clouds, on-prem clusters, and accelerators. Prerequisites ---------------- Before you start, install dstack by following the \`installation instructions \`\_\_. Once dstack server is up, you can initialize your workspace as shown below: .. code:: bash mkdir dstack-qwen-deploy && cd dstack-qwen-deploy dstack init Deploy Qwen3-30B-A3B ----------------------------------------------- Deploy \`\`Qwen3-30B-A3B\`\` on instances available with cloud providers configured in your \`\`~/.dstack/server/config.yml\`\` file. You can use \`\`SgLang\`\`, \`\`TGI\`\` or \`\`vLLM\`\` to serve the model. Here we use \`\`SgLang\`\` as an example. Create a \`service \`\_\_ configuration file named \`\`serve-30b.dstack.yml\`\` with the following content: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 .. note:: For other inference backends such as vLLM or TGI, visit the \`dstack Inference Examples \`\_\_ documentation. Go ahead and apply the service configuration: .. code:: bash dstack apply -f serve-30b.dstack.yml Access the Service -------------------- After the service is successfully deployed, you can access the service's endpoint in the following ways: .. tab-set:: .. tab-item:: CURL Access through service endpoint at \`\`/proxy/services///\`\` .. code:: bash curl http://localhost:3000/proxy/services/main/qwen3-30b-a3b/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -H 'Authorization: Bearer ' \\ -d '{ "model": "Qwen/Qwen3-30B-A3B", "messages": \[ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } \] }' .. note:: When starting the dstack server, an admin token is automatically generated: .. code:: bash The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ .. tab-item:: Chat UI Access through dstack's Chat UI at \`\`/projects//models//\`\` .. image:: https://dstack.ai/static-assets/static-assets/images//dstack-qwen-ui.png .. dropdown:: Gateway :icon: info :animate: fade-in Running services for development purposes doesn't require setting up a gateway. However, you'll need a gateway in the following cases: \* To use auto-scaling or rate limits \* To enable HTTPS for the endpoint and map it to your domain \* If your service requires WebSockets \* If your service cannot work with a path prefix For detailed information about gateway configuration and usage, refer to the \`dstack documentation on gateways \`\_\_. Replicas and Auto Scaling ---------------------------------------- You can auto scale the service by specifying additional configurations in the \`\`serve-30b.dstack.yml\`\`. - Set \`\`replicas: min..max\`\` to define the minimum and maximum number of replicas - Configure \`\`scaling\`\` rules to determine when to scale up or down Below is a complete configuration example with auto-scaling enabled: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 # Minimum and maximum number of replicas replicas: 1..4 scaling: # Requests per seconds metric: rps # Target metric value target: 10 .. note:: The scaling property requires a gateway to be set up. See also ------------ - \*\*Fleets\*\*: Create cloud and on-prem clusters using \`Fleets \`\_\_. - \*\*Dev Environments\*\*: Experiment and test before deploying to production using \`Dev Environments \`\_\_. - \*\*Tasks\*\*: Schedule single node or distributed training using \`Tasks \`\_\_. - \*\*Services\*\*: Deploy models as secure, auto-scaling OpenAI-compatible endpoints using \`Services \`\_\_. - \*\*Metrics\*\*: Monitor performance with automatically tracked metrics via CLI or UI using \`Metrics \`\_\_. --- # Unknown \# SGLang \[SGLang\](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. To learn more about SGLang, please refer to the \[documentation\](https://docs.sglang.ai/). ## Environment Setup By default, you can install \`sglang\` with pip in a clean environment: \`\`\`shell pip install "sglang\[all\]>=0.4.6.post1" \`\`\` If you have encountered issues in installation, please feel free to check the official document for installation (\[link\](https://docs.sglang.ai/start/install.html)). ## API Service It is easy to build an OpenAI-compatible API service with SGLang, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:30000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B \`\`\` By default, if the \`--model-path\` does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export SGLANG\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in SGLang through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": True}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content SGLang supports parsing the thinking content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} \`enable\_thinking=False\` may not be compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, please consider disabling parsing thinking content. ::: ### Parsing Tool Calls SGLang supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25 \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md). ### Structured/JSON Output SGLang supports structured/JSON output. Please refer to \[SGLang's documentation\](https://docs.sglang.ai/backend/structured\_outputs.html#OpenAI-Compatible-API). Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-FP8 # For AWQ quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-AWQ \`\`\` ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. SGLang supports YaRN, which can be configured as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --json-model-override-args '{"rope\_scaling":{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}}' --context-length 131072 \`\`\` :::{note} SGLang implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which is used by SGLang. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: --- # Unknown TGI ===================== .. attention:: To be updated for Qwen3. Hugging Face's Text Generation Inference (TGI) is a production-ready framework specifically designed for deploying and serving large language models (LLMs) for text generation tasks. It offers a seamless deployment experience, powered by a robust set of features: \* \`Speculative Decoding \`\_: Accelerates generation speeds. \* \`Tensor Parallelism\`\_: Enables efficient deployment across multiple GPUs. \* \`Token Streaming\`\_: Allows for the continuous generation of text. \* Versatile Device Support: Works seamlessly with \`AMD\`\_, \`Gaudi\`\_ and \`AWS Inferentia\`\_. .. \_AMD: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/deploy-your-model.html#serving-using-hugging-face-tgi .. \_Gaudi: https://github.com/huggingface/tgi-gaudi .. \_AWS Inferentia: https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/#:~:text=Get%20started%20with%20TGI%20on%20SageMaker%20Hosting .. \_Tensor Parallelism: https://huggingface.co/docs/text-generation-inference/conceptual/tensor\_parallelism .. \_Token Streaming: https://huggingface.co/docs/text-generation-inference/conceptual/streaming Installation ----------------- The easiest way to use TGI is via the TGI docker image. In this guide, we show how to use TGI with docker. It's possible to run it locally via Conda or build locally. Please refer to \`Installation Guide \`\_ and \`CLI tool \`\_ for detailed instructions. Deploy Qwen2.5 with TGI ----------------------- 1. \*\*Find a Qwen2.5 Model:\*\* Choose a model from \`the Qwen2.5 collection \`\_. 2. \*\*Deployment Command:\*\* Run the following command in your terminal, replacing \`\`model\`\` with your chosen Qwen2.5 model ID and \`\`volume\`\` with the path to your local data directory: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model Using TGI API ------------- Once deployed, the model will be available on the mapped port (8080). TGI comes with a handy API for streaming response: .. code:: bash curl http://localhost:8080/generate\_stream -H 'Content-Type: application/json' \\ -d '{"inputs":"Tell me something about large language models.","parameters":{"max\_new\_tokens":512}}' It's also available on OpenAI style API: .. code:: bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' .. note:: The model field in the JSON is not used by TGI, you can put anything. Refer to the \`TGI Swagger UI \`\_ for a complete API reference. You can also use Python API: .. code:: python from openai import OpenAI # initialize the client but point it to TGI client = OpenAI( base\_url="http://localhost:8080/v1/", # replace with your endpoint url api\_key="", # this field is not used when running locally ) chat\_completion = client.chat.completions.create( model="", # it is not used by TGI, you can put anything messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], stream=True, temperature=0.7, top\_p=0.8, max\_tokens=512, ) # iterate and print stream for message in chat\_completion: print(message.choices\[0\].delta.content, end="") Quantization for Performance ---------------------------- 1. Data-dependent quantization (GPTQ and AWQ) Both GPTQ and AWQ models are data-dependent. The official quantized models can be found from \`the Qwen2.5 collection\`\_ and you can also quantize models with your own dataset to make it perform better on your use case. The following shows the command to start TGI with Qwen2.5-7B-Instruct-GPTQ-Int4: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize gptq If the model is quantized with AWQ, e.g. Qwen/Qwen2.5-7B-Instruct-AWQ, please use \`\`--quantize awq\`\`. 2. Data-agnostic quantization EETQ on the other side is not data dependent and can be used with any model. Note that we're passing in the original model (instead of a quantized model) with the \`\`--quantize eetq\`\` flag. .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq Multi-Accelerators Deployment ----------------------------- Use the \`\`--num-shard\`\` flag to specify the number of accelerators. Please also use \`\`--shm-size 1g\`\` to enable shared memory for optimal NCCL performance (\`reference \`\_\_): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --num-shard 2 Speculative Decoding -------------------- Speculative decoding can reduce the time per token by speculating on the next token. Use the \`\`--speculative-decoding\`\` flag, setting the value to the number of tokens to speculate on (default: 0 for no speculation): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 2 The overall performance of speculative decoding highly depends on the type of task. It works best for code or highly repetitive text. More context on speculative decoding can be found \`here \`\_\_. Zero-Code Deployment with HF Inference Endpoints --------------------------------------------------- For effortless deployment, leverage Hugging Face Inference Endpoints: - \*\*GUI interface:\*\* \`\`\_\_ - \*\*Coding interface:\*\* \`\`\_\_ Once deployed, the endpoint can be used as usual. Common Issues ---------------- Qwen2.5 supports long context lengths, so carefully choose the values for \`\`--max-batch-prefill-tokens\`\`, \`\`--max-total-tokens\`\`, and \`\`--max-input-tokens\`\` to avoid potential out-of-memory (OOM) issues. If an OOM occurs, you'll receive an error message upon startup. The following shows an example to modify those parameters: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-batch-prefill-tokens 4096 --max-total-tokens 4096 --max-input-tokens 2048 --- # Unknown \# llama.cpp Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program \`llama-quantize\` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using \`llama-imatrix\`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don't, check our guide \[here\](../run\_locally/llama.cpp.html#getting-the-program){.external}. ## Getting the GGUF Now, suppose you would like to quantize \`Qwen2.5-7B-Instruct\`. You need to first make a GGUF file as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen2.5-7B-Instruct --outfile qwen2.5-7b-instruct-f16.gguf \`\`\` Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen2.5-7B-Instruct --outtype f32 --outfile qwen2.5-7b-instruct-f32.gguf \`\`\` ## Quantizing the GGUF without Calibration For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: \`\`\`bash ./llama-quantize qwen2.5-7b-instruct-f16.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 \`\`\` \`Q8\_0\` is a code for a quantization preset. You can find all the presets in \[the source code of \`llama-quantize\`\](https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/quantize.cpp). Look for the variable \`QUANT\_OPTIONS\`. Common ones used for 7B models include \`Q8\_0\`, \`Q5\_0\`, and \`Q4\_K\_M\`. The letter case doesn't matter, so \`q8\_0\` or \`q4\_K\_m\` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. ## Quantizing the GGUF with AWQ Scale To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following \[this script\](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export). First, when you run \`model.quantize()\` with \`autoawq\`, remember to add \`export\_compatible=True\` as shown below: \`\`\`python ... model.quantize( tokenizer, quant\_config=quant\_config, export\_compatible=True ) model.save\_pretrained(quant\_path) ... \`\`\` The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are "easier" to quantize.\[^AWQ\] Then, when you run \`convert-hf-to-gguf.py\`, remember to replace the model path with the path to the new model: \`\`\`bash python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf \`\`\` Finally, you can quantize the model as in the last example: \`\`\`bash ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 \`\`\` In this way, it should be possible to achieve similar quality with lower bit-per-weight. \[^AWQ\]: If you are interested in what this means, refer to \[the AWQ paper\](https://arxiv.org/abs/2306.00978). Basically, important weights (called salient weights in the paper) are identified based on activations across data examples. The weights are scaled accordingly such that the salient weights are protected even after quantization. ## Quantizing the GGUF with Importance Matrix Another possible solution is to use the "important matrix"\[^imatrix\], following \[this\](https://github.com/ggerganov/llama.cpp/tree/master/examples/imatrix). First, you need to compute the importance matrix data of the weights of a model (\`-m\`) using a calibration dataset (\`-f\`): \`\`\`bash ./llama-imatrix -m qwen2.5-7b-instruct-f16.gguf -f calibration-text.txt --chunk 512 -o qwen2.5-7b-instruct-imatrix.dat -ngl 80 \`\`\` The text is cut in chunks of length \`--chunk\` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named \`qwen2.5-7b-instruct-imatrix.dat\` (\`-o\`), which can then be used: \`\`\`bash ./llama-quantize --imatrix qwen2.5-7b-instruct-imatrix.data \\ qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q4\_k\_m.gguf Q4\_K\_M \`\`\` For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide \`--imatrix\`, a helpful warning will be printed by \`llama-quantize\`. \[^imatrix\]: Here, the importance matrix keeps record of how weights affect the output: the weight should be important is a slight change in its value causes huge difference in the results, akin to the \[GPTQ\](https://arxiv.org/abs/2210.17323) algorithm. ## Perplexity Evaluation \`llama.cpp\` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say "wiki test"\[^wiki\]. You can download the dataset with: \`\`\`bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip \`\`\` Then you can run the test with the following command: \`\`\`bash ./llama-perplexity -m qwen2.5-7b-instruct-q8\_0.gguf -f wiki.test.raw -ngl 80 \`\`\` Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture \[here\](https://github.com/ggerganov/llama.cpp/blob/master/examples/perplexity/README.md). It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. \[^wiki\]: It is not a good evaluation dataset for instruct models though, but it is very common and easily accessible. You probably want to use a dataset similar to your target domain. ## Finally In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the \[llama.cpp GitHub repo\](https://github.com/ggerganov/llama.cpp). We usually quantize the fp16 model to 2, 3, 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don't provide those in our HuggingFace Hub. However, others in the community may have success, so if you haven't found what you need in our repos, look around. Enjoy your freshly quantized models! --- # Unknown TGI ===================== Hugging Face's Text Generation Inference (TGI) is a production-ready framework specifically designed for deploying and serving large language models (LLMs) for text generation tasks. It offers a seamless deployment experience, powered by a robust set of features: \* \`Speculative Decoding \`\_: Accelerates generation speeds. \* \`Tensor Parallelism\`\_: Enables efficient deployment across multiple GPUs. \* \`Token Streaming\`\_: Allows for the continuous generation of text. \* Versatile Device Support: Works seamlessly with \`AMD\`\_, \`Gaudi\`\_ and \`AWS Inferentia\`\_. .. \_AMD: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/deploy-your-model.html#serving-using-hugging-face-tgi .. \_Gaudi: https://github.com/huggingface/tgi-gaudi .. \_AWS Inferentia: https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/#:~:text=Get%20started%20with%20TGI%20on%20SageMaker%20Hosting .. \_Tensor Parallelism: https://huggingface.co/docs/text-generation-inference/conceptual/tensor\_parallelism .. \_Token Streaming: https://huggingface.co/docs/text-generation-inference/conceptual/streaming Installation ----------------- The easiest way to use TGI is via the TGI docker image. In this guide, we show how to use TGI with docker. It's possible to run it locally via Conda or build locally. Please refer to \`Installation Guide \`\_ and \`CLI tool \`\_ for detailed instructions. Deploy Qwen2.5 with TGI ----------------------- 1. \*\*Find a Qwen2.5 Model:\*\* Choose a model from \`the Qwen2.5 collection \`\_. 2. \*\*Deployment Command:\*\* Run the following command in your terminal, replacing \`\`model\`\` with your chosen Qwen2.5 model ID and \`\`volume\`\` with the path to your local data directory: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model Using TGI API ------------- Once deployed, the model will be available on the mapped port (8080). TGI comes with a handy API for streaming response: .. code:: bash curl http://localhost:8080/generate\_stream -H 'Content-Type: application/json' \\ -d '{"inputs":"Tell me something about large language models.","parameters":{"max\_new\_tokens":512}}' It's also available on OpenAI style API: .. code:: bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' .. note:: The model field in the JSON is not used by TGI, you can put anything. Refer to the \`TGI Swagger UI \`\_ for a complete API reference. You can also use Python API: .. code:: python from openai import OpenAI # initialize the client but point it to TGI client = OpenAI( base\_url="http://localhost:8080/v1/", # replace with your endpoint url api\_key="", # this field is not used when running locally ) chat\_completion = client.chat.completions.create( model="", # it is not used by TGI, you can put anything messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], stream=True, temperature=0.7, top\_p=0.8, max\_tokens=512, ) # iterate and print stream for message in chat\_completion: print(message.choices\[0\].delta.content, end="") Quantization for Performance ---------------------------- 1. Data-dependent quantization (GPTQ and AWQ) Both GPTQ and AWQ models are data-dependent. The official quantized models can be found from \`the Qwen2.5 collection\`\_ and you can also quantize models with your own dataset to make it perform better on your use case. The following shows the command to start TGI with Qwen2.5-7B-Instruct-GPTQ-Int4: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize gptq If the model is quantized with AWQ, e.g. Qwen/Qwen2.5-7B-Instruct-AWQ, please use \`\`--quantize awq\`\`. 2. Data-agnostic quantization EETQ on the other side is not data dependent and can be used with any model. Note that we're passing in the original model (instead of a quantized model) with the \`\`--quantize eetq\`\` flag. .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq Multi-Accelerators Deployment ----------------------------- Use the \`\`--num-shard\`\` flag to specify the number of accelerators. Please also use \`\`--shm-size 1g\`\` to enable shared memory for optimal NCCL performance (\`reference \`\_\_): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --num-shard 2 Speculative Decoding -------------------- Speculative decoding can reduce the time per token by speculating on the next token. Use the \`\`--speculative-decoding\`\` flag, setting the value to the number of tokens to speculate on (default: 0 for no speculation): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 2 The overall performance of speculative decoding highly depends on the type of task. It works best for code or highly repetitive text. More context on speculative decoding can be found \`here \`\_\_. Zero-Code Deployment with HF Inference Endpoints --------------------------------------------------- For effortless deployment, leverage Hugging Face Inference Endpoints: - \*\*GUI interface:\*\* \`\`\_\_ - \*\*Coding interface:\*\* \`\`\_\_ Once deployed, the endpoint can be used as usual. Common Issues ---------------- Qwen2.5 supports long context lengths, so carefully choose the values for \`\`--max-batch-prefill-tokens\`\`, \`\`--max-total-tokens\`\`, and \`\`--max-input-tokens\`\` to avoid potential out-of-memory (OOM) issues. If an OOM occurs, you'll receive an error message upon startup. The following shows an example to modify those parameters: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-batch-prefill-tokens 4096 --max-total-tokens 4096 --max-input-tokens 2048 --- # Unknown OpenLLM ======= .. attention:: To be updated for Qwen3. OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit \`the OpenLLM repository \`\_ to learn more. Installation ------------ Install OpenLLM using \`\`pip\`\`. .. code:: bash pip install openllm Verify the installation and display the help information: .. code:: bash openllm --help Quickstart ---------- Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository. .. code:: bash openllm repo update List the supported Qwen2.5 models: .. code:: bash openllm model list --tag qwen2.5 The results also display the required GPU resources and supported platforms: .. code:: bash model version repo required GPU RAM platforms ------- --------------------- ------- ------------------ ----------- qwen2.5 qwen2.5:0.5b default 12G linux qwen2.5:1.5b default 12G linux qwen2.5:3b default 12G linux qwen2.5:7b default 24G linux qwen2.5:14b default 80G linux qwen2.5:14b-ggml-q4 default macos qwen2.5:14b-ggml-q8 default macos qwen2.5:32b default 80G linux qwen2.5:32b-ggml-fp16 default macos qwen2.5:72b default 80Gx2 linux qwen2.5:72b-ggml-q4 default macos To start a server with one of the models, use \`\`openllm serve\`\` like this: .. code:: bash openllm serve qwen2.5:7b By default, the server starts at \`\`http://localhost:3000/\`\`. Interact with the model server ------------------------------ With the model server up and running, you can call its APIs in the following ways: .. tab-set:: .. tab-item:: CURL Send an HTTP request to its \`\`/generate\`\` endpoint via CURL: .. code-block:: bash curl -X 'POST' \\ 'http://localhost:3000/api/generate' \\ -H 'accept: text/event-stream' \\ -H 'Content-Type: application/json' \\ -d '{ "prompt": "Tell me something about large language models.", "model": "Qwen/Qwen2.5-7B-Instruct", "max\_tokens": 2048, "stop": null }' .. tab-item:: Python client Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example: .. code-block:: python from openai import OpenAI client = OpenAI(base\_url='http://localhost:3000/v1', api\_key='na') # Use the following func to get the available models # model\_list = client.models.list() # print(model\_list) chat\_completion = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ { "role": "user", "content": "Tell me something about large language models." } \], stream=True, ) for chunk in chat\_completion: print(chunk.choices\[0\].delta.content or "", end="") .. tab-item:: Chat UI OpenLLM provides a chat UI at the \`\`/chat\`\` endpoint for the LLM server at http://localhost:3000/chat. .. image:: ../../source/assets/qwen-openllm-ui-demo.png Model repository ---------------- A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our \`documentation to learn details \`\_. --- # Unknown \# llama.cpp \[^GGUF\]: GPT-Generated Unified Format :::{dropdown} llama.cpp as a C++ library Before starting, let's first discuss what is llama.cpp and what you should expect, and why we say "use" llama.cpp, with "use" in quotes. llama.cpp is essentially a different ecosystem with a different design philosophy that targets light-weight footprint, minimal external dependency, multi-platform, and extensive, flexible hardware support: - Plain C/C++ implementation without external dependencies - Support a wide variety of hardware: - AVX, AVX2 and AVX512 support for x86\_64 CPU - Apple Silicon via Metal and Accelerate (CPU and GPU) - NVIDIA GPU (via CUDA), AMD GPU (via hipBLAS), Intel GPU (via SYCL), Ascend NPU (via CANN), and Moore Threads GPU (via MUSA) - Vulkan backend for GPU - Various quantization schemes for faster inference and reduced memory footprint - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity It's like the Python frameworks \`torch\`+\`transformers\` or \`torch\`+\`vllm\` but in C++. However, this difference is crucial: - Python is an interpreted language: The code you write is executed line-by-line on-the-fly by an interpreter. You can run the example code snippet or script with an interpreter or a natively interactive interpreter shell. In addition, Python is learner friendly, and even if you don't know much before, you can tweak the source code here and there. - C++ is a compiled language: The source code you write needs to be compiled beforehand, and it is translated to machine code and an executable program by a compiler. The overhead from the language side is minimal. You do have source code for example programs showcasing how to use the library. But it is not very easy to modify the source code if you are not verse in C++ or C. To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of \[Ollama\](https://ollama.com/), \[LM Studio\](https://lmstudio.ai/), \[GPT4ALL\](https://www.nomic.ai/gpt4all), \[llamafile\](https://llamafile.ai/) etc. But that's not what this guide is intended or could do. Instead, here we introduce how to use the \`llama-cli\` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. ::: In this guide, we will show how to "use" \[llama.cpp\](https://github.com/ggml-org/llama.cpp) to run models on your local machine, in particular, the \`llama-cli\` and the \`llama-server\` example program, which comes with the library. The main steps are: 1. Get the programs 2. Get the Qwen3 models in GGUF\[^GGUF\] format 3. Run the program with the model :::{note} llama.cpp supports Qwen3 and Qwen3MoE from version \`b5092\`. ::: ## Getting the Program You can get the programs in various ways. For optimal efficiency, we recommend compiling the programs locally, so you get the CPU optimizations for free. However, if you don't have C++ compilers locally, you can also install using package managers or downloading pre-built binaries. They could be less efficient but for non-production example use, they are fine. :::::{tab-set} ::::{tab-item} Compile Locally Here, we show the basic command to compile \`llama-cli\` locally on \*\*macOS\*\* or \*\*Linux\*\*. For Windows or GPU users, please refer to \[the guide from llama.cpp\](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md). :::{rubric} Installing Build Tools :heading-level: 5 ::: To build locally, a C++ compiler and a build system tool are required. To see if they have been installed already, type \`cc --version\` or \`cmake --version\` in a terminal window. - If installed, the build configuration of the tool will be printed to the terminal, and you are good to go! - If errors are raised, you need to first install the related tools: - On macOS, install with the command \`xcode-select --install\` - On Ubuntu, install with the command \`sudo apt install build-essential\`. For other Linux distributions, the command may vary; the essential packages needed for this guide are \`gcc\` and \`cmake\`. :::{rubric} Compiling the Program :heading-level: 5 ::: For the first step, clone the repo and enter the directory: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Then, build llama.cpp using CMake: \`\`\`bash cmake -B build cmake --build build --config Release \`\`\` The first command will check the local environment and determine which backends and features should be included. The second command will actually build the programs. To shorten the time, you can also enable parallel compiling based on the CPU cores you have, for example: \`\`\`bash cmake --build build --config Release -j 8 \`\`\` This will build the programs with 8 parallel compiling jobs. The built programs will be in \`./build/bin/\`. :::: ::::{tab-item} Package Managers For \*\*macOS\*\* and \*\*Linux\*\* users, \`llama-cli\` and \`llama-server\` can be installed with package managers including Homebrew, Nix, and Flox. Here, we show how to install \`llama-cli\` and \`llama-server\` with Homebrew. For other package managers, please check the instructions \[here\](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). Installing with Homebrew is very simple: 1. Ensure that Homebrew is available on your operating system. If you don't have Homebrew, you can install it as in \[its website\](https://brew.sh/). 2. Second, you can install the pre-built binaries, \`llama-cli\` and \`llama-server\` included, with a single command: \`\`\`bash brew install llama.cpp \`\`\` Note that the installed binaries might not be built with the optimal compile options for your hardware, which can lead to poor performance. They also don't support GPU on Linux systems. :::: ::::{tab-item} Binary Release You can also download pre-built binaries from \[GitHub Releases\](https://github.com/ggml-org/llama.cpp/releases). Please note that those pre-built binaries files are architecture-, backend-, and os-specific. If you are not sure what those mean, you probably don't want to use them and running with incompatible versions will most likely fail or lead to poor performance. The file name is like \`llama-\-bin-\-\-.zip\`. There are three simple parts: - \`\`: the version of llama.cpp. The latest is preferred, but as llama.cpp is updated and released frequently, the latest may contain bugs. If the latest version does not work, try the previous release until it works. - \`\`: the operating system. \`win\` for Windows; \`macos\` for macOS; \`linux\` for Linux. - \`\`: the system architecture. \`x64\` for \`x86\_64\`, e.g., most Intel and AMD systems, including Intel Mac; \`arm64\` for \`arm64\`, e.g., Apple Silicon or Snapdragon-based systems. The \`\` part is somewhat complicated for Windows: - Running on CPU - x86\_64 CPUs: We suggest try the \`avx2\` one first. - \`noavx\`: No hardware acceleration at all. - \`avx2\`, \`avx\`, \`avx512\`: SIMD-based acceleration. Most modern desktop CPUs should support avx2, and some CPUs support \`avx512\`. - \`openblas\`: Relying on OpenBLAS for acceleration for prompt processing but not generation. - arm64 CPUs: We suggest try the \`llvm\` one first. - \[\`llvm\` and \`msvc\`\](https://github.com/ggml-org/llama.cpp/pull/7191) are different compilers - Running on GPU: We suggest try the \`cu\` one for NVIDIA GPUs, \`kompute\` for AMD GPUs, and \`sycl\` for Intel GPUs first. Ensure that you have related drivers installed. - \[\`vulcan\`\](https://github.com/ggml-org/llama.cpp/pull/2059): support certain NVIDIA and AMD GPUs - \[\`kompute\`\](https://github.com/ggml-org/llama.cpp/pull/4456): support certain NVIDIA and AMD GPUs - \[\`sycl\`\](https://github.com/ggml-org/llama.cpp/discussions/5138): Intel GPUs, oneAPI runtime is included - \`cu\`: NVIDIA GPUs, CUDA runtime is not included. You can download the \`cudart-llama-bin-win-cu\-x64.zip\` and unzip it to the same directory if you don't have the corresponding CUDA toolkit installed. You don't have much choice for macOS or Linux. - Linux: only one prebuilt binary, \`llama-\-bin-linux-x64.zip\`, supporting CPU. - macOS: \`llama-\-bin-macos-x64.zip\` for Intel Mac with no GPU support; \`llama-\-bin-macos-arm64.zip\` for Apple Silicon with GPU support. After downloading the \`.zip\` file, unzip them into a directory and open a terminal at that directory. :::: ::::: ## Getting the GGUF GGUF\[^GGUF\] is a file format for storing information needed to run a model, including but not limited to model weights, model hyperparameters, default generation configuration, and tokenizer. You can use the official Qwen GGUFs from our Hugging Face Hub or prepare your own GGUF file. ### Using the Official Qwen3 GGUFs We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with \`-GGUF\`. Download the GGUF model that you want with \`huggingface-cli\` (you need to install it first with \`pip install huggingface\_hub\`): \`\`\`bash huggingface-cli download \--local-dir \`\`\` For example: \`\`\`bash huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4\_k\_m.gguf --local-dir . \`\`\` This will download the Qwen3-8B model in GGUF format quantized with the scheme Q4\_K\_M. ### Preparing Your Own GGUF Model files from Hugging Face Hub can be converted to GGUF, using the \`convert-hf-to-gguf.py\` Python script. It does require you to have a working Python environment with at least \`transformers\` installed. Obtain the source file if you haven't already: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Suppose you would like to use Qwen3-8B you can make a GGUF file for the fp16 model as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile qwen3-8b-f16.gguf \`\`\` The first argument to the script refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file. Remember to create the output directory before you run the command. The fp16 model could be a bit heavy for running locally, and you can quantize the model as needed. We introduce the method of creating and quantizing GGUF files in \[this guide\](../quantization/llama.cpp). You can refer to that document for more information. ## Run Qwen with llama.cpp :::{note} Regarding switching between thinking and non-thinking modes, while the soft switch is always available, the hard switch implemented in the chat template is not exposed in llama.cpp. The quick workaround is to pass \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) equivalent to always \`enable\_thinking=False\` via \`--chat-template-file\`. ::: ### llama-cli \[llama-cli\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main) is a console program which can be used to chat with LLMs. Simple run the following command where you place the llama.cpp programs: \`\`\`shell ./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` Here are some explanations to the above command: - \*\*Model\*\*: llama-cli supports using model files from local path, remote URL, or Hugging Face hub. - \`-hf Qwen/Qwen3-8B-GGUF:Q8\_0\` in the above indicates we are using the model file from Hugging Face hub - To use a local path, pass \`-m qwen3-8b-q8\_0.gguf\` instead - To use a remote URL, pass \`-mu https://hf.co/Qwen/Qwen3-8B-GGUF/resolve/main/qwen3-8b-Q8\_0.gguf?download=true\` instead - \*\*Speed Optimization\*\*: - CPU: llama-cli by default will use CPU and you can change \`-t\` to specify how many threads you would like it to use, e.g., \`-t 8\` means using 8 threads. - GPU: If the programs are built with GPU support, you can use \`-ngl\`, which allows offloading some layers to the GPU for computation. If there are multiple GPUs, it will offload to all the GPUs. You can use \`-dev\` to control the devices used and \`-sm\` to control which kinds of parallelism is used. For example, \`-ngl 99 -dev cuda0,cuda1 -sm row\` means offload all layers to GPU 0 and GPU1 using the split mode row. Adding \`-fa\` may also speed up the generation. - \*\*Sampling Parameters\*\*: llama.cpp supports \[a variety of sampling methods\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main#generation-flags) and has default configuration for many of them. It is recommended to adjust those parameters according to the actual case and the recommended parameters from Qwen3 modelcard could be used as a reference. If you encounter repetition and endless generation, it is recommended to pass in addition \`--presence-penalty\` up to \`2.0\`. - \*\*Context Management\*\*: llama.cpp adopts the "rotating" context management by default. The \`-c\` controls the maximum context length (default 4096, 0 means loaded from model), and \`-n\` controls the maximum generation length each time (default -1 means infinite until ending, -2 means until context full). When the context is full but the generation doesn't end, the first \`--keep\` tokens (default 0, -1 means all) from the initial prompt is kept, and the first half of the rest is discarded. Then, the model continues to generate based on the new context tokens. You can set \`--no-context-shift\` to prevent this rotating behavior and the generation will stop once \`-c\` is reached. llama.cpp supports YaRN, which can be enabled by \`-c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768\`. - \*\*Chat\*\*: \`--jinja\` indicates using the chat template embedded in the GGUF which is preferred and \`--color\` indicates coloring the texts so that user input and model output can be better differentiated. If there is a chat template, like in Qwen3 models, llama-cli will enter chat mode automatically. To stop generation or exit press "Ctrl+C". You can use \`-sys\` to add a system prompt. ### llama-server \[llama-server\](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) is a simple HTTP server, including a set of LLM REST APIs and a simple web front end to interact with LLMs using llama.cpp. The core command is similar to that of llama-cli. In addition, it supports thinking content parsing and tool call parsing. \`\`\`shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` By default, the server will listen at \`http://localhost:8080\` which can be changed by passing \`--host\` and \`--port\`. The web front end can be assessed from a browser at \`http://localhost:8080/\`. The OpenAI compatible API is at \`http://localhost:8080/v1/\`. ## What's More If you still find it difficult to use llama.cpp, don't worry, just check out other llama.cpp-based applications. For example, Qwen3 has already been officially part of Ollama and LM Studio, which are platforms for your to search and run local LLMs. Have fun! --- # Unknown \# llama.cpp \[^GGUF\]: GPT-Generated Unified Format :::{dropdown} llama.cpp as a C++ library Before starting, let's first discuss what is llama.cpp and what you should expect, and why we say "use" llama.cpp, with "use" in quotes. llama.cpp is essentially a different ecosystem with a different design philosophy that targets light-weight footprint, minimal external dependency, multi-platform, and extensive, flexible hardware support: - Plain C/C++ implementation without external dependencies - Support a wide variety of hardware: - AVX, AVX2 and AVX512 support for x86\_64 CPU - Apple Silicon via Metal and Accelerate (CPU and GPU) - NVIDIA GPU (via CUDA), AMD GPU (via hipBLAS), Intel GPU (via SYCL), Ascend NPU (via CANN), and Moore Threads GPU (via MUSA) - Vulkan backend for GPU - Various quantization schemes for faster inference and reduced memory footprint - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity It's like the Python frameworks \`torch\`+\`transformers\` or \`torch\`+\`vllm\` but in C++. However, this difference is crucial: - Python is an interpreted language: The code you write is executed line-by-line on-the-fly by an interpreter. You can run the example code snippet or script with an interpreter or a natively interactive interpreter shell. In addition, Python is learner friendly, and even if you don't know much before, you can tweak the source code here and there. - C++ is a compiled language: The source code you write needs to be compiled beforehand, and it is translated to machine code and an executable program by a compiler. The overhead from the language side is minimal. You do have source code for example programs showcasing how to use the library. But it is not very easy to modify the source code if you are not verse in C++ or C. To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of \[Ollama\](https://ollama.com/), \[LM Studio\](https://lmstudio.ai/), \[GPT4ALL\](https://www.nomic.ai/gpt4all), \[llamafile\](https://llamafile.ai/) etc. But that's not what this guide is intended or could do. Instead, here we introduce how to use the \`llama-cli\` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. ::: In this guide, we will show how to "use" \[llama.cpp\](https://github.com/ggml-org/llama.cpp) to run models on your local machine, in particular, the \`llama-cli\` and the \`llama-server\` example program, which comes with the library. The main steps are: 1. Get the programs 2. Get the Qwen3 models in GGUF\[^GGUF\] format 3. Run the program with the model :::{note} llama.cpp supports Qwen3 and Qwen3MoE from version \`b5092\`. ::: ## Getting the Program You can get the programs in various ways. For optimal efficiency, we recommend compiling the programs locally, so you get the CPU optimizations for free. However, if you don't have C++ compilers locally, you can also install using package managers or downloading pre-built binaries. They could be less efficient but for non-production example use, they are fine. :::::{tab-set} ::::{tab-item} Compile Locally Here, we show the basic command to compile \`llama-cli\` locally on \*\*macOS\*\* or \*\*Linux\*\*. For Windows or GPU users, please refer to \[the guide from llama.cpp\](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md). :::{rubric} Installing Build Tools :heading-level: 5 ::: To build locally, a C++ compiler and a build system tool are required. To see if they have been installed already, type \`cc --version\` or \`cmake --version\` in a terminal window. - If installed, the build configuration of the tool will be printed to the terminal, and you are good to go! - If errors are raised, you need to first install the related tools: - On macOS, install with the command \`xcode-select --install\` - On Ubuntu, install with the command \`sudo apt install build-essential\`. For other Linux distributions, the command may vary; the essential packages needed for this guide are \`gcc\` and \`cmake\`. :::{rubric} Compiling the Program :heading-level: 5 ::: For the first step, clone the repo and enter the directory: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Then, build llama.cpp using CMake: \`\`\`bash cmake -B build cmake --build build --config Release \`\`\` The first command will check the local environment and determine which backends and features should be included. The second command will actually build the programs. To shorten the time, you can also enable parallel compiling based on the CPU cores you have, for example: \`\`\`bash cmake --build build --config Release -j 8 \`\`\` This will build the programs with 8 parallel compiling jobs. The built programs will be in \`./build/bin/\`. :::: ::::{tab-item} Package Managers For \*\*macOS\*\* and \*\*Linux\*\* users, \`llama-cli\` and \`llama-server\` can be installed with package managers including Homebrew, Nix, and Flox. Here, we show how to install \`llama-cli\` and \`llama-server\` with Homebrew. For other package managers, please check the instructions \[here\](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). Installing with Homebrew is very simple: 1. Ensure that Homebrew is available on your operating system. If you don't have Homebrew, you can install it as in \[its website\](https://brew.sh/). 2. Second, you can install the pre-built binaries, \`llama-cli\` and \`llama-server\` included, with a single command: \`\`\`bash brew install llama.cpp \`\`\` Note that the installed binaries might not be built with the optimal compile options for your hardware, which can lead to poor performance. They also don't support GPU on Linux systems. :::: ::::{tab-item} Binary Release You can also download pre-built binaries from \[GitHub Releases\](https://github.com/ggml-org/llama.cpp/releases). Please note that those pre-built binaries files are architecture-, backend-, and os-specific. If you are not sure what those mean, you probably don't want to use them and running with incompatible versions will most likely fail or lead to poor performance. The file name is like \`llama-\-bin-\-\-.zip\`. There are three simple parts: - \`\`: the version of llama.cpp. The latest is preferred, but as llama.cpp is updated and released frequently, the latest may contain bugs. If the latest version does not work, try the previous release until it works. - \`\`: the operating system. \`win\` for Windows; \`macos\` for macOS; \`linux\` for Linux. - \`\`: the system architecture. \`x64\` for \`x86\_64\`, e.g., most Intel and AMD systems, including Intel Mac; \`arm64\` for \`arm64\`, e.g., Apple Silicon or Snapdragon-based systems. The \`\` part is somewhat complicated for Windows: - Running on CPU - x86\_64 CPUs: We suggest try the \`avx2\` one first. - \`noavx\`: No hardware acceleration at all. - \`avx2\`, \`avx\`, \`avx512\`: SIMD-based acceleration. Most modern desktop CPUs should support avx2, and some CPUs support \`avx512\`. - \`openblas\`: Relying on OpenBLAS for acceleration for prompt processing but not generation. - arm64 CPUs: We suggest try the \`llvm\` one first. - \[\`llvm\` and \`msvc\`\](https://github.com/ggml-org/llama.cpp/pull/7191) are different compilers - Running on GPU: We suggest try the \`cu\` one for NVIDIA GPUs, \`kompute\` for AMD GPUs, and \`sycl\` for Intel GPUs first. Ensure that you have related drivers installed. - \[\`vulcan\`\](https://github.com/ggml-org/llama.cpp/pull/2059): support certain NVIDIA and AMD GPUs - \[\`kompute\`\](https://github.com/ggml-org/llama.cpp/pull/4456): support certain NVIDIA and AMD GPUs - \[\`sycl\`\](https://github.com/ggml-org/llama.cpp/discussions/5138): Intel GPUs, oneAPI runtime is included - \`cu\`: NVIDIA GPUs, CUDA runtime is not included. You can download the \`cudart-llama-bin-win-cu\-x64.zip\` and unzip it to the same directory if you don't have the corresponding CUDA toolkit installed. You don't have much choice for macOS or Linux. - Linux: only one prebuilt binary, \`llama-\-bin-linux-x64.zip\`, supporting CPU. - macOS: \`llama-\-bin-macos-x64.zip\` for Intel Mac with no GPU support; \`llama-\-bin-macos-arm64.zip\` for Apple Silicon with GPU support. After downloading the \`.zip\` file, unzip them into a directory and open a terminal at that directory. :::: ::::: ## Getting the GGUF GGUF\[^GGUF\] is a file format for storing information needed to run a model, including but not limited to model weights, model hyperparameters, default generation configuration, and tokenizer. You can use the official Qwen GGUFs from our Hugging Face Hub or prepare your own GGUF file. ### Using the Official Qwen3 GGUFs We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with \`-GGUF\`. Download the GGUF model that you want with \`huggingface-cli\` (you need to install it first with \`pip install huggingface\_hub\`): \`\`\`bash huggingface-cli download \--local-dir \`\`\` For example: \`\`\`bash huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4\_k\_m.gguf --local-dir . \`\`\` This will download the Qwen3-8B model in GGUF format quantized with the scheme Q4\_K\_M. ### Preparing Your Own GGUF Model files from Hugging Face Hub can be converted to GGUF, using the \`convert-hf-to-gguf.py\` Python script. It does require you to have a working Python environment with at least \`transformers\` installed. Obtain the source file if you haven't already: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Suppose you would like to use Qwen3-8B you can make a GGUF file for the fp16 model as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile qwen3-8b-f16.gguf \`\`\` The first argument to the script refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file. Remember to create the output directory before you run the command. The fp16 model could be a bit heavy for running locally, and you can quantize the model as needed. We introduce the method of creating and quantizing GGUF files in \[this guide\](../quantization/llama.cpp). You can refer to that document for more information. ## Run Qwen with llama.cpp :::{note} Regarding switching between thinking and non-thinking modes, while the soft switch is always available, the hard switch implemented in the chat template is not exposed in llama.cpp. The quick workaround is to pass \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) equivalent to always \`enable\_thinking=False\` via \`--chat-template-file\`. ::: ### llama-cli \[llama-cli\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main) is a console program which can be used to chat with LLMs. Simple run the following command where you place the llama.cpp programs: \`\`\`shell ./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` Here are some explanations to the above command: - \*\*Model\*\*: llama-cli supports using model files from local path, remote URL, or Hugging Face hub. - \`-hf Qwen/Qwen3-8B-GGUF:Q8\_0\` in the above indicates we are using the model file from Hugging Face hub - To use a local path, pass \`-m qwen3-8b-q8\_0.gguf\` instead - To use a remote URL, pass \`-mu https://hf.co/Qwen/Qwen3-8B-GGUF/resolve/main/qwen3-8b-Q8\_0.gguf?download=true\` instead - \*\*Speed Optimization\*\*: - CPU: llama-cli by default will use CPU and you can change \`-t\` to specify how many threads you would like it to use, e.g., \`-t 8\` means using 8 threads. - GPU: If the programs are built with GPU support, you can use \`-ngl\`, which allows offloading some layers to the GPU for computation. If there are multiple GPUs, it will offload to all the GPUs. You can use \`-dev\` to control the devices used and \`-sm\` to control which kinds of parallelism is used. For example, \`-ngl 99 -dev cuda0,cuda1 -sm row\` means offload all layers to GPU 0 and GPU1 using the split mode row. Adding \`-fa\` may also speed up the generation. - \*\*Sampling Parameters\*\*: llama.cpp supports \[a variety of sampling methods\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main#generation-flags) and has default configuration for many of them. It is recommended to adjust those parameters according to the actual case and the recommended parameters from Qwen3 modelcard could be used as a reference. If you encounter repetition and endless generation, it is recommended to pass in addition \`--presence-penalty\` up to \`2.0\`. - \*\*Context Management\*\*: llama.cpp adopts the "rotating" context management by default. The \`-c\` controls the maximum context length (default 4096, 0 means loaded from model), and \`-n\` controls the maximum generation length each time (default -1 means infinite until ending, -2 means until context full). When the context is full but the generation doesn't end, the first \`--keep\` tokens (default 0, -1 means all) from the initial prompt is kept, and the first half of the rest is discarded. Then, the model continues to generate based on the new context tokens. You can set \`--no-context-shift\` to prevent this rotating behavior and the generation will stop once \`-c\` is reached. llama.cpp supports YaRN, which can be enabled by \`-c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768\`. - \*\*Chat\*\*: \`--jinja\` indicates using the chat template embedded in the GGUF which is preferred and \`--color\` indicates coloring the texts so that user input and model output can be better differentiated. If there is a chat template, like in Qwen3 models, llama-cli will enter chat mode automatically. To stop generation or exit press "Ctrl+C". You can use \`-sys\` to add a system prompt. ### llama-server \[llama-server\](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) is a simple HTTP server, including a set of LLM REST APIs and a simple web front end to interact with LLMs using llama.cpp. The core command is similar to that of llama-cli. In addition, it supports thinking content parsing and tool call parsing. \`\`\`shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` By default, the server will listen at \`http://localhost:8080\` which can be changed by passing \`--host\` and \`--port\`. The web front end can be assessed from a browser at \`http://localhost:8080/\`. The OpenAI compatible API is at \`http://localhost:8080/v1/\`. ## What's More If you still find it difficult to use llama.cpp, don't worry, just check out other llama.cpp-based applications. For example, Qwen3 has already been officially part of Ollama and LM Studio, which are platforms for your to search and run local LLMs. Have fun! --- # Unknown \# llama.cpp \[^GGUF\]: GPT-Generated Unified Format :::{dropdown} llama.cpp as a C++ library Before starting, let's first discuss what is llama.cpp and what you should expect, and why we say "use" llama.cpp, with "use" in quotes. llama.cpp is essentially a different ecosystem with a different design philosophy that targets light-weight footprint, minimal external dependency, multi-platform, and extensive, flexible hardware support: - Plain C/C++ implementation without external dependencies - Support a wide variety of hardware: - AVX, AVX2 and AVX512 support for x86\_64 CPU - Apple Silicon via Metal and Accelerate (CPU and GPU) - NVIDIA GPU (via CUDA), AMD GPU (via hipBLAS), Intel GPU (via SYCL), Ascend NPU (via CANN), and Moore Threads GPU (via MUSA) - Vulkan backend for GPU - Various quantization schemes for faster inference and reduced memory footprint - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity It's like the Python frameworks \`torch\`+\`transformers\` or \`torch\`+\`vllm\` but in C++. However, this difference is crucial: - Python is an interpreted language: The code you write is executed line-by-line on-the-fly by an interpreter. You can run the example code snippet or script with an interpreter or a natively interactive interpreter shell. In addition, Python is learner friendly, and even if you don't know much before, you can tweak the source code here and there. - C++ is a compiled language: The source code you write needs to be compiled beforehand, and it is translated to machine code and an executable program by a compiler. The overhead from the language side is minimal. You do have source code for example programs showcasing how to use the library. But it is not very easy to modify the source code if you are not verse in C++ or C. To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of \[Ollama\](https://ollama.com/), \[LM Studio\](https://lmstudio.ai/), \[GPT4ALL\](https://www.nomic.ai/gpt4all), \[llamafile\](https://llamafile.ai/) etc. But that's not what this guide is intended or could do. Instead, here we introduce how to use the \`llama-cli\` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. ::: In this guide, we will show how to "use" \[llama.cpp\](https://github.com/ggml-org/llama.cpp) to run models on your local machine, in particular, the \`llama-cli\` and the \`llama-server\` example program, which comes with the library. The main steps are: 1. Get the programs 2. Get the Qwen3 models in GGUF\[^GGUF\] format 3. Run the program with the model :::{note} llama.cpp supports Qwen3 and Qwen3MoE from version \`b5092\`. ::: ## Getting the Program You can get the programs in various ways. For optimal efficiency, we recommend compiling the programs locally, so you get the CPU optimizations for free. However, if you don't have C++ compilers locally, you can also install using package managers or downloading pre-built binaries. They could be less efficient but for non-production example use, they are fine. :::::{tab-set} ::::{tab-item} Compile Locally Here, we show the basic command to compile \`llama-cli\` locally on \*\*macOS\*\* or \*\*Linux\*\*. For Windows or GPU users, please refer to \[the guide from llama.cpp\](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md). :::{rubric} Installing Build Tools :heading-level: 5 ::: To build locally, a C++ compiler and a build system tool are required. To see if they have been installed already, type \`cc --version\` or \`cmake --version\` in a terminal window. - If installed, the build configuration of the tool will be printed to the terminal, and you are good to go! - If errors are raised, you need to first install the related tools: - On macOS, install with the command \`xcode-select --install\` - On Ubuntu, install with the command \`sudo apt install build-essential\`. For other Linux distributions, the command may vary; the essential packages needed for this guide are \`gcc\` and \`cmake\`. :::{rubric} Compiling the Program :heading-level: 5 ::: For the first step, clone the repo and enter the directory: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Then, build llama.cpp using CMake: \`\`\`bash cmake -B build cmake --build build --config Release \`\`\` The first command will check the local environment and determine which backends and features should be included. The second command will actually build the programs. To shorten the time, you can also enable parallel compiling based on the CPU cores you have, for example: \`\`\`bash cmake --build build --config Release -j 8 \`\`\` This will build the programs with 8 parallel compiling jobs. The built programs will be in \`./build/bin/\`. :::: ::::{tab-item} Package Managers For \*\*macOS\*\* and \*\*Linux\*\* users, \`llama-cli\` and \`llama-server\` can be installed with package managers including Homebrew, Nix, and Flox. Here, we show how to install \`llama-cli\` and \`llama-server\` with Homebrew. For other package managers, please check the instructions \[here\](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). Installing with Homebrew is very simple: 1. Ensure that Homebrew is available on your operating system. If you don't have Homebrew, you can install it as in \[its website\](https://brew.sh/). 2. Second, you can install the pre-built binaries, \`llama-cli\` and \`llama-server\` included, with a single command: \`\`\`bash brew install llama.cpp \`\`\` Note that the installed binaries might not be built with the optimal compile options for your hardware, which can lead to poor performance. They also don't support GPU on Linux systems. :::: ::::{tab-item} Binary Release You can also download pre-built binaries from \[GitHub Releases\](https://github.com/ggml-org/llama.cpp/releases). Please note that those pre-built binaries files are architecture-, backend-, and os-specific. If you are not sure what those mean, you probably don't want to use them and running with incompatible versions will most likely fail or lead to poor performance. The file name is like \`llama-\-bin-\-\-.zip\`. There are three simple parts: - \`\`: the version of llama.cpp. The latest is preferred, but as llama.cpp is updated and released frequently, the latest may contain bugs. If the latest version does not work, try the previous release until it works. - \`\`: the operating system. \`win\` for Windows; \`macos\` for macOS; \`linux\` for Linux. - \`\`: the system architecture. \`x64\` for \`x86\_64\`, e.g., most Intel and AMD systems, including Intel Mac; \`arm64\` for \`arm64\`, e.g., Apple Silicon or Snapdragon-based systems. The \`\` part is somewhat complicated for Windows: - Running on CPU - x86\_64 CPUs: We suggest try the \`avx2\` one first. - \`noavx\`: No hardware acceleration at all. - \`avx2\`, \`avx\`, \`avx512\`: SIMD-based acceleration. Most modern desktop CPUs should support avx2, and some CPUs support \`avx512\`. - \`openblas\`: Relying on OpenBLAS for acceleration for prompt processing but not generation. - arm64 CPUs: We suggest try the \`llvm\` one first. - \[\`llvm\` and \`msvc\`\](https://github.com/ggml-org/llama.cpp/pull/7191) are different compilers - Running on GPU: We suggest try the \`cu\` one for NVIDIA GPUs, \`kompute\` for AMD GPUs, and \`sycl\` for Intel GPUs first. Ensure that you have related drivers installed. - \[\`vulcan\`\](https://github.com/ggml-org/llama.cpp/pull/2059): support certain NVIDIA and AMD GPUs - \[\`kompute\`\](https://github.com/ggml-org/llama.cpp/pull/4456): support certain NVIDIA and AMD GPUs - \[\`sycl\`\](https://github.com/ggml-org/llama.cpp/discussions/5138): Intel GPUs, oneAPI runtime is included - \`cu\`: NVIDIA GPUs, CUDA runtime is not included. You can download the \`cudart-llama-bin-win-cu\-x64.zip\` and unzip it to the same directory if you don't have the corresponding CUDA toolkit installed. You don't have much choice for macOS or Linux. - Linux: only one prebuilt binary, \`llama-\-bin-linux-x64.zip\`, supporting CPU. - macOS: \`llama-\-bin-macos-x64.zip\` for Intel Mac with no GPU support; \`llama-\-bin-macos-arm64.zip\` for Apple Silicon with GPU support. After downloading the \`.zip\` file, unzip them into a directory and open a terminal at that directory. :::: ::::: ## Getting the GGUF GGUF\[^GGUF\] is a file format for storing information needed to run a model, including but not limited to model weights, model hyperparameters, default generation configuration, and tokenizer. You can use the official Qwen GGUFs from our Hugging Face Hub or prepare your own GGUF file. ### Using the Official Qwen3 GGUFs We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with \`-GGUF\`. Download the GGUF model that you want with \`huggingface-cli\` (you need to install it first with \`pip install huggingface\_hub\`): \`\`\`bash huggingface-cli download \--local-dir \`\`\` For example: \`\`\`bash huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4\_k\_m.gguf --local-dir . \`\`\` This will download the Qwen3-8B model in GGUF format quantized with the scheme Q4\_K\_M. ### Preparing Your Own GGUF Model files from Hugging Face Hub can be converted to GGUF, using the \`convert-hf-to-gguf.py\` Python script. It does require you to have a working Python environment with at least \`transformers\` installed. Obtain the source file if you haven't already: \`\`\`bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp \`\`\` Suppose you would like to use Qwen3-8B you can make a GGUF file for the fp16 model as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile qwen3-8b-f16.gguf \`\`\` The first argument to the script refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file. Remember to create the output directory before you run the command. The fp16 model could be a bit heavy for running locally, and you can quantize the model as needed. We introduce the method of creating and quantizing GGUF files in \[this guide\](../quantization/llama.cpp). You can refer to that document for more information. ## Run Qwen with llama.cpp :::{note} Regarding switching between thinking and non-thinking modes, while the soft switch is always available, the hard switch implemented in the chat template is not exposed in llama.cpp. The quick workaround is to pass \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) equivalent to always \`enable\_thinking=False\` via \`--chat-template-file\`. ::: ### llama-cli \[llama-cli\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main) is a console program which can be used to chat with LLMs. Simple run the following command where you place the llama.cpp programs: \`\`\`shell ./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` Here are some explanations to the above command: - \*\*Model\*\*: llama-cli supports using model files from local path, remote URL, or Hugging Face hub. - \`-hf Qwen/Qwen3-8B-GGUF:Q8\_0\` in the above indicates we are using the model file from Hugging Face hub - To use a local path, pass \`-m qwen3-8b-q8\_0.gguf\` instead - To use a remote URL, pass \`-mu https://hf.co/Qwen/Qwen3-8B-GGUF/resolve/main/qwen3-8b-Q8\_0.gguf?download=true\` instead - \*\*Speed Optimization\*\*: - CPU: llama-cli by default will use CPU and you can change \`-t\` to specify how many threads you would like it to use, e.g., \`-t 8\` means using 8 threads. - GPU: If the programs are built with GPU support, you can use \`-ngl\`, which allows offloading some layers to the GPU for computation. If there are multiple GPUs, it will offload to all the GPUs. You can use \`-dev\` to control the devices used and \`-sm\` to control which kinds of parallelism is used. For example, \`-ngl 99 -dev cuda0,cuda1 -sm row\` means offload all layers to GPU 0 and GPU1 using the split mode row. Adding \`-fa\` may also speed up the generation. - \*\*Sampling Parameters\*\*: llama.cpp supports \[a variety of sampling methods\](https://github.com/ggml-org/llama.cpp/tree/master/tools/main#generation-flags) and has default configuration for many of them. It is recommended to adjust those parameters according to the actual case and the recommended parameters from Qwen3 modelcard could be used as a reference. If you encounter repetition and endless generation, it is recommended to pass in addition \`--presence-penalty\` up to \`2.0\`. - \*\*Context Management\*\*: llama.cpp adopts the "rotating" context management by default. The \`-c\` controls the maximum context length (default 4096, 0 means loaded from model), and \`-n\` controls the maximum generation length each time (default -1 means infinite until ending, -2 means until context full). When the context is full but the generation doesn't end, the first \`--keep\` tokens (default 0, -1 means all) from the initial prompt is kept, and the first half of the rest is discarded. Then, the model continues to generate based on the new context tokens. You can set \`--no-context-shift\` to prevent this rotating behavior and the generation will stop once \`-c\` is reached. llama.cpp supports YaRN, which can be enabled by \`-c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768\`. - \*\*Chat\*\*: \`--jinja\` indicates using the chat template embedded in the GGUF which is preferred and \`--color\` indicates coloring the texts so that user input and model output can be better differentiated. If there is a chat template, like in Qwen3 models, llama-cli will enter chat mode automatically. To stop generation or exit press "Ctrl+C". You can use \`-sys\` to add a system prompt. ### llama-server \[llama-server\](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) is a simple HTTP server, including a set of LLM REST APIs and a simple web front end to interact with LLMs using llama.cpp. The core command is similar to that of llama-cli. In addition, it supports thinking content parsing and tool call parsing. \`\`\`shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8\_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift \`\`\` By default, the server will listen at \`http://localhost:8080\` which can be changed by passing \`--host\` and \`--port\`. The web front end can be assessed from a browser at \`http://localhost:8080/\`. The OpenAI compatible API is at \`http://localhost:8080/v1/\`. ## What's More If you still find it difficult to use llama.cpp, don't worry, just check out other llama.cpp-based applications. For example, Qwen3 has already been officially part of Ollama and LM Studio, which are platforms for your to search and run local LLMs. Have fun! --- # Unknown dstack ======== \`dstack \`\_\_ is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML teams across top clouds, on-prem clusters, and accelerators. Prerequisites ---------------- Before you start, install dstack by following the \`installation instructions \`\_\_. Once dstack server is up, you can initialize your workspace as shown below: .. code:: bash mkdir dstack-qwen-deploy && cd dstack-qwen-deploy dstack init Deploy Qwen3-30B-A3B ----------------------------------------------- Deploy \`\`Qwen3-30B-A3B\`\` on instances available with cloud providers configured in your \`\`~/.dstack/server/config.yml\`\` file. You can use \`\`SgLang\`\`, \`\`TGI\`\` or \`\`vLLM\`\` to serve the model. Here we use \`\`SgLang\`\` as an example. Create a \`service \`\_\_ configuration file named \`\`serve-30b.dstack.yml\`\` with the following content: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 .. note:: For other inference backends such as vLLM or TGI, visit the \`dstack Inference Examples \`\_\_ documentation. Go ahead and apply the service configuration: .. code:: bash dstack apply -f serve-30b.dstack.yml Access the Service -------------------- After the service is successfully deployed, you can access the service's endpoint in the following ways: .. tab-set:: .. tab-item:: CURL Access through service endpoint at \`\`/proxy/services///\`\` .. code:: bash curl http://localhost:3000/proxy/services/main/qwen3-30b-a3b/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -H 'Authorization: Bearer ' \\ -d '{ "model": "Qwen/Qwen3-30B-A3B", "messages": \[ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } \] }' .. note:: When starting the dstack server, an admin token is automatically generated: .. code:: bash The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ .. tab-item:: Chat UI Access through dstack's Chat UI at \`\`/projects//models//\`\` .. image:: https://dstack.ai/static-assets/static-assets/images//dstack-qwen-ui.png .. dropdown:: Gateway :icon: info :animate: fade-in Running services for development purposes doesn't require setting up a gateway. However, you'll need a gateway in the following cases: \* To use auto-scaling or rate limits \* To enable HTTPS for the endpoint and map it to your domain \* If your service requires WebSockets \* If your service cannot work with a path prefix For detailed information about gateway configuration and usage, refer to the \`dstack documentation on gateways \`\_\_. Replicas and Auto Scaling ---------------------------------------- You can auto scale the service by specifying additional configurations in the \`\`serve-30b.dstack.yml\`\`. - Set \`\`replicas: min..max\`\` to define the minimum and maximum number of replicas - Configure \`\`scaling\`\` rules to determine when to scale up or down Below is a complete configuration example with auto-scaling enabled: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 # Minimum and maximum number of replicas replicas: 1..4 scaling: # Requests per seconds metric: rps # Target metric value target: 10 .. note:: The scaling property requires a gateway to be set up. See also ------------ - \*\*Fleets\*\*: Create cloud and on-prem clusters using \`Fleets \`\_\_. - \*\*Dev Environments\*\*: Experiment and test before deploying to production using \`Dev Environments \`\_\_. - \*\*Tasks\*\*: Schedule single node or distributed training using \`Tasks \`\_\_. - \*\*Services\*\*: Deploy models as secure, auto-scaling OpenAI-compatible endpoints using \`Services \`\_\_. - \*\*Metrics\*\*: Monitor performance with automatically tracked metrics via CLI or UI using \`Metrics \`\_\_. --- # Unknown TGI ===================== .. attention:: To be updated for Qwen3. Hugging Face's Text Generation Inference (TGI) is a production-ready framework specifically designed for deploying and serving large language models (LLMs) for text generation tasks. It offers a seamless deployment experience, powered by a robust set of features: \* \`Speculative Decoding \`\_: Accelerates generation speeds. \* \`Tensor Parallelism\`\_: Enables efficient deployment across multiple GPUs. \* \`Token Streaming\`\_: Allows for the continuous generation of text. \* Versatile Device Support: Works seamlessly with \`AMD\`\_, \`Gaudi\`\_ and \`AWS Inferentia\`\_. .. \_AMD: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/deploy-your-model.html#serving-using-hugging-face-tgi .. \_Gaudi: https://github.com/huggingface/tgi-gaudi .. \_AWS Inferentia: https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/#:~:text=Get%20started%20with%20TGI%20on%20SageMaker%20Hosting .. \_Tensor Parallelism: https://huggingface.co/docs/text-generation-inference/conceptual/tensor\_parallelism .. \_Token Streaming: https://huggingface.co/docs/text-generation-inference/conceptual/streaming Installation ----------------- The easiest way to use TGI is via the TGI docker image. In this guide, we show how to use TGI with docker. It's possible to run it locally via Conda or build locally. Please refer to \`Installation Guide \`\_ and \`CLI tool \`\_ for detailed instructions. Deploy Qwen2.5 with TGI ----------------------- 1. \*\*Find a Qwen2.5 Model:\*\* Choose a model from \`the Qwen2.5 collection \`\_. 2. \*\*Deployment Command:\*\* Run the following command in your terminal, replacing \`\`model\`\` with your chosen Qwen2.5 model ID and \`\`volume\`\` with the path to your local data directory: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model Using TGI API ------------- Once deployed, the model will be available on the mapped port (8080). TGI comes with a handy API for streaming response: .. code:: bash curl http://localhost:8080/generate\_stream -H 'Content-Type: application/json' \\ -d '{"inputs":"Tell me something about large language models.","parameters":{"max\_new\_tokens":512}}' It's also available on OpenAI style API: .. code:: bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' .. note:: The model field in the JSON is not used by TGI, you can put anything. Refer to the \`TGI Swagger UI \`\_ for a complete API reference. You can also use Python API: .. code:: python from openai import OpenAI # initialize the client but point it to TGI client = OpenAI( base\_url="http://localhost:8080/v1/", # replace with your endpoint url api\_key="", # this field is not used when running locally ) chat\_completion = client.chat.completions.create( model="", # it is not used by TGI, you can put anything messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], stream=True, temperature=0.7, top\_p=0.8, max\_tokens=512, ) # iterate and print stream for message in chat\_completion: print(message.choices\[0\].delta.content, end="") Quantization for Performance ---------------------------- 1. Data-dependent quantization (GPTQ and AWQ) Both GPTQ and AWQ models are data-dependent. The official quantized models can be found from \`the Qwen2.5 collection\`\_ and you can also quantize models with your own dataset to make it perform better on your use case. The following shows the command to start TGI with Qwen2.5-7B-Instruct-GPTQ-Int4: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize gptq If the model is quantized with AWQ, e.g. Qwen/Qwen2.5-7B-Instruct-AWQ, please use \`\`--quantize awq\`\`. 2. Data-agnostic quantization EETQ on the other side is not data dependent and can be used with any model. Note that we're passing in the original model (instead of a quantized model) with the \`\`--quantize eetq\`\` flag. .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq Multi-Accelerators Deployment ----------------------------- Use the \`\`--num-shard\`\` flag to specify the number of accelerators. Please also use \`\`--shm-size 1g\`\` to enable shared memory for optimal NCCL performance (\`reference \`\_\_): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --num-shard 2 Speculative Decoding -------------------- Speculative decoding can reduce the time per token by speculating on the next token. Use the \`\`--speculative-decoding\`\` flag, setting the value to the number of tokens to speculate on (default: 0 for no speculation): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 2 The overall performance of speculative decoding highly depends on the type of task. It works best for code or highly repetitive text. More context on speculative decoding can be found \`here \`\_\_. Zero-Code Deployment with HF Inference Endpoints --------------------------------------------------- For effortless deployment, leverage Hugging Face Inference Endpoints: - \*\*GUI interface:\*\* \`\`\_\_ - \*\*Coding interface:\*\* \`\`\_\_ Once deployed, the endpoint can be used as usual. Common Issues ---------------- Qwen2.5 supports long context lengths, so carefully choose the values for \`\`--max-batch-prefill-tokens\`\`, \`\`--max-total-tokens\`\`, and \`\`--max-input-tokens\`\` to avoid potential out-of-memory (OOM) issues. If an OOM occurs, you'll receive an error message upon startup. The following shows an example to modify those parameters: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-batch-prefill-tokens 4096 --max-total-tokens 4096 --max-input-tokens 2048 --- # Unknown \# SGLang \[SGLang\](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. To learn more about SGLang, please refer to the \[documentation\](https://docs.sglang.ai/). ## Environment Setup By default, you can install \`sglang\` with pip in a clean environment: \`\`\`shell pip install "sglang\[all\]>=0.4.6.post1" \`\`\` If you have encountered issues in installation, please feel free to check the official document for installation (\[link\](https://docs.sglang.ai/start/install.html)). ## API Service It is easy to build an OpenAI-compatible API service with SGLang, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:30000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B \`\`\` By default, if the \`--model-path\` does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export SGLANG\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in SGLang through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:30000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": True}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content SGLang supports parsing the thinking content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} \`enable\_thinking=False\` may not be compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, please consider disabling parsing thinking content. ::: ### Parsing Tool Calls SGLang supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25 \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md). ### Structured/JSON Output SGLang supports structured/JSON output. Please refer to \[SGLang's documentation\](https://docs.sglang.ai/backend/structured\_outputs.html#OpenAI-Compatible-API). Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-FP8 # For AWQ quantized model python -m sglang.launch\_server --model-path Qwen/Qwen3-8B-AWQ \`\`\` ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. SGLang supports YaRN, which can be configured as \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --json-model-override-args '{"rope\_scaling":{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}}' --context-length 131072 \`\`\` :::{note} SGLang implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which is used by SGLang. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: --- # Unknown \# vLLM We recommend you trying \[vLLM\](https://github.com/vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the \[paper\](https://arxiv.org/abs/2309.06180) and \[documentation\](https://vllm.readthedocs.io/). ## Installation By default, you can install \`vllm\` by pip in a clean environment: \`\`\`bash pip install vllm \`\`\` Please note that the prebuilt \`vllm\` has strict dependencies on \`torch\` and its CUDA versions. Check the note in the official document for installation (\[link\](https://docs.vllm.ai/en/latest/getting\_started/installation.html)) for some help. We also advise you to install ray by \`pip install ray\` for distributed serving. ## Offline Batched Inference Models supported by Qwen2.5 codes are supported by vLLM. The simplest usage of vLLM is offline batched inference as demonstrated below. \`\`\`python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen2.5-7B-Instruct") # Pass the default decoding hyperparameters of Qwen2.5-7B-Instruct # max\_tokens is for the maximum length for generation. sampling\_params = SamplingParams(temperature=0.7, top\_p=0.8, repetition\_penalty=1.05, max\_tokens=512) # Input the model name or path. Can be GPTQ or AWQ models. llm = LLM(model="Qwen/Qwen2.5-7B-Instruct") # Prepare your prompts prompt = "Tell me something about large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) # generate outputs outputs = llm.generate(\[text\], sampling\_params) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` ## OpenAI-Compatible API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:8000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct \`\`\` You don't need to worry about chat template as it by default uses the chat template provided by the tokenizer. Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} The OpenAI-compatible server in \`vllm\` comes with \[a default set of sampling parameters\](https://github.com/vllm-project/vllm/blob/v0.5.2/vllm/entrypoints/openai/protocol.py#L130), which are not suitable for Qwen2.5 models and prone to repetition. We advise you to always pass sampling parameters to the API. ::: ### Tool Use The OpenAI-compatible API could be configured to support tool call of Qwen2.5. For information, please refer to \[our guide on Function Calling\](../framework/function\_call.md#vllm). ### Structured/JSON Output Qwen 2.5, when used with vLLM, supports structured/JSON output. Please refer to \[vllm's documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#extra-parameters-for-chat-api) for the \`guided\_json\` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ## Multi-GPU Distributed Serving To scale up your serving throughput, distributed serving helps you by leveraging more GPU devices. Besides, for large models like \`Qwen2.5-72B-Instruct\`, it is impossible to serve it on a single GPU. Here, we demonstrate how to run \`Qwen2.5-72B-Instruct\` with tensor parallelism just by passing in the argument \`tensor\_parallel\_size\`: ::::{tab-set} :::{tab-item} Offline \`\`\`python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5-72B-Instruct", tensor\_parallel\_size=4) \`\`\` ::: :::{tab-item} API \`\`\`bash vllm serve Qwen/Qwen2.5-72B-Instruct --tensor-parallel-size 4 \`\`\` ::: :::: ## Extended Context Support By default, the context length for Qwen2.5 models are set to 32,768 tokens. To handle extensive inputs exceeding 32,768 tokens, we utilize \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YARN and it can be enabled by add a \`rope\_scaling\` field to the \`config.json\` file of the model. For example, \`\`\`json { ..., "rope\_scaling": { "factor": 4.0, "original\_max\_position\_embeddings": 32768, "type": "yarn" } } \`\`\` However, vLLM only supports \_static\_ YARN at present, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. ## Serving Quantized Models vLLM supports different types of quantized models, including AWQ, GPTQ, SqueezeLLM, etc. Here we show how to deploy AWQ and GPTQ models. The usage is almost the same as above except for an additional argument for quantization. For example, to run an AWQ model. e.g., \`Qwen2.5-7B-Instruct-AWQ\`: ::::{tab-set} :::{tab-item} Offline \`\`\`python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5-7B-Instruct-AWQ", quantization="awq") \`\`\` ::: :::{tab-item} API \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --quantization awq \`\`\` ::: :::: or GPTQ models like \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: ::::{tab-set} :::{tab-item} Offline \`\`\`python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4", quantization="gptq") \`\`\` ::: :::{tab-item} API \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --quantization gptq \`\`\` ::: :::: Additionally, vLLM supports the combination of AWQ or GPTQ models with KV cache quantization, namely FP8 E5M2 KV Cache. For example, ::::{tab-set} :::{tab-item} Offline \`\`\`python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4", quantization="gptq", kv\_cache\_dtype="fp8\_e5m2") \`\`\` ::: :::{tab-item} API \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --quantization gptq --kv-cache-dtype fp8\_e5m2 \`\`\` ::: :::: ## Troubleshooting You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is \`--max-model-len\`. Our provided default \`max\_position\_embedding\` is \`32768\` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is \`--gpu-memory-utilization\`. vLLM will pre-allocate this much GPU memory. By default, it is \`0.9\`. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try \`--enforce-eager\`, which may slow down infernece, or reduce the \`--max-model-len\`. --- # Unknown SkyPilot ======== What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, the highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the \`instruction \`\_\_ to install SkyPilot. Here we provide a simple example of using \`\`pip\`\` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly\[aws,gcp\]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the \`official document \`\_\_ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple Silicon Macs docker run --platform linux/amd64 \\ -td --rm --name sky \\ -v "$HOME/.sky:/root/.sky:rw" \\ -v "$HOME/.aws:/root/.aws:rw" \\ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \\ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen2.5-72B-Instruct with SkyPilot ------------------------------------------ 1. Start serving Qwen2.5-72B-Instruct on a single instance with any available GPU in the list specified in \`serve-72b.yaml \`\_\_ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "prompt": "My favorite food is", "max\_tokens": 512 }' | jq -r '.choices\[0\].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With \`SkyPilot Serve \`\_\_, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE\_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be \*\*the cheapest available one\*\* on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a \`\`curl -L\`\` command (\`\`-L\`\` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Accessing Qwen2.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen2.5 service with GUI by connecting a \`FastChat GUI server \`\_\_ to the endpoint launched above (see \`gui.yaml \`\_\_). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) \*\*Before launching, make sure you have changed Qwen/Qwen1.5-72B-Chat to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top\_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen2.5 on any cloud. We advise you to read the official doc for more usages and updates. Check \`this \`\_\_ out! --- # Unknown dstack ======== \`dstack \`\_\_ is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML teams across top clouds, on-prem clusters, and accelerators. Prerequisites ---------------- Before you start, install dstack by following the \`installation instructions \`\_\_. Once dstack server is up, you can initialize your workspace as shown below: .. code:: bash mkdir dstack-qwen-deploy && cd dstack-qwen-deploy dstack init Deploy Qwen3-30B-A3B ----------------------------------------------- Deploy \`\`Qwen3-30B-A3B\`\` on instances available with cloud providers configured in your \`\`~/.dstack/server/config.yml\`\` file. You can use \`\`SgLang\`\`, \`\`TGI\`\` or \`\`vLLM\`\` to serve the model. Here we use \`\`SgLang\`\` as an example. Create a \`service \`\_\_ configuration file named \`\`serve-30b.dstack.yml\`\` with the following content: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 .. note:: For other inference backends such as vLLM or TGI, visit the \`dstack Inference Examples \`\_\_ documentation. Go ahead and apply the service configuration: .. code:: bash dstack apply -f serve-30b.dstack.yml Access the Service -------------------- After the service is successfully deployed, you can access the service's endpoint in the following ways: .. tab-set:: .. tab-item:: CURL Access through service endpoint at \`\`/proxy/services///\`\` .. code:: bash curl http://localhost:3000/proxy/services/main/qwen3-30b-a3b/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -H 'Authorization: Bearer ' \\ -d '{ "model": "Qwen/Qwen3-30B-A3B", "messages": \[ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } \] }' .. note:: When starting the dstack server, an admin token is automatically generated: .. code:: bash The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ .. tab-item:: Chat UI Access through dstack's Chat UI at \`\`/projects//models//\`\` .. image:: https://dstack.ai/static-assets/static-assets/images//dstack-qwen-ui.png .. dropdown:: Gateway :icon: info :animate: fade-in Running services for development purposes doesn't require setting up a gateway. However, you'll need a gateway in the following cases: \* To use auto-scaling or rate limits \* To enable HTTPS for the endpoint and map it to your domain \* If your service requires WebSockets \* If your service cannot work with a path prefix For detailed information about gateway configuration and usage, refer to the \`dstack documentation on gateways \`\_\_. Replicas and Auto Scaling ---------------------------------------- You can auto scale the service by specifying additional configurations in the \`\`serve-30b.dstack.yml\`\`. - Set \`\`replicas: min..max\`\` to define the minimum and maximum number of replicas - Configure \`\`scaling\`\` rules to determine when to scale up or down Below is a complete configuration example with auto-scaling enabled: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL\_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch\_server --model-path $MODEL\_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 # Minimum and maximum number of replicas replicas: 1..4 scaling: # Requests per seconds metric: rps # Target metric value target: 10 .. note:: The scaling property requires a gateway to be set up. See also ------------ - \*\*Fleets\*\*: Create cloud and on-prem clusters using \`Fleets \`\_\_. - \*\*Dev Environments\*\*: Experiment and test before deploying to production using \`Dev Environments \`\_\_. - \*\*Tasks\*\*: Schedule single node or distributed training using \`Tasks \`\_\_. - \*\*Services\*\*: Deploy models as secure, auto-scaling OpenAI-compatible endpoints using \`Services \`\_\_. - \*\*Metrics\*\*: Monitor performance with automatically tracked metrics via CLI or UI using \`Metrics \`\_\_. --- # Unknown \# GPTQ \[GPTQ\](https://arxiv.org/abs/2210.17323) is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model with \[AutoGPTQ\](https://github.com/AutoGPTQ/AutoGPTQ). ## Usage of GPTQ Models with Hugging Face transformers :::{note} To use the official Qwen2.5 GPTQ models with \`transformers\`, please ensure that \`optimum>=1.20.0\` and compatible versions of \`transformers\` and \`auto\_gptq\` are installed. You can do that by \`\`\`bash pip install -U "optimum>=1.20.0" \`\`\` ::: Now, \`transformers\` has officially supported AutoGPTQ, which means that you can directly use the quantized model with \`transformers\`. For each size of Qwen2.5, we provide both Int4 and Int8 GPTQ quantized models. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of GPTQ Models with vLLM vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with \`AutoGPTQ\` with vLLM. If possible, it will automatically use the GPTQ Marlin kernel, which is more efficient. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4 \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen2.5-7B-Instruct-GPTQ-Int4", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoGPTQ If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: \`\`\`bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from auto\_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quantize\_config = BaseQuantizeConfig( bits=8, # 4 or 8 group\_size=128, damp\_percent=0.01, desc\_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static\_groups=False, sym=True, true\_sequential=True, model\_name\_or\_path=None, model\_file\_base\_name="model" ) max\_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoGPTQForCausalLM.from\_pretrained(model\_path, quantize\_config) \`\`\` However, if you would like to load the model on multiple GPUs, you need to use \`max\_memory\` instead of \`device\_map\`. Here is an example: \`\`\`python model = AutoGPTQForCausalLM.from\_pretrained( model\_path, quantize\_config, max\_memory={i: "20GB" for i in range(4)} ) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python import torch data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) model\_inputs = tokenizer(\[text\]) input\_ids = torch.tensor(model\_inputs.input\_ids\[:max\_len\], dtype=torch.int) data.append(dict(input\_ids=input\_ids, attention\_mask=input\_ids.ne(tokenizer.pad\_token\_id))) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python import logging logging.basicConfig( format="%(asctime)s %(levelname)s \[%(name)s\] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache\_examples\_on\_gpu=False) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, use\_safetensors=True) tokenizer.save\_pretrained(quant\_path) \`\`\` It is unfortunate that the \`save\_quantized\` method does not support sharding. For sharding, you need to load the model and use \`save\_pretrained\` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! ## Known Issues ### Qwen2.5-72B-Instruct-GPTQ-Int4 cannot stop generation properly :Model: Qwen2.5-72B-Instruct-GPTQ-Int4 :Framework: vLLM, AutoGPTQ (including Hugging Face transformers) :Description: Generation cannot stop properly. Continual generation after where it should stop, then repeated texts, either single character, a phrase, or paragraphs, are generated. :Workaround: The following workaround could be considered 1. Using the original model in 16-bit floating point 2. Using the AWQ variants or llama.cpp-based models for reduced chances of abnormal generation ### Qwen2.5-32B-Instruct-GPTQ-Int4 broken with vLLM on multiple GPUs :Model: Qwen2.5-32B-Instruct-GPTQ-Int4 :Framework: vLLM :Description: Deployment on multiple GPUs and only garbled text like \`!!!!!!!!!!!!!!!!!!\` could be generated. :Workaround: Each of the following workaround could be considered 1. Using the AWQ or GPTQ-Int8 variants 2. Using a single GPU 3. Using Hugging Face \`transformers\` if latency and throughput are not major concerns ## Troubleshooting :::{dropdown} With \`transformers\` and \`auto\_gptq\`, the logs suggest \`CUDA extension not installed.\` and the inference is slow. \`auto\_gptq\` fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its \[installation guide\](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md) to install a pre-built wheel or try installing \`auto\_gptq\` from source. ::: :::{dropdown} Self-quantized Qwen2.5-72B-Instruct-GPTQ with \`vllm\`, \`ValueError: ... must be divisible by ...\` is raised. The intermediate size of the self-quantized model is different from the official Qwen2.5-72B-Instruct-GPTQ models. After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2.5-72B is 29568. Unfortunately, {math}\`29568 \\div 128 = 231\`. Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with \`tensor\_parallel\_size=1\`, i.e., one GPU card. A workaround is to make the intermediate size divisible by {math}\`128 \\times 8 = 1024\`. To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality. Try the following: \`\`\`python import torch from torch.nn import functional as F from transformers import AutoModelForCausalLM # must use AutoModelForCausalLM model = AutoModelForCausalLM.from\_pretrained("Qwen/Qwen2.5-72B-Instruct", torch\_dtype="auto") # this size is Qwen2.5-72B only pad\_size = 128 sd = model.state\_dict() for i, k in enumerate(sd): v = sd\[k\] print(k, i) # interleaving the padded zeros if ('mlp.up\_proj.weight' in k) or ('mlp.gate\_proj.weight' in k): prev\_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568\*2, -1)\[:pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[pad\_size:\]\], dim=0) sd\[k\] = new\_v elif 'mlp.down\_proj.weight' in k: prev\_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568\*2)\[:, :pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[:, pad\_size:\]\], dim=1) sd\[k\] = new\_v # this is a very large file; make sure your RAM is enough to load the model torch.save(sd, '/path/to/padded\_model/pytorch\_model.bin') \`\`\` This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the \`intermediate\_size\` in \`config.json\` to \`29696\`. Finally, you can quantize the saved model checkpoint. ::: --- # Unknown \# vLLM We recommend you trying \[vLLM\](https://github.com/vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the \[paper\](https://arxiv.org/abs/2309.06180) and \[documentation\](https://docs.vllm.ai/). ## Environment Setup By default, you can install \`vllm\` with pip in a clean environment: \`\`\`shell pip install "vllm>=0.8.5" \`\`\` Please note that the prebuilt \`vllm\` has strict dependencies on \`torch\` and its CUDA versions. Check the note in the official document for installation (\[link\](https://docs.vllm.ai/en/latest/getting\_started/installation.html)) for more help. ## API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:8000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell vllm serve Qwen/Qwen3-8B \`\`\` By default, if the model does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export VLLM\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} \`vllm\` will use the sampling parameters from the \`generation\_config.json\` in the model files. While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in vLLM through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": False}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell vllm serve Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content vLLM supports parsing the thinking content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` Since vLLM 0.9.0, one can also use \`\`\`shell vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} As of vLLM 0.8.5, \`enable\_thinking=False\` is not compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, you should disable parsing thinking content. This is resolved in vLLM 0.9.0 with the \`qwen3\` reasoning parser. ::: ### Parsing Tool Calls vLLM supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md#vllm). ### Structured/JSON Output vLLM supports structured/JSON output. Please refer to \[vLLM's documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#extra-parameters-for-chat-api) for the \`guided\_json\` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model vllm serve Qwen/Qwen3-8B-FP8 # For AWQ quantized model vllm serve Qwen/Qwen3-8B-AWQ \`\`\` :::{note} The FP8 models of Qwen3 are block-wise quant, which is supported on NVIDIA GPUs with compute capability > 8.9, that is, Ada Lovelace, Hopper, and later GPUs and runs as w8a8. Since vLLM v0.9.0, FP8 Marlin has supported block-wise quants (running as w8a16) and you can also run Qwen3 FP8 models on Ampere cards. ::: :::{note} If you encountered the following error when deploying the FP8 models, it indicates that the tensor parallel size does not agree with the model weights: \`\`\` File ".../vllm/vllm/model\_executor/layers/quantization/fp8.py", line 477, in create\_weights raise ValueError( ValueError: The output\_size of gate's and up's weight = 192 is not divisible by weight quantization block\_n = 128. \`\`\` We recommend lowering the degree of tensor parallel, e.g., \`--tensor-parallel-size 4\` or enabling expert parallel, e.g., \`--tensor-parallel-size 8 --enable-expert-parallel\`. ::: ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YaRN, which can be configured as \`\`\`shell vllm serve Qwen/Qwen3-8B --rope-scaling '{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}' --max-model-len 131072 \`\`\` :::{note} vLLM implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which used by vLLM, if \`--max-model-len\` is not specified. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: ## Python Library vLLM can also be directly used as a Python library, which is convenient for offline batch inference but lack some API-only features, such as parsing model generation to structure messages. The following shows the basic usage of vLLM as a library: \`\`\`python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen3-8B") # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Set to False to strictly disable thinking ) # Generate outputs outputs = llm.generate(\[text\], sampling\_params) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` Since vLLM v0.9.0, you can also use the \`LLM.chat\` interface which includes support for \`chat\_template\_kwargs\`: \`\`\`python from vllm import LLM, SamplingParams # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] # Generate outputs outputs = llm.chat( \[messages\], sampling\_params, chat\_template\_kwargs={"enable\_thinking": True}, # Set to False to strictly disable thinking ) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` ## FAQ You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is \`--max-model-len\`. Our provided default \`max\_position\_embedding\` is \`40960\` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is \`--gpu-memory-utilization\`. vLLM will pre-allocate this much GPU memory. By default, it is \`0.9\`. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try \`--enforce-eager\`, which may slow down inference, or reduce the \`--max-model-len\`. For more usage guide with vLLM, please see vLLM's \[Qwen3 Usage Guide\](https://github.com/vllm-project/vllm/issues/17327). --- # Unknown OpenLLM ======= OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit \`the OpenLLM repository \`\_ to learn more. Installation ------------ Install OpenLLM using \`\`pip\`\`. .. code:: bash pip install openllm Verify the installation and display the help information: .. code:: bash openllm --help Quickstart ---------- Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository. .. code:: bash openllm repo update List the supported Qwen2.5 models: .. code:: bash openllm model list --tag qwen2.5 The results also display the required GPU resources and supported platforms: .. code:: bash model version repo required GPU RAM platforms ------- --------------------- ------- ------------------ ----------- qwen2.5 qwen2.5:0.5b default 12G linux qwen2.5:1.5b default 12G linux qwen2.5:3b default 12G linux qwen2.5:7b default 24G linux qwen2.5:14b default 80G linux qwen2.5:14b-ggml-q4 default macos qwen2.5:14b-ggml-q8 default macos qwen2.5:32b default 80G linux qwen2.5:32b-ggml-fp16 default macos qwen2.5:72b default 80Gx2 linux qwen2.5:72b-ggml-q4 default macos To start a server with one of the models, use \`\`openllm serve\`\` like this: .. code:: bash openllm serve qwen2.5:7b By default, the server starts at \`\`http://localhost:3000/\`\`. Interact with the model server ------------------------------ With the model server up and running, you can call its APIs in the following ways: .. tab-set:: .. tab-item:: CURL Send an HTTP request to its \`\`/generate\`\` endpoint via CURL: .. code-block:: bash curl -X 'POST' \\ 'http://localhost:3000/api/generate' \\ -H 'accept: text/event-stream' \\ -H 'Content-Type: application/json' \\ -d '{ "prompt": "Tell me something about large language models.", "model": "Qwen/Qwen2.5-7B-Instruct", "max\_tokens": 2048, "stop": null }' .. tab-item:: Python client Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example: .. code-block:: python from openai import OpenAI client = OpenAI(base\_url='http://localhost:3000/v1', api\_key='na') # Use the following func to get the available models # model\_list = client.models.list() # print(model\_list) chat\_completion = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ { "role": "user", "content": "Tell me something about large language models." } \], stream=True, ) for chunk in chat\_completion: print(chunk.choices\[0\].delta.content or "", end="") .. tab-item:: Chat UI OpenLLM provides a chat UI at the \`\`/chat\`\` endpoint for the LLM server at http://localhost:3000/chat. .. image:: ../../source/assets/qwen-openllm-ui-demo.png Model repository ---------------- A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our \`documentation to learn details \`\_. --- # Unknown TGI ===================== .. attention:: To be updated for Qwen3. Hugging Face's Text Generation Inference (TGI) is a production-ready framework specifically designed for deploying and serving large language models (LLMs) for text generation tasks. It offers a seamless deployment experience, powered by a robust set of features: \* \`Speculative Decoding \`\_: Accelerates generation speeds. \* \`Tensor Parallelism\`\_: Enables efficient deployment across multiple GPUs. \* \`Token Streaming\`\_: Allows for the continuous generation of text. \* Versatile Device Support: Works seamlessly with \`AMD\`\_, \`Gaudi\`\_ and \`AWS Inferentia\`\_. .. \_AMD: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/deploy-your-model.html#serving-using-hugging-face-tgi .. \_Gaudi: https://github.com/huggingface/tgi-gaudi .. \_AWS Inferentia: https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/#:~:text=Get%20started%20with%20TGI%20on%20SageMaker%20Hosting .. \_Tensor Parallelism: https://huggingface.co/docs/text-generation-inference/conceptual/tensor\_parallelism .. \_Token Streaming: https://huggingface.co/docs/text-generation-inference/conceptual/streaming Installation ----------------- The easiest way to use TGI is via the TGI docker image. In this guide, we show how to use TGI with docker. It's possible to run it locally via Conda or build locally. Please refer to \`Installation Guide \`\_ and \`CLI tool \`\_ for detailed instructions. Deploy Qwen2.5 with TGI ----------------------- 1. \*\*Find a Qwen2.5 Model:\*\* Choose a model from \`the Qwen2.5 collection \`\_. 2. \*\*Deployment Command:\*\* Run the following command in your terminal, replacing \`\`model\`\` with your chosen Qwen2.5 model ID and \`\`volume\`\` with the path to your local data directory: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model Using TGI API ------------- Once deployed, the model will be available on the mapped port (8080). TGI comes with a handy API for streaming response: .. code:: bash curl http://localhost:8080/generate\_stream -H 'Content-Type: application/json' \\ -d '{"inputs":"Tell me something about large language models.","parameters":{"max\_new\_tokens":512}}' It's also available on OpenAI style API: .. code:: bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' .. note:: The model field in the JSON is not used by TGI, you can put anything. Refer to the \`TGI Swagger UI \`\_ for a complete API reference. You can also use Python API: .. code:: python from openai import OpenAI # initialize the client but point it to TGI client = OpenAI( base\_url="http://localhost:8080/v1/", # replace with your endpoint url api\_key="", # this field is not used when running locally ) chat\_completion = client.chat.completions.create( model="", # it is not used by TGI, you can put anything messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], stream=True, temperature=0.7, top\_p=0.8, max\_tokens=512, ) # iterate and print stream for message in chat\_completion: print(message.choices\[0\].delta.content, end="") Quantization for Performance ---------------------------- 1. Data-dependent quantization (GPTQ and AWQ) Both GPTQ and AWQ models are data-dependent. The official quantized models can be found from \`the Qwen2.5 collection\`\_ and you can also quantize models with your own dataset to make it perform better on your use case. The following shows the command to start TGI with Qwen2.5-7B-Instruct-GPTQ-Int4: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize gptq If the model is quantized with AWQ, e.g. Qwen/Qwen2.5-7B-Instruct-AWQ, please use \`\`--quantize awq\`\`. 2. Data-agnostic quantization EETQ on the other side is not data dependent and can be used with any model. Note that we're passing in the original model (instead of a quantized model) with the \`\`--quantize eetq\`\` flag. .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq Multi-Accelerators Deployment ----------------------------- Use the \`\`--num-shard\`\` flag to specify the number of accelerators. Please also use \`\`--shm-size 1g\`\` to enable shared memory for optimal NCCL performance (\`reference \`\_\_): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --num-shard 2 Speculative Decoding -------------------- Speculative decoding can reduce the time per token by speculating on the next token. Use the \`\`--speculative-decoding\`\` flag, setting the value to the number of tokens to speculate on (default: 0 for no speculation): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 2 The overall performance of speculative decoding highly depends on the type of task. It works best for code or highly repetitive text. More context on speculative decoding can be found \`here \`\_\_. Zero-Code Deployment with HF Inference Endpoints --------------------------------------------------- For effortless deployment, leverage Hugging Face Inference Endpoints: - \*\*GUI interface:\*\* \`\`\_\_ - \*\*Coding interface:\*\* \`\`\_\_ Once deployed, the endpoint can be used as usual. Common Issues ---------------- Qwen2.5 supports long context lengths, so carefully choose the values for \`\`--max-batch-prefill-tokens\`\`, \`\`--max-total-tokens\`\`, and \`\`--max-input-tokens\`\` to avoid potential out-of-memory (OOM) issues. If an OOM occurs, you'll receive an error message upon startup. The following shows an example to modify those parameters: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-batch-prefill-tokens 4096 --max-total-tokens 4096 --max-input-tokens 2048 --- # Unknown OpenLLM ======= .. attention:: To be updated for Qwen3. OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit \`the OpenLLM repository \`\_ to learn more. Installation ------------ Install OpenLLM using \`\`pip\`\`. .. code:: bash pip install openllm Verify the installation and display the help information: .. code:: bash openllm --help Quickstart ---------- Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository. .. code:: bash openllm repo update List the supported Qwen2.5 models: .. code:: bash openllm model list --tag qwen2.5 The results also display the required GPU resources and supported platforms: .. code:: bash model version repo required GPU RAM platforms ------- --------------------- ------- ------------------ ----------- qwen2.5 qwen2.5:0.5b default 12G linux qwen2.5:1.5b default 12G linux qwen2.5:3b default 12G linux qwen2.5:7b default 24G linux qwen2.5:14b default 80G linux qwen2.5:14b-ggml-q4 default macos qwen2.5:14b-ggml-q8 default macos qwen2.5:32b default 80G linux qwen2.5:32b-ggml-fp16 default macos qwen2.5:72b default 80Gx2 linux qwen2.5:72b-ggml-q4 default macos To start a server with one of the models, use \`\`openllm serve\`\` like this: .. code:: bash openllm serve qwen2.5:7b By default, the server starts at \`\`http://localhost:3000/\`\`. Interact with the model server ------------------------------ With the model server up and running, you can call its APIs in the following ways: .. tab-set:: .. tab-item:: CURL Send an HTTP request to its \`\`/generate\`\` endpoint via CURL: .. code-block:: bash curl -X 'POST' \\ 'http://localhost:3000/api/generate' \\ -H 'accept: text/event-stream' \\ -H 'Content-Type: application/json' \\ -d '{ "prompt": "Tell me something about large language models.", "model": "Qwen/Qwen2.5-7B-Instruct", "max\_tokens": 2048, "stop": null }' .. tab-item:: Python client Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example: .. code-block:: python from openai import OpenAI client = OpenAI(base\_url='http://localhost:3000/v1', api\_key='na') # Use the following func to get the available models # model\_list = client.models.list() # print(model\_list) chat\_completion = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ { "role": "user", "content": "Tell me something about large language models." } \], stream=True, ) for chunk in chat\_completion: print(chunk.choices\[0\].delta.content or "", end="") .. tab-item:: Chat UI OpenLLM provides a chat UI at the \`\`/chat\`\` endpoint for the LLM server at http://localhost:3000/chat. .. image:: ../../source/assets/qwen-openllm-ui-demo.png Model repository ---------------- A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our \`documentation to learn details \`\_. --- # Unknown \# vLLM We recommend you trying \[vLLM\](https://github.com/vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the \[paper\](https://arxiv.org/abs/2309.06180) and \[documentation\](https://docs.vllm.ai/). ## Environment Setup By default, you can install \`vllm\` with pip in a clean environment: \`\`\`shell pip install "vllm>=0.8.5" \`\`\` Please note that the prebuilt \`vllm\` has strict dependencies on \`torch\` and its CUDA versions. Check the note in the official document for installation (\[link\](https://docs.vllm.ai/en/latest/getting\_started/installation.html)) for more help. ## API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:8000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell vllm serve Qwen/Qwen3-8B \`\`\` By default, if the model does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export VLLM\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} \`vllm\` will use the sampling parameters from the \`generation\_config.json\` in the model files. While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in vLLM through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": False}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell vllm serve Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content vLLM supports parsing the thinking content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` Since vLLM 0.9.0, one can also use \`\`\`shell vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} As of vLLM 0.8.5, \`enable\_thinking=False\` is not compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, you should disable parsing thinking content. This is resolved in vLLM 0.9.0 with the \`qwen3\` reasoning parser. ::: ### Parsing Tool Calls vLLM supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md#vllm). ### Structured/JSON Output vLLM supports structured/JSON output. Please refer to \[vLLM's documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#extra-parameters-for-chat-api) for the \`guided\_json\` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model vllm serve Qwen/Qwen3-8B-FP8 # For AWQ quantized model vllm serve Qwen/Qwen3-8B-AWQ \`\`\` :::{note} The FP8 models of Qwen3 are block-wise quant, which is supported on NVIDIA GPUs with compute capability > 8.9, that is, Ada Lovelace, Hopper, and later GPUs and runs as w8a8. Since vLLM v0.9.0, FP8 Marlin has supported block-wise quants (running as w8a16) and you can also run Qwen3 FP8 models on Ampere cards. ::: :::{note} If you encountered the following error when deploying the FP8 models, it indicates that the tensor parallel size does not agree with the model weights: \`\`\` File ".../vllm/vllm/model\_executor/layers/quantization/fp8.py", line 477, in create\_weights raise ValueError( ValueError: The output\_size of gate's and up's weight = 192 is not divisible by weight quantization block\_n = 128. \`\`\` We recommend lowering the degree of tensor parallel, e.g., \`--tensor-parallel-size 4\` or enabling expert parallel, e.g., \`--tensor-parallel-size 8 --enable-expert-parallel\`. ::: ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YaRN, which can be configured as \`\`\`shell vllm serve Qwen/Qwen3-8B --rope-scaling '{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}' --max-model-len 131072 \`\`\` :::{note} vLLM implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which used by vLLM, if \`--max-model-len\` is not specified. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: ## Python Library vLLM can also be directly used as a Python library, which is convenient for offline batch inference but lack some API-only features, such as parsing model generation to structure messages. The following shows the basic usage of vLLM as a library: \`\`\`python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen3-8B") # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Set to False to strictly disable thinking ) # Generate outputs outputs = llm.generate(\[text\], sampling\_params) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` Since vLLM v0.9.0, you can also use the \`LLM.chat\` interface which includes support for \`chat\_template\_kwargs\`: \`\`\`python from vllm import LLM, SamplingParams # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] # Generate outputs outputs = llm.chat( \[messages\], sampling\_params, chat\_template\_kwargs={"enable\_thinking": True}, # Set to False to strictly disable thinking ) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` ## FAQ You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is \`--max-model-len\`. Our provided default \`max\_position\_embedding\` is \`40960\` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is \`--gpu-memory-utilization\`. vLLM will pre-allocate this much GPU memory. By default, it is \`0.9\`. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try \`--enforce-eager\`, which may slow down inference, or reduce the \`--max-model-len\`. For more usage guide with vLLM, please see vLLM's \[Qwen3 Usage Guide\](https://github.com/vllm-project/vllm/issues/17327). --- # Unknown OpenLLM ======= .. attention:: To be updated for Qwen3. OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit \`the OpenLLM repository \`\_ to learn more. Installation ------------ Install OpenLLM using \`\`pip\`\`. .. code:: bash pip install openllm Verify the installation and display the help information: .. code:: bash openllm --help Quickstart ---------- Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository. .. code:: bash openllm repo update List the supported Qwen2.5 models: .. code:: bash openllm model list --tag qwen2.5 The results also display the required GPU resources and supported platforms: .. code:: bash model version repo required GPU RAM platforms ------- --------------------- ------- ------------------ ----------- qwen2.5 qwen2.5:0.5b default 12G linux qwen2.5:1.5b default 12G linux qwen2.5:3b default 12G linux qwen2.5:7b default 24G linux qwen2.5:14b default 80G linux qwen2.5:14b-ggml-q4 default macos qwen2.5:14b-ggml-q8 default macos qwen2.5:32b default 80G linux qwen2.5:32b-ggml-fp16 default macos qwen2.5:72b default 80Gx2 linux qwen2.5:72b-ggml-q4 default macos To start a server with one of the models, use \`\`openllm serve\`\` like this: .. code:: bash openllm serve qwen2.5:7b By default, the server starts at \`\`http://localhost:3000/\`\`. Interact with the model server ------------------------------ With the model server up and running, you can call its APIs in the following ways: .. tab-set:: .. tab-item:: CURL Send an HTTP request to its \`\`/generate\`\` endpoint via CURL: .. code-block:: bash curl -X 'POST' \\ 'http://localhost:3000/api/generate' \\ -H 'accept: text/event-stream' \\ -H 'Content-Type: application/json' \\ -d '{ "prompt": "Tell me something about large language models.", "model": "Qwen/Qwen2.5-7B-Instruct", "max\_tokens": 2048, "stop": null }' .. tab-item:: Python client Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example: .. code-block:: python from openai import OpenAI client = OpenAI(base\_url='http://localhost:3000/v1', api\_key='na') # Use the following func to get the available models # model\_list = client.models.list() # print(model\_list) chat\_completion = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=\[ { "role": "user", "content": "Tell me something about large language models." } \], stream=True, ) for chunk in chat\_completion: print(chunk.choices\[0\].delta.content or "", end="") .. tab-item:: Chat UI OpenLLM provides a chat UI at the \`\`/chat\`\` endpoint for the LLM server at http://localhost:3000/chat. .. image:: ../../source/assets/qwen-openllm-ui-demo.png Model repository ---------------- A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our \`documentation to learn details \`\_. --- # Unknown SkyPilot ======== .. attention:: To be updated for Qwen3. What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, the highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the \`instruction \`\_\_ to install SkyPilot. Here we provide a simple example of using \`\`pip\`\` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly\[aws,gcp\]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the \`official document \`\_\_ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple Silicon Macs docker run --platform linux/amd64 \\ -td --rm --name sky \\ -v "$HOME/.sky:/root/.sky:rw" \\ -v "$HOME/.aws:/root/.aws:rw" \\ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \\ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen2.5-72B-Instruct with SkyPilot ------------------------------------------ 1. Start serving Qwen2.5-72B-Instruct on a single instance with any available GPU in the list specified in \`serve-72b.yaml \`\_\_ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "prompt": "My favorite food is", "max\_tokens": 512 }' | jq -r '.choices\[0\].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With \`SkyPilot Serve \`\_\_, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE\_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be \*\*the cheapest available one\*\* on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a \`\`curl -L\`\` command (\`\`-L\`\` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Accessing Qwen2.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen2.5 service with GUI by connecting a \`FastChat GUI server \`\_\_ to the endpoint launched above (see \`gui.yaml \`\_\_). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) \*\*Before launching, make sure you have changed Qwen/Qwen1.5-72B-Chat to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top\_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen2.5 on any cloud. We advise you to read the official doc for more usages and updates. Check \`this \`\_\_ out! --- # Unknown \# llama.cpp \[^GGUF\]: GPT-Generated Unified Format In this guide, we will talk about how to "use" \[llama.cpp\](https://github.com/ggerganov/llama.cpp) to run Qwen2.5 models on your local machine, in particular, the \`llama-cli\` example program, which comes with the library. --- Before starting, let's first discuss what is llama.cpp and what you should expect, and why we say "use" llama.cpp, with "use" in quotes. llama.cpp is essentially a different ecosystem with a different design philosophy that targets light-weight footprint, minimal external dependency, multi-platform, and extensive, flexible hardware support: - Plain C/C++ implementation without external dependencies - Support a wide variety of hardware: - AVX, AVX2 and AVX512 support for x86\_64 CPU - Apple Silicon via Metal and Accelerate (CPU and GPU) - NVIDIA GPU (via CUDA), AMD GPU (via hipBLAS), Intel GPU (via SYCL), Ascend NPU (via CANN), and Moore Threads GPU (via MUSA) - Vulkan backend for GPU - Various quantization scheme for faster inference and reduced memory footprint - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity It's like the Python frameworks \`torch\`+\`transformers\` or \`torch\`+\`vllm\` but in C++. However, this difference is crucial: - Python is an interpreted language: The code you write is executed line-by-line on-the-fly by an interpreter. You can run the example code snippet or script with an interpreter or a natively interactive interpreter shell. In addition, Python is learner friendly, and even if you don't know much before, you can tweak the source code here and there. - C++ is a compiled language: The source code you write needs to be compiled beforehand, and it is translated to machine code and an executable program by a compiler. The overhead from the language side is minimal. You do have source code for example programs showcasing how to use the library. But it is not very easy to modify the source code if you are not verse in C++ or C. To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of \[Ollama\](https://ollama.com/), \[LM Studio\](https://lmstudio.ai/), \[GPT4ALL\](https://www.nomic.ai/gpt4all), \[llamafile\](https://llamafile.ai/) etc. But that's not what this guide is intended or could do. Instead, here we introduce how to use the \`llama-cli\` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. --- The main steps are: 1. Get the \`llama-cli\` program 2. Get the Qwen2.5 models in GGUF\[^GGUF\] format 3. Run the program with the model Remember that \`llama-cli\` is an example program, not a full-blown application. Sometimes it just does not work in the way you would like. This guide could also get quite technical sometimes. If you would like a smooth experience, check out the application mentioned above, which are much easier to "use". ## Getting the Program You can get the \`llama-cli\` program in various ways. For optimal efficiency, we recommend compiling the program locally, so you get the CPU optimizations for free. However, if you don't have C++ compilers locally, you can also install using package managers or downloading pre-built binaries. They could be less efficient but for non-production example use, they are fine. :::::{tab-set} ::::{tab-item} Compile Locally Here, we show the basic command to compile \`llama-cli\` locally on \*\*macOS\*\* or \*\*Linux\*\*. For Windows or GPU users, please refer to \[the guide from llama.cpp\](https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md). :::{rubric} Installing Build Tools :heading-level: 5 ::: To build locally, a C++ compiler and a build system tool are required. To see if they have been installed already, type \`cc --version\` or \`make --version\` in a terminal window. - If installed, the build configuration of the tool will be printed to the terminal, and you are good to go! - If errors are raised, you need to first install the related tools: - On macOS, install with the command \`xcode-select --install\` - On Ubuntu, install with the command \`sudo apt install build-essential\`. For other Linux distributions, the command may vary; the essential packages needed for this guide are \`gcc\` and \`make\`. :::{rubric} Compiling the Program :heading-level: 5 ::: For the first step, clone the repo and enter the directory: \`\`\`bash git clone https://github.com/ggerganov/llama.cpp cd llama.cpp \`\`\` Then use \`make\`: \`\`\`bash make llama-cli \`\`\` The command will only compile the parts needed for \`llama-cli\`. On macOS, it will enable Metal and Accelerate by default, so you can run with GPUs. On Linux, you won't get GPU support by default, but SIMD-optimization is enabled if available. To shorten the time, you can also enable parallel compiling based on the CPU cores you have, for example: \`\`\`bash make -j 8 llama-cli \`\`\` This will build the \`llama-cli\` target with 8 parallel compiling jobs. :::{note} There are other \[example programs\](https://github.com/ggerganov/llama.cpp/tree/master/examples) in llama.cpp. You can build them at once with simply (it may take some time): \`\`\`bash make -j 8 \`\`\` or you can also compile only the one you need, for example: \`\`\`bash make -j 8 llama-cli llama-quantize \`\`\` ::: :::: ::::{tab-item} Package Managers For \*\*macOS\*\* and \*\*Linux\*\* users, \`llama-cli\` can be installed with package managers including Homebrew, Nix, and Flox. Here, we show how to install \`llama-cli\` with Homebrew. For other package managers, please check the instructions \[here\](https://github.com/ggerganov/llama.cpp/blob/master/docs/install.md). Installing with Homebrew is very simple: 1. Ensure that Homebrew is available on your operating system. If you don't have Homebrew, you can install it as in \[its website\](https://brew.sh/). 2. Second, you can install the pre-built binaries, \`llama-cli\` included, with a single command: \`\`\`bash brew install llama.cpp \`\`\` Note that the installed binaries might not be built with the optimal compile options for your hardware, which can lead to poor performance. They also don't support GPU on Linux systems. :::: ::::{tab-item} Binary Release You can also download pre-built binaries from \[GitHub Releases\](https://github.com/ggerganov/llama.cpp/releases). Please note that those pre-built binaries files are architecture-, backend-, and os-specific. If you are not sure what those mean, you probably don't want to use them and running with incompatible versions will most likely fail or lead to poor performance. The file name is like \`llama-\-bin-\-\-.zip\`. There are three simple parts: - \`\`: the version of llama.cpp. The latest is preferred, but as llama.cpp is updated and released frequently, the latest may contain bugs. If the latest version does not work, try the previous release until it works. - \`\`: the operating system. \`win\` for Windows; \`macos\` for macOS; \`linux\` for Linux. - \`\`: the system architecture. \`x64\` for \`x86\_64\`, e.g., most Intel and AMD systems, including Intel Mac; \`arm64\` for \`arm64\`, e.g., Apple Silicon or Snapdragon-based systems. The \`\` part is somewhat complicated for Windows: - Running on CPU - x86\_64 CPUs: We suggest try the \`avx2\` one first. - \`noavx\`: No hardware acceleration at all. - \`avx2\`, \`avx\`, \`avx512\`: SIMD-based acceleration. Most modern desktop CPUs should support avx2, and some CPUs support \`avx512\`. - \`openblas\`: Relying on OpenBLAS for acceleration for prompt processing but not generation. - arm64 CPUs: We suggest try the \`llvm\` one first. - \[\`llvm\` and \`msvc\`\](https://github.com/ggerganov/llama.cpp/pull/7191) are different compilers - Running on GPU: We suggest try the \`cu\` one for NVIDIA GPUs, \`kompute\` for AMD GPUs, and \`sycl\` for Intel GPUs first. Ensure that you have related drivers installed. - \[\`vulcan\`\](https://github.com/ggerganov/llama.cpp/pull/2059): support certain NVIDIA and AMD GPUs - \[\`kompute\`\](https://github.com/ggerganov/llama.cpp/pull/4456): support certain NVIDIA and AMD GPUs - \[\`sycl\`\](https://github.com/ggerganov/llama.cpp/discussions/5138): Intel GPUs, oneAPI runtime is included - \`cu\`: NVIDIA GPUs, CUDA runtime is not included. You can download the \`cudart-llama-bin-win-cu\-x64.zip\` and unzip it to the same directory if you don't have the corresponding CUDA toolkit installed. You don't have much choice for macOS or Linux. - Linux: only one prebuilt binary, \`llama-\-bin-linux-x64.zip\`, supporting CPU. - macOS: \`llama-\-bin-macos-x64.zip\` for Intel Mac with no GPU support; \`llama-\-bin-macos-arm64.zip\` for Apple Silicon with GPU support. After downloading the \`.zip\` file, unzip them into a directory and open a terminal at that directory. :::: ::::: ## Getting the GGUF GGUF\[^GGUF\] is a file format for storing information needed to run a model, including but not limited to model weights, model hyperparameters, default generation configuration, and tokenizer. You can use the official Qwen2.5 GGUFs from our HuggingFace Hub or prepare your own GGUF file. ### Using the Official Qwen2.5 GGUFs We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with \`-GGUF\`. Download the GGUF model that you want with \`huggingface-cli\` (you need to install it first with \`pip install huggingface\_hub\`): \`\`\`bash huggingface-cli download \--local-dir \`\`\` For example: \`\`\`bash huggingface-cli download Qwen/Qwen2.5-7B-Instruct-GGUF qwen2.5-7b-instruct-q5\_k\_m.gguf --local-dir . \`\`\` This will download the Qwen2.5-7B-Instruct model in GGUF format quantized with the scheme Q5\_K\_M. ### Preparing Your Own Qwen2.5 GGUF Model files from HuggingFace Hub can be converted to GGUF, using the \`convert-hf-to-gguf.py\` Python script. It does require you to have a working Python environment with at least \`transformers\` installed. Obtain the source file if you haven't already: \`\`\`bash git clone https://github.com/ggerganov/llama.cpp cd llama.cpp \`\`\` Suppose you would like to use Qwen2.5-7B-Instruct, you can make a GGUF file for the fp16 model as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen2.5-7B-Instruct --outfile Qwen2.5-7b-instruct-f16.gguf \`\`\` The first argument to the script refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file. Remember to create the output directory before you run the command. The fp16 model could be a bit heavy for running locally, and you can quantize the model as needed. We introduce the method of creating and quantizing GGUF files in \[this guide\](../quantization/llama.cpp). You can refer to that document for more information. ## Running the Model :::{note} Due to random sampling and source code updates, the generated content with the same command as given in this section may be different from what is shown in the examples. ::: \`llama-cli\` provide multiple "mode" to "interact" with the model. Here, we demonstrate three ways to run the model, with increasing difficulty. ### Conversation Mode For users, to achieve chatbot-like experience, it is recommended to commence in the conversation mode \`\`\`bash ./llama-cli -m qwen2.5-7b-instruct-q5\_k\_m.gguf \\ -co -cnv -p "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." \\ -fa -ngl 80 -n 512 \`\`\` The program will first print metadata to the screen until you see the following: ... === Running in interactive mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to the AI. - To return control without starting a new line, end your input with '/'. - If you want to submit another line, end your input with '\\'. system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. > Now, the model is waiting for your input, and you can chat with the model: ... === Running in interactive mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to the AI. - To return control without starting a new line, end your input with '/'. - If you want to submit another line, end your input with '\\'. system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. > explain large language models in simple English Large language models are complex computer programs that have been trained to understand and generate human language. They are like giant dictionaries or thesauruses, but instead of just having words and definitions, they can understand the context, meaning and relationships between words. Imagine a really smart person who has read and understood every book on the planet. This person can understand what you are saying, know the meanings of words, understand the context, and even predict what you might say next. A large language model does something similar but it is based on algorithms and has been taught using huge amounts of text data. They're used in various applications like translating text, answerxng questions, writing stories, summarizing texts, detecting emotions in text, and even generating new text. These mdels are quite powerful but also quite complex and require a lot of computational resources to run. That's something, isn't it? You can stop the model generation anytime by Ctrl+C or Command+. However, if the model generation is ended and the control is returned to you, pressing the combination will exit the program. So what does the command we used actually do? Let's explain a little: :\`-m\` or \`--model\`: Model path, obviously. :\`-co\` or \`--color\`: Colorize output to distinguish prompt and user input from generations. Prompt text is dark yellow; user text is green; generated text is white; error text is red. :\`-cnv\` or \`--conversation\`: Run in conversation mode. The program will apply the chat template accordingly. :\`-p\` or \`--prompt\`: In conversation mode, it acts as the system message. :\`-fa\` or \`--flash-attn\`: Enable Flash Attention if the program is compiled with GPU support. :\`-ngl\` or \`--n-gpu-layers\`: Layers to the GPU for computation if the program is compiled with GPU support. :\`-n\` or \`--predict\`: Number of tokens to predict. You can also explore other options by \`\`\`bash ./llama-cli -h \`\`\` ### Interactive Mode The conversation mode hides the inner workings of LLMs. With interactive mode, you are made aware how LLMs work in the way to completion or continuation. The workflow is like 1. Give the model an initial prompt, and the model generates a completion. 2. Interrupt the model generation any time or wait until the model generates a reverse prompt or an eos token. 3. Append new texts (with optional prefix and suffix), and then let the model continues the generation. 4. Repeat Step 2. and Step 3. This workflow requires a different set of options, since you have to mind the chat template yourselves. To proper run the Qwen2.5 models, try the following: \`\`\`bash ./llama-cli -m Qwen2.5-7b-instruct-q5\_k\_m.gguf \\ -co -sp -i -if -p "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" \\ --in-prefix "<|im\_start|>user\\n" --in-suffix "<|im\_end|>\\n<|im\_start|>assistant\\n" \\ -fa -ngl 80 -n 512 \`\`\` We use some new options here: :\`-sp\` or \`--special\`: Show the special tokens. :\`-i\` or \`--interactive\`: Enter interactive mode. You can interrupt model generation and append new texts. :\`-if\` or \`--interactive-first\`: Immediately wait for user input. Otherwise, the model will run at once and generate based on the prompt. :\`-p\` or \`--prompt\`: In interactive mode, it is the contexts based on which the model predicts the continuation. :\`--in-prefix\`: String to prefix user inputs with. :\`--in-suffix\`: String to suffix after user inputs with. The result is like this: ... === Running in interactive mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to the AI. - To return control without starting a new line, end your input with '/'. - If you want to submit another line, end your input with '\\'. <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user explain large language models in simple English<|im\_end|> <|im\_start|>assistant Large Language Models (LLMs) are sophisticated algorithms designed to process and generate human-like text. They're trained on vast amounts of text data from the internet, books, and other sources, enabling them to understand language in various domains and generate responses that often closely resemble human output. LLMs can be used for various tasks such as language translation, text summarization, answering questions, and even creative writing. They are typically characterized by their size, with some models containing billions of parameters, making them quite complex and powerful.<|im\_end|> We use \`prompt\`, \`in-prefix\`, and \`in-suffix\` together to implement the chat template (ChatML-like) used by Qwen2.5 with a system message. So the experience is very similar to the conversation mode: you just need to type in the things you want to ask the model and don't need to worry about the chat template once the program starts. Note that, there should not be a new line after user input according to the template, so remember to end your input with \`/\`. :::{dropdown} Advanced Usage Interactive mode can achieve a lot more flexible workflows, under the condition that the chat template is maintained properly throughout. The following is an example: ... === Running in interactive mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to the AI. - To return control without starting a new line, end your input with '/'. - If you want to submit another line, end your input with '\\'. <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user give me a short introduction to LLMs.<|im\_end|> <|im\_start|>assistant A Large Language Model (LLM) is not what you think it is. It's not just a model of the world like a globe or a map. It's a type of artificial intelligence that has been trained on vast amounts of text data to understand and generate human language. These models can be used for a variety of tasks, such as language translation, text summarization, and even generating new text that is similar to a given piece of text. Some of the most well-known LLM applications are not what you think they are. They might include chatbots, content generation, and language translation, among others. The capabilities of these models are constantly expanding as researchers continue to improve and refine them. <|im\_end|> In the above example, I set \`--reverse-prompt\` to \`"LLM"\` so that the generation is interrupted whenever the model generates \`"LLM"\`\[^rp\]. The in prefix and in suffix are also set to empty so that I can add content exactly I want. After every generation of \`"LLM"\`, I added the part \`"...not what you think..."\` which are not likely to be generated by the model. Yet the model can continue generation just as fluent, although the logic is broken the second time around. I think it's fun to play around. ::: \[^rp\]: There are some gotchas in using \`--reverse-prompt\` as it matches tokens instead of strings. Since the same string can be tokenized differently in different contexts in BPE tokenization, some reverse prompts are never matched even though the string does exist in generation. ### Non-interactive Mode You can also use \`llama-cli\` for text completion by using just the prompt. However, it also means you have to format the input properly and only one turn can be generated. The following is an example: \`\`\` ./llama-cli -m Qwen2.5-7b-instruct-q5\_k\_m.gguf \\ -co -sp -p "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n<|im\_start|>user\\ngive me a short introduction to LLMs.<|im\_end|>\\n<|im\_start|>assistant\\n" \\ -fa -ngl 80 -n 512 \`\`\` The main output is as follows: <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user give me a short introduction to LLMs.<|im\_end|> <|im\_start|>assistant Large Language Models (LLMs) are sophisticated artificial intelligence models capable of generating human-like text. These models, trained on vast amounts of textual data, are designed to understand and generate text with varying degrees of complexity, including paragraphs, stories, articles, and even scripts. LLMs are trained using deep learning algorithms, often involving techniques like transformer architectures, which help them to learn the patterns and structures within language. Some of the most prominent examples of LLMs are: 1. \*\*GPT (Generative Pre-trained Transformer)\*\*: Developed by OpenAI, GPT models are known for their ability to generate text that is difficult to distinguish from human-written text. There are several versions of GPT, including GPT-2, GPT-3, and GPT-3.5. 2. \*\*Bert (Bidirectional Encoder Representations from Transformers)\*\*: Developed by Google, Bert is another influential model that uses transformers for natural language processing tasks, such as text classification and question answering. 3. \*\*LaMDA (Language Model for Dialogue Applications)\*\*: Developed by Google, this model is designed for conversational applications, capable of understanding context and providing responses in a natural and coherent manner. 4. \*\*T5 (Text to Text Transfer Transformer)\*\*: Developed by Google, T5 is designed to handle a wide range of natural language processing tasks by learning a universal representation of language. These models have applications in various fields, including but not limited to, content generation, text summarization, code generation, chatbot development, and more. They can significantly enhance human productivity and provide solutions for tasks that were previously difficult or impossible to automate. However, they also raise ethical concerns about privacy, bias, and misinformation, emphasizing the importance of responsible AI development and deployment.<|im\_end|> \[end of text\] In fact, you can start completion anywhere you want, even in the middle of an assistant message: <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user give me a short introduction to LLMs.<|im\_end|> <|im\_start|>assistant Large Language Models (LLMs) areare sophisticated artificial intelligence models designed to understand and generate human-like text. These models are typically trained on massive amounts of textual data, which enables them to learn patterns and linguistic structures, allowing them to produce coherent and contextually relevant text. LLMs can be used for various applications, including but not limited to, language translation, text summarization, chatbot interactions, content generation, and personalized recommendations. Some popular examples of LLMs include GPT-3, developed by OpenAI, and BERT, created by Google. These models are continuously refined and updated to enhance their performance and provide more accurate and nuanced outputs.<|im\_end|> \[end of text\] Now you can use \`llama-cli\` in three very different ways! Try talk to Qwen2.5 and share your experience with the community! ## What's More If you still find it difficult to use \`llama-cli\`, don't worry, just check out other llama.cpp-based applications. For example, Qwen2.5 has already been officially part of Ollama and LM Studio, which are platforms for your to search and run local LLMs. Have fun! --- # Unknown SkyPilot ======== .. attention:: To be updated for Qwen3. What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, the highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the \`instruction \`\_\_ to install SkyPilot. Here we provide a simple example of using \`\`pip\`\` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly\[aws,gcp\]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the \`official document \`\_\_ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple Silicon Macs docker run --platform linux/amd64 \\ -td --rm --name sky \\ -v "$HOME/.sky:/root/.sky:rw" \\ -v "$HOME/.aws:/root/.aws:rw" \\ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \\ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen2.5-72B-Instruct with SkyPilot ------------------------------------------ 1. Start serving Qwen2.5-72B-Instruct on a single instance with any available GPU in the list specified in \`serve-72b.yaml \`\_\_ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "prompt": "My favorite food is", "max\_tokens": 512 }' | jq -r '.choices\[0\].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With \`SkyPilot Serve \`\_\_, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE\_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be \*\*the cheapest available one\*\* on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a \`\`curl -L\`\` command (\`\`-L\`\` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Accessing Qwen2.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen2.5 service with GUI by connecting a \`FastChat GUI server \`\_\_ to the endpoint launched above (see \`gui.yaml \`\_\_). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) \*\*Before launching, make sure you have changed Qwen/Qwen1.5-72B-Chat to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top\_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen2.5 on any cloud. We advise you to read the official doc for more usages and updates. Check \`this \`\_\_ out! --- # Unknown \# AWQ :::{attention} To be updated for Qwen3. ::: For quantized models, one of our recommendations is the usage of \[AWQ\](https://arxiv.org/abs/2306.00978) with \[AutoAWQ\](https://github.com/casper-hansen/AutoAWQ). \*\*AWQ\*\* refers to Activation-aware Weight Quantization, a hardware-friendly approach for LLM low-bit weight-only quantization. \*\*AutoAWQ\*\* is an easy-to-use Python library for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model. ## Usage of AWQ Models with Hugging Face transformers Now, \`transformers\` has officially supported AutoAWQ, which means that you can directly use the quantized model with \`transformers\`. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-AWQ\` with the quantized model: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-AWQ" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto", ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of AWQ Models with vLLM vLLM has supported AWQ, which means that you can directly use our provided AWQ models or those quantized with \`AutoAWQ\` with vLLM. We recommend using the latest version of vLLM (\`vllm>=0.6.1\`) which brings performance improvements to AWQ models; otherwise, the performance might not be well-optimized. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-AWQ\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-AWQ", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct-AWQ", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoAWQ If you want to quantize your own model to AWQ quantized models, we advise you to use AutoAWQ. \`\`\`bash pip install "autoawq<0.2.7" \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own AWQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quant\_config = { "zero\_point": True, "q\_group\_size": 128, "w\_bit": 4, "version": "GEMM" } # Load your tokenizer and model with AutoAWQ tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoAWQForCausalLM.from\_pretrained(model\_path, device\_map="auto", safetensors=True) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) data.append(text.strip()) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python model.quantize(tokenizer, quant\_config=quant\_config, calib\_data=data) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, safetensors=True, shard\_size="4GB") tokenizer.save\_pretrained(quant\_path) \`\`\` Then you can obtain your own AWQ quantized model for deployment. Enjoy! --- # Unknown \# vLLM We recommend you trying \[vLLM\](https://github.com/vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the \[paper\](https://arxiv.org/abs/2309.06180) and \[documentation\](https://docs.vllm.ai/). ## Environment Setup By default, you can install \`vllm\` with pip in a clean environment: \`\`\`shell pip install "vllm>=0.8.5" \`\`\` Please note that the prebuilt \`vllm\` has strict dependencies on \`torch\` and its CUDA versions. Check the note in the official document for installation (\[link\](https://docs.vllm.ai/en/latest/getting\_started/installation.html)) for more help. ## API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`http://localhost:8000\`. You can specify the address with \`--host\` and \`--port\` arguments. Run the command as shown below: \`\`\`shell vllm serve Qwen/Qwen3-8B \`\`\` By default, if the model does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: \`\`\`shell export VLLM\_USE\_MODELSCOPE=true \`\`\` For distributed inference with tensor parallelism, it is as simple as \`\`\`shell vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 \`\`\` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{tip} \`vllm\` will use the sampling parameters from the \`generation\_config.json\` in the model files. While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in vLLM through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.7, "top\_p": 0.8, "top\_k": 20, "max\_tokens": 8192, "presence\_penalty": 1.5, "chat\_template\_kwargs": {"enable\_thinking": false} }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=8192, temperature=0.7, top\_p=0.8, presence\_penalty=1.5, extra\_body={ "top\_k": 20, "chat\_template\_kwargs": {"enable\_thinking": False}, }, ) print("Chat response:", chat\_response) \`\`\` :::: :::{note} Please note that passing \`enable\_thinking\` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use \[a custom chat template\](../../source/assets/qwen3\_nonthinking.jinja) when starting the model: \`\`\`shell vllm serve Qwen/Qwen3-8B --chat-template ./qwen3\_nonthinking.jinja \`\`\` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with \`/think\`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content vLLM supports parsing the thinking content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` Since vLLM 0.9.0, one can also use \`\`\`shell vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 \`\`\` The response message will have a field named \`reasoning\_content\` in addition to \`content\`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} As of vLLM 0.8.5, \`enable\_thinking=False\` is not compatible with this feature. If you need to pass \`enable\_thinking=False\` to the API, you should disable parsing thinking content. This is resolved in vLLM 0.9.0 with the \`qwen3\` reasoning parser. ::: ### Parsing Tool Calls vLLM supports parsing the tool calling content from the model generation into structured messages: \`\`\`shell vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes \`\`\` For more information, please refer to \[our guide on Function Calling\](../framework/function\_call.md#vllm). ### Structured/JSON Output vLLM supports structured/JSON output. Please refer to \[vLLM's documentation\](https://docs.vllm.ai/en/stable/serving/openai\_compatible\_server.html#extra-parameters-for-chat-api) for the \`guided\_json\` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: \`\`\`shell # For FP8 quantized model vllm serve Qwen/Qwen3-8B-FP8 # For AWQ quantized model vllm serve Qwen/Qwen3-8B-AWQ \`\`\` :::{note} The FP8 models of Qwen3 are block-wise quant, which is supported on NVIDIA GPUs with compute capability > 8.9, that is, Ada Lovelace, Hopper, and later GPUs and runs as w8a8. Since vLLM v0.9.0, FP8 Marlin has supported block-wise quants (running as w8a16) and you can also run Qwen3 FP8 models on Ampere cards. ::: :::{note} If you encountered the following error when deploying the FP8 models, it indicates that the tensor parallel size does not agree with the model weights: \`\`\` File ".../vllm/vllm/model\_executor/layers/quantization/fp8.py", line 477, in create\_weights raise ValueError( ValueError: The output\_size of gate's and up's weight = 192 is not divisible by weight quantization block\_n = 128. \`\`\` We recommend lowering the degree of tensor parallel, e.g., \`--tensor-parallel-size 4\` or enabling expert parallel, e.g., \`--tensor-parallel-size 8 --enable-expert-parallel\`. ::: ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of \[YaRN\](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YaRN, which can be configured as \`\`\`shell vllm serve Qwen/Qwen3-8B --rope-scaling '{"rope\_type":"yarn","factor":4.0,"original\_max\_position\_embeddings":32768}' --max-model-len 131072 \`\`\` :::{note} vLLM implements static YaRN, which means the scaling factor remains constant regardless of input length, \*\*potentially impacting performance on shorter texts.\*\* We advise adding the \`rope\_scaling\` configuration only when processing long contexts is required. It is also recommended to modify the \`factor\` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set \`factor\` as 2.0. ::: :::{note} The default \`max\_position\_embeddings\` in \`config.json\` is set to 40,960, which used by vLLM, if \`--max-model-len\` is not specified. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: ## Python Library vLLM can also be directly used as a Python library, which is convenient for offline batch inference but lack some API-only features, such as parsing model generation to structure messages. The following shows the basic usage of vLLM as a library: \`\`\`python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen3-8B") # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Set to False to strictly disable thinking ) # Generate outputs outputs = llm.generate(\[text\], sampling\_params) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` Since vLLM v0.9.0, you can also use the \`LLM.chat\` interface which includes support for \`chat\_template\_kwargs\`: \`\`\`python from vllm import LLM, SamplingParams # Configurae the sampling parameters (for thinking mode) sampling\_params = SamplingParams(temperature=0.6, top\_p=0.95, top\_k=20, max\_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt} \] # Generate outputs outputs = llm.chat( \[messages\], sampling\_params, chat\_template\_kwargs={"enable\_thinking": True}, # Set to False to strictly disable thinking ) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") \`\`\` ## FAQ You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is \`--max-model-len\`. Our provided default \`max\_position\_embedding\` is \`40960\` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is \`--gpu-memory-utilization\`. vLLM will pre-allocate this much GPU memory. By default, it is \`0.9\`. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try \`--enforce-eager\`, which may slow down inference, or reduce the \`--max-model-len\`. For more usage guide with vLLM, please see vLLM's \[Qwen3 Usage Guide\](https://github.com/vllm-project/vllm/issues/17327). --- # Unknown \# AWQ :::{attention} To be updated for Qwen3. ::: For quantized models, one of our recommendations is the usage of \[AWQ\](https://arxiv.org/abs/2306.00978) with \[AutoAWQ\](https://github.com/casper-hansen/AutoAWQ). \*\*AWQ\*\* refers to Activation-aware Weight Quantization, a hardware-friendly approach for LLM low-bit weight-only quantization. \*\*AutoAWQ\*\* is an easy-to-use Python library for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model. ## Usage of AWQ Models with Hugging Face transformers Now, \`transformers\` has officially supported AutoAWQ, which means that you can directly use the quantized model with \`transformers\`. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-AWQ\` with the quantized model: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-AWQ" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto", ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of AWQ Models with vLLM vLLM has supported AWQ, which means that you can directly use our provided AWQ models or those quantized with \`AutoAWQ\` with vLLM. We recommend using the latest version of vLLM (\`vllm>=0.6.1\`) which brings performance improvements to AWQ models; otherwise, the performance might not be well-optimized. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-AWQ\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-AWQ", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct-AWQ", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoAWQ If you want to quantize your own model to AWQ quantized models, we advise you to use AutoAWQ. \`\`\`bash pip install "autoawq<0.2.7" \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own AWQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quant\_config = { "zero\_point": True, "q\_group\_size": 128, "w\_bit": 4, "version": "GEMM" } # Load your tokenizer and model with AutoAWQ tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoAWQForCausalLM.from\_pretrained(model\_path, device\_map="auto", safetensors=True) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) data.append(text.strip()) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python model.quantize(tokenizer, quant\_config=quant\_config, calib\_data=data) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, safetensors=True, shard\_size="4GB") tokenizer.save\_pretrained(quant\_path) \`\`\` Then you can obtain your own AWQ quantized model for deployment. Enjoy! --- # Unknown \# llama.cpp Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program \`llama-quantize\` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using \`llama-imatrix\`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don't, check our guide \[here\](../run\_locally/llama.cpp.html#getting-the-program){.external}. ## Getting the GGUF Now, suppose you would like to quantize \`Qwen3-8B\`. You need to first make a GGUF file as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile Qwen3-8B-F16.gguf \`\`\` Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype bf16 --outfile Qwen3-8B-BF16.gguf \`\`\` Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype f32 --outfile Qwen3-8B-F32.gguf \`\`\` ## Quantizing the GGUF without Calibration For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: \`\`\`bash ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8\_0.gguf Q8\_0 \`\`\` \`Q8\_0\` is a code for a quantization preset. You can find all the presets in \[the source code of \`llama-quantize\`\](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp). Look for the variable \`QUANT\_OPTIONS\`. Common ones used for 8B models include \`Q8\_0\`, \`Q5\_K\_M\`, and \`Q4\_K\_M\`. The letter case doesn't matter, so \`q8\_0\` or \`q4\_K\_m\` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. ## Quantizing the GGUF with AWQ Scale :::{attention} To be updated for Qwen3. ::: To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following \[this script\](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export). First, when you run \`model.quantize()\` with \`autoawq\`, remember to add \`export\_compatible=True\` as shown below: \`\`\`python ... model.quantize( tokenizer, quant\_config=quant\_config, export\_compatible=True ) model.save\_pretrained(quant\_path) ... \`\`\` The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are "easier" to quantize.\[^AWQ\] Then, when you run \`convert-hf-to-gguf.py\`, remember to replace the model path with the path to the new model: \`\`\`bash python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf \`\`\` Finally, you can quantize the model as in the last example: \`\`\`bash ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 \`\`\` In this way, it should be possible to achieve similar quality with lower bit-per-weight. \[^AWQ\]: If you are interested in what this means, refer to \[the AWQ paper\](https://arxiv.org/abs/2306.00978). Basically, important weights (called salient weights in the paper) are identified based on activations across data examples. The weights are scaled accordingly such that the salient weights are protected even after quantization. ## Quantizing the GGUF with Importance Matrix Another possible solution is to use the "important matrix"\[^imatrix\], following \[this\](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix). First, you need to compute the importance matrix data of the weights of a model (\`-m\`) using a calibration dataset (\`-f\`): \`\`\`bash ./llama-imatrix -m Qwen3-8B-F16.gguf -f calibration-text.txt --chunk 512 -o Qwen3-8B-imatrix.dat -ngl 80 \`\`\` The text is cut in chunks of length \`--chunk\` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named \`Qwen3-8B-imatrix.dat\` (\`-o\`), which can then be used: \`\`\`bash ./llama-quantize --imatrix Qwen3-8B-imatrix.dat \\ Qwen3-8B-F16.gguf Qwen3-8B-Q4\_K\_M.gguf Q4\_K\_M \`\`\` For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide \`--imatrix\`, a helpful warning will be printed by \`llama-quantize\`. \[^imatrix\]: Here, the importance matrix keeps record of how weights affect the output: the weight should be important is a slight change in its value causes huge difference in the results, akin to the \[GPTQ\](https://arxiv.org/abs/2210.17323) algorithm. ## Perplexity Evaluation \`llama.cpp\` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say "wiki test"\[^wiki\]. You can download the dataset with: \`\`\`bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip \`\`\` Then you can run the test with the following command: \`\`\`bash ./llama-perplexity -m Qwen3-8B-Q8\_0.gguf -f wiki.test.raw -ngl 80 \`\`\` Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture \[here\](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md). It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. \[^wiki\]: It is not a good evaluation dataset for instruct models though, but it is very common and easily accessible. You probably want to use a dataset similar to your target domain. ## Finally In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the \[llama.cpp GitHub repo\](https://github.com/ggml-org/llama.cpp). We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don't provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven't found what you need in our repos, look around. Enjoy your freshly quantized models! --- # Unknown \# LLaMA-Factory :::{attention} To be updated for Qwen3. ::: Here we provide a script for supervised finetuning Qwen2.5 with \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory). This script for supervised finetuning (SFT) has the following features: - Support single-GPU and multi-GPU training; - Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. ## Installation Before you start, make sure you have installed the following packages: 1. Follow the instructions of \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory), and build the environment. 2. Install these packages (Optional): \`\`\` pip install deepspeed pip install flash-attn --no-build-isolation \`\`\` 3. If you want to use \[FlashAttention-2\](https://github.com/Dao-AILab/flash-attention), make sure your CUDA is 11.6 and above. ## Data Preparation LLaMA-Factory provides several training datasets in \`data\` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a \*\*json\*\* file and put your data in \`data\` folder. LLaMA-Factory supports dataset in \`alpaca\` or \`sharegpt\` format. - The dataset in \`alpaca\` format should follow the below format: \`\`\`json \[ { "instruction": "user instruction (required)", "input": "user input (optional)", "output": "model response (required)", "system": "system prompt (optional)", "history": \[ \["user instruction in the first round (optional)", "model response in the first round (optional)"\], \["user instruction in the second round (optional)", "model response in the second round (optional)"\] \] } \] \`\`\` - The dataset in \`sharegpt\` format should follow the below format: \`\`\`json \[ { "conversations": \[ { "from": "human", "value": "user instruction" }, { "from": "gpt", "value": "model response" } \], "system": "system prompt (optional)", "tools": "tool description (optional)" } \] \`\`\` 2. Provide your dataset definition in \`data/dataset\_info.json\` in the following format . - For \`alpaca\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } \`\`\` - For \`sharegpt\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } \`\`\` ## Training Execute the following training command: \`\`\`bash DISTRIBUTED\_ARGS=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ --deepspeed $DS\_CONFIG\_PATH \\ --stage sft \\ --do\_train \\ --use\_fast\_tokenizer \\ --flash\_attn \\ --model\_name\_or\_path $MODEL\_PATH \\ --dataset your\_dataset \\ --template qwen \\ --finetuning\_type lora \\ --lora\_target q\_proj,v\_proj\\ --output\_dir $OUTPUT\_PATH \\ --overwrite\_cache \\ --overwrite\_output\_dir \\ --warmup\_steps 100 \\ --weight\_decay 0.1 \\ --per\_device\_train\_batch\_size 4 \\ --gradient\_accumulation\_steps 4 \\ --ddp\_timeout 9000 \\ --learning\_rate 5e-6 \\ --lr\_scheduler\_type cosine \\ --logging\_steps 1 \\ --cutoff\_len 4096 \\ --save\_steps 1000 \\ --plot\_loss \\ --num\_train\_epochs 3 \\ --bf16 \`\`\` and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is \`cutoff\_len\`, which is the maximum length of the training data. Control this parameter to avoid OOM error. ## Merge LoRA If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 llamafactory-cli export \\ --model\_name\_or\_path path\_to\_base\_model \\ --adapter\_name\_or\_path path\_to\_adapter \\ --template qwen \\ --finetuning\_type lora \\ --export\_dir path\_to\_export \\ --export\_size 2 \\ --export\_legacy\_format False \`\`\` ## Conclusion The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # Unknown \# verl verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of \[HybridFlow: A Flexible and Efficient RLHF Framework\](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: \[verl\](https://github.com/volcengine/verl) verl is flexible and easy to use with: - \*\*Easy extension of diverse RL algorithms\*\*: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. - \*\*Seamless integration of existing LLM infra with modular APIs\*\*: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc - \*\*Flexible device mapping\*\*: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. - \*\*Ready integration with popular HuggingFace models\*\*: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: - \*\*State-of-the-art throughput\*\*: SOTA LLM training and inference engine integrations and SOTA RL throughput. - \*\*Efficient actor model resharding with 3D-HybridEngine\*\*: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. ## Reinforcement Learning (RL) Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation You can follow verl's \[installation guide\](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: \`\`\`shell git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py --local\_dir ~/data/gsm8k \`\`\` Model download can be done using the following command: \`\`\`shell python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" \`\`\` ### Step2: Start Training In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. \`\`\`bash python3 -m verl.trainer.main\_ppo \\ algorithm.adv\_estimator=grpo \\ data.train\_files=$HOME/data/gsm8k/train.parquet \\ data.val\_files=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size=1024 \\ data.max\_prompt\_length=512 \\ data.max\_response\_length=1024 \\ data.filter\_overlong\_prompts=True \\ data.truncation='error' \\ actor\_rollout\_ref.model.path=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size=1 \\ actor\_rollout\_ref.rollout.name=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization=0.6 \\ actor\_rollout\_ref.rollout.n=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload=True \\ algorithm.use\_kl\_in\_reward=False \\ trainer.critic\_warmup=0 \\ trainer.logger=\['console'\] \\ trainer.project\_name='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node=1 \\ trainer.nnodes=1 \\ trainer.save\_freq=-1 \\ trainer.test\_freq=5 \\ trainer.total\_epochs=15 $@ \`\`\` ## Finally If you encounter any difficulties during use, please join the discussion at \[GitHub\](https://github.com/volcengine/verl/discussions). --- # Unknown \# LLaMA-Factory :::{attention} To be updated for Qwen3. ::: Here we provide a script for supervised finetuning Qwen2.5 with \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory). This script for supervised finetuning (SFT) has the following features: - Support single-GPU and multi-GPU training; - Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. ## Installation Before you start, make sure you have installed the following packages: 1. Follow the instructions of \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory), and build the environment. 2. Install these packages (Optional): \`\`\` pip install deepspeed pip install flash-attn --no-build-isolation \`\`\` 3. If you want to use \[FlashAttention-2\](https://github.com/Dao-AILab/flash-attention), make sure your CUDA is 11.6 and above. ## Data Preparation LLaMA-Factory provides several training datasets in \`data\` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a \*\*json\*\* file and put your data in \`data\` folder. LLaMA-Factory supports dataset in \`alpaca\` or \`sharegpt\` format. - The dataset in \`alpaca\` format should follow the below format: \`\`\`json \[ { "instruction": "user instruction (required)", "input": "user input (optional)", "output": "model response (required)", "system": "system prompt (optional)", "history": \[ \["user instruction in the first round (optional)", "model response in the first round (optional)"\], \["user instruction in the second round (optional)", "model response in the second round (optional)"\] \] } \] \`\`\` - The dataset in \`sharegpt\` format should follow the below format: \`\`\`json \[ { "conversations": \[ { "from": "human", "value": "user instruction" }, { "from": "gpt", "value": "model response" } \], "system": "system prompt (optional)", "tools": "tool description (optional)" } \] \`\`\` 2. Provide your dataset definition in \`data/dataset\_info.json\` in the following format . - For \`alpaca\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } \`\`\` - For \`sharegpt\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } \`\`\` ## Training Execute the following training command: \`\`\`bash DISTRIBUTED\_ARGS=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ --deepspeed $DS\_CONFIG\_PATH \\ --stage sft \\ --do\_train \\ --use\_fast\_tokenizer \\ --flash\_attn \\ --model\_name\_or\_path $MODEL\_PATH \\ --dataset your\_dataset \\ --template qwen \\ --finetuning\_type lora \\ --lora\_target q\_proj,v\_proj\\ --output\_dir $OUTPUT\_PATH \\ --overwrite\_cache \\ --overwrite\_output\_dir \\ --warmup\_steps 100 \\ --weight\_decay 0.1 \\ --per\_device\_train\_batch\_size 4 \\ --gradient\_accumulation\_steps 4 \\ --ddp\_timeout 9000 \\ --learning\_rate 5e-6 \\ --lr\_scheduler\_type cosine \\ --logging\_steps 1 \\ --cutoff\_len 4096 \\ --save\_steps 1000 \\ --plot\_loss \\ --num\_train\_epochs 3 \\ --bf16 \`\`\` and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is \`cutoff\_len\`, which is the maximum length of the training data. Control this parameter to avoid OOM error. ## Merge LoRA If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 llamafactory-cli export \\ --model\_name\_or\_path path\_to\_base\_model \\ --adapter\_name\_or\_path path\_to\_adapter \\ --template qwen \\ --finetuning\_type lora \\ --export\_dir path\_to\_export \\ --export\_size 2 \\ --export\_legacy\_format False \`\`\` ## Conclusion The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # Unknown Qwen-Agent ========== \`Qwen-Agent \`\_\_ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant. Installation ------------ .. code:: bash git clone https://github.com/QwenLM/Qwen-Agent.git cd Qwen-Agent pip install -e ./ Developing Your Own Agent ------------------------- Qwen-Agent provides atomic components such as LLMs and prompts, as well as high-level components such as Agents. The example below uses the Assistant component as an illustration, demonstrating how to add custom tools and quickly develop an agent that uses tools. .. code:: py import json import os import json5 import urllib.parse from qwen\_agent.agents import Assistant from qwen\_agent.tools.base import BaseTool, register\_tool llm\_cfg = { # Use the model service provided by DashScope: 'model': 'qwen-max', 'model\_server': 'dashscope', # 'api\_key': 'YOUR\_DASHSCOPE\_API\_KEY', # It will use the \`DASHSCOPE\_API\_KEY' environment variable if 'api\_key' is not set here. # Use your own model service compatible with OpenAI API: # 'model': 'Qwen/Qwen2.5-7B-Instruct', # 'model\_server': 'http://localhost:8000/v1', # api\_base # 'api\_key': 'EMPTY', # (Optional) LLM hyperparameters for generation: 'generate\_cfg': { 'top\_p': 0.8 } } system = 'According to the user\\'s request, you first draw a picture and then automatically run code to download the picture ' + \\ 'and select an image operation from the given document to process the image' # Add a custom tool named my\_image\_gen： @register\_tool('my\_image\_gen') class MyImageGen(BaseTool): description = 'AI painting (image generation) service, input text description, and return the image URL drawn based on text information.' parameters = \[{ 'name': 'prompt', 'type': 'string', 'description': 'Detailed description of the desired image content, in English', 'required': True }\] def call(self, params: str, \*\*kwargs) -> str: prompt = json5.loads(params)\['prompt'\] prompt = urllib.parse.quote(prompt) return json.dumps( {'image\_url': f'https://image.pollinations.ai/prompt/{prompt}'}, ensure\_ascii=False) tools = \['my\_image\_gen', 'code\_interpreter'\] # code\_interpreter is a built-in tool in Qwen-Agent bot = Assistant(llm=llm\_cfg, system\_message=system, function\_list=tools, files=\[os.path.abspath('doc.pdf')\]) messages = \[\] while True: query = input('user question: ') messages.append({'role': 'user', 'content': query}) response = \[\] for response in bot.run(messages=messages): print('bot response:', response) messages.extend(response) The framework also provides more atomic components for developers to combine. For additional showcases, please refer to \`examples \`\_\_. --- # Unknown \# llama.cpp Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program \`llama-quantize\` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using \`llama-imatrix\`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don't, check our guide \[here\](../run\_locally/llama.cpp.html#getting-the-program){.external}. ## Getting the GGUF Now, suppose you would like to quantize \`Qwen3-8B\`. You need to first make a GGUF file as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile Qwen3-8B-F16.gguf \`\`\` Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype bf16 --outfile Qwen3-8B-BF16.gguf \`\`\` Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype f32 --outfile Qwen3-8B-F32.gguf \`\`\` ## Quantizing the GGUF without Calibration For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: \`\`\`bash ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8\_0.gguf Q8\_0 \`\`\` \`Q8\_0\` is a code for a quantization preset. You can find all the presets in \[the source code of \`llama-quantize\`\](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp). Look for the variable \`QUANT\_OPTIONS\`. Common ones used for 8B models include \`Q8\_0\`, \`Q5\_K\_M\`, and \`Q4\_K\_M\`. The letter case doesn't matter, so \`q8\_0\` or \`q4\_K\_m\` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. ## Quantizing the GGUF with AWQ Scale :::{attention} To be updated for Qwen3. ::: To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following \[this script\](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export). First, when you run \`model.quantize()\` with \`autoawq\`, remember to add \`export\_compatible=True\` as shown below: \`\`\`python ... model.quantize( tokenizer, quant\_config=quant\_config, export\_compatible=True ) model.save\_pretrained(quant\_path) ... \`\`\` The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are "easier" to quantize.\[^AWQ\] Then, when you run \`convert-hf-to-gguf.py\`, remember to replace the model path with the path to the new model: \`\`\`bash python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf \`\`\` Finally, you can quantize the model as in the last example: \`\`\`bash ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 \`\`\` In this way, it should be possible to achieve similar quality with lower bit-per-weight. \[^AWQ\]: If you are interested in what this means, refer to \[the AWQ paper\](https://arxiv.org/abs/2306.00978). Basically, important weights (called salient weights in the paper) are identified based on activations across data examples. The weights are scaled accordingly such that the salient weights are protected even after quantization. ## Quantizing the GGUF with Importance Matrix Another possible solution is to use the "important matrix"\[^imatrix\], following \[this\](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix). First, you need to compute the importance matrix data of the weights of a model (\`-m\`) using a calibration dataset (\`-f\`): \`\`\`bash ./llama-imatrix -m Qwen3-8B-F16.gguf -f calibration-text.txt --chunk 512 -o Qwen3-8B-imatrix.dat -ngl 80 \`\`\` The text is cut in chunks of length \`--chunk\` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named \`Qwen3-8B-imatrix.dat\` (\`-o\`), which can then be used: \`\`\`bash ./llama-quantize --imatrix Qwen3-8B-imatrix.dat \\ Qwen3-8B-F16.gguf Qwen3-8B-Q4\_K\_M.gguf Q4\_K\_M \`\`\` For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide \`--imatrix\`, a helpful warning will be printed by \`llama-quantize\`. \[^imatrix\]: Here, the importance matrix keeps record of how weights affect the output: the weight should be important is a slight change in its value causes huge difference in the results, akin to the \[GPTQ\](https://arxiv.org/abs/2210.17323) algorithm. ## Perplexity Evaluation \`llama.cpp\` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say "wiki test"\[^wiki\]. You can download the dataset with: \`\`\`bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip \`\`\` Then you can run the test with the following command: \`\`\`bash ./llama-perplexity -m Qwen3-8B-Q8\_0.gguf -f wiki.test.raw -ngl 80 \`\`\` Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture \[here\](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md). It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. \[^wiki\]: It is not a good evaluation dataset for instruct models though, but it is very common and easily accessible. You probably want to use a dataset similar to your target domain. ## Finally In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the \[llama.cpp GitHub repo\](https://github.com/ggml-org/llama.cpp). We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don't provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven't found what you need in our repos, look around. Enjoy your freshly quantized models! --- # Unknown \# verl verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of \[HybridFlow: A Flexible and Efficient RLHF Framework\](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: \[verl\](https://github.com/volcengine/verl) verl is flexible and easy to use with: - \*\*Easy extension of diverse RL algorithms\*\*: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. - \*\*Seamless integration of existing LLM infra with modular APIs\*\*: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc - \*\*Flexible device mapping\*\*: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. - \*\*Ready integration with popular HuggingFace models\*\*: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: - \*\*State-of-the-art throughput\*\*: SOTA LLM training and inference engine integrations and SOTA RL throughput. - \*\*Efficient actor model resharding with 3D-HybridEngine\*\*: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. ## Reinforcement Learning (RL) Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation You can follow verl's \[installation guide\](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: \`\`\`shell git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py --local\_dir ~/data/gsm8k \`\`\` Model download can be done using the following command: \`\`\`shell python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" \`\`\` ### Step2: Start Training In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. \`\`\`bash python3 -m verl.trainer.main\_ppo \\ algorithm.adv\_estimator=grpo \\ data.train\_files=$HOME/data/gsm8k/train.parquet \\ data.val\_files=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size=1024 \\ data.max\_prompt\_length=512 \\ data.max\_response\_length=1024 \\ data.filter\_overlong\_prompts=True \\ data.truncation='error' \\ actor\_rollout\_ref.model.path=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size=1 \\ actor\_rollout\_ref.rollout.name=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization=0.6 \\ actor\_rollout\_ref.rollout.n=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload=True \\ algorithm.use\_kl\_in\_reward=False \\ trainer.critic\_warmup=0 \\ trainer.logger=\['console'\] \\ trainer.project\_name='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node=1 \\ trainer.nnodes=1 \\ trainer.save\_freq=-1 \\ trainer.test\_freq=5 \\ trainer.total\_epochs=15 $@ \`\`\` ## Finally If you encounter any difficulties during use, please join the discussion at \[GitHub\](https://github.com/volcengine/verl/discussions). --- # Unknown \# Axolotl This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. ## Requirements - \*\*GPU:\*\* NVIDIA Ampere (or newer) for \`bf16\` and \`Flash Attention\`, or AMD GPU - \*\*Python:\*\* ≥3.11 - \*\*CUDA:\*\* ≥12.4 (for NVIDIA GPUs) ## Installation You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. :::{important} Install PyTorch \*before\* installing Axolotl to ensure CUDA compatibility. ::: For the latest instructions, see the official \[Axolotl Installation Guide\](https://docs.axolotl.ai/docs/installation.html). ## Quickstart ### SFT We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: \[SFT 32B QLoRA config\](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml). \`\`\`shell # Train the model axolotl train path/to/32b-qlora.yaml # Merge LoRA weights with the base model # This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml \`\`\` :::{tip} To train a smaller model, edit the \`base\_model\` in your config: \`\`\`yaml base\_model: Qwen/Qwen3-8B \`\`\` ::: Qwen3 works with all Axolotl features including \`Flash Attention\`, \`bf16\`, \`LoRA\`, \`torch\_compile\`, and \`QLoRA\`. To run on more than single GPU, please take a look at the \[Multi-GPU Training Guide\](https://docs.axolotl.ai/docs/multi-gpu.html) or \[Multi-node Training Guide\](https://docs.axolotl.ai/docs/multi-node.html). ### RLHF See the \[RLHF Guide\](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM Please refer to the \[Reward Modelling Guide\](https://docs.axolotl.ai/docs/reward\_modelling.html) for required dataset formats and config examples. ## Dataset By default, the example config uses the \`mlabonne/FineTome-100k\` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format Axolotl handles various SFT dataset formats, but the current \*\*recommended\*\* format (for use with \`chat\_template\`) is the OpenAI Messages format: \`\`\`json \[ { "messages": \[ { "role": "user", "content": "What is Qwen3?" }, { "role": "assistant", "content": "Qwen3 is a language model..." } \] } \] \`\`\` Use this in your config: \`\`\`yaml datasets: - path: path/to/your/dataset.json type: chat\_template \`\`\` You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the \[Dataset Loading Guide\](https://docs.axolotl.ai/docs/dataset\_loading.html) for more details. To load different dataset formats, refer to the \[SFT Dataset Formats Guide\](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft). ## Optimizations With Qwen3/Qwen3\_MOE, you can leverage Axolotl's custom optimizations for improved speed and reduced memory usage: - \[Cut Cross Entropy\](https://docs.axolotl.ai/docs/custom\_integrations.html#cut-cross-entropy) - \[Liger Kernels\](https://docs.axolotl.ai/docs/custom\_integrations.html#liger-kernels) - (LoRA/QLoRA only): \[LoRA Kernels Optimization\](https://docs.axolotl.ai/docs/lora\_optims.html) ## Additional Suggestions ### Troubleshooting - Ensure your CUDA version matches your GPU and PyTorch version. - If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. - Qwen3 MoE may have slower training due to the upstream transformer's handling of MoE layers. - For help, check the help channel on \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) or create a Discussion on \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl). ### Links - \[Axolotl Documentation\](https://docs.axolotl.ai/) - \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) - \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl) - \[Axolotl Website\](https://axolotl.ai) --- # Unknown \# Axolotl This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. ## Requirements - \*\*GPU:\*\* NVIDIA Ampere (or newer) for \`bf16\` and \`Flash Attention\`, or AMD GPU - \*\*Python:\*\* ≥3.11 - \*\*CUDA:\*\* ≥12.4 (for NVIDIA GPUs) ## Installation You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. :::{important} Install PyTorch \*before\* installing Axolotl to ensure CUDA compatibility. ::: For the latest instructions, see the official \[Axolotl Installation Guide\](https://docs.axolotl.ai/docs/installation.html). ## Quickstart ### SFT We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: \[SFT 32B QLoRA config\](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml). \`\`\`shell # Train the model axolotl train path/to/32b-qlora.yaml # Merge LoRA weights with the base model # This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml \`\`\` :::{tip} To train a smaller model, edit the \`base\_model\` in your config: \`\`\`yaml base\_model: Qwen/Qwen3-8B \`\`\` ::: Qwen3 works with all Axolotl features including \`Flash Attention\`, \`bf16\`, \`LoRA\`, \`torch\_compile\`, and \`QLoRA\`. To run on more than single GPU, please take a look at the \[Multi-GPU Training Guide\](https://docs.axolotl.ai/docs/multi-gpu.html) or \[Multi-node Training Guide\](https://docs.axolotl.ai/docs/multi-node.html). ### RLHF See the \[RLHF Guide\](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM Please refer to the \[Reward Modelling Guide\](https://docs.axolotl.ai/docs/reward\_modelling.html) for required dataset formats and config examples. ## Dataset By default, the example config uses the \`mlabonne/FineTome-100k\` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format Axolotl handles various SFT dataset formats, but the current \*\*recommended\*\* format (for use with \`chat\_template\`) is the OpenAI Messages format: \`\`\`json \[ { "messages": \[ { "role": "user", "content": "What is Qwen3?" }, { "role": "assistant", "content": "Qwen3 is a language model..." } \] } \] \`\`\` Use this in your config: \`\`\`yaml datasets: - path: path/to/your/dataset.json type: chat\_template \`\`\` You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the \[Dataset Loading Guide\](https://docs.axolotl.ai/docs/dataset\_loading.html) for more details. To load different dataset formats, refer to the \[SFT Dataset Formats Guide\](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft). ## Optimizations With Qwen3/Qwen3\_MOE, you can leverage Axolotl's custom optimizations for improved speed and reduced memory usage: - \[Cut Cross Entropy\](https://docs.axolotl.ai/docs/custom\_integrations.html#cut-cross-entropy) - \[Liger Kernels\](https://docs.axolotl.ai/docs/custom\_integrations.html#liger-kernels) - (LoRA/QLoRA only): \[LoRA Kernels Optimization\](https://docs.axolotl.ai/docs/lora\_optims.html) ## Additional Suggestions ### Troubleshooting - Ensure your CUDA version matches your GPU and PyTorch version. - If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. - Qwen3 MoE may have slower training due to the upstream transformer's handling of MoE layers. - For help, check the help channel on \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) or create a Discussion on \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl). ### Links - \[Axolotl Documentation\](https://docs.axolotl.ai/) - \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) - \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl) - \[Axolotl Website\](https://axolotl.ai) --- # Unknown SkyPilot ======== .. attention:: To be updated for Qwen3. What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, the highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the \`instruction \`\_\_ to install SkyPilot. Here we provide a simple example of using \`\`pip\`\` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly\[aws,gcp\]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the \`official document \`\_\_ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple Silicon Macs docker run --platform linux/amd64 \\ -td --rm --name sky \\ -v "$HOME/.sky:/root/.sky:rw" \\ -v "$HOME/.aws:/root/.aws:rw" \\ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \\ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen2.5-72B-Instruct with SkyPilot ------------------------------------------ 1. Start serving Qwen2.5-72B-Instruct on a single instance with any available GPU in the list specified in \`serve-72b.yaml \`\_\_ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "prompt": "My favorite food is", "max\_tokens": 512 }' | jq -r '.choices\[0\].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With \`SkyPilot Serve \`\_\_, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml \*\*Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE\_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be \*\*the cheapest available one\*\* on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a \`\`curl -L\`\` command (\`\`-L\`\` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": \[ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Accessing Qwen2.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen2.5 service with GUI by connecting a \`FastChat GUI server \`\_\_ to the endpoint launched above (see \`gui.yaml \`\_\_). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) \*\*Before launching, make sure you have changed Qwen/Qwen1.5-72B-Chat to Qwen/Qwen2.5-72B-Instruct in the YAML file.\*\* 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top\_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen2.5 on any cloud. We advise you to read the official doc for more usages and updates. Check \`this \`\_\_ out! --- # Unknown Qwen-Agent ========== \`Qwen-Agent \`\_\_ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to \`Qwen-Agent \`\_\_ repository. Installation ------------ - Install the stable version from PyPI: .. code:: bash pip install -U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" # Or use \`pip install -U qwen-agent\` for the minimal requirements. # The optional requirements, specified in double brackets, are: # \[gui\] for Gradio-based GUI support; # \[rag\] for RAG support; # \[code\_interpreter\] for Code Interpreter support; # \[mcp\] for MCP support. Developing Your Own Agent ------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. .. code:: python import os from qwen\_agent.agents import Assistant # Define LLM llm\_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', # api\_base 'api\_key': 'EMPTY', # 'generate\_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra\_body': { # 'chat\_template\_kwargs': {'enable\_thinking': False} # }, # # # Add: When the content is \`this is the thoughtthis is the answer\` # # Do not add: When the response has been separated by reasoning\_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought\_in\_content': True, # }, } # llm\_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_type': 'qwen\_dashscope', # # # 'generate\_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable\_thinking': False, # # }, # } # llm\_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # # # 'generate\_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra\_body': { # # 'enable\_thinking': False # # }, # # }, # } # Define Tools tools = \[ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\] }, "fetch": { "command": "uvx", "args": \["mcp-server-fetch"\] } } }, 'code\_interpreter', # Built-in tools \] # Define Agent bot = Assistant(llm=llm\_cfg, function\_list=tools) # Streaming generation messages = \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to \`Qwen-Agent \`\_\_ repository. --- # Unknown Performance of Quantized Models ================================== .. attention:: To be updated for Qwen2.5. This section reports the generation performance of quantized models (including GPTQ and AWQ) of the Qwen2 series. Specifically, we report: \* MMLU (Accuracy) \* C-Eval (Accuracy) \* IFEval (Strict Prompt-Level Accuracy) We use greedy decoding in evaluating all models. +---------------------+--------------+---------+-------+--------+--------+ | | Quantization | Average | MMLU | C-Eval | IFEval | +=====================+==============+=========+=======+========+========+ | Qwen2-72B-Instruct | BF16 | 81.3 | 82.3 | 83.8 | 77.6 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 80.7 | 81.3 | 83.4 | 77.5 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 81.2 | 80.8 | 83.9 | 78.9 | + +--------------+---------+-------+--------+--------+ | | AWQ | 80.4 | 80.5 | 83.9 | 76.9 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-7B-Instruct | BF16 | 66.9 | 70.5 | 77.2 | 53.1 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 66.2 | 69.1 | 76.7 | 52.9 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 64.1 | 67.8 | 75.2 | 49.4 | + +--------------+---------+-------+--------+--------+ | | AWQ | 64.1 | 67.4 | 73.6 | 51.4 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-1.5B-Instruct | BF16 | 48.4 | 52.4 | 63.8 | 29.0 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 48.1 | 53.0 | 62.5 | 28.8 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 45.0 | 50.7 | 57.4 | 27.0 | + +--------------+---------+-------+--------+--------+ | | AWQ | 46.5 | 51.6 | 58.1 | 29.9 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-0.5B-Instruct | BF16 | 34.4 | 37.9 | 45.2 | 20.0 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 32.6 | 35.6 | 43.9 | 18.1 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 29.7 | 33.0 | 39.2 | 16.8 | + +--------------+---------+-------+--------+--------+ | | AWQ | 31.1 | 34.4 | 42.1 | 16.7 | +---------------------+--------------+---------+-------+--------+--------+ --- # Unknown \# AWQ :::{attention} To be updated for Qwen3. ::: For quantized models, one of our recommendations is the usage of \[AWQ\](https://arxiv.org/abs/2306.00978) with \[AutoAWQ\](https://github.com/casper-hansen/AutoAWQ). \*\*AWQ\*\* refers to Activation-aware Weight Quantization, a hardware-friendly approach for LLM low-bit weight-only quantization. \*\*AutoAWQ\*\* is an easy-to-use Python library for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model. ## Usage of AWQ Models with Hugging Face transformers Now, \`transformers\` has officially supported AutoAWQ, which means that you can directly use the quantized model with \`transformers\`. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-AWQ\` with the quantized model: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-AWQ" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto", ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of AWQ Models with vLLM vLLM has supported AWQ, which means that you can directly use our provided AWQ models or those quantized with \`AutoAWQ\` with vLLM. We recommend using the latest version of vLLM (\`vllm>=0.6.1\`) which brings performance improvements to AWQ models; otherwise, the performance might not be well-optimized. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-AWQ\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-AWQ", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct-AWQ", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoAWQ If you want to quantize your own model to AWQ quantized models, we advise you to use AutoAWQ. \`\`\`bash pip install "autoawq<0.2.7" \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own AWQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quant\_config = { "zero\_point": True, "q\_group\_size": 128, "w\_bit": 4, "version": "GEMM" } # Load your tokenizer and model with AutoAWQ tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoAWQForCausalLM.from\_pretrained(model\_path, device\_map="auto", safetensors=True) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) data.append(text.strip()) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python model.quantize(tokenizer, quant\_config=quant\_config, calib\_data=data) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, safetensors=True, shard\_size="4GB") tokenizer.save\_pretrained(quant\_path) \`\`\` Then you can obtain your own AWQ quantized model for deployment. Enjoy! --- # Unknown \# GPTQ :::{attention} To be updated for Qwen3. ::: \[GPTQ\](https://arxiv.org/abs/2210.17323) is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model with \[AutoGPTQ\](https://github.com/AutoGPTQ/AutoGPTQ). ## Usage of GPTQ Models with Hugging Face transformers :::{note} To use the official Qwen2.5 GPTQ models with \`transformers\`, please ensure that \`optimum>=1.20.0\` and compatible versions of \`transformers\` and \`auto\_gptq\` are installed. You can do that by \`\`\`bash pip install -U "optimum>=1.20.0" \`\`\` ::: Now, \`transformers\` has officially supported AutoGPTQ, which means that you can directly use the quantized model with \`transformers\`. For each size of Qwen2.5, we provide both Int4 and Int8 GPTQ quantized models. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of GPTQ Models with vLLM vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with \`AutoGPTQ\` with vLLM. If possible, it will automatically use the GPTQ Marlin kernel, which is more efficient. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4 \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen2.5-7B-Instruct-GPTQ-Int4", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoGPTQ If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: \`\`\`bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from auto\_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quantize\_config = BaseQuantizeConfig( bits=8, # 4 or 8 group\_size=128, damp\_percent=0.01, desc\_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static\_groups=False, sym=True, true\_sequential=True, model\_name\_or\_path=None, model\_file\_base\_name="model" ) max\_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoGPTQForCausalLM.from\_pretrained(model\_path, quantize\_config) \`\`\` However, if you would like to load the model on multiple GPUs, you need to use \`max\_memory\` instead of \`device\_map\`. Here is an example: \`\`\`python model = AutoGPTQForCausalLM.from\_pretrained( model\_path, quantize\_config, max\_memory={i: "20GB" for i in range(4)} ) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python import torch data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) model\_inputs = tokenizer(\[text\]) input\_ids = torch.tensor(model\_inputs.input\_ids\[:max\_len\], dtype=torch.int) data.append(dict(input\_ids=input\_ids, attention\_mask=input\_ids.ne(tokenizer.pad\_token\_id))) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python import logging logging.basicConfig( format="%(asctime)s %(levelname)s \[%(name)s\] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache\_examples\_on\_gpu=False) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, use\_safetensors=True) tokenizer.save\_pretrained(quant\_path) \`\`\` It is unfortunate that the \`save\_quantized\` method does not support sharding. For sharding, you need to load the model and use \`save\_pretrained\` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! ## Known Issues ### Qwen2.5-72B-Instruct-GPTQ-Int4 cannot stop generation properly :Model: Qwen2.5-72B-Instruct-GPTQ-Int4 :Framework: vLLM, AutoGPTQ (including Hugging Face transformers) :Description: Generation cannot stop properly. Continual generation after where it should stop, then repeated texts, either single character, a phrase, or paragraphs, are generated. :Workaround: The following workaround could be considered 1. Using the original model in 16-bit floating point 2. Using the AWQ variants or llama.cpp-based models for reduced chances of abnormal generation ### Qwen2.5-32B-Instruct-GPTQ-Int4 broken with vLLM on multiple GPUs :Model: Qwen2.5-32B-Instruct-GPTQ-Int4 :Framework: vLLM :Description: Deployment on multiple GPUs and only garbled text like \`!!!!!!!!!!!!!!!!!!\` could be generated. :Workaround: Each of the following workaround could be considered 1. Using the AWQ or GPTQ-Int8 variants 2. Using a single GPU 3. Using Hugging Face \`transformers\` if latency and throughput are not major concerns ## Troubleshooting :::{dropdown} With \`transformers\` and \`auto\_gptq\`, the logs suggest \`CUDA extension not installed.\` and the inference is slow. \`auto\_gptq\` fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its \[installation guide\](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md) to install a pre-built wheel or try installing \`auto\_gptq\` from source. ::: :::{dropdown} Self-quantized Qwen2.5-72B-Instruct-GPTQ with \`vllm\`, \`ValueError: ... must be divisible by ...\` is raised. The intermediate size of the self-quantized model is different from the official Qwen2.5-72B-Instruct-GPTQ models. After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2.5-72B is 29568. Unfortunately, {math}\`29568 \\div 128 = 231\`. Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with \`tensor\_parallel\_size=1\`, i.e., one GPU card. A workaround is to make the intermediate size divisible by {math}\`128 \\times 8 = 1024\`. To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality. Try the following: \`\`\`python import torch from torch.nn import functional as F from transformers import AutoModelForCausalLM # must use AutoModelForCausalLM model = AutoModelForCausalLM.from\_pretrained("Qwen/Qwen2.5-72B-Instruct", torch\_dtype="auto") # this size is Qwen2.5-72B only pad\_size = 128 sd = model.state\_dict() for i, k in enumerate(sd): v = sd\[k\] print(k, i) # interleaving the padded zeros if ('mlp.up\_proj.weight' in k) or ('mlp.gate\_proj.weight' in k): prev\_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568\*2, -1)\[:pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[pad\_size:\]\], dim=0) sd\[k\] = new\_v elif 'mlp.down\_proj.weight' in k: prev\_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568\*2)\[:, :pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[:, pad\_size:\]\], dim=1) sd\[k\] = new\_v # this is a very large file; make sure your RAM is enough to load the model torch.save(sd, '/path/to/padded\_model/pytorch\_model.bin') \`\`\` This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the \`intermediate\_size\` in \`config.json\` to \`29696\`. Finally, you can quantize the saved model checkpoint. ::: --- # Unknown \# Axolotl This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3\_MOE using Axolotl, and covers optimizations to enable for better performance. ## Requirements - \*\*GPU:\*\* NVIDIA Ampere (or newer) for \`bf16\` and \`Flash Attention\`, or AMD GPU - \*\*Python:\*\* ≥3.11 - \*\*CUDA:\*\* ≥12.4 (for NVIDIA GPUs) ## Installation You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. :::{important} Install PyTorch \*before\* installing Axolotl to ensure CUDA compatibility. ::: For the latest instructions, see the official \[Axolotl Installation Guide\](https://docs.axolotl.ai/docs/installation.html). ## Quickstart ### SFT We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: \[SFT 32B QLoRA config\](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml). \`\`\`shell # Train the model axolotl train path/to/32b-qlora.yaml # Merge LoRA weights with the base model # This will create a new \`merged\` directory under \`{output\_dir}\` axolotl merge-lora path/to/32b-qlora.yaml \`\`\` :::{tip} To train a smaller model, edit the \`base\_model\` in your config: \`\`\`yaml base\_model: Qwen/Qwen3-8B \`\`\` ::: Qwen3 works with all Axolotl features including \`Flash Attention\`, \`bf16\`, \`LoRA\`, \`torch\_compile\`, and \`QLoRA\`. To run on more than single GPU, please take a look at the \[Multi-GPU Training Guide\](https://docs.axolotl.ai/docs/multi-gpu.html) or \[Multi-node Training Guide\](https://docs.axolotl.ai/docs/multi-node.html). ### RLHF See the \[RLHF Guide\](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM Please refer to the \[Reward Modelling Guide\](https://docs.axolotl.ai/docs/reward\_modelling.html) for required dataset formats and config examples. ## Dataset By default, the example config uses the \`mlabonne/FineTome-100k\` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format Axolotl handles various SFT dataset formats, but the current \*\*recommended\*\* format (for use with \`chat\_template\`) is the OpenAI Messages format: \`\`\`json \[ { "messages": \[ { "role": "user", "content": "What is Qwen3?" }, { "role": "assistant", "content": "Qwen3 is a language model..." } \] } \] \`\`\` Use this in your config: \`\`\`yaml datasets: - path: path/to/your/dataset.json type: chat\_template \`\`\` You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the \[Dataset Loading Guide\](https://docs.axolotl.ai/docs/dataset\_loading.html) for more details. To load different dataset formats, refer to the \[SFT Dataset Formats Guide\](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft). ## Optimizations With Qwen3/Qwen3\_MOE, you can leverage Axolotl's custom optimizations for improved speed and reduced memory usage: - \[Cut Cross Entropy\](https://docs.axolotl.ai/docs/custom\_integrations.html#cut-cross-entropy) - \[Liger Kernels\](https://docs.axolotl.ai/docs/custom\_integrations.html#liger-kernels) - (LoRA/QLoRA only): \[LoRA Kernels Optimization\](https://docs.axolotl.ai/docs/lora\_optims.html) ## Additional Suggestions ### Troubleshooting - Ensure your CUDA version matches your GPU and PyTorch version. - If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. - Qwen3 MoE may have slower training due to the upstream transformer's handling of MoE layers. - For help, check the help channel on \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) or create a Discussion on \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl). ### Links - \[Axolotl Documentation\](https://docs.axolotl.ai/) - \[Axolotl Discord\](https://discord.gg/7m9sfhzaf3) - \[Axolotl GitHub\](https://github.com/axolotl-ai-cloud/axolotl) - \[Axolotl Website\](https://axolotl.ai) --- # Unknown \# llama.cpp Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program \`llama-quantize\` supports many quantization presets, such as Q4\_K\_M and Q8\_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using \`llama-imatrix\`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don't, check our guide \[here\](../run\_locally/llama.cpp.html#getting-the-program){.external}. ## Getting the GGUF Now, suppose you would like to quantize \`Qwen3-8B\`. You need to first make a GGUF file as shown below: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile Qwen3-8B-F16.gguf \`\`\` Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype bf16 --outfile Qwen3-8B-BF16.gguf \`\`\` Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use \`\`\`bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype f32 --outfile Qwen3-8B-F32.gguf \`\`\` ## Quantizing the GGUF without Calibration For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: \`\`\`bash ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8\_0.gguf Q8\_0 \`\`\` \`Q8\_0\` is a code for a quantization preset. You can find all the presets in \[the source code of \`llama-quantize\`\](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp). Look for the variable \`QUANT\_OPTIONS\`. Common ones used for 8B models include \`Q8\_0\`, \`Q5\_K\_M\`, and \`Q4\_K\_M\`. The letter case doesn't matter, so \`q8\_0\` or \`q4\_K\_m\` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. ## Quantizing the GGUF with AWQ Scale :::{attention} To be updated for Qwen3. ::: To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following \[this script\](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export). First, when you run \`model.quantize()\` with \`autoawq\`, remember to add \`export\_compatible=True\` as shown below: \`\`\`python ... model.quantize( tokenizer, quant\_config=quant\_config, export\_compatible=True ) model.save\_pretrained(quant\_path) ... \`\`\` The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are "easier" to quantize.\[^AWQ\] Then, when you run \`convert-hf-to-gguf.py\`, remember to replace the model path with the path to the new model: \`\`\`bash python convert-hf-to-gguf.py \--outfile qwen2.5-7b-instruct-f16-awq.gguf \`\`\` Finally, you can quantize the model as in the last example: \`\`\`bash ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8\_0.gguf Q8\_0 \`\`\` In this way, it should be possible to achieve similar quality with lower bit-per-weight. \[^AWQ\]: If you are interested in what this means, refer to \[the AWQ paper\](https://arxiv.org/abs/2306.00978). Basically, important weights (called salient weights in the paper) are identified based on activations across data examples. The weights are scaled accordingly such that the salient weights are protected even after quantization. ## Quantizing the GGUF with Importance Matrix Another possible solution is to use the "important matrix"\[^imatrix\], following \[this\](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix). First, you need to compute the importance matrix data of the weights of a model (\`-m\`) using a calibration dataset (\`-f\`): \`\`\`bash ./llama-imatrix -m Qwen3-8B-F16.gguf -f calibration-text.txt --chunk 512 -o Qwen3-8B-imatrix.dat -ngl 80 \`\`\` The text is cut in chunks of length \`--chunk\` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named \`Qwen3-8B-imatrix.dat\` (\`-o\`), which can then be used: \`\`\`bash ./llama-quantize --imatrix Qwen3-8B-imatrix.dat \\ Qwen3-8B-F16.gguf Qwen3-8B-Q4\_K\_M.gguf Q4\_K\_M \`\`\` For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide \`--imatrix\`, a helpful warning will be printed by \`llama-quantize\`. \[^imatrix\]: Here, the importance matrix keeps record of how weights affect the output: the weight should be important is a slight change in its value causes huge difference in the results, akin to the \[GPTQ\](https://arxiv.org/abs/2210.17323) algorithm. ## Perplexity Evaluation \`llama.cpp\` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say "wiki test"\[^wiki\]. You can download the dataset with: \`\`\`bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip \`\`\` Then you can run the test with the following command: \`\`\`bash ./llama-perplexity -m Qwen3-8B-Q8\_0.gguf -f wiki.test.raw -ngl 80 \`\`\` Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture \[here\](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md). It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. \[^wiki\]: It is not a good evaluation dataset for instruct models though, but it is very common and easily accessible. You probably want to use a dataset similar to your target domain. ## Finally In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the \[llama.cpp GitHub repo\](https://github.com/ggml-org/llama.cpp). We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don't provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven't found what you need in our repos, look around. Enjoy your freshly quantized models! --- # Unknown Qwen-Agent ========== \`Qwen-Agent \`\_\_ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to \`Qwen-Agent \`\_\_ repository. Installation ------------ - Install the stable version from PyPI: .. code:: bash pip install -U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" # Or use \`pip install -U qwen-agent\` for the minimal requirements. # The optional requirements, specified in double brackets, are: # \[gui\] for Gradio-based GUI support; # \[rag\] for RAG support; # \[code\_interpreter\] for Code Interpreter support; # \[mcp\] for MCP support. Developing Your Own Agent ------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. .. code:: python import os from qwen\_agent.agents import Assistant # Define LLM llm\_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', # api\_base 'api\_key': 'EMPTY', # 'generate\_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra\_body': { # 'chat\_template\_kwargs': {'enable\_thinking': False} # }, # # # Add: When the content is \`this is the thoughtthis is the answer\` # # Do not add: When the response has been separated by reasoning\_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought\_in\_content': True, # }, } # llm\_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_type': 'qwen\_dashscope', # # # 'generate\_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable\_thinking': False, # # }, # } # llm\_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # # # 'generate\_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra\_body': { # # 'enable\_thinking': False # # }, # # }, # } # Define Tools tools = \[ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\] }, "fetch": { "command": "uvx", "args": \["mcp-server-fetch"\] } } }, 'code\_interpreter', # Built-in tools \] # Define Agent bot = Assistant(llm=llm\_cfg, function\_list=tools) # Streaming generation messages = \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to \`Qwen-Agent \`\_\_ repository. --- # Unknown \# LLaMA-Factory :::{attention} To be updated for Qwen3. ::: Here we provide a script for supervised finetuning Qwen2.5 with \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory). This script for supervised finetuning (SFT) has the following features: - Support single-GPU and multi-GPU training; - Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. ## Installation Before you start, make sure you have installed the following packages: 1. Follow the instructions of \[LLaMA-Factory\](https://github.com/hiyouga/LLaMA-Factory), and build the environment. 2. Install these packages (Optional): \`\`\` pip install deepspeed pip install flash-attn --no-build-isolation \`\`\` 3. If you want to use \[FlashAttention-2\](https://github.com/Dao-AILab/flash-attention), make sure your CUDA is 11.6 and above. ## Data Preparation LLaMA-Factory provides several training datasets in \`data\` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a \*\*json\*\* file and put your data in \`data\` folder. LLaMA-Factory supports dataset in \`alpaca\` or \`sharegpt\` format. - The dataset in \`alpaca\` format should follow the below format: \`\`\`json \[ { "instruction": "user instruction (required)", "input": "user input (optional)", "output": "model response (required)", "system": "system prompt (optional)", "history": \[ \["user instruction in the first round (optional)", "model response in the first round (optional)"\], \["user instruction in the second round (optional)", "model response in the second round (optional)"\] \] } \] \`\`\` - The dataset in \`sharegpt\` format should follow the below format: \`\`\`json \[ { "conversations": \[ { "from": "human", "value": "user instruction" }, { "from": "gpt", "value": "model response" } \], "system": "system prompt (optional)", "tools": "tool description (optional)" } \] \`\`\` 2. Provide your dataset definition in \`data/dataset\_info.json\` in the following format . - For \`alpaca\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } \`\`\` - For \`sharegpt\` format dataset, the columns in \`dataset\_info.json\` should be: \`\`\`json "dataset\_name": { "file\_name": "dataset\_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role\_tag": "from", "content\_tag": "value", "user\_tag": "user", "assistant\_tag": "assistant" } } \`\`\` ## Training Execute the following training command: \`\`\`bash DISTRIBUTED\_ARGS=" --nproc\_per\_node $NPROC\_PER\_NODE \\ --nnodes $NNODES \\ --node\_rank $NODE\_RANK \\ --master\_addr $MASTER\_ADDR \\ --master\_port $MASTER\_PORT " torchrun $DISTRIBUTED\_ARGS src/train.py \\ --deepspeed $DS\_CONFIG\_PATH \\ --stage sft \\ --do\_train \\ --use\_fast\_tokenizer \\ --flash\_attn \\ --model\_name\_or\_path $MODEL\_PATH \\ --dataset your\_dataset \\ --template qwen \\ --finetuning\_type lora \\ --lora\_target q\_proj,v\_proj\\ --output\_dir $OUTPUT\_PATH \\ --overwrite\_cache \\ --overwrite\_output\_dir \\ --warmup\_steps 100 \\ --weight\_decay 0.1 \\ --per\_device\_train\_batch\_size 4 \\ --gradient\_accumulation\_steps 4 \\ --ddp\_timeout 9000 \\ --learning\_rate 5e-6 \\ --lr\_scheduler\_type cosine \\ --logging\_steps 1 \\ --cutoff\_len 4096 \\ --save\_steps 1000 \\ --plot\_loss \\ --num\_train\_epochs 3 \\ --bf16 \`\`\` and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is \`cutoff\_len\`, which is the maximum length of the training data. Control this parameter to avoid OOM error. ## Merge LoRA If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 llamafactory-cli export \\ --model\_name\_or\_path path\_to\_base\_model \\ --adapter\_name\_or\_path path\_to\_adapter \\ --template qwen \\ --finetuning\_type lora \\ --export\_dir path\_to\_export \\ --export\_size 2 \\ --export\_legacy\_format False \`\`\` ## Conclusion The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- # Unknown \# GPTQ :::{attention} To be updated for Qwen3. ::: \[GPTQ\](https://arxiv.org/abs/2210.17323) is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model with \[AutoGPTQ\](https://github.com/AutoGPTQ/AutoGPTQ). ## Usage of GPTQ Models with Hugging Face transformers :::{note} To use the official Qwen2.5 GPTQ models with \`transformers\`, please ensure that \`optimum>=1.20.0\` and compatible versions of \`transformers\` and \`auto\_gptq\` are installed. You can do that by \`\`\`bash pip install -U "optimum>=1.20.0" \`\`\` ::: Now, \`transformers\` has officially supported AutoGPTQ, which means that you can directly use the quantized model with \`transformers\`. For each size of Qwen2.5, we provide both Int4 and Int8 GPTQ quantized models. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of GPTQ Models with vLLM vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with \`AutoGPTQ\` with vLLM. If possible, it will automatically use the GPTQ Marlin kernel, which is more efficient. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4 \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen2.5-7B-Instruct-GPTQ-Int4", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoGPTQ If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: \`\`\`bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from auto\_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quantize\_config = BaseQuantizeConfig( bits=8, # 4 or 8 group\_size=128, damp\_percent=0.01, desc\_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static\_groups=False, sym=True, true\_sequential=True, model\_name\_or\_path=None, model\_file\_base\_name="model" ) max\_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoGPTQForCausalLM.from\_pretrained(model\_path, quantize\_config) \`\`\` However, if you would like to load the model on multiple GPUs, you need to use \`max\_memory\` instead of \`device\_map\`. Here is an example: \`\`\`python model = AutoGPTQForCausalLM.from\_pretrained( model\_path, quantize\_config, max\_memory={i: "20GB" for i in range(4)} ) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python import torch data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) model\_inputs = tokenizer(\[text\]) input\_ids = torch.tensor(model\_inputs.input\_ids\[:max\_len\], dtype=torch.int) data.append(dict(input\_ids=input\_ids, attention\_mask=input\_ids.ne(tokenizer.pad\_token\_id))) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python import logging logging.basicConfig( format="%(asctime)s %(levelname)s \[%(name)s\] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache\_examples\_on\_gpu=False) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, use\_safetensors=True) tokenizer.save\_pretrained(quant\_path) \`\`\` It is unfortunate that the \`save\_quantized\` method does not support sharding. For sharding, you need to load the model and use \`save\_pretrained\` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! ## Known Issues ### Qwen2.5-72B-Instruct-GPTQ-Int4 cannot stop generation properly :Model: Qwen2.5-72B-Instruct-GPTQ-Int4 :Framework: vLLM, AutoGPTQ (including Hugging Face transformers) :Description: Generation cannot stop properly. Continual generation after where it should stop, then repeated texts, either single character, a phrase, or paragraphs, are generated. :Workaround: The following workaround could be considered 1. Using the original model in 16-bit floating point 2. Using the AWQ variants or llama.cpp-based models for reduced chances of abnormal generation ### Qwen2.5-32B-Instruct-GPTQ-Int4 broken with vLLM on multiple GPUs :Model: Qwen2.5-32B-Instruct-GPTQ-Int4 :Framework: vLLM :Description: Deployment on multiple GPUs and only garbled text like \`!!!!!!!!!!!!!!!!!!\` could be generated. :Workaround: Each of the following workaround could be considered 1. Using the AWQ or GPTQ-Int8 variants 2. Using a single GPU 3. Using Hugging Face \`transformers\` if latency and throughput are not major concerns ## Troubleshooting :::{dropdown} With \`transformers\` and \`auto\_gptq\`, the logs suggest \`CUDA extension not installed.\` and the inference is slow. \`auto\_gptq\` fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its \[installation guide\](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md) to install a pre-built wheel or try installing \`auto\_gptq\` from source. ::: :::{dropdown} Self-quantized Qwen2.5-72B-Instruct-GPTQ with \`vllm\`, \`ValueError: ... must be divisible by ...\` is raised. The intermediate size of the self-quantized model is different from the official Qwen2.5-72B-Instruct-GPTQ models. After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2.5-72B is 29568. Unfortunately, {math}\`29568 \\div 128 = 231\`. Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with \`tensor\_parallel\_size=1\`, i.e., one GPU card. A workaround is to make the intermediate size divisible by {math}\`128 \\times 8 = 1024\`. To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality. Try the following: \`\`\`python import torch from torch.nn import functional as F from transformers import AutoModelForCausalLM # must use AutoModelForCausalLM model = AutoModelForCausalLM.from\_pretrained("Qwen/Qwen2.5-72B-Instruct", torch\_dtype="auto") # this size is Qwen2.5-72B only pad\_size = 128 sd = model.state\_dict() for i, k in enumerate(sd): v = sd\[k\] print(k, i) # interleaving the padded zeros if ('mlp.up\_proj.weight' in k) or ('mlp.gate\_proj.weight' in k): prev\_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568\*2, -1)\[:pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[pad\_size:\]\], dim=0) sd\[k\] = new\_v elif 'mlp.down\_proj.weight' in k: prev\_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568\*2)\[:, :pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[:, pad\_size:\]\], dim=1) sd\[k\] = new\_v # this is a very large file; make sure your RAM is enough to load the model torch.save(sd, '/path/to/padded\_model/pytorch\_model.bin') \`\`\` This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the \`intermediate\_size\` in \`config.json\` to \`29696\`. Finally, you can quantize the saved model checkpoint. ::: --- # Unknown LlamaIndex ========== To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on \`LlamaIndex \`\_\_. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: .. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the \`\`bge-base-en-v1.5\`\` model to retrieve from English documents, and you can download the \`\`bge-base-zh-v1.5\`\` model to retrieve from Chinese documents. You can also choose \`\`bge-large\`\` or \`\`bge-small\`\` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) .. code:: python import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt = "" for message in messages: if message.role == "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role == "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role == "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt = "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt = prompt + "<|im\_start|>assistant\\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings.llm = HuggingFaceLLM( model\_name="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name="Qwen/Qwen2.5-7B-Instruct", context\_window=30000, max\_new\_tokens=2000, generate\_kwargs={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt=messages\_to\_prompt, completion\_to\_prompt=completion\_to\_prompt, device\_map="auto", ) # Set embedding model Settings.embed\_model = HuggingFaceEmbedding( model\_name = "BAAI/bge-base-en-v1.5" ) # Set the size of the text chunk for retrieval Settings.transformations = \[SentenceSplitter(chunk\_size=1024)\] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'. .. code:: python from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load\_data() index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. .. code:: python from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html\_to\_text=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) To save and load the index, you can use the following code snippet. .. code:: python from llama\_index.core import StorageContext, load\_index\_from\_storage # save index storage\_context = StorageContext.from\_defaults(persist\_dir="save") # load index index = load\_index\_from\_storage(storage\_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. .. code:: python query\_engine = index.as\_query\_engine() your\_query = "" print(query\_engine.query(your\_query).response) --- # Unknown \# Unsloth This guide will teach you how to easily train Qwen3 models with Unsloth. Unsloth simplifies local model training, handling everything from loading and quantization to training, evaluation, running, and deployment with inference engines (Ollama, llama.cpp, vLLM). \*\*Train Qwen\*\* models 2× faster using 70% less VRAM. \*\*GitHub repo:\*\* \[Unsloth\](https://github.com/unslothai/unsloth) ## ⭐ Key Features - Supports full fine-tuning, pretraining, LoRA, QLoRA, 8-bit training & more - Single and multi-GPU support (Linux, Windows, Colab, Kaggle; NVIDIA GPUs, soon AMD & Intel) - Compatible with all transformer models: TTS, multimodal, STT, BERT, RL - RLHF support: GRPO, DPO, DAPO, RM, PPO, KTO, etc. - Hand-written Triton kernels and a manual backprop engine ensure no accuracy degradation (0% approximation). ## Quickstart \*\*Local Installation (Linux recommended):\*\* \`\`\`bash pip install unsloth \`\`\` You can view Unsloth’s full \[installation instructions here.\](https://docs.unsloth.ai/get-started/installing-+-updating) ## Fine-tuning Qwen3 with Unsloth Unsloth makes Qwen3 fine-tuning 2× faster, uses 70% less VRAM, with 8× longer contexts. Qwen3 (14B) fits in a free 16 GB Colab Tesla T4 GPU. To retain Qwen3's reasoning capabilities, use a 75% reasoning to 25% non-reasoning dataset ratio (e.g., NVIDIA’s math‑reasoning dataset + Maxime’s FineTome). For more details, see Unsloth’s full \[Qwen3 fine-tuning guide\](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth). ### Colab Notebooks - \[Qwen3 (14B) Reasoning + Conversational\](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(14B)-Reasoning-Conversational.ipynb) - \[Qwen3 (4B) Advanced GRPO LoRA\](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(4B)-GRPO.ipynb) - \[Qwen3 (14B) Alpaca (Base model)\](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(14B)-Alpaca.ipynb) \*\*Update Unsloth locally:\*\* \`\`\`bash pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth\_zoo \`\`\` ### Fine-tuning Qwen3 MoE Models Supported MoE models include 30B‑A3B and 235B‑A22B. Unsloth fine-tunes the 30B‑A3B model with just 17.5 GB VRAM. Router-layer fine-tuning is disabled by default. Use \`FastModel\` for MoE fine-tuning: \`\`\`python from unsloth import FastModel model, tokenizer = FastModel.from\_pretrained( model\_name="unsloth/Qwen3-30B-A3B", max\_seq\_length=2048, load\_in\_4bit=True, load\_in\_8bit=False, full\_finetuning=False, ) \`\`\` ### Notebook Guide For an end-to-end walkthrough, see Unsloth’s \[full end-to-end fine-tuning guide\](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide). - Open the notebook → click \*\*Runtime ▸ Run all\*\*. - Adjust settings (e.g., model name, context length) directly in the notebook: - \`max\_seq\_length\`: Recommended 2048 (Qwen3 supports up to 40960). - \`load\_in\_4bit=True\`: reduces memory usage by 4×. - Enable full fine-tuning (\`full\_finetuning=True\`) or 8-bit training (\`load\_in\_8bit=True\`). If you want to use models directly from \[ModelScope\](https://modelscope.cn/organization/unsloth), use: \`\`\`bash pip install modelscope -qqq \`\`\` \`\`\`python import os os.environ\["UNSLOTH\_USE\_MODELSCOPE"\] = "1" from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from\_pretrained( model\_name="unsloth/Qwen3-4B-Base", max\_seq\_length=2048, ) \`\`\` ## RL & GRPO with Qwen3 You can also train Qwen models with reinforcement learning (RL) using Unsloth. Explore Unsloth’s advanced GRPO notebook, featuring proximity-based reward scoring and Hugging Face's Open‑R1 math dataset: \[Qwen3 (4B) Advanced GRPO LoRA notebook\](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3\_(4B)-GRPO.ipynb). - Proximity-based rewards for closer answers - Custom GRPO formatting and templates - Enhanced evaluation accuracy with regex matching ## Resources & Links That’s how you can easily train Qwen models with Unsloth. If you need any help, join the discussion on Unsloth’s \[Discord\](https://discord.com/invite/unsloth) or \[GitHub\](https://github.com/unslothai/unsloth) pages. \*\*Links:\*\* - \[Unsloth Documentation\](https://docs.unsloth.ai/) - \[Unsloth Discord\](https://discord.com/invite/unsloth) - \[Unsloth Website\](https://unsloth.ai/) - \[Unsloth Reddit\](https://www.reddit.com/r/unsloth/) --- # Unknown \# verl verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of \[HybridFlow: A Flexible and Efficient RLHF Framework\](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: \[verl\](https://github.com/volcengine/verl) verl is flexible and easy to use with: - \*\*Easy extension of diverse RL algorithms\*\*: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. - \*\*Seamless integration of existing LLM infra with modular APIs\*\*: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc - \*\*Flexible device mapping\*\*: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. - \*\*Ready integration with popular HuggingFace models\*\*: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: - \*\*State-of-the-art throughput\*\*: SOTA LLM training and inference engine integrations and SOTA RL throughput. - \*\*Efficient actor model resharding with 3D-HybridEngine\*\*: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. ## Reinforcement Learning (RL) Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation You can follow verl's \[installation guide\](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: \`\`\`shell git clone https://github.com/volcengine/verl.git cd verl python3 examples/data\_preprocess/gsm8k.py --local\_dir ~/data/gsm8k \`\`\` Model download can be done using the following command: \`\`\`shell python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" \`\`\` ### Step2: Start Training In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. \`\`\`bash python3 -m verl.trainer.main\_ppo \\ algorithm.adv\_estimator=grpo \\ data.train\_files=$HOME/data/gsm8k/train.parquet \\ data.val\_files=$HOME/data/gsm8k/test.parquet \\ data.train\_batch\_size=1024 \\ data.max\_prompt\_length=512 \\ data.max\_response\_length=1024 \\ data.filter\_overlong\_prompts=True \\ data.truncation='error' \\ actor\_rollout\_ref.model.path=Qwen/Qwen3-1.7B \\ actor\_rollout\_ref.actor.optim.lr=1e-6 \\ actor\_rollout\_ref.model.use\_remove\_padding=True \\ actor\_rollout\_ref.actor.ppo\_mini\_batch\_size=80 \\ actor\_rollout\_ref.actor.ppo\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.actor.use\_kl\_loss=True \\ actor\_rollout\_ref.actor.kl\_loss\_coef=0.001 \\ actor\_rollout\_ref.actor.kl\_loss\_type=low\_var\_kl \\ actor\_rollout\_ref.actor.entropy\_coeff=0 \\ actor\_rollout\_ref.model.enable\_gradient\_checkpointing=True \\ actor\_rollout\_ref.actor.fsdp\_config.param\_offload=False \\ actor\_rollout\_ref.actor.fsdp\_config.optimizer\_offload=False \\ actor\_rollout\_ref.rollout.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.rollout.tensor\_model\_parallel\_size=1 \\ actor\_rollout\_ref.rollout.name=vllm \\ actor\_rollout\_ref.rollout.gpu\_memory\_utilization=0.6 \\ actor\_rollout\_ref.rollout.n=3 \\ actor\_rollout\_ref.ref.log\_prob\_micro\_batch\_size\_per\_gpu=20 \\ actor\_rollout\_ref.ref.fsdp\_config.param\_offload=True \\ algorithm.use\_kl\_in\_reward=False \\ trainer.critic\_warmup=0 \\ trainer.logger=\['console'\] \\ trainer.project\_name='verl\_grpo\_example\_gsm8k' \\ trainer.experiment\_name='qwen3\_1\_7b\_function\_rm' \\ trainer.n\_gpus\_per\_node=1 \\ trainer.nnodes=1 \\ trainer.save\_freq=-1 \\ trainer.test\_freq=5 \\ trainer.total\_epochs=15 $@ \`\`\` ## Finally If you encounter any difficulties during use, please join the discussion at \[GitHub\](https://github.com/volcengine/verl/discussions). --- # Unknown Qwen-Agent ========== \`Qwen-Agent \`\_\_ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to \`Qwen-Agent \`\_\_ repository. Installation ------------ - Install the stable version from PyPI: .. code:: bash pip install -U "qwen-agent\[gui,rag,code\_interpreter,mcp\]" # Or use \`pip install -U qwen-agent\` for the minimal requirements. # The optional requirements, specified in double brackets, are: # \[gui\] for Gradio-based GUI support; # \[rag\] for RAG support; # \[code\_interpreter\] for Code Interpreter support; # \[mcp\] for MCP support. Developing Your Own Agent ------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. .. code:: python import os from qwen\_agent.agents import Assistant # Define LLM llm\_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model\_server': 'http://localhost:8000/v1', # api\_base 'api\_key': 'EMPTY', # 'generate\_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra\_body': { # 'chat\_template\_kwargs': {'enable\_thinking': False} # }, # # # Add: When the content is \`this is the thoughtthis is the answer\` # # Do not add: When the response has been separated by reasoning\_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought\_in\_content': True, # }, } # llm\_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_type': 'qwen\_dashscope', # # # 'generate\_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable\_thinking': False, # # }, # } # llm\_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model\_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api\_key': os.getenv('DASHSCOPE\_API\_KEY'), # # # 'generate\_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra\_body': { # # 'enable\_thinking': False # # }, # # }, # } # Define Tools tools = \[ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': \['mcp-server-time', '--local-timezone=Asia/Shanghai'\] }, "fetch": { "command": "uvx", "args": \["mcp-server-fetch"\] } } }, 'code\_interpreter', # Built-in tools \] # Define Agent bot = Assistant(llm=llm\_cfg, function\_list=tools) # Streaming generation messages = \[{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}\] for responses in bot.run(messages=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to \`Qwen-Agent \`\_\_ repository. --- # Unknown \# GPTQ :::{attention} To be updated for Qwen3. ::: \[GPTQ\](https://arxiv.org/abs/2210.17323) is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with Hugging Face \`transformers\` and also how to quantize your own model with \[AutoGPTQ\](https://github.com/AutoGPTQ/AutoGPTQ). ## Usage of GPTQ Models with Hugging Face transformers :::{note} To use the official Qwen2.5 GPTQ models with \`transformers\`, please ensure that \`optimum>=1.20.0\` and compatible versions of \`transformers\` and \`auto\_gptq\` are installed. You can do that by \`\`\`bash pip install -U "optimum>=1.20.0" \`\`\` ::: Now, \`transformers\` has officially supported AutoGPTQ, which means that you can directly use the quantized model with \`transformers\`. For each size of Qwen2.5, we provide both Int4 and Int8 GPTQ quantized models. The following is a very simple code snippet showing how to run \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4" model = AutoModelForCausalLM.from\_pretrained( model\_name, device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) prompt = "Give me a short introduction to large language models." messages = \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` ## Usage of GPTQ Models with vLLM vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with \`AutoGPTQ\` with vLLM. If possible, it will automatically use the GPTQ Marlin kernel, which is more efficient. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`Qwen2.5-7B-Instruct-GPTQ-Int4\`: Run the following in a shell to start an OpenAI-compatible API service: \`\`\`bash vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4 \`\`\` Then, you can call the API as \`\`\`bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], "temperature": 0.7, "top\_p": 0.8, "repetition\_penalty": 1.05, "max\_tokens": 512 }' \`\`\` or you can use the API client with the \`openai\` Python package as shown below: \`\`\`python from openai import OpenAI openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen2.5-7B-Instruct-GPTQ-Int4", messages=\[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \], temperature=0.7, top\_p=0.8, max\_tokens=512, extra\_body={ "repetition\_penalty": 1.05, }, ) print("Chat response:", chat\_response) \`\`\` ## Quantize Your Own Model with AutoGPTQ If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: \`\`\`bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . \`\`\` Suppose you have finetuned a model based on \`Qwen2.5-7B\`, which is named \`Qwen2.5-7B-finetuned\`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: \`\`\`python from auto\_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quantize\_config = BaseQuantizeConfig( bits=8, # 4 or 8 group\_size=128, damp\_percent=0.01, desc\_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static\_groups=False, sym=True, true\_sequential=True, model\_name\_or\_path=None, model\_file\_base\_name="model" ) max\_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoGPTQForCausalLM.from\_pretrained(model\_path, quantize\_config) \`\`\` However, if you would like to load the model on multiple GPUs, you need to use \`max\_memory\` instead of \`device\_map\`. Here is an example: \`\`\`python model = AutoGPTQForCausalLM.from\_pretrained( model\_path, quantize\_config, max\_memory={i: "20GB" for i in range(4)} ) \`\`\` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, \`\`\`python import torch data = \[\] for msg in dataset: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) model\_inputs = tokenizer(\[text\]) input\_ids = torch.tensor(model\_inputs.input\_ids\[:max\_len\], dtype=torch.int) data.append(dict(input\_ids=input\_ids, attention\_mask=input\_ids.ne(tokenizer.pad\_token\_id))) \`\`\` where each \`msg\` is a typical chat message as shown below: \`\`\`json \[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] \`\`\` Then just run the calibration process by one line of code: \`\`\`python import logging logging.basicConfig( format="%(asctime)s %(levelname)s \[%(name)s\] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache\_examples\_on\_gpu=False) \`\`\` Finally, save the quantized model: \`\`\`python model.save\_quantized(quant\_path, use\_safetensors=True) tokenizer.save\_pretrained(quant\_path) \`\`\` It is unfortunate that the \`save\_quantized\` method does not support sharding. For sharding, you need to load the model and use \`save\_pretrained\` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! ## Known Issues ### Qwen2.5-72B-Instruct-GPTQ-Int4 cannot stop generation properly :Model: Qwen2.5-72B-Instruct-GPTQ-Int4 :Framework: vLLM, AutoGPTQ (including Hugging Face transformers) :Description: Generation cannot stop properly. Continual generation after where it should stop, then repeated texts, either single character, a phrase, or paragraphs, are generated. :Workaround: The following workaround could be considered 1. Using the original model in 16-bit floating point 2. Using the AWQ variants or llama.cpp-based models for reduced chances of abnormal generation ### Qwen2.5-32B-Instruct-GPTQ-Int4 broken with vLLM on multiple GPUs :Model: Qwen2.5-32B-Instruct-GPTQ-Int4 :Framework: vLLM :Description: Deployment on multiple GPUs and only garbled text like \`!!!!!!!!!!!!!!!!!!\` could be generated. :Workaround: Each of the following workaround could be considered 1. Using the AWQ or GPTQ-Int8 variants 2. Using a single GPU 3. Using Hugging Face \`transformers\` if latency and throughput are not major concerns ## Troubleshooting :::{dropdown} With \`transformers\` and \`auto\_gptq\`, the logs suggest \`CUDA extension not installed.\` and the inference is slow. \`auto\_gptq\` fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its \[installation guide\](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md) to install a pre-built wheel or try installing \`auto\_gptq\` from source. ::: :::{dropdown} Self-quantized Qwen2.5-72B-Instruct-GPTQ with \`vllm\`, \`ValueError: ... must be divisible by ...\` is raised. The intermediate size of the self-quantized model is different from the official Qwen2.5-72B-Instruct-GPTQ models. After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2.5-72B is 29568. Unfortunately, {math}\`29568 \\div 128 = 231\`. Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with \`tensor\_parallel\_size=1\`, i.e., one GPU card. A workaround is to make the intermediate size divisible by {math}\`128 \\times 8 = 1024\`. To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality. Try the following: \`\`\`python import torch from torch.nn import functional as F from transformers import AutoModelForCausalLM # must use AutoModelForCausalLM model = AutoModelForCausalLM.from\_pretrained("Qwen/Qwen2.5-72B-Instruct", torch\_dtype="auto") # this size is Qwen2.5-72B only pad\_size = 128 sd = model.state\_dict() for i, k in enumerate(sd): v = sd\[k\] print(k, i) # interleaving the padded zeros if ('mlp.up\_proj.weight' in k) or ('mlp.gate\_proj.weight' in k): prev\_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568\*2, -1)\[:pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[pad\_size:\]\], dim=0) sd\[k\] = new\_v elif 'mlp.down\_proj.weight' in k: prev\_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568\*2)\[:, :pad\_size\*2\] new\_v = torch.cat(\[prev\_v, v\[:, pad\_size:\]\], dim=1) sd\[k\] = new\_v # this is a very large file; make sure your RAM is enough to load the model torch.save(sd, '/path/to/padded\_model/pytorch\_model.bin') \`\`\` This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the \`intermediate\_size\` in \`config.json\` to \`29696\`. Finally, you can quantize the saved model checkpoint. ::: --- # Unknown \# MS-SWIFT ModelScope SWIFT (\*\*ms-swift\*\*) is the large model and multimodal large model training and deployment framework provided by the \[ModelScope community\](https://modelscope.cn/). GitHub repository: \[ms-swift\](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: - \*\*Model Types\*\*: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. - \*\*Hardware Support\*\*: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. - \*\*Training Methods\*\*: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. - \*\*Distributed Training\*\*: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. - \*\*RLHF Training\*\*: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift's integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. \`\`\`shell pip install ms-swift -U # Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers -U # Optional packages pip install deepspeed # multi-GPU training pip install liger-kernel # save GPU memory resources pip install flash-attn --no-build-isolation \`\`\` ## Supervised Fine-Tuning (SFT) ### Data Preparation The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify \`--dataset \` in the training script. For complete dataset formatting guidelines, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) - General format \`\`\`json {"messages": \[ {"role": "system", "content": ""}, {"role": "user", "content": ""}, {"role": "assistant", "content": ""} \]} \`\`\` - Format with think \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` If you want to train using data without a chain of thought but retain the model's reasoning ability, there are two approaches to minimize disruption during fine-tuning: \*\*Option 1\*\*: During training, specify \`--loss\_scale ignore\_empty\_think\` to ignore the loss calculation for \`\\n\\n\\n\\n\`, preventing the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo1.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` \*\*Option 2\*\*: Add \`/no\_think\` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo2.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` ### 30-Minute Self-Cognition Fine-Tuning This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by \[ModelScope's free compute resources\](https://modelscope.cn/my/mynotebook). After training, the model will identify itself as "swift-robot," trained by "swift," instead of its original self-cognition as "Qwen," trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify \`--model \` and \`--dataset \`. The dataset can be found on \[Modelscope Hub\](https://modelscope.cn/datasets/swift/self-cognition). For the meaning of each parameter in the training script, please refer to the \[Command-line parameters documentation\](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type lora \\ --dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-4 \\ --lora\_rank 8 \\ --lora\_alpha 32 \\ --target\_modules all-linear \\ --gradient\_accumulation\_steps 16 \\ --eval\_steps 50 \\ --save\_steps 50 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --max\_length 2048 \\ --output\_dir output \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --model\_author swift \\ --model\_name swift-robot \`\`\` After fine-tuning, you can use the following script to test the fine-tuning results. Note that the \`--adapters\` section needs to be modified to the directory path of the last saved checkpoint: \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift infer \\ --adapters output/vx-xxx/checkpoint-xxx \\ --stream true \\ --temperature 0 \\ --max\_new\_tokens 2048 \`\`\` \`\`\`text <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! \`\`\` By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify \`--use\_hf true\`. Merge LoRA weights: \`\`\`shell swift export \\ --adapters output/checkpoint-xxx \\ --merge\_lora true \`\`\` Push the model to ModelScope/HuggingFace: \`\`\`shell # If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. # The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ --adapters output/checkpoint-xxx \\ --push\_to\_hub true \\ --hub\_model\_id '' \\ --hub\_token '' \\ --use\_hf false \`\`\` If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \`\`\`shell # 4 \* 60GB # You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. # Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE=4 \\ CUDA\_VISIBLE\_DEVICES=0,1,2,3 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset '' \\ --torch\_dtype bfloat16 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-5 \\ --gradient\_accumulation\_steps 4 \\ --packing true \\ --eval\_steps 100 \\ --save\_steps 100 \\ --logging\_steps 5 \\ --max\_length 8192 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 8 \\ --dataset\_num\_proc 8 \\ --save\_total\_limit 2 \\ --save\_only\_model true \\ --output\_dir output \\ --deepspeed zero3 \\ --use\_liger\_kernel true \\ --attn\_impl flash\_attn \`\`\` ## Reinforcement Learning (RL) ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: \[Supported Features\](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). ### Environment Setup In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: \`\`\`shell pip install "math\_verify==0.5.2" pip install vllm \`\`\` ### Data Preparation The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a \`solution\` column is needed to calculate the accuracy. Example Dataset Formats: \`\`\`json {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} \`\`\` For dataset preparation for other RLHF algorithms, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf). Notes on Dataset Requirements: - \*\*Reward Function Calculation\*\*: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: - When using the built-in accuracy or cosine similarity reward, the dataset must include a \`solution\` column to calculate the accuracy of the responses. - Other columns in the dataset will be passed as \`\*\*kwargs\` to the reward function for additional customization. - \*\*Customizing the Reward Function\*\*: To adapt the reward function to your specific needs, you can refer to the following resource: \[External Reward Plugin\](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin). This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting \`num\_infer\_workers=8\`, we deploy a vLLM engine for each device to speed up the sampling process. \`\`\`shell # 70G\*8 CUDA\_VISIBLE\_DEVICES=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE=8 \\ swift rlhf \\ --rlhf\_type grpo \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset 'AI-MO/NuminaMath-TIR#5000' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 2 \\ --per\_device\_eval\_batch\_size 2 \\ --learning\_rate 1e-6 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --output\_dir output \\ --gradient\_accumulation\_steps 1 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --max\_completion\_length 4096 \\ --vllm\_max\_model\_len 8192 \\ --reward\_funcs accuracy \\ --num\_generations 16 \\ --use\_vllm true \\ --vllm\_gpu\_memory\_utilization 0.4 \\ --sleep\_level 1 \\ --offload\_model true \\ --offload\_optimizer true \\ --gc\_collect\_after\_offload true \\ --deepspeed zero3 \\ --num\_infer\_workers 8 \\ --tensor\_parallel\_size 1 \\ --temperature 1.0 \\ --top\_p 0.85 \\ --log\_completions true \\ --overlong\_filter true \`\`\` ## Megatron-SWIFT ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the \[Supported Models Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For environment preparation and the conversion between HF and MCore model weights, you can refer to the \[Megatron-SWIFT Training Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html). These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to \[here\](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). \`\`\`shell # https://help.aliyun.com/zh/pai/user-guide/general-environment-variables # Ensure that the weight-saving paths on the two nodes are identical. NNODES=$WORLD\_SIZE \\ NODE\_RANK=$RANK \\ megatron sft \\ --load Qwen3-30B-A3B-Base-mcore \\ --dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ --tensor\_model\_parallel\_size 2 \\ --expert\_model\_parallel\_size 8 \\ --moe\_grouped\_gemm true \\ --moe\_shared\_expert\_overlap true \\ --moe\_aux\_loss\_coeff 0.01 \\ --micro\_batch\_size 1 \\ --global\_batch\_size 16 \\ --packing true \\ --recompute\_granularity full \\ --recompute\_method uniform \\ --recompute\_num\_layers 1 \\ --train\_iters 2000 \\ --eval\_iters 50 \\ --finetune true \\ --cross\_entropy\_loss\_fusion true \\ --lr 1e-5 \\ --lr\_warmup\_iters 100 \\ --min\_lr 1e-6 \\ --save megatron\_output/Qwen3-30B-A3B-Base \\ --eval\_interval 200 \\ --save\_interval 200 \\ --max\_length 8192 \\ --num\_workers 8 \\ --dataset\_num\_proc 8 \\ --no\_save\_optim true \\ --no\_save\_rng true \\ --sequence\_parallel true \\ --use\_flash\_attn true \`\`\` The custom dataset format is the same as \`swift sft\`, which can be found in the previous section. Simply specify \`--dataset \`. The following is a comparison of training speed and GPU memory usage between \`megatron sft\` and \`swift sft\` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | ---------------- | ----------- | --------------- | --------------- | | Training Speed | 9.6s/it | - | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | ## Conclusion The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in \[this issue\](https://github.com/modelscope/ms-swift/issues/4030). --- # Unknown \# Using Transformers to Chat The most significant but also the simplest usage of Qwen2 is to chat with it using the \`transformers\` library. In this document, we show how to chat with \`Qwen2-7B-Instruct\`, in either streaming mode or not. Select the interface you would like to use: ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual Using \`AutoTokenizer\` and \`AutoModelForCausalLM\`. ::: :::{tab-item} Pipeline :sync: pipeline Using \`pipeline\`. ::: :::: ## Basic Usage ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual You can just write several lines of code with \`transformers\` to chat with Qwen2-Instruct. Essentially, we build the tokenizer and the model with \`from\_pretrained\` method, and we use \`generate\` method to perform chatting with the help of chat template provided by the tokenizer. Below is an example of how to chat with Qwen2-7B-Instruct: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer # Now you do not need to add "trust\_remote\_code=True" model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen2-7B-Instruct") # Instead of using model.chat(), we directly use model.generate() # But you need to use tokenizer.apply\_chat\_template() to format your inputs as shown below prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # Directly use generate() and tokenizer.decode() to get the output. # Use \`max\_new\_tokens\` to control the maximum output length. generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` To continue the chat, simply append the response to the messages with the role assistant and repeat the procedure. The following shows and example: \`\`\`python messages.append({"role": "assistant", "content": response}) prompt = "Tell me more." messages.append({"role": "user", "content": prompt}) text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device) # Directly use generate() and tokenizer.decode() to get the output. # Use \`max\_new\_tokens\` to control the maximum output length. generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=512, ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] \`\`\` Note that the previous method in the original Qwen repo \`chat()\` is now replaced by \`generate()\`. The \`apply\_chat\_template()\` function is used to convert the messages into a format that the model can understand. The \`add\_generation\_prompt\` argument is used to add a generation prompt, which refers to \`<|im\_start|>assistant\\n\` to the input. Notably, we apply ChatML template for chat models following our previous practice. The \`max\_new\_tokens\` argument is used to set the maximum length of the response. The \`tokenizer.batch\_decode()\` function is used to decode the response. In terms of the input, the above \`messages\` is an example to show how to format your dialog history and system prompt. By default, if you do not specify system prompt, we directly use \`You are a helpful assistant.\`. ::: :::{tab-item} Pipeline :sync: pipeline \`transformers\` provides a functionality called "pipeline" that encapsulates the many operations in common tasks. You can chat with the model in just 4 lines of code: \`\`\`python from transformers import pipeline pipe = pipeline("text-generation", "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto") # the default system message will be used messages = \[{"role": "user", "content": "Give me a short introduction to large language model."}\] response\_message = pipe(messages, max\_new\_tokens=512)\[0\]\["generated\_text"\]\[-1\] \`\`\` To continue the chat, simply append the response to the messages with the role assistant and repeat the procedure. The following shows and example: \`\`\`python messages.append(response\_message) prompt = "Tell me more." messages.append({"role": "user", "content": prompt}) response\_message = pipe(messages, max\_new\_tokens=512)\[0\]\["generated\_text"\]\[-1\] \`\`\` ::: :::: ## Batching :::{note} Batching is not automatically a win for performance. ::: All common \`transformers\` methods support batched input and output. For basic usage, the following is an example: ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen2-7B-Instruct", padding\_side="left") message\_batch = \[ \[{"role": "user", "content": "Give me a detailed introduction to large language model."}\], \[{"role": "user", "content": "Hello!"}\], \] text\_batch = tokenizer.apply\_chat\_template( message\_batch, tokenize=False, add\_generation\_prompt=True, ) model\_inputs\_batch = tokenizer(text\_batch, return\_tensors="pt", padding=True).to(model.device) generated\_ids\_batch = model.generate( \*\*model\_inputs\_batch, max\_new\_tokens=512, ) generated\_ids\_batch = generated\_ids\_batch\[:, model\_inputs\_batch.input\_ids.shape\[1\]:\] response\_batch = tokenizer.batch\_decode(generated\_ids\_batch, skip\_special\_tokens=True) \`\`\` ::: :::{tab-item} Pipeline :sync: pipeline With pipeline, it is simpler: \`\`\`python from transformers import pipeline pipe = pipeline("text-generation", "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto") pipe.tokenizer.padding\_side="left" message\_batch = \[ \[{"role": "user", "content": "Give me a detailed introduction to large language model."}\], \[{"role": "user", "content": "Hello!"}\], \] result\_batch = pipe(message\_batch, max\_new\_tokens=512, batch\_size=2) response\_message\_batch = \[result\[0\]\["generated\_text"\]\[-1\] for result in result\_batch\] \`\`\` ::: :::: ## Streaming Mode With the help of \`TextStreamer\`, you can modify your chatting with Qwen to streaming mode. It will print the response as being generated to the console or the terminal. Below we show you an example of how to use it: ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual \`\`\`python # Repeat the code above before model.generate() # Starting here, we add streamer for text generation. from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) generated\_ids = model.generate( model\_inputs, max\_new\_tokens=512, streamer=streamer, ) \`\`\` ::: :::{tab-item} Pipeline :sync: pipeline \`\`\`python from transformers import pipeline, TextStreamer pipe = pipeline( "text-generation", "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", ) streamer = TextStreamer(pipe.tokenizer, skip\_prompt=True, skip\_special\_tokens=True) response\_message = pipe(messages, max\_new\_tokens=512, streamer=streamer)\[0\]\["generated\_text"\]\[-1\] \`\`\` ::: :::: Besides using \`TextStreamer\`, we can also use \`TextIteratorStreamer\` which stores print-ready text in a queue, to be used by a downstream application as an iterator: ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual \`\`\`python # Repeat the code above before model.generate() # Starting here, we add streamer for text generation. from transformers import TextIteratorStreamer streamer = TextIteratorStreamer(tokenizer, skip\_prompt=True, skip\_special\_tokens=True) # Use Thread to run generation in background # Otherwise, the process is blocked until generation is complete # and no streaming effect can be observed. from threading import Thread generation\_kwargs = dict(model\_inputs, streamer=streamer, max\_new\_tokens=512) thread = Thread(target=model.generate, kwargs=generation\_kwargs) thread.start() generated\_text = "" for new\_text in streamer: generated\_text += new\_text print(generated\_text) \`\`\` ::: :::{tab-item} Pipeline :sync: pipeline \`\`\`python from transformers import pipeline, TextIteratorStreamer pipe = pipeline( "text-generation", "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", ) streamer = TextIteratorStreamer(pipe.tokenizer, skip\_prompt=True, skip\_special\_tokens=True) # Use Thread to run generation in background # Otherwise, the process is blocked until generation is complete # and no streaming effect can be observed. from threading import Thread generation\_kwargs = dict(text\_inputs=messages, max\_new\_tokens=512, streamer=streamer) thread = Thread(target=pipe, kwargs=generation\_kwargs) thread.start() generated\_text = "" for new\_text in streamer: generated\_text += new\_text print(generated\_text) \`\`\` ::: :::: ## Using Flash Attention 2 to Accelerate Generation :::{note} With the latest \`transformers\` and \`torch\`, Flash Attention 2 will be applied by default if applicable.\[^fa2\] You do not need to request the use of Flash Attention 2 in \`transformers\` or install the \`flash\_attn\` package. The following is intended for users that cannot use the latest versions for various reasons. ::: If you would like to apply Flash Attention 2, you need to install an appropriate version of \`flash\_attn\`. You can find pre-built wheels at \[its GitHub repository\](https://github.com/Dao-AILab/flash-attention/releases), and you should make sure the Python version, the torch version, and the CUDA version of torch are a match. Otherwise, you need to install from source. Please follow the guides at \[its GitHub README\](https://github.com/Dao-AILab/flash-attention). After a successful installation, you can load the model as shown below: ::::{tab-set} :sync-group: interface :::{tab-item} Manual :sync: manual \`\`\`python model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", attn\_implementation="flash\_attention\_2", ) \`\`\` ::: :::{tab-item} Pipeline :sync: pipeline \`\`\`python pipe = pipeline( "text-generation", "Qwen/Qwen2-7B-Instruct", torch\_dtype="auto", device\_map="auto", model\_kwargs=dict(attn\_implementation="flash\_attention\_2"), ) \`\`\` ::: :::: \[^fa2\]: The attention module for a model in \`transformers\` typically has three variants: \`sdpa\`, \`flash\_attention\_2\`, and \`eager\`. The first two are wrappers around related functions in the \`torch\` and the \`flash\_attn\` packages. It defaults to \`sdpa\` if available. In addition, \`torch\` has integrated three implementations for \`sdpa\`: \`FLASH\_ATTENTION\` (indicating Flash Attention 2 since version 2.2), \`EFFICIENT\_ATTENTION\` (Memory Efficient Attention), and \`MATH\`. It attempts to automatically select the most optimal implementation based on the inputs. You don't need to install extra packages to use them. Hence, if applicable, by default, \`transformers\` uses \`sdpa\` and \`torch\` selects \`FLASH\_ATTENTION\`. If you wish to explicitly select the implementations in \`torch\`, refer to \[this tutorial\](https://pytorch.org/tutorials/intermediate/scaled\_dot\_product\_attention\_tutorial.html). ## Troubleshooting :::{dropdown} Loading models takes a lot of memory Normally, memory usage after loading the model can be roughly taken as twice the parameter count. For example, a 7B model will take 14GB memory to load. It is because for large language models, the compute dtype is often 16-bit floating point number. Of course, you will need more memory in inference to store the activations. For \`transformers\`, \`torch\_dtype="auto"\` is recommended and the model will be loaded in \`bfloat16\` automatically. Otherwise, the model will be loaded in \`float32\` and it will need double memory. You can also pass \`torch.bfloat16\` as \`torch\_dtype\` explicitly. ::: :::{dropdown} Multi-GPU inference is slow \`transformers\` relies on \`accelerate\` for multi-GPU inference and the implementation is a kind of naive model parallelism: different GPUs computes different layers of the model. It is enabled by the use of \`device\_map="auto"\` or a customized \`device\_map\` for multiple GPUs. However, this kind of implementation is not efficient as for a single request, only one GPU computes at the same time and the other GPUs just wait. To use all the GPUs, you need to arrange multiple sequences as on a pipeline, making sure each GPU has some work to do. However, that will require concurrency management and load balancing, which is out of the scope of \`transformers\`. Even if all things are implemented, you can make use of concurrency to improve the total throughput but the latency for each request is not great. For Multi-GPU inference, we recommend using specialized inference framework, such as vLLM and TGI, which support tensor parallelism. ::: :::{dropdown} The inference of Qwen2 MoE models is slow All MoE models in \`transformers\` compute the results of the expert FFNs in loops, and it is less efficient for GPUs by nature. The performance is even worse for model with fine-grained experts, where the model has a lot of experts and each expert is relatively small, which is the case for Qwen2 MoE. To optimize that, a fused kernel implementation (as in \`vllm\`) or methods like expert parallel (as in \`mcore\`) is needed. For now, we recommend using \`vllm\` for Qwen2 MoE. ::: :::{dropdown} \`\`RuntimeError: probability tensor contains either \`inf\`, \`nan\` or element < 0\`\` or generating repeating \`!!!!...\` We don't recommend using \`float16\` for Qwen2 models or numerical instability may occur, especially for cards without support of fp16 matmul with fp32 accumulate. If you have to use \`float16\`, consider using \[this fork\](https://github.com/jklj077/transformers/tree/qwen2-patch) and force \`attn\_implementation="eager"\`. If it works with single GPU but not multiple GPUs, especially if there are PCI-E switches in your system, please also refer to the next issue. ::: :::{dropdown} \`RuntimeError: CUDA error: device-side assert triggered\`, \`Assertion -sizes\[i\] <= index && index < sizes\[i\] && "index out of bounds" failed.\` If it works with single GPU but not multiple GPUs, especially if there are PCI-E switches in your system, it could be related to drivers. 1. Try upgrading the GPU driver. For data center GPUs (e.g., A800, H800, and L40s), please use the data center GPU drivers and upgrade to the latest subrelease, e.g., 535.104.05 to 535.183.01. You can check the release note at , where the issues fixed and known issues are presented. For consumer GPUs (e.g., RTX 3090 and RTX 4090), their GPU drivers are released more frequently and focus more on gaming optimization. There are online reports that 545.29.02 breaks \`vllm\` and \`torch\` but 545.29.06 works. Their release notes are also less helpful in identifying the real issues. However, in general, the advice is still upgrading the GPU driver. 2. Try disabling P2P for process hang, but it has negative effect on speed. \`\`\` export NCCL\_P2P\_DISABLE=1 \`\`\` ::: ## Next Step Now you can chat with Qwen2 in either streaming mode or not. Continue to read the documentation and try to figure out more advanced usages of model inference! --- # Unknown \# Quickstart This guide helps you quickly start using Qwen3. We provide examples of \[Hugging Face Transformers\](https://github.com/huggingface/transformers) as well as \[ModelScope\](https://github.com/modelscope/modelscope), and \[vLLM\](https://github.com/vllm-project/vllm) for deployment. You can find Qwen3 models in \[the Qwen3 collection\](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) at Hugging Face Hub and \[the Qwen3 collection\](https://www.modelscope.cn/collections/Qwen3-9743180bdc6b48) at ModelScope. ## Transformers To get a quick start with Qwen3, you can try the inference with \`transformers\` first. Make sure that you have installed \`transformers>=4.51.0\`. We advise you to use Python 3.10 or higher, and PyTorch 2.6 or higher. The following is a very simple code snippet showing how to run Qwen3-8B: \`\`\`python from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) # conduct text completion generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=32768 ) output\_ids = generated\_ids\[0\]\[len(model\_inputs.input\_ids\[0\]):\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` Qwen3 will think before respond, similar to QwQ models. This means the model will use its reasoning abilities to enhance the quality of generated responses. The model will first generate thinking content wrapped in a \`...\` block, followed by the final response. - Hard Switch: To strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5-Instruct models, you can set \`enable\_thinking=False\` when formatting the text. \`\`\`python text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=False, # Setting enable\_thinking=False disables thinking mode ) \`\`\` It can be particularly useful in scenarios where disabling thinking is essential for enhancing efficiency. - Soft Switch: Qwen3 also understands the user's instruction on its thinking behavior, in particular, the soft switch \`/think\` and \`/no\_think\`. You can add them to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations. :::{note} For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in \`generation\_config.json\`). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section. For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. ::: ## ModelScope To tackle with downloading issues, we advise you to try \[ModelScope\](https://github.com/modelscope/modelscope). Before starting, you need to install \`modelscope\` with \`pip\`. \`modelscope\` adopts a programmatic interface similar (but not identical) to \`transformers\`. For basic usage, you can simply change the first line of code above to the following: \`\`\`python from modelscope import AutoModelForCausalLM, AutoTokenizer \`\`\` For more information, please refer to \[the documentation of \`modelscope\`\](https://www.modelscope.cn/docs). ## OpenAI API Compatibility You can serve Qwen3 via OpenAI-compatible APIs using frameworks such as vLLM, SGLang, and interact with the API using common HTTP clients or the OpenAI SDKs. Here we take Qwen3-8B as an example to start the API: - SGLang (\`sglang>=0.4.6.post1\` is required): \`\`\`shell python -m sglang.launch\_server --model-path Qwen/Qwen3-8B --port 8000 --reasoning-parser qwen3 \`\`\` - vLLM (\`vllm>=0.8.5\` is recommended): \`\`\`shell vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser deepseek\_r1 \`\`\` Then, you can use the \[create chat interface\](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl \`\`\`shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": \[ {"role": "user", "content": "Give me a short introduction to large language models."} \], "temperature": 0.6, "top\_p": 0.95, "top\_k": 20, "max\_tokens": 32768 }' \`\`\` ::: :::{tab-item} Python You can use the API client with the \`openai\` Python SDK as shown below: \`\`\`python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=\[ {"role": "user", "content": "Give me a short introduction to large language models."}, \], max\_tokens=32768, temperature=0.6, top\_p=0.95, extra\_body={ "top\_k": 20, } ) print("Chat response:", chat\_response) \`\`\` :::: While the soft switch is always available, the hard switch is also available in the API through the following configuration to the API call. For more usage, please refer to our document on \[SGLang\](../deployment/sglang) and \[vLLM\](../deployment/vllm). ## Thinking Budget Qwen3 supports the configuration of thinking budget. It is achieved by ending the thinking process once the budget is reached and guiding the model to generate the "summary" with an early-stopping prompt. Since this feature involves customization specific to each model, it is currently not available in the open-source frameworks and only implemented by \[the Alibaba Cloud Model Studio API\](https://www.alibabacloud.com/help/en/model-studio/deep-thinking#6f0633b9cdts1). However, with existing open-source frameworks, one can generate twice to implement this feature as follows: 1. For the first time, generate tokens up to the thinking budget and check if the thinking process is finished. If the thinking process is not finished, append the early-stopping prompt. 2. For the second time, continue generation until the end of the content or the upper length limit is fulfilled. The following snippet shows the implementation with Hugging Face Transformers: \`\`\`python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model\_name = "Qwen/Qwen3-8B" thinking\_budget = 16 max\_new\_tokens = 32768 # load the tokenizer and the model model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained(model\_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = \[ {"role": "user", "content": prompt}, \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True, enable\_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(model.device) input\_length = model\_inputs.input\_ids.size(-1) # first generation until thinking budget generated\_ids = model.generate( \*\*model\_inputs, max\_new\_tokens=thinking\_budget ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # check if the generation has already finished (151645 is <|im\_end|>) if 151645 not in output\_ids: # check if the thinking process has finished (151668 is ) # and prepare the second model input if 151668 not in output\_ids: print("thinking budget is reached") early\_stopping\_text = "\\n\\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\\n\\n\\n" early\_stopping\_ids = tokenizer(\[early\_stopping\_text\], return\_tensors="pt", return\_attention\_mask=False).input\_ids.to(model.device) input\_ids = torch.cat(\[generated\_ids, early\_stopping\_ids\], dim=-1) else: input\_ids = generated\_ids attention\_mask = torch.ones\_like(input\_ids, dtype=torch.int64) # second generation generated\_ids = model.generate( input\_ids=input\_ids, attention\_mask=attention\_mask, max\_new\_tokens=input\_length + max\_new\_tokens - input\_ids.size(-1) # could be negative if max\_new\_tokens is not large enough (early stopping text is 24 tokens) ) output\_ids = generated\_ids\[0\]\[input\_length:\].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output\_ids) - output\_ids\[::-1\].index(151668) except ValueError: index = 0 thinking\_content = tokenizer.decode(output\_ids\[:index\], skip\_special\_tokens=True).strip("\\n") content = tokenizer.decode(output\_ids\[index:\], skip\_special\_tokens=True).strip("\\n") print("thinking content:", thinking\_content) print("content:", content) \`\`\` You should see the output in the console like the following \`\`\`text thinking budget is reached thinking content: Okay, the user is asking for a short introduction to large language models Considering the limited time by the user, I have to give the solution based on the thinking directly now. content: Large language models (LLMs) are advanced artificial intelligence systems trained on vast amounts of text data to understand and generate human-like language. They can perform tasks such as answering questions, writing stories, coding, and translating languages. LLMs are powered by deep learning techniques and have revolutionized natural language processing by enabling more context-aware and versatile interactions with text. Examples include models like GPT, BERT, and others developed by companies like OpenAI and Alibaba. \`\`\` :::{note} For purpose of demonstration only, \`thinking\_budget\` is set to 16. However, \`thinking\_budget\` should not be set to that low in practice. We recommend tuning \`thinking\_budget\` based on the latency users can accept and setting it higher than 1024 for meaningful improvements across tasks. If thinking is not desired at all, developers should make use of the hard switch instead. ::: ## Next Step Now, you can have fun with Qwen3 models. Would love to know more about its usage? Feel free to check other documents in this documentation. --- # Unknown LlamaIndex ========== .. attention:: To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on \`LlamaIndex \`\_\_. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: .. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the \`\`bge-base-en-v1.5\`\` model to retrieve from English documents, and you can download the \`\`bge-base-zh-v1.5\`\` model to retrieve from Chinese documents. You can also choose \`\`bge-large\`\` or \`\`bge-small\`\` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) .. code:: python import torch from llama\_index.core import Settings from llama\_index.core.node\_parser import SentenceSplitter from llama\_index.llms.huggingface import HuggingFaceLLM from llama\_index.embeddings.huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama\_index.core import PromptTemplate def completion\_to\_prompt(completion): return f"<|im\_start|>system\\n<|im\_end|>\\n<|im\_start|>user\\n{completion}<|im\_end|>\\n<|im\_start|>assistant\\n" def messages\_to\_prompt(messages): prompt = "" for message in messages: if message.role == "system": prompt += f"<|im\_start|>system\\n{message.content}<|im\_end|>\\n" elif message.role == "user": prompt += f"<|im\_start|>user\\n{message.content}<|im\_end|>\\n" elif message.role == "assistant": prompt += f"<|im\_start|>assistant\\n{message.content}<|im\_end|>\\n" if not prompt.startswith("<|im\_start|>system"): prompt = "<|im\_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|>\\n" + prompt prompt = prompt + "<|im\_start|>assistant\\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings.llm = HuggingFaceLLM( model\_name="Qwen/Qwen2.5-7B-Instruct", tokenizer\_name="Qwen/Qwen2.5-7B-Instruct", context\_window=30000, max\_new\_tokens=2000, generate\_kwargs={"temperature": 0.7, "top\_k": 50, "top\_p": 0.95}, messages\_to\_prompt=messages\_to\_prompt, completion\_to\_prompt=completion\_to\_prompt, device\_map="auto", ) # Set embedding model Settings.embed\_model = HuggingFaceEmbedding( model\_name = "BAAI/bge-base-en-v1.5" ) # Set the size of the text chunk for retrieval Settings.transformations = \[SentenceSplitter(chunk\_size=1024)\] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'. .. code:: python from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load\_data() index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. .. code:: python from llama\_index.readers.web import SimpleWebPageReader from llama\_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html\_to\_text=True).load\_data( \["web\_address\_1","web\_address\_2",...\] ) index = VectorStoreIndex.from\_documents( documents, embed\_model=Settings.embed\_model, transformations=Settings.transformations ) To save and load the index, you can use the following code snippet. .. code:: python from llama\_index.core import StorageContext, load\_index\_from\_storage # save index storage\_context = StorageContext.from\_defaults(persist\_dir="save") # load index index = load\_index\_from\_storage(storage\_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. .. code:: python query\_engine = index.as\_query\_engine() your\_query = "" print(query\_engine.query(your\_query).response) --- # Unknown Text Generation Web UI ======================================== \`Text Generation Web UI \`\_\_ (TGW, or usually referred to “oobabooga”) is a popular web UI for text generation, similar to \`AUTOMATIC1111/stable-diffusion-webui \`\_\_. It has multiple interfaces, and supports multiple model backends, including \`Transformers \`\_\_, \`llama.cpp \`\_\_ (through \`llama-cpp-python \`\_\_), \`ExLlamaV2 \`\_\_, \`AutoGPTQ \`\_\_, \`AutoAWQ \`\_\_, \`GPTQ-for-LLaMa \`\_\_, \`CTransformers \`\_\_, \`QuIP# \`\_\_. In this section, we introduce how to run Qwen locally with TGW. Quickstart ---------- The simplest way to run TGW is to use the provided shell scripts in the \`repo \`\_\_. For the first step, clone the repo and enter the directory: .. code:: bash git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui You can directly run the \`\`start\_linux.sh\`\`, \`\`start\_windows.bat\`\`, \`\`start\_macos.sh\`\`, or \`\`start\_wsl.bat\`\` script depending on your OS. Alternatively you can manually install the requirements in your conda environment. Here I take the practice on MacOS as an example. .. code:: bash conda create -n textgen python=3.11 conda activate textgen pip install torch torchvision torchaudio Then you can install the requirements by running \`\`pip install -r\`\` based on your OS, e.g., .. code:: bash pip install -r requirements\_apple\_silicon.txt For \`\`bitsandbytes\`\` and \`\`llama-cpp-python\`\` inside the requirements, I advise you to install them through \`\`pip\`\` directly. However, temporarily please do not use GGUF as the performance with TGW is unsatisfactory. After finishing the installation of required packages, you need to prepare your models by putting the model files or directories in the folder \`\`./models\`\`. For example, you should put the transformers model directory of \`\`Qwen1.5-7B-Chat\`\` in the way shown below: :: text-generation-webui ├── models │   ├── Qwen1.5-7B-Chat │   │   ├── config.json │   │   ├── generation\_config.json │   │   ├── model-00001-of-00004.safetensor │   │   ├── model-00002-of-00004.safetensor │   │   ├── model-00003-of-00004.safetensor │   │   ├── model-00004-of-00004.safetensor │   │   ├── model.safetensor.index.json │   │   ├── merges.txt │   │   ├── tokenizer\_config.json │   │   └── vocab.json Then you just need to run .. code:: bash python server.py to launch your web UI service. Please browse to :: \`http://localhost:7860/?\_\_theme=dark\` and enjoy playing with Qwen in a web UI! Next Step --------- There are a lot more usages in TGW, where you can even enjoy role play, use different types of quantized models, train LoRA, incorporate extensions like stable diffusion and whisper, etc. Go to figure out more advanced usages and apply them to Qwen models! --- # Unknown \# MS-SWIFT ModelScope SWIFT (\*\*ms-swift\*\*) is the large model and multimodal large model training and deployment framework provided by the \[ModelScope community\](https://modelscope.cn/). GitHub repository: \[ms-swift\](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: - \*\*Model Types\*\*: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. - \*\*Hardware Support\*\*: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. - \*\*Training Methods\*\*: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. - \*\*Distributed Training\*\*: Supports distributed training technologies such as DDP, device\_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. - \*\*RLHF Training\*\*: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift's integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. \`\`\`shell pip install ms-swift -U # Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers -U # Optional packages pip install deepspeed # multi-GPU training pip install liger-kernel # save GPU memory resources pip install flash-attn --no-build-isolation \`\`\` ## Supervised Fine-Tuning (SFT) ### Data Preparation The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify \`--dataset \` in the training script. For complete dataset formatting guidelines, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) - General format \`\`\`json {"messages": \[ {"role": "system", "content": ""}, {"role": "user", "content": ""}, {"role": "assistant", "content": ""} \]} \`\`\` - Format with think \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n...\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` If you want to train using data without a chain of thought but retain the model's reasoning ability, there are two approaches to minimize disruption during fine-tuning: \*\*Option 1\*\*: During training, specify \`--loss\_scale ignore\_empty\_think\` to ignore the loss calculation for \`\\n\\n\\n\\n\`, preventing the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo1.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` \*\*Option 2\*\*: Add \`/no\_think\` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script \[here\](https://github.com/modelscope/ms-swift/blob/main/examples/train/think\_model/qwen3\_demo2.sh). The custom dataset format is as follows: \`\`\`json {"messages": \[ {"role": "user", "content": "Where is the capital of Zhejiang? /no\_think"}, {"role": "assistant", "content": "\\n\\n\\n\\nThe capital of Zhejiang is Hangzhou."} \]} \`\`\` ### 30-Minute Self-Cognition Fine-Tuning This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by \[ModelScope's free compute resources\](https://modelscope.cn/my/mynotebook). After training, the model will identify itself as "swift-robot," trained by "swift," instead of its original self-cognition as "Qwen," trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify \`--model \` and \`--dataset \`. The dataset can be found on \[Modelscope Hub\](https://modelscope.cn/datasets/swift/self-cognition). For the meaning of each parameter in the training script, please refer to the \[Command-line parameters documentation\](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type lora \\ --dataset 'swift/Qwen3-SFT-Mixin#2000' \\ 'swift/self-cognition:qwen3#600' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-4 \\ --lora\_rank 8 \\ --lora\_alpha 32 \\ --target\_modules all-linear \\ --gradient\_accumulation\_steps 16 \\ --eval\_steps 50 \\ --save\_steps 50 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --max\_length 2048 \\ --output\_dir output \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --model\_author swift \\ --model\_name swift-robot \`\`\` After fine-tuning, you can use the following script to test the fine-tuning results. Note that the \`--adapters\` section needs to be modified to the directory path of the last saved checkpoint: \`\`\`bash CUDA\_VISIBLE\_DEVICES=0 \\ swift infer \\ --adapters output/vx-xxx/checkpoint-xxx \\ --stream true \\ --temperature 0 \\ --max\_new\_tokens 2048 \`\`\` \`\`\`text <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! \`\`\` By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify \`--use\_hf true\`. Merge LoRA weights: \`\`\`shell swift export \\ --adapters output/checkpoint-xxx \\ --merge\_lora true \`\`\` Push the model to ModelScope/HuggingFace: \`\`\`shell # If you are pushing the complete weights, you need to change \`--adapters\` to \`--model\`. # The Modelscope hub\_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \\ --adapters output/checkpoint-xxx \\ --push\_to\_hub true \\ --hub\_model\_id '' \\ --hub\_token '' \\ --use\_hf false \`\`\` If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: \`\`\`shell # 4 \* 60GB # You can run the experiment by setting \`--dataset AI-ModelScope/alpaca-gpt4-data-en\`. # Note: If you want to specify \`--packing true\`, you must additionally set \`--attn\_impl flash\_attn\`. NPROC\_PER\_NODE=4 \\ CUDA\_VISIBLE\_DEVICES=0,1,2,3 \\ swift sft \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset '' \\ --torch\_dtype bfloat16 \\ --per\_device\_train\_batch\_size 1 \\ --per\_device\_eval\_batch\_size 1 \\ --learning\_rate 1e-5 \\ --gradient\_accumulation\_steps 4 \\ --packing true \\ --eval\_steps 100 \\ --save\_steps 100 \\ --logging\_steps 5 \\ --max\_length 8192 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 8 \\ --dataset\_num\_proc 8 \\ --save\_total\_limit 2 \\ --save\_only\_model true \\ --output\_dir output \\ --deepspeed zero3 \\ --use\_liger\_kernel true \\ --attn\_impl flash\_attn \`\`\` ## Reinforcement Learning (RL) ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: \[Supported Features\](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). ### Environment Setup In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: \`\`\`shell pip install "math\_verify==0.5.2" pip install vllm \`\`\` ### Data Preparation The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a \`solution\` column is needed to calculate the accuracy. Example Dataset Formats: \`\`\`json {"messages": \[{"role": "user", "content": "Tell me tomorrow's weather"}\]} {"messages": \[{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}\]} {"messages": \[{"role": "user", "content": "What is your name?"}\]} \`\`\` For dataset preparation for other RLHF algorithms, see: \[Custom Dataset Documentation\](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf). Notes on Dataset Requirements: - \*\*Reward Function Calculation\*\*: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: - When using the built-in accuracy or cosine similarity reward, the dataset must include a \`solution\` column to calculate the accuracy of the responses. - Other columns in the dataset will be passed as \`\*\*kwargs\` to the reward function for additional customization. - \*\*Customizing the Reward Function\*\*: To adapt the reward function to your specific needs, you can refer to the following resource: \[External Reward Plugin\](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin). This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting \`num\_infer\_workers=8\`, we deploy a vLLM engine for each device to speed up the sampling process. \`\`\`shell # 70G\*8 CUDA\_VISIBLE\_DEVICES=0,1,2,3,4,5,6,7 \\ NPROC\_PER\_NODE=8 \\ swift rlhf \\ --rlhf\_type grpo \\ --model Qwen/Qwen3-8B \\ --train\_type full \\ --dataset 'AI-MO/NuminaMath-TIR#5000' \\ --torch\_dtype bfloat16 \\ --num\_train\_epochs 1 \\ --per\_device\_train\_batch\_size 2 \\ --per\_device\_eval\_batch\_size 2 \\ --learning\_rate 1e-6 \\ --save\_total\_limit 2 \\ --logging\_steps 5 \\ --output\_dir output \\ --gradient\_accumulation\_steps 1 \\ --warmup\_ratio 0.05 \\ --dataloader\_num\_workers 4 \\ --max\_completion\_length 4096 \\ --vllm\_max\_model\_len 8192 \\ --reward\_funcs accuracy \\ --num\_generations 16 \\ --use\_vllm true \\ --vllm\_gpu\_memory\_utilization 0.4 \\ --sleep\_level 1 \\ --offload\_model true \\ --offload\_optimizer true \\ --gc\_collect\_after\_offload true \\ --deepspeed zero3 \\ --num\_infer\_workers 8 \\ --tensor\_parallel\_size 1 \\ --temperature 1.0 \\ --top\_p 0.85 \\ --log\_completions true \\ --overlong\_filter true \`\`\` ## Megatron-SWIFT ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the \[Supported Models Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For environment preparation and the conversion between HF and MCore model weights, you can refer to the \[Megatron-SWIFT Training Documentation\](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html). These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 \* 80GiB A800 GPUs. For more information on multi-node startup methods, refer to \[here\](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). \`\`\`shell # https://help.aliyun.com/zh/pai/user-guide/general-environment-variables # Ensure that the weight-saving paths on the two nodes are identical. NNODES=$WORLD\_SIZE \\ NODE\_RANK=$RANK \\ megatron sft \\ --load Qwen3-30B-A3B-Base-mcore \\ --dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \\ --tensor\_model\_parallel\_size 2 \\ --expert\_model\_parallel\_size 8 \\ --moe\_grouped\_gemm true \\ --moe\_shared\_expert\_overlap true \\ --moe\_aux\_loss\_coeff 0.01 \\ --micro\_batch\_size 1 \\ --global\_batch\_size 16 \\ --packing true \\ --recompute\_granularity full \\ --recompute\_method uniform \\ --recompute\_num\_layers 1 \\ --train\_iters 2000 \\ --eval\_iters 50 \\ --finetune true \\ --cross\_entropy\_loss\_fusion true \\ --lr 1e-5 \\ --lr\_warmup\_iters 100 \\ --min\_lr 1e-6 \\ --save megatron\_output/Qwen3-30B-A3B-Base \\ --eval\_interval 200 \\ --save\_interval 200 \\ --max\_length 8192 \\ --num\_workers 8 \\ --dataset\_num\_proc 8 \\ --no\_save\_optim true \\ --no\_save\_rng true \\ --sequence\_parallel true \\ --use\_flash\_attn true \`\`\` The custom dataset format is the same as \`swift sft\`, which can be found in the previous section. Simply specify \`--dataset \`. The following is a comparison of training speed and GPU memory usage between \`megatron sft\` and \`swift sft\` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | ---------------- | ----------- | --------------- | --------------- | | Training Speed | 9.6s/it | - | 91.2s/it | | GPU Memory Usage | 16 \* 60GiB | OOM | 16 \* 80GiB | ## Conclusion The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in \[this issue\](https://github.com/modelscope/ms-swift/issues/4030). --- # Unknown MLX-LM ====== \`mlx-lm \`\_\_ helps you run LLMs locally on Apple Silicon. It is available at MacOS. It has already supported Qwen models and this time, we have also provided checkpoints that you can directly use with it. Prerequisites ------------- The easiest way to get started is to install the \`\`mlx-lm\`\` package: - with \`\`pip\`\`: .. code:: bash pip install mlx-lm - with \`\`conda\`\`: .. code:: bash conda install -c conda-forge mlx-lm Runnig with Qwen MLX Files -------------------------- We provide model checkpoints with \`\`mlx-lm\`\` in our Hugging Face organization, and to search for what you need you can search the repo names with \`\`-MLX\`\`. Here provides a code snippet with \`\`apply\_chat\_template\`\` to show you how to load the tokenizer and model and how to generate contents. .. code:: python from mlx\_lm import load, generate model, tokenizer = load('Qwen/Qwen2-7B-Instruct-MLX', tokenizer\_config={"eos\_token": "<|im\_end|>"}) prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) response = generate(model, tokenizer, prompt=text, verbose=True, top\_p=0.8, temp=0.7, repetition\_penalty=1.05, max\_tokens=512) Make Your MLX files ------------------- You can make mlx files with just one command: .. code:: bash mlx\_lm.convert --hf-path Qwen/Qwen2-7B-Instruct --mlx-path mlx/Qwen2-7B-Instruct/ -q where - \`\`--hf-path\`\`: the model name on Hugging Face Hub or the local path - \`\`--mlx-path\`\`: the path for output files - \`\`-q\`\`: enable quantization --- # Unknown GGUF =========================== Recently, running LLMs locally is popular in the community, and running GGUF files with llama.cpp is a typical example. With llama.cpp, you can not only build GGUF files for your models but also perform low-bit quantization. In GGUF, you can directly quantize your models without calibration, or apply the AWQ scale for better quality, or use imatrix with calibration data. In this document, we demonstrate the simplest way to quantize your model as well as the way to apply AWQ scale to your Qwen model quantization. Quantize Your Models and Make GGUF Files ---------------------------------------- Before you move to quantization, make sure you have followed the instruction and started to use llama.cpp. The following guidance will NOT provide instructions about installation and building. Now, suppose you would like to quantize \`\`Qwen1.5-7B-Chat\`\`. You need to first make a GGUF file for the fp16 model as shown below: .. code:: bash python convert-hf-to-gguf.py Qwen/Qwen1.5-7B-Chat --outfile models/7B/qwen1\_5-7b-chat-fp16.gguf where the first argument refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file (here I just put it under the directory \`\`models/7B\`\`. Remember to create the directory before you run the command). In this way, you have generated a GGUF file for your fp16 model, and you then need to quantize it to low bits based on your requirements. An example of quantizing the model to 4 bits is shown below: .. code:: bash ./quantize models/7B/qwen1\_5-7b-chat-fp16.gguf models/7B/qwen1\_5-7b-chat-q4\_0.gguf q4\_0 where we use \`\`q4\_0\`\` for the 4-bit quantization. Until now, you have finished quantizing a model to 4 bits and putting it into a GGUF file, which can be run directly with llama.cpp. Quantize Your Models With AWQ Scales ------------------------------------ To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following \`this script \`\_\_. First of all, when you run \`\`model.quantize()\`\` with AutoAWQ, remember to add \`\`export\_compatible=True\`\` as shown below: .. code:: python ... model.quantize( tokenizer, quant\_config=quant\_config, export\_compatible=True ) model.save\_pretrained(quant\_path) ... With \`\`model.save\_quantzed()\`\` as shown above, a fp16 model with AWQ scales is saved. Then, when you run \`\`convert-hf-to-gguf.py\`\`, remember to replace the model path with the path to the fp16 model with AWQ scales, e.g., .. code:: bash python convert-hf-to-gguf.py ${quant\_path} --outfile models/7B/qwen1\_5-7b-chat-fp16-awq.gguf In this way, you can apply the AWQ scales to your quantized models in GGUF formats, which helps improving the model quality. We usually quantize the fp16 model to 2, 3, 4, 5, 6, and 8-bit models. To perform different low-bit quantization, just replace the quantization method in your command. For example, if you want to quantize your model to 2-bit model, you can replace \`\`q4\_0\`\` to \`\`q2\_k\`\` as demonstrated below: .. code:: bash ./quantize models/7B/qwen1\_5-7b-chat-fp16.gguf models/7B/qwen1\_5-7b-chat-q2\_k.gguf q2\_k We now provide GGUF models in the following quantization levels: \`\`q2\_k\`\`, \`\`q3\_k\_m\`\`, \`\`q4\_0\`\`, \`\`q4\_k\_m\`\`, \`\`q5\_0\`\`, \`\`q5\_k\_m\`\`, \`\`q6\_k\`\`, and \`\`q8\_0\`\`. For more information, please visit \`llama.cpp \`\_\_. --- # Unknown AWQ ===================== For quantized models, one of our recommendations is the usage of \`AWQ \`\_\_ with \`AutoAWQ \`\_\_. AWQ refers to Activation-aware Weight Quantization, a hardware-friendly approach for LLM low-bit weight-only quantization. AutoAWQ is an easy-to-use package for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. In this document, we show you how to use the quantized model with Transformers and also how to quantize your own model. Usage of AWQ Quantized Models with Transformers ----------------------------------------------- Now, Transformers has officially supported AutoAWQ, which means that you can directly use the quantized model with Transformers. The following is a very simple code snippet showing how to run \`\`Qwen1.5-7B-Chat-AWQ\`\` with the quantized model: .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" # the device to load the model onto model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat-AWQ", # the quantized model device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat-AWQ") prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device) generated\_ids = model.generate( model\_inputs.input\_ids, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] Usage of AWQ Quantized Models with vLLM --------------------------------------- vLLM has supported AWQ, which means that you can directly use our provided AWQ models or those trained with \`\`AutoAWQ\`\` with vLLM. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`\`Qwen1.5-7B-Chat-AWQ\`\`: .. code:: bash python -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen1.5-7B-Chat-AWQ .. code:: bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen1.5-7B-Chat-AWQ", "messages": \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], }' or you can use python client with \`\`openai\`\` python package as shown below: .. code:: python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen1.5-7B-Chat-AWQ", messages=\[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \] ) print("Chat response:", chat\_response) Quantize Your Own Model with AutoAWQ ------------------------------------ If you want to quantize your own model to AWQ quantized models, we advise you to use AutoAWQ. It is suggested installing the latest version of the package by installing from source code: .. code:: bash git clone https://github.com/casper-hansen/AutoAWQ.git cd AutoAWQ pip install -e . Suppose you have finetuned a model based on \`\`Qwen1.5-7B\`\`, which is named \`\`Qwen1.5-7B-finetuned\`\`, with your own dataset, e.g., Alpaca. To build your own AWQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: .. code:: python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quant\_config = { "zero\_point": True, "q\_group\_size": 128, "w\_bit": 4, "version": "GEMM" } # Load your tokenizer and model with AutoAWQ tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoAWQForCausalLM.from\_pretrained(model\_path, device\_map="auto", safetensors=True) Then you need to prepare your data for calibaration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example: .. code:: python data = \[\] for msg in messages: msg = c\['messages'\] text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) data.append(text.strip()) where each \`\`msg\`\` is a typical chat message as shown below: .. code:: json \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] Then just run the calibration process by one line of code: .. code:: python model.quantize(tokenizer, quant\_config=quant\_config, calib\_data=data) Finally, save the quantized model: .. code:: python model.save\_quantized(quant\_path, safetensors=True, shard\_size="4GB") tokenizer.save\_pretrained(quant\_path) Then you can obtain your own AWQ quantized model for deployment. Enjoy! --- # Unknown vLLM ===================== We recommend you trying with \`vLLM \`\_\_ for your deployement of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughtput, efficienct management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the \`paper \`\_\_ and \`documentation \`\_\_. Installation ------------ By default, you can install \`\`vLLM\`\` by pip: \`\`pip install vLLM>=0.3.0\`\`, but if you are using CUDA 11.8, check the note in the official document for installation (\`link \`\_\_) for some help. We also advise you to install ray by \`\`pip install ray\`\` for distributed serving. Offline Batched Inference ------------------------- Models supported by Qwen2 codes, e.g., Qwen1.5, are supported by vLLM. The simplest usage of vLLM is offline batched inference as demonstrated below. .. code:: python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat") # Pass the default decoding hyperparameters of Qwen1.5-7B-Chat # max\_tokens is for the maximum length for generation. sampling\_params = SamplingParams(temperature=0.7, top\_p=0.8, repetition\_penalty=1.05, max\_tokens=512) # Input the model name or path. Can be GPTQ or AWQ models. llm = LLM(model="Qwen/Qwen1.5-7B-Chat") # Prepare your prompts prompt = "Tell me something about large language models." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) # generate outputs outputs = llm.generate(\[text\], sampling\_params) # Print the outputs. for output in outputs: prompt = output.prompt generated\_text = output.outputs\[0\].text print(f"Prompt: {prompt!r}, Generated text: {generated\_text!r}") OpenAI-API Compatible API Service --------------------------------- It is easy to build an OpenAI-API compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at \`\`http://localhost:8000\`\`. You can specify the address with \`\`--host\`\` and \`\`--port\`\` arguments. Run the command as shown below: .. code:: bash python -m vllm.entrypoints.openai.api\_server \\ --model Qwen/Qwen1.5-7B-Chat You don’t need to worry about chat template as it by default uses the chat template provided by the tokenizer. Then, you can use the \`create chat interface \`\_\_ to communicate with Qwen: .. code:: bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen1.5-7B-Chat", "messages": \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \] }' or you can use python client with \`\`openai\`\` python package as shown below: .. code:: python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen1.5-7B-Chat", messages=\[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \] ) print("Chat response:", chat\_response) Multi-GPU Distributred Serving ------------------------------ To scale up your serving throughputs, distributed serving helps you by leveraging more GPU devices. Besides, for large models like \`\`Qwen1.5-72B-Chat\`\`, it is impossible to serve it on a single GPU. Here, we demonstrate how to run \`\`Qwen1.5-72B-Chat\`\` with tensor parallelism just by passing in the argument \`\`tensor\_parallel\_size\`\`: .. code:: python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen1.5-72B-Chat", tensor\_parallel\_size=4) You can run multi-GPU serving by passing in the argument \`\`--tensor-parallel-size\`\`: .. code:: bash python -m vllm.entrypoints.api\_server \\ --model Qwen/Qwen1.5-72B-Chat \\ --tensor-parallel-size 4 Serving Quantized Models ------------------------ vLLM supports different types of quantized models, including AWQ, GPTQ, SqueezeLLM, etc. Here we show how to deploy AWQ and GPTQ models. The usage is almost the same as above except for an additional argument for quantization. For example, to run an AWQ model. e.g., \`\`Qwen1.5-7B-Chat-AWQ\`\`: .. code:: python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen1.5-7B-Chat-AWQ", quantization="awq") or GPTQ models like \`\`Qwen1.5-7B-Chat-GPTQ-Int8\`\`: .. code:: python llm = LLM(model="Qwen/Qwen1.5-7B-Chat-GPTQ-Int4", quantization="gptq") Similarly, you can run serving adding the argument \`\`--quantization\`\` as shown below: .. code:: bash python -m vllm.entrypoints.openai.api\_server \\ --model Qwen/Qwen1.5-7B-Chat-AWQ \\ --quantization awq or .. code:: bash python -m vllm.entrypoints.openai.api\_server \\ --model Qwen/Qwen1.5-7B-Chat-GPTQ-Int8 \\ --quantization gptq Additionally, vLLM supports the combination of AWQ or GPTQ models with KV cache quantization, namely FP8 E5M2 KV Cache. For example: .. code:: python llm = LLM(model="Qwen/Qwen1.5-7B-Chat-GPTQ-Int8", quantization="gptq", kv\_cache\_dtype="fp8\_e5m2") .. code:: bash python -m vllm.entrypoints.openai.api\_server \\ --model Qwen/Qwen1.5-7B-Chat-GPTQ-Int8 \\ --quantization gptq \\ --kv-cache-dtype fp8\_e5m2 Troubleshooting --------------- You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. The first one is \`\`--max-model-len\`\`. Our provided default \`\`max\_postiion\_embedding\`\` is \`\`32768\`\` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. Another argument you can pay attention to is \`\`--gpu-memory-utilization\`\`. By default it is \`\`0.9\`\` and you can level it up to tackle the OOM problem. This is also why you find a vLLM service always takes so much memory. --- # Unknown GPTQ ====================== \`GPTQ \`\_\_ is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with transformers and also how to quantize your own model with \`AutoGPTQ \`\_\_. Usage of GPTQ Models with Transformers -------------------------------------- Now, Transformers has officially supported AutoGPTQ, which means that you can directly use the quantized model with Transformers. The following is a very simple code snippet showing how to run \`\`Qwen1.5-7B-Chat-GPTQ-Int8\`\` (note that for each size of Qwen1.5, we provide both Int4 and Int8 quantized models) with the quantized model: .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" # the device to load the model onto model = AutoModelForCausalLM.from\_pretrained( "Qwen/Qwen1.5-7B-Chat-GPTQ-Int8", # the quantized model device\_map="auto" ) tokenizer = AutoTokenizer.from\_pretrained("Qwen/Qwen1.5-7B-Chat-GPTQ-Int8") prompt = "Give me a short introduction to large language model." messages = \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt} \] text = tokenizer.apply\_chat\_template( messages, tokenize=False, add\_generation\_prompt=True ) model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device) generated\_ids = model.generate( model\_inputs.input\_ids, max\_new\_tokens=512 ) generated\_ids = \[ output\_ids\[len(input\_ids):\] for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids) \] response = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[0\] Usage of GPTQ Quantized Models with vLLM ---------------------------------------- vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with \`\`AutoGPTQ\`\` with vLLM. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and \`\`Qwen1.5-7B-Chat-GPTQ-Int8\`\`: .. code:: bash python -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen1.5-7B-Chat-GPTQ-Int8 .. code:: bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen1.5-7B-Chat-GPTQ-Int8", "messages": \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} \], }' or you can use python client with \`\`openai\`\` python package as shown below: .. code:: python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai\_api\_key = "EMPTY" openai\_api\_base = "http://localhost:8000/v1" client = OpenAI( api\_key=openai\_api\_key, base\_url=openai\_api\_base, ) chat\_response = client.chat.completions.create( model="Qwen/Qwen1.5-7B-Chat-GPTQ-Int8", messages=\[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, \] ) print("Chat response:", chat\_response) Quantize Your Own Model with AutoGPTQ ------------------------------------- If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: .. code:: bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . Suppose you have finetuned a model based on \`\`Qwen1.5-7B\`\`, which is named \`\`Qwen1.5-7B-finetuned\`\`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: .. code:: python from auto\_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model\_path = "your\_model\_path" quant\_path = "your\_quantized\_model\_path" quantize\_config = BaseQuantizeConfig( bits=8, # 4 or 8 group\_size=128, damp\_percent=0.01, desc\_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static\_groups=False, sym=True, true\_sequential=True, model\_name\_or\_path=None, model\_file\_base\_name="model" ) max\_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from\_pretrained(model\_path) model = AutoGPTQForCausalLM.from\_pretrained(model\_path, quantize\_config) However, if you would like to load the model on multiple GPUs, you need to use \`\`max\_memory\`\` instead of \`\`device\_map\`\`. Here is an example: .. code:: python model = AutoGPTQForCausalLM.from\_pretrained( model\_path, quantize\_config, max\_memory={i:"20GB" for i in range(4)} ) Then you need to prepare your data for calibaration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example: .. code:: python import torch data = \[\] for msg in messages: text = tokenizer.apply\_chat\_template(msg, tokenize=False, add\_generation\_prompt=False) model\_inputs = tokenizer(\[text\]) input\_ids = torch.tensor(model\_inputs.input\_ids\[:max\_len\], dtype=torch.int) data.append(dict(input\_ids=input\_ids, attention\_mask=input\_ids.ne(tokenizer.pad\_token\_id))) where each \`\`msg\`\` is a typical chat message as shown below: .. code:: json \[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} \] Then just run the calibration process by one line of code: .. code:: python import logging logging.basicConfig( format="%(asctime)s %(levelname)s \[%(name)s\] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache\_examples\_on\_gpu=False) Finally, save the quantized model: .. code:: python model.save\_quantized(quant\_path, use\_safetensors=True) tokenizer.save\_pretrained(quant\_path) It is unfortunate that the \`\`save\_quantized\`\` method does not support sharding. For sharding, you need to load the model and use \`\`save\_pretrained\`\` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! --- # Unknown \# Key Concepts :::{attention} To be updated for Qwen3. ::: ## Qwen Qwen (Chinese: 通义千问; pinyin: \_Tongyi Qianwen\_) is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Both language models and multimodal models are pre-trained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. There is the proprietary version and the open-weight version. The proprietary versions include - Qwen: the language models - Qwen Max - Qwen Plus - Qwen Turbo - Qwen-VL: the vision-language models - Qwen-VL Max - Qwen-VL Plus - Qwen-VL OCR - Qwen-Audio: the audio-language models - Qwen-Audio Turbo - Qwen-Audio ASR You can learn more about them at Alibaba Cloud Model Studio (\[China Site\](https://help.aliyun.com/zh/model-studio/getting-started/models#9f8890ce29g5u) \\\[zh\\\], \[International Site\](https://www.alibabacloud.com/en/product/modelstudio)). The spectrum for the open-weight models spans over - Qwen: the language models - \[Qwen\](https://github.com/QwenLM/Qwen): 1.8B, 7B, 14B, and 72B models - \[Qwen1.5\](https://github.com/QwenLM/Qwen1.5/tree/v1.5): 0.5B, 1.8B, 4B, 14BA2.7B, 7B, 14B, 32B, 72B, and 110B models - \[Qwen2\](https://github.com/QwenLM/Qwen2/tree/v2.0): 0.5B, 1.5B, 7B, 57A14B, and 72B models - \[Qwen2.5\](https://github.com/QwenLM/Qwen2.5/): 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B models - Qwen-VL: the vision-language models - \[Qwen-VL\](https://github.com/QwenLM/Qwen-VL): 7B-based models - \[Qwen2-VL\](https://github.com/QwenLM/Qwen2-VL): 2B, 7B, and 72B-based models - Qwen-Audio: the audio-language models - \[Qwen-Audio\](https://github.com/QwenLM/Qwen-Audio): 7B-based model - \[Qwen2-Audio\](https://github.com/QwenLM/Qwen2-Audio): 7B-based models - Q\*Q: the reasoning models - \[QwQ-Preview\](https://github.com/QwenLM/Qwen2.5/): 32B LLM - \[QVQ-Preview\](https://github.com/QwenLM/Qwen2-VL): 72B VLM - CodeQwen/Qwen-Coder: the language models for coding - \[CodeQwen1.5\](https://github.com/QwenLM/CodeQwen1.5): 7B models - \[Qwen2.5-Coder\](https://github.com/QwenLM/Qwen2.5-Coder): 0.5B, 1.5B, 3B, 7B, 14B, and 32B models - Qwen-Math: the language models for mathematics - \[Qwen2-Math\](https://github.com/QwenLM/Qwen2-Math): 1.5B, 7B, and 72B models - \[Qwen2.5-Math\](https://github.com/QwenLM/Qwen2.5-Math): 1.5B, 7B, and 72B models - Qwen-Math-RM: the reward models for mathematics - \[Qwen2-Math-RM\](https://github.com/QwenLM/Qwen2-Math): 72B models - \[Qwen2.5-Math-RM\](https://github.com/QwenLM/Qwen2.5-Math): 72B models - \[Qwen2.5-Math-PRM\](https://github.com/QwenLM/Qwen2.5-Math): 7B and 72B models \*\*In this document, our focus is Qwen, the language models.\*\* ## Causal Language Models Causal language models, also known as autoregressive language models or decoder-only language models, are a type of machine learning model designed to predict the next token in a sequence based on the preceding tokens. In other words, they generate text one token at a time, using the previously generated tokens as context. The "causal" aspect refers to the fact that the model only considers the past context (the already generated tokens) when predicting the next token, not any future tokens. Causal language models are widely used for various natural language processing tasks involving text completion and generation. They have been particularly successful in generating coherent and contextually relevant text, making them a cornerstone of modern natural language understanding and generation systems. \*\*Takeaway: Qwen models are causal language models suitable for text completion.\*\* :::{dropdown} Learn more about language models They are three main kinds of models that are commonly referred to as language models in deep learning: - Sequence-to-sequence models: T5 and the likes Sequence-to-sequence models use both an encoder to capture the entire input sequence and a decoder to generate an output sequence. They are widely used for tasks like machine translation, text summarization, etc. - Bidirectional models or encoder-only models: BERT and the likes Bidirectional models can access both past and future context in a sequence during training. They cannot generate sequential outputs in real-time due to the need for future context. They are widely used as embedding models and subsequently used for text classification. - Casual language models or decoder-only models: GPT and the likes Causal language models operate unidirectionally in a strictly forward direction, predicting each subsequent word based only on the previous words in the sequence. This unidirectional nature ensures that the model's predictions do not rely on future context, making them suitable for tasks like text completion and generation. ::: ### Pre-training & Base models Base language models are foundational models trained on extensive corpora of text to predict the next word in a sequence. Their main goal is to capture the statistical patterns and structures of language, enabling them to generate coherent and contextually relevant text. These models are versatile and can be adapted to various natural language processing tasks through fine-tuning. While adept at producing fluent text, they may require in-context learning or additional training to follow specific instructions or perform complex reasoning tasks effectively. For Qwen models, the base models are those without "-Instruct" indicators, such as Qwen2.5-7B and Qwen2.5-72B. \*\*Takeaway: Use base models for in-context learning, downstream fine-tuning, etc.\*\* ### Post-training & Instruction-tuned models Instruction-tuned language models are specialized models designed to understand and execute specific instructions in conversational styles. These models are fine-tuned to interpret user commands accurately and can perform tasks such as summarization, translation, and question answering with improved accuracy and consistency. Unlike base models, which are trained on large corpora of text, instruction-tuned models undergo additional training using datasets that contain examples of instructions and their desired outcomes, often in multiple turns. This kind of training makes them ideal for applications requiring targeted functionalities while maintaining the ability to generate fluent and coherent text. For Qwen models, the instruction-tuned models are those with the "-Instruct" suffix, such as Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct. \[^instruct-chat\] \*\*Takeaway: Use instruction-tuned models for conducting tasks in conversations, downstream fine-tuning, etc.\*\* \[^instruct-chat\]: Previously, they are known as the chat models and with the "-Chat" suffix. Starting from Qwen2, the name is changed to follow the common practice. For Qwen, "-Instruct" and "-Chat" should be regarded as synonymous. ## Tokens & Tokenization Tokens represent the fundamental units that models process and generate. They can represent texts in human languages (regular tokens) or represent specific functionality like keywords in programming languages (control tokens \[^special\]). Typically, a tokenizer is used to split text into regular tokens, which can be words, subwords, or characters depending on the specific tokenization scheme employed, and furnish the token sequence with control tokens as needed. The vocabulary size, or the total number of unique tokens a model recognizes, significantly impacts its performance and versatility. Larger language models often use sophisticated tokenization methods to handle the vast diversity of human language while keeping the vocabulary size manageable. Qwen use a relatively large vocabulary of 151,646 tokens in total. \[^special\]: Control tokens can be called special tokens. However, the meaning of special tokens need to be interpreted based on the contexts: special tokens may contain extra regular tokens. \*\*Takeaway: Tokenization method and vocabulary size is important.\*\* ### Byte-level Byte Pair Encoding Qwen adopts a subword tokenization method called Byte Pair Encoding (BPE), which attempts to learn the composition of tokens that can represent the text with the fewest tokens. For example, the string " tokenization" is decomposed as " token" and "ization" (note that the space is part of the token). Especially, the tokenization of Qwen ensures that there is no unknown words and all texts can be transformed to token sequences. There are 151,643 tokens as a result of BPE in the vocabulary of Qwen, which is a large vocabulary efficient for diverse languages. As a rule of thumb, 1 token is 3~4 characters for English texts and 1.5~1.8 characters for Chinese texts. \*\*Takeaway: Qwen processes texts in subwords and there are no unknown words.\*\* :::{dropdown} Learn more about tokenization in Qwen Qwen uses byte-level BPE (BBPE) on UTF-8 encoded texts. It starts by treating each byte as a token and then iteratively merges the most frequent pairs of tokens occurring the texts into larger tokens until the desired vocabulary size is met. In byte-level BPE, minimum 256 tokens are needed to tokenize every piece of text and avoid the out of vocabulary (OOV) problem. In comparison, character-level BPE needs every Unicode character in its vocabulary to avoid OOV and the Unicode Standard contains 154,998 characters as of Unicode Version 16.0. One limitation to keep in mind for byte-level BPE is that the individual tokens in the vocabulary may not be seemingly semantically meaningful or even valid UTF-8 byte sequences, and in certain aspects, they should be viewed as a text compression scheme. ::: ### Control Tokens & Chat Template Control tokens and chat templates both serve as mechanisms to guide the model's behavior and outputs. Control tokens are special tokens inserted into the sequence that signifies meta information. For example, in pre-training, multiple documents may be packed into a single sequence. For Qwen, the control token "<|endoftext|>" is inserted after each document to signify that the document has ended and a new document will proceed. Chat templates provide a structured format for conversational interactions, where predefined placeholders or prompts are used to elicit responses from the model that adhere to a desired dialogue flow or context. Different models may use different kinds of chat template to format the conversations. It is crucial to use the designated one to ensure the precise control over the LLM's generation process. Qwen uses the following format (ChatML\[^chatml\]), making use of control tokens to format each turn in the conversations \`\`\`text <|im\_start|>{{role}} {{content}}<|im\_end|> \`\`\` The user input take the role of \`user\` and the model generation takes the role of \`assistant\`. Qwen also supports the meta message that instruct the model to perform specific actions or generate text with certain characteristics, such as altering tone, style, or content, which takes the role of \`system\` and the content defaults to "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." The following is a full example: \`\`\`text <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user hello<|im\_end|> <|im\_start|>assistant Hello! How can I assist you today? Let me know if you have any questions or need help with something. I'm here to help!<|im\_end|> <|im\_start|>user Explain large language models like I'm 5.<|im\_end|> <|im\_start|>assistant Sure, I'd be happy to explain large language models in a simple way! Imagine that you have a friend who is really good at playing with words. This friend has read so many books and talked to so many people that they know how to use words in all sorts of different ways. They can tell stories, answer questions, and even make up new words if they need to. Now imagine that this friend is actually a computer program, called a "large language model". It's been trained on lots and lots of text, like books, articles, and conversations, so it knows how to use words just like your word-loving friend does. When you ask the model a question or give it a task, it uses all the knowledge it's learned to come up with a response that makes sense. Just like your friend might sometimes make mistakes or say things in a funny way, the large language model isn't perfect either. But it's still really impressive, because it can understand and generate human language in a way that was once thought impossible for machines!<|im\_end|><|endoftext|> \`\`\` Starting from Qwen2.5, the Qwen model family including multimodal and specialized models will use a unified vocabulary, which contains control tokens from all subfamilies. There are 22 control tokens in the vocabulary of Qwen2.5, making the vocabulary size totaling 151,665: - 1 general: \`<|endoftext|>\` - 2 for chat: \`<|im\_start|>\` and \`<|im\_end|>\` - 2 for tool use: \`\` and \`\` - 11 for vision - 6 for coding \*\*Takeaway: Qwen uses ChatML with control tokens for chat template.\*\* \[^chatml\]: For historical reference only, ChatML is first described by the OpenAI Python SDK. The last available version is \[this\](https://github.com/openai/openai-python/blob/v0.28.1/chatml.md). Please also be aware that that document lists use cases intended for OpenAI models. For Qwen2.5 models, please only use as in our guide. ## Length Limit As Qwen models are causal language models, in theory there is only one length limit of the entire sequence. However, since there is often packing in training and each sequence may contain multiple individual pieces of texts. \*\*How long the model can generate or complete ultimately depends on the use case and in that case how long each document (for pre-training) or each turn (for post-training) is in training.\*\* For Qwen2.5, the packed sequence length in training is 32,768 tokens.\[^yarn\] The maximum document length in pre-training is this length. The maximum message length for user and assistant is different in post-training. In general, the assistant message could be up to 8192 tokens. \[^yarn\]: The sequence length can be extended to 131,072 tokens for Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B, and Qwen2.5-72B models with YaRN. Please refer to the model card on how to enable YaRN in vLLM. \*\*Takeaway: Qwen2.5 models can process texts of 32K or 128K tokens and up to 8K tokens can be assistant output.\*\* --- # Unknown \# Key Concepts :::{attention} To be updated for Qwen3. ::: ## Qwen Qwen (Chinese: 通义千问; pinyin: \_Tongyi Qianwen\_) is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Both language models and multimodal models are pre-trained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. There is the proprietary version and the open-weight version. The proprietary versions include - Qwen: the language models - Qwen Max - Qwen Plus - Qwen Turbo - Qwen-VL: the vision-language models - Qwen-VL Max - Qwen-VL Plus - Qwen-VL OCR - Qwen-Audio: the audio-language models - Qwen-Audio Turbo - Qwen-Audio ASR You can learn more about them at Alibaba Cloud Model Studio (\[China Site\](https://help.aliyun.com/zh/model-studio/getting-started/models#9f8890ce29g5u) \\\[zh\\\], \[International Site\](https://www.alibabacloud.com/en/product/modelstudio)). The spectrum for the open-weight models spans over - Qwen: the language models - \[Qwen\](https://github.com/QwenLM/Qwen): 1.8B, 7B, 14B, and 72B models - \[Qwen1.5\](https://github.com/QwenLM/Qwen1.5/tree/v1.5): 0.5B, 1.8B, 4B, 14BA2.7B, 7B, 14B, 32B, 72B, and 110B models - \[Qwen2\](https://github.com/QwenLM/Qwen2/tree/v2.0): 0.5B, 1.5B, 7B, 57A14B, and 72B models - \[Qwen2.5\](https://github.com/QwenLM/Qwen2.5/): 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B models - Qwen-VL: the vision-language models - \[Qwen-VL\](https://github.com/QwenLM/Qwen-VL): 7B-based models - \[Qwen2-VL\](https://github.com/QwenLM/Qwen2-VL): 2B, 7B, and 72B-based models - Qwen-Audio: the audio-language models - \[Qwen-Audio\](https://github.com/QwenLM/Qwen-Audio): 7B-based model - \[Qwen2-Audio\](https://github.com/QwenLM/Qwen2-Audio): 7B-based models - Q\*Q: the reasoning models - \[QwQ-Preview\](https://github.com/QwenLM/Qwen2.5/): 32B LLM - \[QVQ-Preview\](https://github.com/QwenLM/Qwen2-VL): 72B VLM - CodeQwen/Qwen-Coder: the language models for coding - \[CodeQwen1.5\](https://github.com/QwenLM/CodeQwen1.5): 7B models - \[Qwen2.5-Coder\](https://github.com/QwenLM/Qwen2.5-Coder): 0.5B, 1.5B, 3B, 7B, 14B, and 32B models - Qwen-Math: the language models for mathematics - \[Qwen2-Math\](https://github.com/QwenLM/Qwen2-Math): 1.5B, 7B, and 72B models - \[Qwen2.5-Math\](https://github.com/QwenLM/Qwen2.5-Math): 1.5B, 7B, and 72B models - Qwen-Math-RM: the reward models for mathematics - \[Qwen2-Math-RM\](https://github.com/QwenLM/Qwen2-Math): 72B models - \[Qwen2.5-Math-RM\](https://github.com/QwenLM/Qwen2.5-Math): 72B models - \[Qwen2.5-Math-PRM\](https://github.com/QwenLM/Qwen2.5-Math): 7B and 72B models \*\*In this document, our focus is Qwen, the language models.\*\* ## Causal Language Models Causal language models, also known as autoregressive language models or decoder-only language models, are a type of machine learning model designed to predict the next token in a sequence based on the preceding tokens. In other words, they generate text one token at a time, using the previously generated tokens as context. The "causal" aspect refers to the fact that the model only considers the past context (the already generated tokens) when predicting the next token, not any future tokens. Causal language models are widely used for various natural language processing tasks involving text completion and generation. They have been particularly successful in generating coherent and contextually relevant text, making them a cornerstone of modern natural language understanding and generation systems. \*\*Takeaway: Qwen models are causal language models suitable for text completion.\*\* :::{dropdown} Learn more about language models They are three main kinds of models that are commonly referred to as language models in deep learning: - Sequence-to-sequence models: T5 and the likes Sequence-to-sequence models use both an encoder to capture the entire input sequence and a decoder to generate an output sequence. They are widely used for tasks like machine translation, text summarization, etc. - Bidirectional models or encoder-only models: BERT and the likes Bidirectional models can access both past and future context in a sequence during training. They cannot generate sequential outputs in real-time due to the need for future context. They are widely used as embedding models and subsequently used for text classification. - Casual language models or decoder-only models: GPT and the likes Causal language models operate unidirectionally in a strictly forward direction, predicting each subsequent word based only on the previous words in the sequence. This unidirectional nature ensures that the model's predictions do not rely on future context, making them suitable for tasks like text completion and generation. ::: ### Pre-training & Base models Base language models are foundational models trained on extensive corpora of text to predict the next word in a sequence. Their main goal is to capture the statistical patterns and structures of language, enabling them to generate coherent and contextually relevant text. These models are versatile and can be adapted to various natural language processing tasks through fine-tuning. While adept at producing fluent text, they may require in-context learning or additional training to follow specific instructions or perform complex reasoning tasks effectively. For Qwen models, the base models are those without "-Instruct" indicators, such as Qwen2.5-7B and Qwen2.5-72B. \*\*Takeaway: Use base models for in-context learning, downstream fine-tuning, etc.\*\* ### Post-training & Instruction-tuned models Instruction-tuned language models are specialized models designed to understand and execute specific instructions in conversational styles. These models are fine-tuned to interpret user commands accurately and can perform tasks such as summarization, translation, and question answering with improved accuracy and consistency. Unlike base models, which are trained on large corpora of text, instruction-tuned models undergo additional training using datasets that contain examples of instructions and their desired outcomes, often in multiple turns. This kind of training makes them ideal for applications requiring targeted functionalities while maintaining the ability to generate fluent and coherent text. For Qwen models, the instruction-tuned models are those with the "-Instruct" suffix, such as Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct. \[^instruct-chat\] \*\*Takeaway: Use instruction-tuned models for conducting tasks in conversations, downstream fine-tuning, etc.\*\* \[^instruct-chat\]: Previously, they are known as the chat models and with the "-Chat" suffix. Starting from Qwen2, the name is changed to follow the common practice. For Qwen, "-Instruct" and "-Chat" should be regarded as synonymous. ## Tokens & Tokenization Tokens represent the fundamental units that models process and generate. They can represent texts in human languages (regular tokens) or represent specific functionality like keywords in programming languages (control tokens \[^special\]). Typically, a tokenizer is used to split text into regular tokens, which can be words, subwords, or characters depending on the specific tokenization scheme employed, and furnish the token sequence with control tokens as needed. The vocabulary size, or the total number of unique tokens a model recognizes, significantly impacts its performance and versatility. Larger language models often use sophisticated tokenization methods to handle the vast diversity of human language while keeping the vocabulary size manageable. Qwen use a relatively large vocabulary of 151,646 tokens in total. \[^special\]: Control tokens can be called special tokens. However, the meaning of special tokens need to be interpreted based on the contexts: special tokens may contain extra regular tokens. \*\*Takeaway: Tokenization method and vocabulary size is important.\*\* ### Byte-level Byte Pair Encoding Qwen adopts a subword tokenization method called Byte Pair Encoding (BPE), which attempts to learn the composition of tokens that can represent the text with the fewest tokens. For example, the string " tokenization" is decomposed as " token" and "ization" (note that the space is part of the token). Especially, the tokenization of Qwen ensures that there is no unknown words and all texts can be transformed to token sequences. There are 151,643 tokens as a result of BPE in the vocabulary of Qwen, which is a large vocabulary efficient for diverse languages. As a rule of thumb, 1 token is 3~4 characters for English texts and 1.5~1.8 characters for Chinese texts. \*\*Takeaway: Qwen processes texts in subwords and there are no unknown words.\*\* :::{dropdown} Learn more about tokenization in Qwen Qwen uses byte-level BPE (BBPE) on UTF-8 encoded texts. It starts by treating each byte as a token and then iteratively merges the most frequent pairs of tokens occurring the texts into larger tokens until the desired vocabulary size is met. In byte-level BPE, minimum 256 tokens are needed to tokenize every piece of text and avoid the out of vocabulary (OOV) problem. In comparison, character-level BPE needs every Unicode character in its vocabulary to avoid OOV and the Unicode Standard contains 154,998 characters as of Unicode Version 16.0. One limitation to keep in mind for byte-level BPE is that the individual tokens in the vocabulary may not be seemingly semantically meaningful or even valid UTF-8 byte sequences, and in certain aspects, they should be viewed as a text compression scheme. ::: ### Control Tokens & Chat Template Control tokens and chat templates both serve as mechanisms to guide the model's behavior and outputs. Control tokens are special tokens inserted into the sequence that signifies meta information. For example, in pre-training, multiple documents may be packed into a single sequence. For Qwen, the control token "<|endoftext|>" is inserted after each document to signify that the document has ended and a new document will proceed. Chat templates provide a structured format for conversational interactions, where predefined placeholders or prompts are used to elicit responses from the model that adhere to a desired dialogue flow or context. Different models may use different kinds of chat template to format the conversations. It is crucial to use the designated one to ensure the precise control over the LLM's generation process. Qwen uses the following format (ChatML\[^chatml\]), making use of control tokens to format each turn in the conversations \`\`\`text <|im\_start|>{{role}} {{content}}<|im\_end|> \`\`\` The user input take the role of \`user\` and the model generation takes the role of \`assistant\`. Qwen also supports the meta message that instruct the model to perform specific actions or generate text with certain characteristics, such as altering tone, style, or content, which takes the role of \`system\` and the content defaults to "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." The following is a full example: \`\`\`text <|im\_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im\_end|> <|im\_start|>user hello<|im\_end|> <|im\_start|>assistant Hello! How can I assist you today? Let me know if you have any questions or need help with something. I'm here to help!<|im\_end|> <|im\_start|>user Explain large language models like I'm 5.<|im\_end|> <|im\_start|>assistant Sure, I'd be happy to explain large language models in a simple way! Imagine that you have a friend who is really good at playing with words. This friend has read so many books and talked to so many people that they know how to use words in all sorts of different ways. They can tell stories, answer questions, and even make up new words if they need to. Now imagine that this friend is actually a computer program, called a "large language model". It's been trained on lots and lots of text, like books, articles, and conversations, so it knows how to use words just like your word-loving friend does. When you ask the model a question or give it a task, it uses all the knowledge it's learned to come up with a response that makes sense. Just like your friend might sometimes make mistakes or say things in a funny way, the large language model isn't perfect either. But it's still really impressive, because it can understand and generate human language in a way that was once thought impossible for machines!<|im\_end|><|endoftext|> \`\`\` Starting from Qwen2.5, the Qwen model family including multimodal and specialized models will use a unified vocabulary, which contains control tokens from all subfamilies. There are 22 control tokens in the vocabulary of Qwen2.5, making the vocabulary size totaling 151,665: - 1 general: \`<|endoftext|>\` - 2 for chat: \`<|im\_start|>\` and \`<|im\_end|>\` - 2 for tool use: \`\` and \`\` - 11 for vision - 6 for coding \*\*Takeaway: Qwen uses ChatML with control tokens for chat template.\*\* \[^chatml\]: For historical reference only, ChatML is first described by the OpenAI Python SDK. The last available version is \[this\](https://github.com/openai/openai-python/blob/v0.28.1/chatml.md). Please also be aware that that document lists use cases intended for OpenAI models. For Qwen2.5 models, please only use as in our guide. ## Length Limit As Qwen models are causal language models, in theory there is only one length limit of the entire sequence. However, since there is often packing in training and each sequence may contain multiple individual pieces of texts. \*\*How long the model can generate or complete ultimately depends on the use case and in that case how long each document (for pre-training) or each turn (for post-training) is in training.\*\* For Qwen2.5, the packed sequence length in training is 32,768 tokens.\[^yarn\] The maximum document length in pre-training is this length. The maximum message length for user and assistant is different in post-training. In general, the assistant message could be up to 8192 tokens. \[^yarn\]: The sequence length can be extended to 131,072 tokens for Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B, and Qwen2.5-72B models with YaRN. Please refer to the model card on how to enable YaRN in vLLM. \*\*Takeaway: Qwen2.5 models can process texts of 32K or 128K tokens and up to 8K tokens can be assistant output.\*\* --- # Unknown SkyPilot ======== What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the \`instruction \`\_\_ to install Skypilot. Here we provide a simple example of using \`\`pip\`\` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly\[aws,gcp\]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the \`official document \`\_\_ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple silicon Macs docker run --platform linux/amd64 \\ -td --rm --name sky \\ -v "$HOME/.sky:/root/.sky:rw" \\ -v "$HOME/.aws:/root/.aws:rw" \\ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \\ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen1.5-72B-Chat with SkyPilot -------------------------------------- 1. Start serving Qwen1.5-72B-Chat on a single instance with any available GPU in the list specified in \`serve-72b.yaml \`\_\_ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen1.5-72B-Chat", "prompt": "My favorite food is", "max\_tokens": 512 }' | jq -r '.choices\[0\].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen1.5-72B-Chat", "messages": \[ { "role": "system", "content": "You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With \`SkyPilot Serve \`\_\_, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE\_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be \*\*the cheapest available one\*\* on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a \`\`curl -L\`\` command (\`\`-L\`\` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{ "model": "Qwen/Qwen1.5-72B-Chat", "messages": \[ { "role": "system", "content": "You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } \], "max\_tokens": 512 }' | jq -r '.choices\[0\].message.content' Accessing Qwen1.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen1.5 service with GUI by connecting a \`FastChat GUI server \`\_\_ to the endpoint launched above (see \`gui.yaml \`\_\_). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top\_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen1.5 on any cloud. We advise you to read the official doc for more usages and more updates. Check \`this \`\_\_ out! --- # Unknown Text Generation Web UI ======================================== \`Text Generation Web UI \`\_\_ (TGW, or usually referred to "oobabooga") is a popular web UI for text generation, similar to \`AUTOMATIC1111/stable-diffusion-webui \`\_\_. It has multiple interfaces, and supports multiple model backends, including \`Transformers \`\_\_, \`llama.cpp \`\_\_ (through \`llama-cpp-python \`\_\_), \`ExLlamaV2 \`\_\_, \`AutoGPTQ \`\_\_, \`AutoAWQ \`\_\_, \`GPTQ-for-LLaMa \`\_\_, \`CTransformers \`\_\_, \`QuIP# \`\_\_. In this section, we introduce how to run Qwen locally with TGW. Quickstart ---------- The simplest way to run TGW is to use the provided shell scripts in the \`repo \`\_\_. For the first step, clone the repo and enter the directory: .. code:: bash git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui You can directly run the \`\`start\_linux.sh\`\`, \`\`start\_windows.bat\`\`, \`\`start\_macos.sh\`\`, or \`\`start\_wsl.bat\`\` script depending on your OS. Alternatively you can manually install the requirements in your conda environment. Here I take the practice on MacOS as an example. .. code:: bash conda create -n textgen python=3.11 conda activate textgen pip install torch torchvision torchaudio Then you can install the requirements by running \`\`pip install -r\`\` based on your OS, e.g., .. code:: bash pip install -r requirements\_apple\_silicon.txt For \`\`bitsandbytes\`\` and \`\`llama-cpp-python\`\` inside the requirements, I advise you to install them through \`\`pip\`\` directly. However, temporarily please do not use GGUF as the performance with TGW is unsatisfactory. After finishing the installation of required packages, you need to prepare your models by putting the model files or directories in the folder \`\`./models\`\`. For example, you should put the transformers model directory of \`\`Qwen2-7B-Instruct\`\` in the way shown below: :: text-generation-webui ├── models │   ├── Qwen2-7B-Instruct │   │   ├── config.json │   │   ├── generation\_config.json │   │   ├── model-00001-of-00004.safetensor │   │   ├── model-00002-of-00004.safetensor │   │   ├── model-00003-of-00004.safetensor │   │   ├── model-00004-of-00004.safetensor │   │   ├── model.safetensor.index.json │   │   ├── merges.txt │   │   ├── tokenizer\_config.json │   │   └── vocab.json Then you just need to run .. code:: bash python server.py to launch your web UI service. Please browse to :: http://localhost:7860/?\_\_theme=dark and enjoy playing with Qwen in a web UI! Next Step --------- There are a lot more usages in TGW, where you can even enjoy role play, use different types of quantized models, train LoRA, incorporate extensions like stable diffusion and whisper, etc. Go to figure out more advanced usages and apply them to Qwen models! ---