# Table of Contents - [Using in Electron | node-llama-cpp](#using-in-electron-node-llama-cpp) - [Blog | node-llama-cpp](#blog-node-llama-cpp) - [node-llama-cpp | Run AI models locally on your machine](#node-llama-cpp-run-ai-models-locally-on-your-machine) - [node-llama-cpp v3.0 | node-llama-cpp](#node-llama-cpp-v3-0-node-llama-cpp) - [gpt-oss is here! | node-llama-cpp](#gpt-oss-is-here-node-llama-cpp) - [DeepSeek R1 with function calling | node-llama-cpp](#deepseek-r1-with-function-calling-node-llama-cpp) - [Awesome node-llama-cpp | node-llama-cpp](#awesome-node-llama-cpp-node-llama-cpp) - [Metal Support | node-llama-cpp](#metal-support-node-llama-cpp) - [Opening a PR on node-llama-cpp | node-llama-cpp](#opening-a-pr-on-node-llama-cpp-node-llama-cpp) - [Using Batching | node-llama-cpp](#using-batching-node-llama-cpp) - [Using node-llama-cpp in Docker | node-llama-cpp](#using-node-llama-cpp-in-docker-node-llama-cpp) - [CLI | node-llama-cpp](#cli-node-llama-cpp) - [init command | node-llama-cpp](#init-command-node-llama-cpp) - [inspect command | node-llama-cpp](#inspect-command-node-llama-cpp) - [inspect gpu command | node-llama-cpp](#inspect-gpu-command-node-llama-cpp) - [source command | node-llama-cpp](#source-command-node-llama-cpp) - [source clear command | node-llama-cpp](#source-clear-command-node-llama-cpp) - [source build command | node-llama-cpp](#source-build-command-node-llama-cpp) - [Variable: specializedChatWrapperTypeNames | node-llama-cpp](#variable-specializedchatwrappertypenames-node-llama-cpp) - [Variable: templateChatWrapperTypeNames | node-llama-cpp](#variable-templatechatwrappertypenames-node-llama-cpp) - [Variable: resolvableChatWrapperTypeNames | node-llama-cpp](#variable-resolvablechatwrappertypenames-node-llama-cpp) - [Variable: chatWrappers | node-llama-cpp](#variable-chatwrappers-node-llama-cpp) --- # Using in Electron | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/guide/electron#VPContent) On this page Are you an LLM? You can read better optimized documentation at /guide/electron.md for this page in Markdown format Using in Electron [​](https://node-llama-cpp.withcat.ai/guide/electron#using-in-electron) ========================================================================================== `node-llama-cpp` is fully supported in [Electron](https://www.electronjs.org/) , and also includes custom Electron-specific adaptations. You can only use `node-llama-cpp` on the main process in Electron applications. Trying to use `node-llama-cpp` on a renderer process will crash the application. You can scaffold an example Electron app that uses `node-llama-cpp` with complete configuration for packaging and distribution by running the following command: shell npm create node-llama-cpp@latest -- --template electron-typescript-react TIP Even if you intend to integrate `node-llama-cpp` into your existing Electron app, it's still recommended that you scaffold a new Electron project and investigate the `electron-builder.ts` file to see how to configure your existing Electron app to work well with `node-llama-cpp`. Electron Support [​](https://node-llama-cpp.withcat.ai/guide/electron#electron-support) ---------------------------------------------------------------------------------------- In Electron, when there's no binary available for the current platform, `node-llama-cpp` won't build from source by default, since we cannot assume that the user has the necessary build tools installed. You can customize this behavior by using the [`build`](https://node-llama-cpp.withcat.ai/api/type-aliases/LlamaOptions#build) option when calling [`getLlama`](https://node-llama-cpp.withcat.ai/api/functions/getLlama) . When running from an asar archive, building from source is always disabled, since the asar archive is read-only. It's important to make sure that the native binaries are not packed into the asar archive. If you're using the scaffolded Electron app, this is already taken care of. Customizing Prebuilt Binaries [​](https://node-llama-cpp.withcat.ai/guide/electron#customizing-prebuilt-binaries) ------------------------------------------------------------------------------------------------------------------ If you'd like to use `llama.cpp` with custom CMake build options, you need to build all the binaries you want to ship to users before packaging your Electron app. You also need to call [`getLlama`](https://node-llama-cpp.withcat.ai/api/functions/getLlama) with the CMake build options you used to build the binaries, so that `node-llama-cpp` can find them. Cross Compilation [​](https://node-llama-cpp.withcat.ai/guide/electron#cross-compilation) ------------------------------------------------------------------------------------------ Cross packaging from one platform to another is not supported, since binaries for other platforms are not downloaded to your machine when you run `npm install`. Packaging an `arm64` app on an `x64` machine is supported, but packaging an `x64` app on an `arm64` machine is not. GitHub Actions template for cross-compilation yml name: Build on: [push] jobs: build-electron: name: Build Electron app - ${{ matrix.config.name }} runs-on: ${{ matrix.config.os }} strategy: fail-fast: false matrix: config: - name: "Windows" os: windows-2022 - name: "Ubuntu" os: ubuntu-22.04 - name: "macOS" os: macos-13 steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: "20" - name: Install dependencies on Ubuntu if: matrix.config.name == 'Ubuntu' run: | sudo apt-get update sudo apt-get install libarchive-tools rpm sudo snap install snapcraft --classic - name: Install modules run: npm ci - name: Build electron app id: build shell: bash timeout-minutes: 480 run: npm run build - name: Upload artifacts uses: actions/upload-artifact@v4 with: include-hidden-files: true name: "electron-app-${{ matrix.config.name }}" path: "./release" Bundling [​](https://node-llama-cpp.withcat.ai/guide/electron#bundling) ------------------------------------------------------------------------ When bundling your code for Electron using [Electron Vite](https://electron-vite.org/) or Webpack, ensure that `node-llama-cpp` is not bundled, and is instead treated as an external module. Marking `node-llama-cpp` as an external module will prevent its code from being bundled with your application code, and instead, it'll be loaded from the `node_modules` directory at runtime (which should be packed into a `.asar` archive). The file structure of `node-llama-cpp` is crucial for it to function correctly, so bundling it will break its functionality. Moreover, since `node-llama-cpp` includes prebuilt binaries (and also local builds from source), those files must be retained in their original structure for it to work. Electron has [its own bundling solution called ASAR](https://www.electronjs.org/docs/latest/tutorial/asar-archives) that is designed to work with node modules. ASAR retains the original file structure of node modules by packing all the files into a single `.asar` archive file that Electron will read from at runtime like it would from the file system. This method ensures node modules work as intended in Electron applications, even though they are bundled into a single file. Using ASAR is the recommended way to bundle `node-llama-cpp` in your Electron app. If you're using the scaffolded Electron app, this is already taken care of. NOTE We recommend using [Electron Vite](https://electron-vite.org/) over Webpack for your Electron app due to to Vite's speed and Webpack's lack of proper ESM support in the output bundle, which complicates the bundling process. Last edited 8 months agoView full history [`ee94403`](https://github.com/withcatai/node-llama-cpp/commit/ee94403bb1ae7cb8814a459e6db8259da6c2da8a) \-docs: fix electron command ([#442](https://github.com/withcatai/node-llama-cpp/issues/442) )![The avatar of contributor named as Hyeonggeun Yun](https://gravatar.com/avatar/545f70f4653ea9be270c860645041d4f732077c6b08528cd724f425d6859e795?d=retro) on 3/20/2025 [`v3.3.1`](https://github.com/withcatai/node-llama-cpp/releases/tag/v3.3.1) on 12/8/2024 [`97abbca`](https://github.com/withcatai/node-llama-cpp/commit/97abbca063f0ccf1b69607638d42f5ccc6ee1e2d) \-feat: Llama 3.2 3B function calling support ([#386](https://github.com/withcatai/node-llama-cpp/issues/386) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) on 11/27/2024 [`ea12dc5`](https://github.com/withcatai/node-llama-cpp/commit/ea12dc5128fce5e486847be378b676d3bca6a30d) \-feat: chat session response prefix ([#375](https://github.com/withcatai/node-llama-cpp/issues/375) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) on 10/29/2024 [`4b3ad61`](https://github.com/withcatai/node-llama-cpp/commit/4b3ad61637c951b955984d2d22cf97c1ed109b71) \-feat: new docs ([#309](https://github.com/withcatai/node-llama-cpp/issues/309) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) on 9/18/2024 --- # Blog | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/blog/#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /blog/index.md for this page in Markdown format August 9, 2025 [gpt-oss is here!\ ----------------](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) [![node-llama-cpp + gpt-oss](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss/cover.jpg)](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) v3.12 is here, with full support for [`gpt-oss`](https://huggingface.co/openai/gpt-oss-20b) models! [Read more](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) February 21, 2025 [DeepSeek R1 with function calling\ ---------------------------------](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1) [![node-llama-cpp + DeepSeek R1](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1/cover.jpg)](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) v3.6 is here, with full support for [DeepSeek R1](https://github.com/deepseek-ai/DeepSeek-R1) , including function calling! [Read more](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1) September 23, 2024 [node-llama-cpp v3.0\ -------------------](https://node-llama-cpp.withcat.ai/blog/v3) [![Celebrate](https://node-llama-cpp.withcat.ai/blog/v3/cover.jpg)](https://node-llama-cpp.withcat.ai/blog/v3) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) 3.0 is finally here. With [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) , you can run large language models locally on your machine using the power of [`llama.cpp`](https://github.com/ggml-org/llama.cpp) with a simple and easy-to-use API. It includes everything you need, from downloading models, to running them in the most optimized way for your hardware, and integrating them in your projects. [Read more](https://node-llama-cpp.withcat.ai/blog/v3) --- # node-llama-cpp | Run AI models locally on your machine [Skip to content](https://node-llama-cpp.withcat.ai/#VPContent) [gpt-oss is here!](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) node-llama-cppRun AI models locally on your machine =================================================== node.js bindings for llama.cpp, and much more [Get Started](https://node-llama-cpp.withcat.ai/guide/) [API Reference](https://node-llama-cpp.withcat.ai/api/functions/getLlama) [gpt-oss is here!](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss) ![node-llama-cpp Logo](https://node-llama-cpp.withcat.ai/logo.jpg) [🌟\ \ Easy to use\ -----------\ \ Zero-config by default. Works in Node.js, Bun, and Electron. Bootstrap a project with a single command\ \ Learn more](https://node-llama-cpp.withcat.ai/guide/) [🚀\ \ Metal, CUDA and Vulkan support\ ------------------------------\ \ Adapts to your hardware automatically to run models with maximum performance\ \ Learn more](https://node-llama-cpp.withcat.ai/guide/#gpu-support) [📦\ \ Native binaries\ ---------------\ \ Pre-built binaries are provided, with a fallback to building from source without `node-gyp` or Python\ \ Learn more](https://node-llama-cpp.withcat.ai/guide/building-from-source) [Powerful features\ -----------------\ \ Force a model to generate output according to a JSON schema, provide a model with functions it can call on demand, and much more\ \ Learn more](https://node-llama-cpp.withcat.ai/guide/grammar#json-schema) Are you an LLM? View /llms.txt for optimized Markdown documentation, or /llms-full.txt for full documentation bundle Chat with many popular models ----------------------------- Experience the ease of running models on your machine shell npx -y node-llama-cpp chat > To chat with models using a UI, try the [example Electron app](https://github.com/withcatai/node-llama-cpp/releases/latest) Inspect your hardware --------------------- Check out your hardware capabilities shell npx -y node-llama-cpp inspect gpu [Learn more](https://node-llama-cpp.withcat.ai/cli/inspect/gpu) A complete package ------------------ Everything you need to use large language models in your project * [Embedding](https://node-llama-cpp.withcat.ai/guide/embedding) * [Grammar](https://node-llama-cpp.withcat.ai/guide/grammar) * [JSON schema grammar](https://node-llama-cpp.withcat.ai/guide/#chatbot-with-json-schema) * [Function calling](https://node-llama-cpp.withcat.ai/guide/#chatbot-with-json-schema) * [CUDA support](https://node-llama-cpp.withcat.ai/guide/CUDA) * [Metal support](https://node-llama-cpp.withcat.ai/guide/Metal) * [Vulkan support](https://node-llama-cpp.withcat.ai/guide/Vulkan) * [Adapts to your hardware](https://node-llama-cpp.withcat.ai/guide/#gpu-support) * [Model downloader](https://node-llama-cpp.withcat.ai/guide/downloading-models) * [Prebuilt binaries](https://node-llama-cpp.withcat.ai/guide/building-from-source) * [Electron support](https://node-llama-cpp.withcat.ai/guide/electron) * [Prompt preloading](https://node-llama-cpp.withcat.ai/guide/chat-session#preload-prompt) * [Automatic chat wrapper](https://node-llama-cpp.withcat.ai/guide/chat-wrapper#chat-wrappers) * [Template chat wrapper](https://node-llama-cpp.withcat.ai/guide/chat-wrapper#template) * [Text completion](https://node-llama-cpp.withcat.ai/guide/text-completion#complete) * [Fill in the middle (infill)](https://node-llama-cpp.withcat.ai/guide/text-completion#infill) * [Jinja support](https://node-llama-cpp.withcat.ai/guide/chat-wrapper#jinja) * [Smart context shift](https://node-llama-cpp.withcat.ai/guide/chat-wrapper#smart-context-shift) * [Token bias](https://node-llama-cpp.withcat.ai/guide/token-bias) * Windows on Arm support * [Apple Silicon support](https://node-llama-cpp.withcat.ai/guide/Metal) * [Inspect GGUF files](https://node-llama-cpp.withcat.ai/cli/inspect/gguf) * [Custom CMake options](https://node-llama-cpp.withcat.ai/guide/building-from-source#customize-build) * [Automatic batching](https://node-llama-cpp.withcat.ai/guide/batching) * [TypeScript type-safety](https://node-llama-cpp.withcat.ai/api/functions/getLlama) * [LoRA](https://node-llama-cpp.withcat.ai/api/type-aliases/LlamaContextOptions#lora) * [Remote GGUF reader](https://node-llama-cpp.withcat.ai/api/functions/readGgufFileInfo) * [User input safety](https://node-llama-cpp.withcat.ai/guide/llama-text#input-safety-in-node-llama-cpp) * [Token prediction](https://node-llama-cpp.withcat.ai/guide/token-prediction) * [Reranking](https://node-llama-cpp.withcat.ai/guide/embedding#reranking) * [Thought segmentation](https://node-llama-cpp.withcat.ai/guide/chat-session#stream-response-segments) Prompt a model -------------- Integrate `node-llama-cpp` in your codebase and prompt models TypeScript import {fileURLToPath} from "url"; import path from "path"; import {getLlama, LlamaChatSession} from "node-llama-cpp"; const __dirname = path.dirname( fileURLToPath(import.meta.url) ); const llama = await getLlama(); const model = await llama.loadModel({ modelPath: path.join(__dirname, "my-model.gguf") }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence() }); const q1 = "Hi there, how are you?"; console.log("User: " + q1); const a1 = await session.prompt(q1); console.log("AI: " + a1); [Learn more](https://node-llama-cpp.withcat.ai/guide/#usage) Embed documents --------------- Get embedding for a given text TypeScript import {fileURLToPath} from "url"; import path from "path"; import {getLlama} from "node-llama-cpp"; const __dirname = path.dirname( fileURLToPath(import.meta.url) ); const llama = await getLlama(); const model = await llama.loadModel({ modelPath: path.join(__dirname, "my-model.gguf") }); const context = await model.createEmbeddingContext(); const text = "Hello world"; console.log("Text:", text); const embedding = await context.getEmbeddingFor(text); console.log("Embedding vector:", embedding.vector); [Learn more](https://node-llama-cpp.withcat.ai/guide/embedding) Enforce a JSON schema --------------------- Force a model response to follow your JSON schema TypeScript import {fileURLToPath} from "url"; import path from "path"; import {getLlama, LlamaChatSession} from "node-llama-cpp"; const __dirname = path.dirname( fileURLToPath(import.meta.url) ); const llama = await getLlama(); const model = await llama.loadModel({ modelPath: path.join(__dirname, "my-model.gguf") }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence() }); const grammar = await llama.createGrammarForJsonSchema({ type: "object", properties: { positiveWordsInUserMessage: { type: "array", items: { type: "string" } }, userMessagePositivityScoreFromOneToTen: { enum: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] }, nameOfUser: { oneOf: [{\ type: "null"\ }, {\ type: "string"\ }] } } }); const prompt = "Hi there! I'm John. Nice to meet you!"; const res = await session.prompt(prompt, { grammar }); const parsedRes = grammar.parse(res); console.log("User name:", parsedRes.nameOfUser); console.log( "Positive words in user message:", parsedRes.positiveWordsInUserMessage ); console.log( "User message positivity score:", parsedRes.userMessagePositivityScoreFromOneToTen ); [Learn more](https://node-llama-cpp.withcat.ai/guide/#chatbot-with-json-schema) Prompt with function calling ---------------------------- Let a model call functions to retrieve data or perform actions TypeScript import {fileURLToPath} from "url"; import path from "path"; import { getLlama, LlamaChatSession, defineChatSessionFunction } from "node-llama-cpp"; const __dirname = path.dirname( fileURLToPath(import.meta.url) ); const llama = await getLlama(); const model = await llama.loadModel({ modelPath: path.join(__dirname, "my-model.gguf") }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence() }); const fruitPrices: Record = { "apple": "$6", "banana": "$4" }; const functions = { getFruitPrice: defineChatSessionFunction({ description: "Get the price of a fruit", params: { type: "object", properties: { name: { type: "string" } } }, async handler(params) { const name = params.name.toLowerCase(); if (Object.keys(fruitPrices).includes(name)) return { name: name, price: fruitPrices[name] }; return `Unrecognized fruit "${params.name}"`; } }) }; const q1 = "Is an apple more expensive than a banana?"; console.log("User: " + q1); const a1 = await session.prompt(q1, {functions}); console.log("AI: " + a1); [Learn more](https://node-llama-cpp.withcat.ai/guide/#chatbot-with-function-calling) --- # node-llama-cpp v3.0 | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/blog/v3#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /blog/v3.md for this page in Markdown format node-llama-cpp v3.0 [​](https://node-llama-cpp.withcat.ai/blog/v3#node-llama-cpp-v3-0) ======================================================================================= September 23, 2024 ![Celebrate](https://node-llama-cpp.withcat.ai/blog/v3/cover.jpg) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) 3.0 is finally here. With [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) , you can run large language models locally on your machine using the power of [`llama.cpp`](https://github.com/ggml-org/llama.cpp) with a simple and easy-to-use API. It includes everything you need, from downloading models, to running them in the most optimized way for your hardware, and integrating them in your projects. * * * Why `node-llama-cpp`? [​](https://node-llama-cpp.withcat.ai/blog/v3#why-node-llama-cpp) ---------------------------------------------------------------------------------------- You might be wondering, why choose `node-llama-cpp` over using an OpenAI API of a service running on your machine? The answer is simple: simplicity, performance, and flexibility. Let's break it down: ### Simplicity [​](https://node-llama-cpp.withcat.ai/blog/v3#simplicity) To use `node-llama-cpp`, you install it like any other npm package, and you're good to go. To run your project, all you have to do is `npm install` and `npm start`. That's it. No installing additional software on your machine, no setting up API keys or environment variables, no setup process at all. Everything is self-contained in your project, giving you complete control over it. With `node-llama-cpp`, you can run large language models on your machine using Node.js and TypeScript, _without_ any Python at all. Say goodbye to setup headaches, "it works on my machine" issues, and all other Python-related problems. While `llama.cpp` is an amazing project, it's also highly technical and can be challenging for beginners. `node-llama-cpp` bridge that gap, making `llama.cpp` accessible to everyone, regardless of their experience level. ### Performance [​](https://node-llama-cpp.withcat.ai/blog/v3#performance) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) is built on top of [`llama.cpp`](https://github.com/ggml-org/llama.cpp) , a highly optimized C++ library for running large language models. `llama.cpp` supports many compute backends, including Metal, CUDA, and Vulkan. It also uses [Accelerate](https://developer.apple.com/accelerate/) on Mac. `node-llama-cpp` automatically adapts to your hardware and adjusts the default settings to give you the best performance, so you don't _have_ to configure anything to use it. By using `node-llama-cpp` you are essentially running models _inside_ your project. With no overhead of network calls or data serializations, you can more effectively take advantage of the stateful nature of inference operations. For example, you can prompt a model on top of an existing conversation inference state, without re-evaluating the entire history just to process the new prompt. This reduces the time it takes to start generating a response, and makes more efficient use of your resources. If you were using an API, you would have to re-evaluate the entire history every time you prompt the model, or have the API store the state for you, which can use huge amounts of disk space. ### Flexibility [​](https://node-llama-cpp.withcat.ai/blog/v3#flexibility) Since `node-llama-cpp` runs inside your project, you can also deploy it together with your project. You can run models in your [Electron](https://node-llama-cpp.withcat.ai/guide/electron) app without requiring any additional setup on the user's machine. You can build libraries that use large language models and distribute them as npm packages, or deploy self-contained Docker images and run them on any hardware you want. You can use [any model you want](https://node-llama-cpp.withcat.ai/guide/choosing-a-model) , or even create your own and use it with `node-llama-cpp`. Download models [as part of `npm install`](https://node-llama-cpp.withcat.ai/guide/downloading-models) or [on-demand from your code](https://node-llama-cpp.withcat.ai/guide/downloading-models#programmatic) . [Tweak inference settings](https://node-llama-cpp.withcat.ai/guide/chat-session#repeat-penalty) to get better results for your particular use case. `node-llama-cpp` is regularly updated with the latest `llama.cpp` release, but you can also [download and build the latest release](https://node-llama-cpp.withcat.ai/guide/building-from-source#download-new-release) at any time with a single command. The possibilities are endless. You have full control over the models you use, how you use them, and where you use them. You can tailor `node-llama-cpp` to your needs in ways that aren't possible with an OpenAI API (at least not efficiently or easily). Powerful Features [​](https://node-llama-cpp.withcat.ai/blog/v3#powerful-features) ----------------------------------------------------------------------------------- `node-llama-cpp` includes a complete suite of everything you need to use large language models in your projects, with convenient wrappers for popular tasks, such as: * [Enforcing a JSON schema](https://node-llama-cpp.withcat.ai/guide/chat-session#response-json-schema) on the output the model generates * Providing the model with [functions it can call on demand](https://node-llama-cpp.withcat.ai/guide/chat-session#function-calling) to retrieve information or perform actions, even with some models that don't officially support it * [Generating completion](https://node-llama-cpp.withcat.ai/guide/text-completion) for a given text * [Embedding text](https://node-llama-cpp.withcat.ai/guide/embedding) for similarity searches or other tasks * Much more Why Node.js? [​](https://node-llama-cpp.withcat.ai/blog/v3#why-node-js) ------------------------------------------------------------------------ JavaScript is the most popular programming language in the world, and Node.js is the most popular runtime for JavaScript server-side applications. Developers choose Node.js for its versatility, reliability, ease of use, forward compatibility, and the vast ecosystem of npm packages. While Python is currently the go-to language for data science and machine learning, the needs of data scientists differ from those of developers building services and applications. `node-llama-cpp` bridges this gap, making it easier to integrate large language models into Node.js and Electron projects, while focusing on the needs of developers building services and applications. Try It Out [​](https://node-llama-cpp.withcat.ai/blog/v3#try-it-out) --------------------------------------------------------------------- `node-llama-cpp` comes with comprehensive documentation, covering everything from installation to advanced usage. It's beginner-friendly, with explanations for every step of the way for those who are new to the world of large language models, while still being flexible enough to allow advanced usage for those who are more experienced and knowledgeable. Experience the ease of running models on your machine with this single command: shell npx -y node-llama-cpp chat Check out the [getting started guide](https://node-llama-cpp.withcat.ai/guide/) to learn how to use `node-llama-cpp`. Thank You [​](https://node-llama-cpp.withcat.ai/blog/v3#thank-you) ------------------------------------------------------------------- `node-llama-cpp` is only possible thanks to the amazing work done on [`llama.cpp`](https://github.com/ggml-org/llama.cpp) by [Georgi Gerganov](https://github.com/ggerganov) , [Slaren](https://github.com/slaren) and all the contributors from the community. What's next? [​](https://node-llama-cpp.withcat.ai/blog/v3#what-s-next) ------------------------------------------------------------------------ Version 3.0 is a major milestone, but there's plenty more planned for the future. Check out the [roadmap](https://github.com/orgs/withcatai/projects/1) to see what's coming next, and [give `node-llama-cpp` a star on GitHub](https://github.com/withcatai/node-llama-cpp) to support the project. --- # gpt-oss is here! | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /blog/v3.12-gpt-oss.md for this page in Markdown format gpt-oss is here! [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#gpt-oss-is-here) =========================================================================================== August 9, 2025 ![node-llama-cpp + gpt-oss](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss/cover.jpg) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) v3.12 is here, with full support for [`gpt-oss`](https://huggingface.co/openai/gpt-oss-20b) models! * * * gpt-oss [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#gpt-oss) -------------------------------------------------------------------------- [`gpt-oss`](https://huggingface.co/openai/gpt-oss-20b) comes in two flavors: * [`gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) - 21B parameters with 3.6B active parameters * [`gpt-oss-120b`](https://huggingface.co/openai/gpt-oss-120b) - 117B parameters with 5.1B active parameters Here are a few highlights of these models: * Due to the low number of active parameters, these models are very fast * These are reasoning models, and you can adjust their reasoning effort * They are very good at function calling, and are built with agentic capabilities in mind * These models were trained with native MXFP4 precision, so no need to quantize them further. They're small compared to their capabilities already * They are provided with an Apache 2.0 license, so you can use them in your commercial applications Recommended Models [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#recommended-models) ------------------------------------------------------------------------------------------------ Here are some recommended model URIs you can use to try out `gpt-oss` right away: | Model | Size | URI | | --- | --- | --- | | [`gpt-oss-20b`](https://huggingface.co/giladgd/gpt-oss-20b-GGUF) | 12.1GB | `hf:giladgd/gpt-oss-20b-GGUF/gpt-oss-20b.MXFP4.gguf` | | [`gpt-oss-120b`](https://huggingface.co/giladgd/gpt-oss-120b-GGUF) | 63.4GB | `hf:giladgd/gpt-oss-120b-GGUF/gpt-oss-120b.MXFP4-00001-of-00002.gguf` | TIP [Estimate the compatibility](https://node-llama-cpp.withcat.ai/cli/inspect/estimate) of a model with your machine before downloading it: shell npx -y node-llama-cpp inspect estimate `MXFP4` Quantization [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#mxfp4-quantization) -------------------------------------------------------------------------------------------------- You might be used to looking for a `Q4_K_M` quantization because of its good balance between quality and size, and be looking for a `Q4_K_M` quantization of `gpt-oss` models. You don't have to, because these models are already natively provided in a similar quantization format called `MXFP4`. Let's break down what `MXFP4` is: * `MXFP4` stands for Microscaling FP4 (Floating Point, 4-bit). `Q4_K_M` is also a 4-bit quantization. * It's a format what was created and standardized by the Open Compute Project (OCP) in early 2024. OCP is backed by big players like OpenAI, NVIDIA, AMD, Microsoft, and Meta, with the goal of lowering the hardware and compute barriers to running AI models. * Designed to dramatically reduce the memory and compute requirements for training and running AI models, while preserving as much precision as possible. This format was used to train the `gpt-oss` models, so the most precise format of these models is `MXFP4`. Since this is a 4-bit precision format, its size footprint is similar to `Q4_K_M` quantization, but it provides better precision and thus better quality. First class support for `MXFP4` in `llama.cpp` was introduced as part of the `gpt-oss` release. The bottom line is that you don't have to find a `Q4_K_M` quantization of `gpt-oss` models, because the `MXFP4` format is as small, efficient, and fast as `Q4_K_M`, but offers better precision and thus better quality. ### Try It Using the CLI [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#try-it-using-the-cli) To quickly try out [`gpt-oss-20b`](https://huggingface.co/giladgd/gpt-oss-20b-GGUF) , you can use the [CLI `chat` command](https://node-llama-cpp.withcat.ai/cli/chat) : shell npx -y node-llama-cpp chat --prompt "Hi there" hf:giladgd/gpt-oss-20b-GGUF/gpt-oss-20b.MXFP4.gguf `thought` Segments [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#thought-segments) ---------------------------------------------------------------------------------------------- Since `gpt-oss` models are reasoning models, they generate thoughts as part of their response. These thoughts are useful for debugging and understanding the model's reasoning process, and can be used to iterate on the system prompt and inputs you provide to the model to improve its responses. However, OpenAI [emphasizes](https://openai.com/index/chain-of-thought-monitoring/#:~:text=leaving%20CoTs%20unrestricted%20may%20make%20them%20unfit%20to%20be%20shown%20to%20end%2Dusers%2C%20as%20they%20might%20violate%20some%20misuse%20policies) that the thoughts generated by these models may not be safe to show to end users as they are unrestricted and might include sensitive information, uncontained language, hallucinations, or other issues. Thus, OpenAI recommends not showing these to users without further filtering, moderation or summarization. Check out the [segment streaming example](https://node-llama-cpp.withcat.ai/guide/chat-session#stream-response-segments) to learn how to use segments. `comment` Segments [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#comment-segments) ---------------------------------------------------------------------------------------------- `gpt-oss` models output "preamble" messages in their response; these are segmented as a new `comment` segment in the model's response. The model might choose to generate those segments to inform the user about the functions it's about to call. For example, when it plans to use multiple functions, it may generate a plan in advance. These are intended for the user to see, but not as part of the main response. Check out the [segment streaming example](https://node-llama-cpp.withcat.ai/guide/chat-session#stream-response-segments) to learn how to use segments. Experiment with `comment` segments The [Electron app template](https://node-llama-cpp.withcat.ai/guide/electron) has been updated to properly segment comments in the response. Try it out by downloading the latest build [from GitHub](https://github.com/withcatai/node-llama-cpp/releases/latest) , or by [scaffolding a new project](https://node-llama-cpp.withcat.ai/guide/#scaffold-new-project) based on the Electron template: shell npm create node-llama-cpp@latest Customizing gpt-oss [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#customizing-gpt-oss) -------------------------------------------------------------------------------------------------- You can adjust `gpt-oss`'s responses by configuring the options of [`HarmonyChatWrapper`](https://node-llama-cpp.withcat.ai/api/classes/HarmonyChatWrapper) : typescript import { getLlama, resolveModelFile, LlamaChatSession, HarmonyChatWrapper } from "node-llama-cpp"; const modelUri = "hf:giladgd/gpt-oss-20b-GGUF/gpt-oss-20b.MXFP4.gguf"; const llama = await getLlama(); const model = await llama.loadModel({ modelPath: await resolveModelFile(modelUri) }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence(), chatWrapper: new HarmonyChatWrapper({ modelIdentity: "You are ChatGPT, a large language model trained by OpenAI.", reasoningEffort: "high" }) }); const q1 = "What is the weather like in SF?"; console.log("User: " + q1); const a1 = await session.prompt(q1); console.log("AI: " + a1); ### Using Function Calling [​](https://node-llama-cpp.withcat.ai/blog/v3.12-gpt-oss#using-function-calling) `gpt-oss` models have great support for function calling. However, these models don't support parallel function calling, so only one function will be called at a time. typescript import { getLlama, resolveModelFile, LlamaChatSession, defineChatSessionFunction } from "node-llama-cpp"; const modelUri = "hf:giladgd/gpt-oss-20b-GGUF/gpt-oss-20b.MXFP4.gguf"; const llama = await getLlama(); const model = await llama.loadModel({ modelPath: await resolveModelFile(modelUri) }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence() }); const functions = { getCurrentWeather: defineChatSessionFunction({ description: "Gets the current weather in the provided location.", params: { type: "object", properties: { location: { type: "string", description: "The city and state, e.g. San Francisco, CA" }, format: { enum: ["celsius", "fahrenheit"] } } }, handler({location, format}) { console.log(`Getting current weather for "${location}" in ${format}`); return { // simulate a weather API response temperature: format === "celsius" ? 20 : 68, format }; } }) }; const q1 = "What is the weather like in SF?"; console.log("User: " + q1); const a1 = await session.prompt(q1, {functions}); console.log("AI: " + a1); --- # DeepSeek R1 with function calling | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /blog/v3.6-deepseek-r1.md for this page in Markdown format DeepSeek R1 with function calling [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#deepseek-r1-with-function-calling) ================================================================================================================================= February 21, 2025 ![node-llama-cpp + DeepSeek R1](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1/cover.jpg) [`node-llama-cpp`](https://node-llama-cpp.withcat.ai/) v3.6 is here, with full support for [DeepSeek R1](https://github.com/deepseek-ai/DeepSeek-R1) , including function calling! * * * Function Calling [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#function-calling) ----------------------------------------------------------------------------------------------- `node-llama-cpp` includes [many tricks](https://node-llama-cpp.withcat.ai/guide/function-calling) used to make function calling work with most models. This release includes special adaptations for DeepSeek R1 to improve function calling performance and stability. Here's a basic example of function calling with DeepSeek R1: typescript import {fileURLToPath} from "url"; import path from "path"; import { getLlama, LlamaChatSession, defineChatSessionFunction, resolveModelFile } from "node-llama-cpp"; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const modelsDir = path.join(__dirname, "..", "models"); const modelUri = "hf:mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q4_K_M"; const llama = await getLlama(); const model = await llama.loadModel({ modelPath: await resolveModelFile(modelUri, modelsDir) }); const context = await model.createContext(); const session = new LlamaChatSession({ contextSequence: context.getSequence() }); const fruitPrices: Record = { "apple": "$6", "banana": "$4" }; const functions = { getFruitPrice: defineChatSessionFunction({ description: "Get the price of a fruit", params: { type: "object", properties: { name: { type: "string" } } }, async handler(params) { const name = params.name.toLowerCase(); if (Object.keys(fruitPrices).includes(name)) return { name: name, price: fruitPrices[name] }; return `Unrecognized fruit "${params.name}"`; } }) }; const q1 = "Is an apple more expensive than a banana?"; console.log("User: " + q1); const a1 = await session.prompt(q1, {functions}); console.log("AI: " + a1.trim()); Recommended Models [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#recommended-models) --------------------------------------------------------------------------------------------------- Here are some recommended model URIs you can use to try out DeepSeek R1 with function calling. | Model | Size | URI | | --- | --- | --- | | [DeepSeek R1 Distill Qwen 7B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF) | 4.68GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q4_K_M` | | [DeepSeek R1 Distill Qwen 14B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-14B-GGUF) | 8.99GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M` | | [DeepSeek R1 Distill Qwen 32B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-GGUF) | 19.9GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M` | > The 7B model works well with function calling in the first prompt, but tends to deteriorate in subsequent queries. > Use a larger model for better performance with multiple prompts. TIP Estimate the compatibility of a model with your machine before downloading it using the [`inspect estimate`](https://node-llama-cpp.withcat.ai/cli/inspect/estimate) command: shell npx -y node-llama-cpp inspect estimate ### Try It Using the CLI [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#try-it-using-the-cli) To try out function calling with a given model using the CLI, you can use the [`chat` command](https://node-llama-cpp.withcat.ai/cli/chat) with the `--ef` flag to provide the model with date and time functions: shell npx -y node-llama-cpp chat --ef --prompt "What is the time?" Chain of Thought Segmentation [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#chain-of-thought-segmentation) ------------------------------------------------------------------------------------------------------------------------- The thoughts generated by a reasoning model are now [separated into `thought` segments](https://node-llama-cpp.withcat.ai/guide/chat-session#stream-response-segments) in the response, so you can choose whether to use them or not. By default, the [`.prompt(...)`](https://node-llama-cpp.withcat.ai/api/classes/LlamaChatSession#prompt) method returns only the main response, without any `thought` segments. Use the [`.promptWithMeta(...)`](https://node-llama-cpp.withcat.ai/api/classes/LlamaChatSession#promptwithmeta) method to get the full response. You can use the new [`onResponseChunk`](https://node-llama-cpp.withcat.ai/api/type-aliases/LLamaChatPromptOptions#onresponsechunk) option to [stream `thought` segments as they are being generated](https://node-llama-cpp.withcat.ai/guide/chat-session#stream-response-segments) . Electron App Template [​](https://node-llama-cpp.withcat.ai/blog/v3.6-deepseek-r1#electron-app-template) --------------------------------------------------------------------------------------------------------- The [Electron app template](https://node-llama-cpp.withcat.ai/guide/electron) has been updated to properly segment the thoughts in the response. Try it out by downloading the latest build [from GitHub](https://github.com/withcatai/node-llama-cpp/releases/latest) , or by [scaffolding a new project](https://node-llama-cpp.withcat.ai/guide/#scaffold-new-project) based on the Electron template: shell npm create node-llama-cpp@latest --- # Awesome node-llama-cpp | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/guide/awesome#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /guide/awesome.md for this page in Markdown format Awesome `node-llama-cpp` [​](https://node-llama-cpp.withcat.ai/guide/awesome#awesome-node-llama-cpp) ===================================================================================================== 😎 Awesome projects that use `node-llama-cpp`. Open Source [​](https://node-llama-cpp.withcat.ai/guide/awesome#open-source) ----------------------------------------------------------------------------- * [CatAI](https://github.com/withcatai/catai) - a simplified AI assistant API for Node.js, with REST API support License MIT * [Manzoni](https://manzoni.app/) ([GitHub](https://github.com/gems-platforms/manzoni-app) ) - a text editor running local LLMs License AGPL-3.0 * [Clippy](https://felixrieseberg.github.io/clippy/) ([GitHub](https://github.com/felixrieseberg/clippy) ) - Clippy, resurrected from the 1990s, now with some AI License MIT Proprietary [​](https://node-llama-cpp.withcat.ai/guide/awesome#proprietary) ----------------------------------------------------------------------------- * [BashBuddy](https://bashbuddy.run/) ([GitHub](https://github.com/wosherco/bashbuddy) ) - write bash commands with natural language Partially open source [Source available](https://github.com/wosherco/bashbuddy/blob/main/LICENSE.md) * [nutshell](https://withnutshell.com/) - Private AI meeting notes processed completely on your device * * * > To add a project to this list, [open a PR](https://github.com/withcatai/node-llama-cpp/edit/master/docs/guide/awesome.md) > . > > To have a project listed here, it should clearly state that it uses `node-llama-cpp`. Last edited 11 months agoView full history [`v3.3.1`](https://github.com/withcatai/node-llama-cpp/releases/tag/v3.3.1) [`97abbca`](https://github.com/withcatai/node-llama-cpp/commit/97abbca063f0ccf1b69607638d42f5ccc6ee1e2d) \-feat: Llama 3.2 3B function calling support ([#386](https://github.com/withcatai/node-llama-cpp/issues/386) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) [`4b3ad61`](https://github.com/withcatai/node-llama-cpp/commit/4b3ad61637c951b955984d2d22cf97c1ed109b71) \-feat: new docs ([#309](https://github.com/withcatai/node-llama-cpp/issues/309) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) --- # Metal Support | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/guide/Metal#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /guide/Metal.md for this page in Markdown format Metal Support [​](https://node-llama-cpp.withcat.ai/guide/Metal#metal-support) =============================================================================== > Metal is a low-level 3D graphics and compute API created by Apple for Apple platforms Metal support is enabled by default on macOS on Apple Silicon Macs, and is disabled by default on Intel Macs. The pre-built binaries of `node-llama-cpp` for macOS are built with Metal support enabled for Apple Silicon Macs, and when building from source on macOS on Apple Silicon Macs, Metal support is enabled by default. `llama.cpp` doesn't support Metal well on Intel Macs, so it is disabled by default on those machines. [Accelerate framework](https://developer.apple.com/accelerate/) is always enabled on Mac. Toggling Metal Support [​](https://node-llama-cpp.withcat.ai/guide/Metal#building) ----------------------------------------------------------------------------------- ### Prerequisites [​](https://node-llama-cpp.withcat.ai/guide/Metal#prerequisites) * [`cmake-js` dependencies](https://github.com/cmake-js/cmake-js#:~:text=projectRoot/build%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5Bstring%5D-,Requirements%3A,-CMake) * [CMake](https://cmake.org/download/) 3.26 or higher (optional, recommended if you have build issues) ### Building `node-llama-cpp` With Metal Support Disabled [​](https://node-llama-cpp.withcat.ai/guide/Metal#building-node-llama-cpp-with-metal-support-disabled) Run this command inside of your project: shell npx --no node-llama-cpp source download --gpu false > If `cmake` is not installed on your machine, `node-llama-cpp` will automatically download `cmake` to an internal directory and try to use it to build `llama.cpp` from source. ### Building `node-llama-cpp` With Metal Support Enabled [​](https://node-llama-cpp.withcat.ai/guide/Metal#building-node-llama-cpp-with-metal-support-enabled) Run this command inside of your project: shell npx --no node-llama-cpp source download --gpu metal > If `cmake` is not installed on your machine, `node-llama-cpp` will automatically download `cmake` to an internal directory and try to use it to build `llama.cpp` from source. Last edited 11 months agoView full history [`v3.3.1`](https://github.com/withcatai/node-llama-cpp/releases/tag/v3.3.1) [`7805fd5`](https://github.com/withcatai/node-llama-cpp/commit/7805fd583da98a3c2cab2ec7e4dba264bf1ed83f) \-docs: improve documentation ([#324](https://github.com/withcatai/node-llama-cpp/issues/324) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) [`4b3ad61`](https://github.com/withcatai/node-llama-cpp/commit/4b3ad61637c951b955984d2d22cf97c1ed109b71) \-feat: new docs ([#309](https://github.com/withcatai/node-llama-cpp/issues/309) )[![The avatar of contributor named as Gilad S.](https://github.com/giladgd.png)](https://github.com/giladgd) [`5fcdf9b`](https://github.com/withcatai/node-llama-cpp/commit/5fcdf9b1cc809ef8f93b97969d305e1613f12c8e) \-feat: function calling ([#139](https://github.com/withcatai/node-llama-cpp/issues/139) )![The avatar of contributor named as Gilad S](https://gravatar.com/avatar/ca00745a6ad53b4c8686895c1b56e12efb417d1121234ce8d1666067677425ef?d=retro) --- # Opening a PR on node-llama-cpp | node-llama-cpp [Skip to content](https://node-llama-cpp.withcat.ai/guide/contributing#VPContent) Return to top Are you an LLM? You can read better optimized documentation at /guide/contributing.md for this page in Markdown format Opening a PR on `node-llama-cpp` [​](https://node-llama-cpp.withcat.ai/guide/contributing#opening-a-pr-on-node-llama-cpp) ========================================================================================================================== This document describes the guidelines of how to open a PR on the `node-llama-cpp` project. Development [​](https://node-llama-cpp.withcat.ai/guide/contributing#development) ---------------------------------------------------------------------------------- To set up your development environment, read the [development guide](https://node-llama-cpp.withcat.ai/guide/development) . Commit Message Guidelines [​](https://node-llama-cpp.withcat.ai/guide/contributing#commit) ------------------------------------------------------------------------------------------- This repository has very precise rules over how git commit messages can be formatted. This leads to **more readable messages** that are easy to follow when looking through the **project history**. But also, git commit messages as used to **generate changelog**. ### Commit Message Format [​](https://node-llama-cpp.withcat.ai/guide/contributing#commit-message-format) Each commit message consists of a **header**, a **body** and a **footer**. The header has a special format that includes a **type** and a **subject**: :