# Table of Contents - [1. NVIDIA GPU Accelerated Computing on WSL 2 — CUDA on WSL 12.8 documentation](#1-nvidia-gpu-accelerated-computing-on-wsl-2-cuda-on-wsl-12-8-documentation) - [Contents — CUDA on WSL 12.8 documentation](#contents-cuda-on-wsl-12-8-documentation) - [CUDA Toolkit Documentation 12.8 Update 1](#cuda-toolkit-documentation-12-8-update-1) - [1. NVIDIA GPU Accelerated Computing on WSL 2 — CUDA on WSL 12.8 documentation](#1-nvidia-gpu-accelerated-computing-on-wsl-2-cuda-on-wsl-12-8-documentation) - [Installing Docker and The Docker Utility Engine for NVIDIA GPUs — NVIDIA AI Enterprise: VMware Deployment Guide](#installing-docker-and-the-docker-utility-engine-for-nvidia-gpus-nvidia-ai-enterprise-vmware-deployment-guide) - [Unknown](#unknown) - [1. CUDA 12.8 Update 1 Release Notes — Release Notes 12.8 documentation](#1-cuda-12-8-update-1-release-notes-release-notes-12-8-documentation) - [1. Introduction — Installation Guide for Linux 12.8 documentation](#1-introduction-installation-guide-for-linux-12-8-documentation) - [1. Introduction — Installation Guide Windows 12.8 documentation](#1-introduction-installation-guide-windows-12-8-documentation) - [1. Introduction — Quick Start Guide 12.8 documentation](#1-introduction-quick-start-guide-12-8-documentation) - [1. CUDA 12.8 Features — CUDA Features Archive 12.8 documentation](#1-cuda-12-8-features-cuda-features-archive-12-8-documentation) - [1. License Agreement for NVIDIA Software Development Kits — EULA](#1-license-agreement-for-nvidia-software-development-kits-eula) - [1. Maxwell Compatibility — Maxwell Compatibility Guide 12.8 documentation](#1-maxwell-compatibility-maxwell-compatibility-guide-12-8-documentation) - [1. Turing Compatibility — Turing Compatibility Guide 12.8 documentation](#1-turing-compatibility-turing-compatibility-guide-12-8-documentation) - [1. Volta Compatibility — Volta Compatibility Guide 12.8 documentation](#1-volta-compatibility-volta-compatibility-guide-12-8-documentation) - [1. NVIDIA Ampere GPU Architecture Compatibility — NVIDIA Ampere Compatibility Guide 12.8 documentation](#1-nvidia-ampere-gpu-architecture-compatibility-nvidia-ampere-compatibility-guide-12-8-documentation) - [1. NVIDIA Ada GPU Architecture Compatibility — Ada Compatibility Guide 12.8 documentation](#1-nvidia-ada-gpu-architecture-compatibility-ada-compatibility-guide-12-8-documentation) - [1. Pascal Tuning Guide — Pascal Tuning Guide 12.8 documentation](#1-pascal-tuning-guide-pascal-tuning-guide-12-8-documentation) - [1. Hopper Architecture Compatibility — Hopper Compatibility Guide 12.8 documentation](#1-hopper-architecture-compatibility-hopper-compatibility-guide-12-8-documentation) - [1. Turing Tuning Guide — Turing Tuning Guide 12.8 documentation](#1-turing-tuning-guide-turing-tuning-guide-12-8-documentation) - [1. Volta Tuning Guide — Volta Tuning Guide 12.8 documentation](#1-volta-tuning-guide-volta-tuning-guide-12-8-documentation) - [1. NVIDIA Ampere GPU Architecture Tuning Guide — NVIDIA Ampere Tuning Guide 12.8 documentation](#1-nvidia-ampere-gpu-architecture-tuning-guide-nvidia-ampere-tuning-guide-12-8-documentation) - [1. NVIDIA Hopper Tuning Guide — Hopper Tuning Guide 12.8 documentation](#1-nvidia-hopper-tuning-guide-hopper-tuning-guide-12-8-documentation) - [1. NVIDIA Ada GPU Architecture Tuning Guide — Ada Tuning Guide 12.8 documentation](#1-nvidia-ada-gpu-architecture-tuning-guide-ada-tuning-guide-12-8-documentation) - [1. NVIDIA Blackwell Tuning Guide — Blackwell Tuning Guide 12.8 documentation](#1-nvidia-blackwell-tuning-guide-blackwell-tuning-guide-12-8-documentation) - [NVIDIA Video Decoder (NVCUVID)](#nvidia-video-decoder-nvcuvid-) - [1. Preface — CUDA C++ Best Practices Guide 12.8 documentation](#1-preface-cuda-c-best-practices-guide-12-8-documentation) - [1. Pascal Compatibility — Pascal Compatibility Guide 12.8 documentation](#1-pascal-compatibility-pascal-compatibility-guide-12-8-documentation) - [1. Blackwell Architecture Compatibility — Blackwell Compatibility Guide 12.8 documentation](#1-blackwell-architecture-compatibility-blackwell-compatibility-guide-12-8-documentation) - [1. Maxwell Tuning Guide — Maxwell Tuning Guide 12.8 documentation](#1-maxwell-tuning-guide-maxwell-tuning-guide-12-8-documentation) - [CUDA Driver API :: CUDA Toolkit Documentation](#cuda-driver-api-cuda-toolkit-documentation) - [CUDA Runtime API :: CUDA Toolkit Documentation](#cuda-runtime-api-cuda-toolkit-documentation) - [CUDA Math API Reference Manual — CUDA Math API Reference Manual 12.8 documentation](#cuda-math-api-reference-manual-cuda-math-api-reference-manual-12-8-documentation) - [1. Introduction — NVBLAS 12.8 documentation](#1-introduction-nvblas-12-8-documentation) - [1. Introduction — nvJPEG 12.8 documentation](#1-introduction-nvjpeg-12-8-documentation) - [1. Introduction — cuFFT 12.8 documentation](#1-introduction-cufft-12-8-documentation) - [cuDLA API :: CUDA Toolkit Documentation](#cudla-api-cuda-toolkit-documentation) - [1. GDS cuFile API Reference — GDS cuFile API Reference](#1-gds-cufile-api-reference-gds-cufile-api-reference) - [cuRAND :: CUDA Toolkit Documentation](#curand-cuda-toolkit-documentation) - [NVIDIA 2D Image and Signal Processing Performance Primitives (NPP) — npp 12.8 documentation](#nvidia-2d-image-and-signal-processing-performance-primitives-npp-npp-12-8-documentation) - [1. Introduction — nvFatbin 12.8 documentation](#1-introduction-nvfatbin-12-8-documentation) - [1. Introduction — nvJitLink 12.8 documentation](#1-introduction-nvjitlink-12-8-documentation) --- # 1. NVIDIA GPU Accelerated Computing on WSL 2 — CUDA on WSL 12.8 documentation * [](../index.html) » * 1\. NVIDIA GPU Accelerated Computing on WSL 2 * v12.8 | [PDF](../pdf/CUDA_on_WSL_User_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA on WSL User Guide The guide for using NVIDIA CUDA on Windows Subsystem for Linux. 1\. NVIDIA GPU Accelerated Computing on WSL 2[](#nvidia-gpu-accelerated-computing-on-wsl-2 "Permalink to this headline") ========================================================================================================================== WSL or Windows Subsystem for Linux is a Windows feature that enables users to run native Linux applications, containers and command-line tools directly on Windows 11 and later OS builds. CUDA support in this user guide is specifically for WSL 2, which is the second generation of WSL that offers the following benefits * Linux applications can run as is in WSL 2. WSL 2 is characteristically a VM with a Linux WSL Kernel in it that provides full compatibility with mainstream Linux kernel allowing support for native Linux applications including popular Linux distros. * Faster file system support and that’s more performant. * WSL 2 is tightly integrated with the Microsoft Windows operating system, which allows it to run Linux applications alongside and even interop with other Windows desktop and modern store apps. For the rest of this user guide, WSL and WSL 2 may be used interchangeably. Typically, developers working across both Linux and Windows environments have a very disruptive workflow. They either have to: * Use different systems for Linux and Windows, or * Dual Boot i.e. install Linux and Windows in separate partitions on the same or different hard disks on the system and boot to the OS of choice. In both cases, developers have to stop all the work and then switch the system or reboot. Also this has historically restricted the development of seamless, well integrated tools and software systems across two dominant ecosystems. WSL enables users to have a seamless transition across the two environments without the need for a resource intensive traditional virtual machine and to improve productivity and develop using tools and integrate their workflow. More importantly WSL 2 enables applications that were hitherto only available on Linux to be available on Windows. WSL 2 support for GPU allows for these applications to benefit from GPU accelerated computing and expands the domain of applications that can be developed on WSL 2. With NVIDIA CUDA support for WSL 2, developers can leverage NVIDIA GPU accelerated computing technology for data science, machine learning and inference on Windows through WSL. GPU acceleration also serves to bring down the performance overhead of running an application inside a WSL like environment close to near-native by being able to pipeline more parallel work on the GPU with less CPU intervention. NVIDIA driver support for WSL 2 includes not only CUDA but also DirectX and Direct ML support. For some helpful examples, see [https://docs.microsoft.com/en-us/windows/win32/direct3d12/gpu-tensorflow-wsl](https://docs.microsoft.com/en-us/windows/win32/direct3d12/gpu-tensorflow-wsl) . WSL 2 is a key enabler in making GPU acceleration to be seamlessly shared between Windows and Linux applications on the same system a reality. This offers flexibility and versatility while also serving to open up GPU accelerated computing by making it more accessible. [![Illustration of the possibilities with NVIDIA CUDA software stack on WSL 2](_images/wsl-launch-upt-0625-rz.png)](_images/wsl-launch-upt-0625-rz.png) Figure 1. Illustration of the possibilities with NVIDIA CUDA software stack on WSL 2[](#gpu-accelerated-computing-cuda-wsl-stack "Permalink to this image") This document describes a workflow for getting started with running CUDA applications or containers in a WSL 2 environment. 1.1. NVIDIA Compute Software Support on WSL 2[](#nvidia-compute-software-support-on-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- This table captures the readiness and suggested software versions for NVIDIA software stack for WSL 2. | Package | Suggested Versions | Installation | | --- | --- | --- | | NVIDIA Windows Driver x86 | Use the latest Windows x86 production driver. R495 and later windows will have CUDA support for WSL 2. NVIDIA-SMI will have a Limited Feature Set on WSL 2.

Legacy CUDA IPC APIs are support from R510. | Windows x86 drivers can be directly downloaded from [https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx)
for WSL 2 support on Pascal or later GPUs. | | Docker support | Supported.

NVIDIA Container Toolkit - Minimum versions - v2.6.0 with libnvidia-container - 1.5.1+

CLI and Docker Desktop Supported. | Refer to [https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html](https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html)
. | | CUDA Toolkit and CUDA Developer Tools | Preview Support

Compute Sanitizer - Pascal and later

Nsight Systems CLI, and CUPTI (Trace) - Volta and later

Developer tools - Debuggers - Pascal and later (Using driver r535+)

Developer tools - Profilers - Volta and later (Using Windows 10 OS build 19044+ with driver r545+ or using Windows 11 with driver r525+ ) | Latest Linux CUDA toolkit package - WSL-Ubuntu from 12.x releases can be downloaded from [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads)
. | | RAPIDS | 22.04 or later 1.10 - Experimental Support for single GPU. | [https://docs.rapids.ai/notices/rgn0024/](https://docs.rapids.ai/notices/rgn0024/) | | NCCL | 2.12 or later 1.4+ | Refer to the [NCCL Installation guide for Linux x86](https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html#down)
. | 2\. Getting Started with CUDA on WSL 2[](#getting-started-with-cuda-on-wsl-2 "Permalink to this headline") ============================================================================================================ To get started with running CUDA on WSL, complete these steps in order: 2.1. Step 1: Install NVIDIA Driver for GPU Support[](#step-1-install-nvidia-driver-for-gpu-support "Permalink to this headline") ---------------------------------------------------------------------------------------------------------------------------------- * Install NVIDIA GeForce Game Ready or NVIDIA RTX Quadro Windows 11 display driver on your system with a compatible GeForce or NVIDIA RTX/Quadro card from [https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx) . Refer to the system requirements in the Appendix.) > Note > > **This is the only driver you need to install. Do not install any Linux display driver in WSL.** 2.2. Step 2: Install WSL 2[](#step-2-install-wsl-2 "Permalink to this headline") ---------------------------------------------------------------------------------- 1. Launch your preferred Windows Terminal / Command Prompt / Powershell and install WSL: wsl.exe --install 2. Ensure you have the latest WSL kernel: wsl.exe --update 2.3. Step 3: Set Up a Linux Development Environment[](#step-3-set-up-a-linux-development-environment "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------ From a Windows terminal, enter WSL: C:\\> wsl.exe The default distro is Ubuntu. To update the distro to your favorite distro from the command line and to review other WSL commands, refer to the following resources: * [https://docs.microsoft.com/en-us/windows/wsl/install](https://docs.microsoft.com/en-us/windows/wsl/install) * [https://docs.microsoft.com/en-us/windows/wsl/basic-commands](https://docs.microsoft.com/en-us/windows/wsl/basic-commands) From this point you should be able to run any existing Linux application which requires CUDA. Do not install any driver within the WSL environment. For building a CUDA application, you will need CUDA Toolkit. Read the next section for further information. 3\. CUDA Support for WSL 2[](#cuda-support-for-wsl-2 "Permalink to this headline") ==================================================================================== The [latest NVIDIA Windows GPU Driver](https://www.nvidia.com/Download/index.aspx?lang=en-us) will fully support WSL 2. With CUDA support in the driver, existing applications (compiled elsewhere on a Linux system for the same target GPU) can run unmodified within the WSL environment. To compile new CUDA applications, a CUDA Toolkit for Linux x86 is needed. CUDA Toolkit support for WSL is still in preview stage as developer tools such as profilers are not available yet. However, CUDA application development is fully supported in the WSL2 environment, as a result, users should be able to compile new CUDA Linux applications with the latest CUDA Toolkit for x86 Linux. Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2. The CUDA driver installed on Windows host will be stubbed inside the WSL 2 as `libcuda.so`, therefore **users must not install any NVIDIA GPU Linux driver within WSL 2**. One has to be very careful here as the default CUDA Toolkit comes packaged with a driver, and it is easy to overwrite the WSL 2 NVIDIA driver with the default installation. We recommend developers to use a separate CUDA Toolkit for WSL 2 (Ubuntu) available from the [CUDA Toolkit Downloads](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0) page to avoid this overwriting. This WSL-Ubuntu CUDA toolkit installer will not overwrite the NVIDIA driver that was already mapped into the WSL 2 environment. To learn how to compile CUDA applications, please read the CUDA documentation for Linux. First, remove the old GPG key: sudo apt-key del 7fa2af80 **Option 1: Installation of Linux x86 CUDA Toolkit using WSL-Ubuntu Package - Recommended** The CUDA WSL-Ubuntu local installer does not contain the NVIDIA Linux GPU driver, so by following the steps on the CUDA [download page for WSL-Ubuntu](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local) , you will be able to get just the CUDA toolkit installed on WSL. **Option 2: Installation of Linux x86 CUDA Toolkit using Meta Package** If you installed the toolkit using the WSL-Ubuntu package, please skip this section. Meta packages do not contain the driver, so by following the steps on the download page for [Ubuntu](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_local) , you will be able to get just the CUDA toolkit installed on WSL. The installation instructions for the CUDA Toolkit can be found in the CUDA Toolkit download page for each installer. But DO NOT choose the “`cuda`”, “`cuda-12-x`”, or “`cuda-drivers`” meta-packages under WSL 2 as these packages will result in an attempt to install the Linux NVIDIA driver under WSL 2. Install the `cuda-toolkit-12-x` metapackage only. You can also install other components of the toolkit by choosing the right [meta-package](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#meta-packages) . 4\. WSL 2 Support Constraints[](#wsl-2-support-constraints "Permalink to this headline") ========================================================================================== * WSL 2 GPU acceleration will be available on Pascal and later GPU architecture on both GeForce and Quadro product SKUs in WDDM mode. It will not be available on Quadro GPUs in TCC mode or Tesla GPUs yet. * Ensure you are on the latest WSL Kernel or at least 4.19.121+. We recommend 5.10.16.3 or later for better performance and functional fixes. * If you are on Windows 11, you no longer need to be on Windows Insider Program to use WSL. Refer to Windows11 [system requirements in the Microsoft Blog](https://blogs.windows.com/windows-insider/2021/06/28/update-on-windows-11-minimum-system-requirements/) . * If you are continuing to use Windows 10, see [Windows Insider Preview and Windows 10 Support](#installing-insider-preview-builds) . 4.1. Known Limitations for Linux CUDA Applications[](#known-limitations-for-linux-cuda-applications "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------- The following table lists the known limitations on WSL 2 that may affect CUDA applications that use some of these features that are fully supported on Linux. | Limitations | Impact | | --- | --- | | Maxwell GPU is not supported. | Maxwell GPUs are not officially supported in WSL 2, but it may still work. Pascal and later GPU is recommended. | | Unified Memory - Full Managed Memory Support is not available on Windows native and therefore WSL 2 will not support it for the foreseeable future. | UVM full features will not be available and therefore applications relying on UVM full features may not work.

If your application is using Managed Memory, your application could see reduced performance and high system memory usage.

Concurrent CPU/GPU access is not supported. CUDA queries will say whether it is supported or not and applications are expected to check this. | | Pinned system memory (example: System memory that an application makes resident for GPU accesses) availability for applications is limited. | For example, some deep learning training workloads, depending on the framework, model and dataset size used, can exceed this limit and may not work. | | Root user on bare metal (not containers) will not find nvidia-smi at the expected location. | Use `/usr/lib/wsl/lib/nvidia-smi` or manually add `/usr/lib/wsl/lib/` to the PATH). | | With the NVIDIA Container Toolkit for Docker 19.03, only `--gpus all` is supported. | On multi-GPU systems it is not possible to filter for specific GPU devices by using specific index numbers to enumerate GPUs. | 4.2. Features Not Yet Supported[](#features-not-yet-supported "Permalink to this headline") --------------------------------------------------------------------------------------------- The following table lists the set of features that are currently not supported. | Limitations | Impact | | --- | --- | | NVML (nvidia-smi) does not support all the queries yet. | GPU utilization, active compute process are some queries that are not yet supported. Modifiable state features (ECC, Compute mode, Persistence mode) will not be supported. | | OpenGL-CUDA Interop is not yet supported. | Applications relying on OpenGL will not work. | 5\. Appendix[](#appendix "Permalink to this headline") ======================================================== 5.1. Windows Insider Preview and Windows 10 Support[](#windows-insider-preview-and-windows-10-support "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------- * If you are on Windows 11 please skip this section. Windows 11 is generally available to the public and therefore does not require special registration. All the instructions at the beginning of this user guide were mainly focused toward Windows 11 users. * If you are looking to use WSL 2 on Windows 10 or to be on the bleeding edge of WSL 2 development, you may want to register for the [Windows Insider Program](https://insider.windows.com/en-us/getting-started/#register) and choose the appropriate [flighting channel](https://docs.microsoft.com/en-us/windows-insider/flighting#switching-between-channels) (previously fast rings) and get the latest build for your needs. * Learn more on [Releasing Windows 10 Build 19043.1263 (21H1) to Release Preview Channel](https://blogs.windows.com/windows-insider/2021/09/23/releasing-windows-10-build-19043-1263-21h1-to-release-preview-channel/) . * You can check your build version number by running `winver` via the Run command. 5.2. Troubleshooting[](#troubleshooting "Permalink to this headline") ----------------------------------------------------------------------- ### 5.2.1. Container Runtime Initialization Errors[](#container-runtime-initialization-errors "Permalink to this headline") In some cases, when running a Docker container, you may encounter `nvidia-container-cli : initialization error`: $ sudo docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark docker: Error response from daemon: OCI runtime create failed: container\_linux.go:349: starting container process caused "process\_linux.go:449: container init caused \\"process\_linux.go:432: running prestart hook 0 caused \\\\\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\\\\\n\\\\\\"\\"": unknown. ERRO\[0000\] error waiting for container: context canceled This usually indicates that the right Windows OS build or Microsoft Windows Insider Preview Builds (Windows 10 only), WSL 2, NVIDIA drivers and NVIDIA Container Toolkit may not be installed correctly. Review the known issues and changelog sections to ensure the right versions of the driver and container toolkit are installed. Ensure you have followed through the steps listed under Setup under Running CUDA containers; especially ensure that the `docker` daemon is still running. $ sudo service docker stop $ sudo service docker start Or start the daemon directly and see if that resolves the issue: $ sudo dockerd If you are still running into this issue, use the `dxdiag` tools from the Run dialog and provide the diagnostic logs to NVIDIA by posting in the [Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/cuda/cuda-on-windows-subsystem-for-linux/303) or by filing a [report](https://forums.developer.nvidia.com/t/how-to-report-a-bug/) . You can also use the CUDA on WSL 2 [Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/cuda/cuda-on-windows-subsystem-for-linux/303) to get in touch with NVIDIA product and engineering teams for help. ### 5.2.2. Checking WSL Kernel Version[](#checking-wsl-kernel-version "Permalink to this headline") 1. Ensure you have the latest kernel by running the following command in PowerShell: $ wsl cat /proc/version Linux version 5.10.16.3-microsoft-standard-WSL2 (x86\_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Fri Apr 2 22:23:49 UTC 2021 2. If you don’t have the latest WSL kernel, you will see the following blocking warning upon trying to launch a Linux distribution within the WSL 2 container: ![_images/wsl2-ubuntu-lts.png](_images/wsl2-ubuntu-lts.png) 5.3. Traditional Virtual Machines vs WSL 2[](#traditional-virtual-machines-vs-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- Whether to efficiently use hardware resources or to improve productivity, virtualization is a more widely used solution in both consumer and enterprise space. There are different types of virtualizations, and it is beyond the scope of this document to delve into the specifics. But traditional virtualization solutions require installation and setup of a virtualization management software to manage the guest virtual machines. Although WSL 2 is itself a Virtual Machine, unlike traditional VMs it is easy to setup as it is provided by the host operating system provider and is quite lightweight. Applications running within WSL see less overhead compared to traditional VMs especially if they require access to the hardware or perform privileged operations compared to when run directly on the system. This is especially important for GPU accelerated workload. While VMs allow applications to be run unmodified, due to constraints from setup and performance overhead, they are not the best option in many situations. 5.4. Containers vs WSL 2[](#containers-vs-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------- While a VM provides a secure self-contained, execution environment with a complete user space for the application, containers enable application composability without the overhead of VMs. Containers compose all the dependencies of the applications such as libraries, files etc., to be bundled together for development and easy and predictable deployment. Containers run on the operating system that is installed on the system directly and therefore do not provide full isolation from other containers like a VM does, but keeps overhead negligible as a result. To learn more about differences between VMs and containers, refer to [https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/containers-vs-vm](https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/containers-vs-vm) . 6\. Notices[](#notices "Permalink to this headline") ====================================================== 6.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 6.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 6.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # Contents — CUDA on WSL 12.8 documentation * [](../index.html) » * Contents * v12.8 | [PDF](../pdf/CUDA_on_WSL_User_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Contents[](#contents "Permalink to this headline") ==================================================== * [1\. NVIDIA GPU Accelerated Computing on WSL 2](index.html) * [1.1. NVIDIA Compute Software Support on WSL 2](index.html#nvidia-compute-software-support-on-wsl-2) * [2\. Getting Started with CUDA on WSL 2](index.html#getting-started-with-cuda-on-wsl-2) * [2.1. Step 1: Install NVIDIA Driver for GPU Support](index.html#step-1-install-nvidia-driver-for-gpu-support) * [2.2. Step 2: Install WSL 2](index.html#step-2-install-wsl-2) * [2.3. Step 3: Set Up a Linux Development Environment](index.html#step-3-set-up-a-linux-development-environment) * [3\. CUDA Support for WSL 2](index.html#cuda-support-for-wsl-2) * [4\. WSL 2 Support Constraints](index.html#wsl-2-support-constraints) * [4.1. Known Limitations for Linux CUDA Applications](index.html#known-limitations-for-linux-cuda-applications) * [4.2. Features Not Yet Supported](index.html#features-not-yet-supported) * [5\. Appendix](index.html#appendix) * [5.1. Windows Insider Preview and Windows 10 Support](index.html#windows-insider-preview-and-windows-10-support) * [5.2. Troubleshooting](index.html#troubleshooting) * [5.2.1. Container Runtime Initialization Errors](index.html#container-runtime-initialization-errors) * [5.2.2. Checking WSL Kernel Version](index.html#checking-wsl-kernel-version) * [5.3. Traditional Virtual Machines vs WSL 2](index.html#traditional-virtual-machines-vs-wsl-2) * [5.4. Containers vs WSL 2](index.html#containers-vs-wsl-2) * [6\. Notices](index.html#notices) * [6.1. Notice](index.html#notice) * [6.2. OpenCL](index.html#opencl) * [6.3. Trademarks](index.html#trademarks) --- # CUDA Toolkit Documentation 12.8 Update 1 * [](#) » * CUDA Toolkit Documentation 12.8 Update 1 * [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive) - [Send Feedback](mailto:CUDAIssues@nvidia.com?subject=CUDA Toolkit Documentation Feedback) * * * CUDA Toolkit Documentation 12.8 Update 1 [](#cuda-toolkit-documentation-v12-8 "Permalink to this headline") ============================================================================================================= **Develop, Optimize and Deploy GPU-Accelerated Apps** The NVIDIA® CUDA® Toolkit provides a development environment for creating high performance GPU-accelerated applications. With the CUDA Toolkit, you can develop, optimize, and deploy your applications on GPU-accelerated embedded systems, desktop workstations, enterprise data centers, cloud-based platforms and HPC supercomputers. The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and a runtime library to deploy your application. Using built-in capabilities for distributing computations across multi-GPU configurations, scientists and researchers can develop applications that scale from single GPU workstations to cloud installations with thousands of GPUs. * * * [Release Notes](cuda-toolkit-release-notes/index.html) The Release Notes for the CUDA Toolkit. [CUDA Features Archive](cuda-features-archive/index.html) The list of CUDA features by release. [EULA](eula/index.html) The CUDA Toolkit End User License Agreement applies to the NVIDIA CUDA Toolkit, the NVIDIA CUDA Samples, the NVIDIA Display Driver, NVIDIA Nsight tools (Visual Studio Edition), and the associated documentation on CUDA APIs, programming model and development tools. If you do not agree with the terms and conditions of the license agreement, then do not download or use the software. * * * Installation Guides[](#installation-guides "Permalink to this headline") -------------------------------------------------------------------------- [Quick Start Guide](cuda-quick-start-guide/index.html) This guide provides the minimal first-steps instructions for installation and verifying CUDA on a standard system. [Installation Guide Windows](cuda-installation-guide-microsoft-windows/index.html) This guide discusses how to install and check for correct operation of the CUDA Development Tools on Microsoft Windows systems. [Installation Guide Linux](cuda-installation-guide-linux/index.html) This guide discusses how to install and check for correct operation of the CUDA Development Tools on GNU/Linux systems. * * * Programming Guides[](#programming-guides "Permalink to this headline") ------------------------------------------------------------------------ [Programming Guide](cuda-c-programming-guide/index.html) This guide provides a detailed discussion of the CUDA programming model and programming interface. It then describes the hardware implementation, and provides guidance on how to achieve maximum performance. The appendices include a list of all CUDA-enabled devices, detailed description of all extensions to the C++ language, listings of supported mathematical functions, C++ features supported in host and device code, details on texture fetching, technical specifications of various devices, and concludes by introducing the low-level driver API. [Best Practices Guide](cuda-c-best-practices-guide/index.html) This guide presents established parallelization and optimization techniques and explains coding metaphors and idioms that can greatly simplify programming for CUDA-capable GPU architectures. The intent is to provide guidelines for obtaining the best performance from NVIDIA GPUs using the CUDA Toolkit. [Maxwell Compatibility Guide](maxwell-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on GPUs based on the NVIDIA Maxwell Architecture. This document provides guidance to ensure that your software applications are compatible with Maxwell. [Pascal Compatibility Guide](pascal-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on GPUs based on the NVIDIA Pascal Architecture. This document provides guidance to ensure that your software applications are compatible with Pascal. [Volta Compatibility Guide](volta-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on GPUs based on the NVIDIA Volta Architecture. This document provides guidance to ensure that your software applications are compatible with Volta. [Turing Compatibility Guide](turing-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on GPUs based on the NVIDIA Turing Architecture. This document provides guidance to ensure that your software applications are compatible with Turing. [NVIDIA Ampere GPU Architecture Compatibility Guide](ampere-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on GPUs based on the NVIDIA Ampere GPU Architecture. This document provides guidance to ensure that your software applications are compatible with NVIDIA Ampere GPU architecture. [Hopper Compatibility Guide](hopper-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on the Hopper GPUs. This document provides guidance to ensure that your software applications are compatible with Hopper architecture. [Ada Compatibility Guide](ada-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on the Ada GPUs. This document provides guidance to ensure that your software applications are compatible with Ada architecture. [Blackwell Compatibility Guide](blackwell-compatibility-guide/index.html) This application note is intended to help developers ensure that their NVIDIA CUDA applications will run properly on the Blackwell GPUs. This document provides guidance to ensure that your software applications are compatible with Blackwell architecture. [Maxwell Tuning Guide](maxwell-tuning-guide/index.html) Maxwell is NVIDIA’s 4th-generation architecture for CUDA compute applications. Applications that follow the best practices for the Kepler architecture should typically see speedups on the Maxwell architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging Maxwell architectural features. [Pascal Tuning Guide](pascal-tuning-guide/index.html) Pascal is NVIDIA’s 5th-generation architecture for CUDA compute applications. Applications that follow the best practices for the Maxwell architecture should typically see speedups on the Pascal architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging Pascal architectural features. [Volta Tuning Guide](volta-tuning-guide/index.html) Volta is NVIDIA’s 6th-generation architecture for CUDA compute applications. Applications that follow the best practices for the Pascal architecture should typically see speedups on the Volta architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging Volta architectural features. [Turing Tuning Guide](turing-tuning-guide/index.html) Turing is NVIDIA’s 7th-generation architecture for CUDA compute applications. Applications that follow the best practices for the Pascal architecture should typically see speedups on the Turing architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging Turing architectural features. [NVIDIA Ampere GPU Architecture Tuning Guide](ampere-tuning-guide/index.html) NVIDIA Ampere GPU Architecture is NVIDIA’s 8th-generation architecture for CUDA compute applications. Applications that follow the best practices for the NVIDIA Volta architecture should typically see speedups on the NVIDIA Ampere GPU Architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging NVIDIA Ampere GPU Architecture’s features. [Hopper Tuning Guide](hopper-tuning-guide/index.html) Hopper GPU Architecture is NVIDIA’s 9th-generation architecture for CUDA compute applications. Applications that follow the best practices for the NVIDIA Volta architecture should typically see speedups on the Hopper GPU Architecture without any code changes. This guide summarizes the ways that applications can be fine-tuned to gain additional speedups by leveraging Hopper GPU Architecture’s features. [Ada Tuning Guide](ada-tuning-guide/index.html) The NVIDIA® Ada GPU architecture is NVIDIA’s 10th-generation architecture for CUDA® compute applications. The NVIDIA Ada GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as NVIDIA Ampere and Turing architectures, and applications that follow the best practices for those architectures should typically see speedups on the NVIDIA Ada architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Ada GPU architecture’s features. [Blackwell Tuning Guide](blackwell-tuning-guide/index.html) The NVIDIA® Blackwell GPU architecture is NVIDIA’s latest architecture for CUDA® compute applications. The NVIDIA Blackwell GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as NVIDIA Ampere and Turing srchitectures, and applications that follow the best practices for those architectures should typically see speedups on the NVIDIA Blackwell architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Blackwell GPU architecture’s features. [PTX ISA](parallel-thread-execution/index.html) This guide provides detailed instructions on the use of PTX, a low-level parallel thread execution virtual machine and instruction set architecture (ISA). PTX exposes the GPU as a data-parallel computing device. [Video Decoder](video-decoder/index.html) NVIDIA Video Decoder (NVCUVID) is deprecated. Instead, use the NVIDIA Video Codec SDK ([https://developer.nvidia.com/nvidia-video-codec-sdk](https://developer.nvidia.com/nvidia-video-codec-sdk) ). [PTX Interoperability](ptx-writers-guide-to-interoperability/index.html) This document shows how to write PTX that is ABI-compliant and interoperable with other CUDA code. [Inline PTX Assembly](inline-ptx-assembly/index.html) This document shows how to inline PTX (parallel thread execution) assembly language statements into CUDA code. It describes available assembler statement parameters and constraints, and the document also provides a list of some pitfalls that you may encounter. * * * CUDA API References[](#cuda-api-references "Permalink to this headline") -------------------------------------------------------------------------- [CUDA Runtime API](cuda-runtime-api/index.html) Fields in structures might appear in order that is different from the order of declaration. [CUDA Driver API](cuda-driver-api/index.html) Fields in structures might appear in order that is different from the order of declaration. [CUDA Math API](cuda-math-api/index.html) The CUDA math API. [cuBLAS](cublas/index.html) The cuBLAS library is an implementation of BLAS (Basic Linear Algebra Subprograms) on top of the NVIDIA CUDA runtime. It allows the user to access the computational resources of NVIDIA Graphical Processing Unit (GPU), but does not auto-parallelize across multiple GPUs. [cuDLA API](cudla-api/index.html) The cuDLA API. [NVBLAS](nvblas/index.html) The NVBLAS library is a multi-GPUs accelerated drop-in BLAS (Basic Linear Algebra Subprograms) built on top of the NVIDIA cuBLAS Library. [nvJPEG](nvjpeg/index.html) The nvJPEG Library provides high-performance GPU accelerated JPEG decoding functionality for image formats commonly used in deep learning and hyperscale multimedia applications. [cuFFT](cufft/index.html) The cuFFT library user guide. [CUB](https://nvlabs.github.io/cub/) The user guide for CUB. [CUDA C++ Standard Library](https://nvidia.github.io/libcudacxx/) The API reference for libcu++, the CUDA C++ standard library. [cuFile API Reference Guide](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html) The NVIDIA® GPUDirect® Storage cuFile API Reference Guide provides information about the preliminary version of the cuFile API reference guide that is used in applications and frameworks to leverage GDS technology and describes the intent, context, and operation of those APIs, which are part of the GDS technology. [cuRAND](curand/index.html) The cuRAND library user guide. [cuSPARSE](cusparse/index.html) The cuSPARSE library user guide. [NPP](npp/index.html) NVIDIA NPP is a library of functions for performing CUDA accelerated processing. The initial set of functionality in the library focuses on imaging and video processing and is widely applicable for developers in these areas. NPP will evolve over time to encompass more of the compute heavy tasks in a variety of problem domains. The NPP library is written to maximize flexibility, while maintaining high performance. [nvJitLink](nvjitlink/index.html) The user guide for the nvJitLink library. [nvFatbin](nvfatbin/index.html) The user guide for the nvFatbin library. [NVRTC (Runtime Compilation)](nvrtc/index.html) NVRTC is a runtime compilation library for CUDA C++. It accepts CUDA C++ source code in character string form and creates handles that can be used to obtain the PTX. The PTX string generated by NVRTC can be loaded by cuModuleLoadData and cuModuleLoadDataEx, and linked with other modules by cuLinkAddData of the CUDA Driver API. This facility can often provide optimizations and performance not possible in a purely offline static compilation. [Thrust](https://nvidia.github.io/cccl/thrust/) The C++ parallel algorithms library. [cuSOLVER](cusolver/index.html) The cuSOLVER library user guide. * * * PTX Compiler API References[](#ptx-compiler-api-references "Permalink to this headline") ------------------------------------------------------------------------------------------ [PTX Compiler APIs](ptx-compiler-api/index.html) This guide shows how to compile a PTX program into GPU assembly code using APIs provided by the static PTX Compiler library. * * * Miscellaneous[](#miscellaneous "Permalink to this headline") -------------------------------------------------------------- [CUDA Demo Suite](demo-suite/index.html) This document describes the demo applications shipped with the CUDA Demo Suite. [CUDA on WSL](wsl-user-guide/index.html) This guide is intended to help users get started with using NVIDIA CUDA on Windows Subsystem for Linux (WSL 2). The guide covers installation and running CUDA applications and containers in this environment. [Multi-Instance GPU (MIG)](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html) This edition of the user guide describes the Multi-Instance GPU feature of the NVIDIA® A100 GPU. [CUDA Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) This document describes CUDA Compatibility, including CUDA Enhanced Compatibility and CUDA Forward Compatible Upgrade. [CUPTI](https://docs.nvidia.com/cupti/index.html) The CUPTI-API. The CUDA Profiling Tools Interface (CUPTI) enables the creation of profiling and tracing tools that target CUDA applications. [Debugger API](debugger-api/index.html) The CUDA debugger API. [GPUDirect RDMA](gpudirect-rdma/index.html) A technology introduced in Kepler-class GPUs and CUDA 5.0, enabling a direct path for communication between the GPU and a third-party peer device on the PCI Express bus when the devices share the same upstream root complex using standard features of PCI Express. This document introduces the technology and describes the steps necessary to enable a GPUDirect RDMA connection to NVIDIA GPUs within the Linux device driver model. [GPUDirect Storage](https://docs.nvidia.com/gpudirect-storage/index.html) The documentation for GPUDirect Storage. [vGPU](vGPU/index.html) vGPUs that support CUDA. * * * Tools[](#tools "Permalink to this headline") ---------------------------------------------- [NVCC](cuda-compiler-driver-nvcc/index.html) This is a reference document for nvcc, the CUDA compiler driver. nvcc accepts a range of conventional compiler options, such as for defining macros and include/library paths, and for steering the compilation process. [CUDA-GDB](cuda-gdb/index.html) The NVIDIA tool for debugging CUDA applications running on Linux and QNX, providing developers with a mechanism for debugging CUDA applications running on actual hardware. CUDA-GDB is an extension to the x86-64 port of GDB, the GNU Project debugger. [Compute Sanitizer](https://docs.nvidia.com/compute-sanitizer/index.html) The user guide for Compute Sanitizer. [Nsight Eclipse Plugins Installation Guide](nsightee-plugins-install-guide/index.html) Nsight Eclipse Plugins Installation Guide [Nsight Eclipse Plugins Edition](nsight-eclipse-plugins-guide/index.html) Nsight Eclipse Plugins Edition getting started guide [Nsight Systems](https://docs.nvidia.com/nsight-systems/index.html) The documentation for Nsight Systems. [Nsight Compute](https://docs.nvidia.com/nsight-compute/index.html) The NVIDIA Nsight Compute is the next-generation interactive kernel profiler for CUDA applications. It provides detailed performance metrics and API debugging via a user interface and command line tool. [Nsight Visual Studio Edition](https://docs.nvidia.com/nsight-visual-studio-edition/index.html) The documentation for Nsight Visual Studio Edition. [Profiler](profiler-users-guide/index.html) This is the guide to the Profiler. [CUDA Binary Utilities](cuda-binary-utilities/index.html) The application notes for cuobjdump, nvdisasm, and nvprune. * * * White Papers[](#white-papers "Permalink to this headline") ------------------------------------------------------------ [Floating Point and IEEE 754](floating-point/index.html) A number of issues related to floating point accuracy and compliance are a frequent source of confusion on both CPUs and GPUs. The purpose of this white paper is to discuss the most common issues related to NVIDIA GPUs and to supplement the documentation in the CUDA C++ Programming Guide. [Incomplete-LU and Cholesky Preconditioned Iterative Methods](incomplete-lu-cholesky/index.html) In this white paper we show how to use the cuSPARSE and cuBLAS libraries to achieve a 2x speedup over CPU in the incomplete-LU and Cholesky preconditioned iterative methods. We focus on the Bi-Conjugate Gradient Stabilized and Conjugate Gradient iterative methods, that can be used to solve large sparse nonsymmetric and symmetric positive definite linear systems, respectively. Also, we comment on the parallel sparse triangular solve, which is an essential building block in these algorithms. * * * Application Notes[](#application-notes "Permalink to this headline") ---------------------------------------------------------------------- [CUDA for Tegra](cuda-for-tegra-appnote/index.html) This application note provides an overview of NVIDIA® Tegra® memory architecture and considerations for porting code from a discrete GPU (dGPU) attached to an x86 system to the Tegra® integrated GPU (iGPU). It also discusses EGL interoperability. * * * Compiler SDK[](#compiler-sdk "Permalink to this headline") ------------------------------------------------------------ [libNVVM API](libnvvm-api/index.html) The libNVVM API. [libdevice User’s Guide](libdevice-users-guide/index.html) The libdevice library is an LLVM bitcode library that implements common functions for GPU kernels. [NVVM IR](nvvm-ir-spec/index.html) NVVM IR is a compiler IR (intermediate representation) based on the LLVM IR. The NVVM IR is designed to represent GPU compute kernels (for example, CUDA kernels). High-level language front-ends, like the CUDA C compiler front-end, can generate NVVM IR. --- # 1. NVIDIA GPU Accelerated Computing on WSL 2 — CUDA on WSL 12.8 documentation * [](../index.html) » * 1\. NVIDIA GPU Accelerated Computing on WSL 2 * v12.8 | [PDF](../pdf/CUDA_on_WSL_User_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA on WSL User Guide The guide for using NVIDIA CUDA on Windows Subsystem for Linux. 1\. NVIDIA GPU Accelerated Computing on WSL 2[](#nvidia-gpu-accelerated-computing-on-wsl-2 "Permalink to this headline") ========================================================================================================================== WSL or Windows Subsystem for Linux is a Windows feature that enables users to run native Linux applications, containers and command-line tools directly on Windows 11 and later OS builds. CUDA support in this user guide is specifically for WSL 2, which is the second generation of WSL that offers the following benefits * Linux applications can run as is in WSL 2. WSL 2 is characteristically a VM with a Linux WSL Kernel in it that provides full compatibility with mainstream Linux kernel allowing support for native Linux applications including popular Linux distros. * Faster file system support and that’s more performant. * WSL 2 is tightly integrated with the Microsoft Windows operating system, which allows it to run Linux applications alongside and even interop with other Windows desktop and modern store apps. For the rest of this user guide, WSL and WSL 2 may be used interchangeably. Typically, developers working across both Linux and Windows environments have a very disruptive workflow. They either have to: * Use different systems for Linux and Windows, or * Dual Boot i.e. install Linux and Windows in separate partitions on the same or different hard disks on the system and boot to the OS of choice. In both cases, developers have to stop all the work and then switch the system or reboot. Also this has historically restricted the development of seamless, well integrated tools and software systems across two dominant ecosystems. WSL enables users to have a seamless transition across the two environments without the need for a resource intensive traditional virtual machine and to improve productivity and develop using tools and integrate their workflow. More importantly WSL 2 enables applications that were hitherto only available on Linux to be available on Windows. WSL 2 support for GPU allows for these applications to benefit from GPU accelerated computing and expands the domain of applications that can be developed on WSL 2. With NVIDIA CUDA support for WSL 2, developers can leverage NVIDIA GPU accelerated computing technology for data science, machine learning and inference on Windows through WSL. GPU acceleration also serves to bring down the performance overhead of running an application inside a WSL like environment close to near-native by being able to pipeline more parallel work on the GPU with less CPU intervention. NVIDIA driver support for WSL 2 includes not only CUDA but also DirectX and Direct ML support. For some helpful examples, see [https://docs.microsoft.com/en-us/windows/win32/direct3d12/gpu-tensorflow-wsl](https://docs.microsoft.com/en-us/windows/win32/direct3d12/gpu-tensorflow-wsl) . WSL 2 is a key enabler in making GPU acceleration to be seamlessly shared between Windows and Linux applications on the same system a reality. This offers flexibility and versatility while also serving to open up GPU accelerated computing by making it more accessible. [![Illustration of the possibilities with NVIDIA CUDA software stack on WSL 2](_images/wsl-launch-upt-0625-rz.png)](_images/wsl-launch-upt-0625-rz.png) Figure 1. Illustration of the possibilities with NVIDIA CUDA software stack on WSL 2[](#gpu-accelerated-computing-cuda-wsl-stack "Permalink to this image") This document describes a workflow for getting started with running CUDA applications or containers in a WSL 2 environment. 1.1. NVIDIA Compute Software Support on WSL 2[](#nvidia-compute-software-support-on-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- This table captures the readiness and suggested software versions for NVIDIA software stack for WSL 2. | Package | Suggested Versions | Installation | | --- | --- | --- | | NVIDIA Windows Driver x86 | Use the latest Windows x86 production driver. R495 and later windows will have CUDA support for WSL 2. NVIDIA-SMI will have a Limited Feature Set on WSL 2.

Legacy CUDA IPC APIs are support from R510. | Windows x86 drivers can be directly downloaded from [https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx)
for WSL 2 support on Pascal or later GPUs. | | Docker support | Supported.

NVIDIA Container Toolkit - Minimum versions - v2.6.0 with libnvidia-container - 1.5.1+

CLI and Docker Desktop Supported. | Refer to [https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html](https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html)
. | | CUDA Toolkit and CUDA Developer Tools | Preview Support

Compute Sanitizer - Pascal and later

Nsight Systems CLI, and CUPTI (Trace) - Volta and later

Developer tools - Debuggers - Pascal and later (Using driver r535+)

Developer tools - Profilers - Volta and later (Using Windows 10 OS build 19044+ with driver r545+ or using Windows 11 with driver r525+ ) | Latest Linux CUDA toolkit package - WSL-Ubuntu from 12.x releases can be downloaded from [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads)
. | | RAPIDS | 22.04 or later 1.10 - Experimental Support for single GPU. | [https://docs.rapids.ai/notices/rgn0024/](https://docs.rapids.ai/notices/rgn0024/) | | NCCL | 2.12 or later 1.4+ | Refer to the [NCCL Installation guide for Linux x86](https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html#down)
. | 2\. Getting Started with CUDA on WSL 2[](#getting-started-with-cuda-on-wsl-2 "Permalink to this headline") ============================================================================================================ To get started with running CUDA on WSL, complete these steps in order: 2.1. Step 1: Install NVIDIA Driver for GPU Support[](#step-1-install-nvidia-driver-for-gpu-support "Permalink to this headline") ---------------------------------------------------------------------------------------------------------------------------------- * Install NVIDIA GeForce Game Ready or NVIDIA RTX Quadro Windows 11 display driver on your system with a compatible GeForce or NVIDIA RTX/Quadro card from [https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx) . Refer to the system requirements in the Appendix.) > Note > > **This is the only driver you need to install. Do not install any Linux display driver in WSL.** 2.2. Step 2: Install WSL 2[](#step-2-install-wsl-2 "Permalink to this headline") ---------------------------------------------------------------------------------- 1. Launch your preferred Windows Terminal / Command Prompt / Powershell and install WSL: wsl.exe --install 2. Ensure you have the latest WSL kernel: wsl.exe --update 2.3. Step 3: Set Up a Linux Development Environment[](#step-3-set-up-a-linux-development-environment "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------ From a Windows terminal, enter WSL: C:\\> wsl.exe The default distro is Ubuntu. To update the distro to your favorite distro from the command line and to review other WSL commands, refer to the following resources: * [https://docs.microsoft.com/en-us/windows/wsl/install](https://docs.microsoft.com/en-us/windows/wsl/install) * [https://docs.microsoft.com/en-us/windows/wsl/basic-commands](https://docs.microsoft.com/en-us/windows/wsl/basic-commands) From this point you should be able to run any existing Linux application which requires CUDA. Do not install any driver within the WSL environment. For building a CUDA application, you will need CUDA Toolkit. Read the next section for further information. 3\. CUDA Support for WSL 2[](#cuda-support-for-wsl-2 "Permalink to this headline") ==================================================================================== The [latest NVIDIA Windows GPU Driver](https://www.nvidia.com/Download/index.aspx?lang=en-us) will fully support WSL 2. With CUDA support in the driver, existing applications (compiled elsewhere on a Linux system for the same target GPU) can run unmodified within the WSL environment. To compile new CUDA applications, a CUDA Toolkit for Linux x86 is needed. CUDA Toolkit support for WSL is still in preview stage as developer tools such as profilers are not available yet. However, CUDA application development is fully supported in the WSL2 environment, as a result, users should be able to compile new CUDA Linux applications with the latest CUDA Toolkit for x86 Linux. Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2. The CUDA driver installed on Windows host will be stubbed inside the WSL 2 as `libcuda.so`, therefore **users must not install any NVIDIA GPU Linux driver within WSL 2**. One has to be very careful here as the default CUDA Toolkit comes packaged with a driver, and it is easy to overwrite the WSL 2 NVIDIA driver with the default installation. We recommend developers to use a separate CUDA Toolkit for WSL 2 (Ubuntu) available from the [CUDA Toolkit Downloads](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0) page to avoid this overwriting. This WSL-Ubuntu CUDA toolkit installer will not overwrite the NVIDIA driver that was already mapped into the WSL 2 environment. To learn how to compile CUDA applications, please read the CUDA documentation for Linux. First, remove the old GPG key: sudo apt-key del 7fa2af80 **Option 1: Installation of Linux x86 CUDA Toolkit using WSL-Ubuntu Package - Recommended** The CUDA WSL-Ubuntu local installer does not contain the NVIDIA Linux GPU driver, so by following the steps on the CUDA [download page for WSL-Ubuntu](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local) , you will be able to get just the CUDA toolkit installed on WSL. **Option 2: Installation of Linux x86 CUDA Toolkit using Meta Package** If you installed the toolkit using the WSL-Ubuntu package, please skip this section. Meta packages do not contain the driver, so by following the steps on the download page for [Ubuntu](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_local) , you will be able to get just the CUDA toolkit installed on WSL. The installation instructions for the CUDA Toolkit can be found in the CUDA Toolkit download page for each installer. But DO NOT choose the “`cuda`”, “`cuda-12-x`”, or “`cuda-drivers`” meta-packages under WSL 2 as these packages will result in an attempt to install the Linux NVIDIA driver under WSL 2. Install the `cuda-toolkit-12-x` metapackage only. You can also install other components of the toolkit by choosing the right [meta-package](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#meta-packages) . 4\. WSL 2 Support Constraints[](#wsl-2-support-constraints "Permalink to this headline") ========================================================================================== * WSL 2 GPU acceleration will be available on Pascal and later GPU architecture on both GeForce and Quadro product SKUs in WDDM mode. It will not be available on Quadro GPUs in TCC mode or Tesla GPUs yet. * Ensure you are on the latest WSL Kernel or at least 4.19.121+. We recommend 5.10.16.3 or later for better performance and functional fixes. * If you are on Windows 11, you no longer need to be on Windows Insider Program to use WSL. Refer to Windows11 [system requirements in the Microsoft Blog](https://blogs.windows.com/windows-insider/2021/06/28/update-on-windows-11-minimum-system-requirements/) . * If you are continuing to use Windows 10, see [Windows Insider Preview and Windows 10 Support](#installing-insider-preview-builds) . 4.1. Known Limitations for Linux CUDA Applications[](#known-limitations-for-linux-cuda-applications "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------- The following table lists the known limitations on WSL 2 that may affect CUDA applications that use some of these features that are fully supported on Linux. | Limitations | Impact | | --- | --- | | Maxwell GPU is not supported. | Maxwell GPUs are not officially supported in WSL 2, but it may still work. Pascal and later GPU is recommended. | | Unified Memory - Full Managed Memory Support is not available on Windows native and therefore WSL 2 will not support it for the foreseeable future. | UVM full features will not be available and therefore applications relying on UVM full features may not work.

If your application is using Managed Memory, your application could see reduced performance and high system memory usage.

Concurrent CPU/GPU access is not supported. CUDA queries will say whether it is supported or not and applications are expected to check this. | | Pinned system memory (example: System memory that an application makes resident for GPU accesses) availability for applications is limited. | For example, some deep learning training workloads, depending on the framework, model and dataset size used, can exceed this limit and may not work. | | Root user on bare metal (not containers) will not find nvidia-smi at the expected location. | Use `/usr/lib/wsl/lib/nvidia-smi` or manually add `/usr/lib/wsl/lib/` to the PATH). | | With the NVIDIA Container Toolkit for Docker 19.03, only `--gpus all` is supported. | On multi-GPU systems it is not possible to filter for specific GPU devices by using specific index numbers to enumerate GPUs. | 4.2. Features Not Yet Supported[](#features-not-yet-supported "Permalink to this headline") --------------------------------------------------------------------------------------------- The following table lists the set of features that are currently not supported. | Limitations | Impact | | --- | --- | | NVML (nvidia-smi) does not support all the queries yet. | GPU utilization, active compute process are some queries that are not yet supported. Modifiable state features (ECC, Compute mode, Persistence mode) will not be supported. | | OpenGL-CUDA Interop is not yet supported. | Applications relying on OpenGL will not work. | 5\. Appendix[](#appendix "Permalink to this headline") ======================================================== 5.1. Windows Insider Preview and Windows 10 Support[](#windows-insider-preview-and-windows-10-support "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------- * If you are on Windows 11 please skip this section. Windows 11 is generally available to the public and therefore does not require special registration. All the instructions at the beginning of this user guide were mainly focused toward Windows 11 users. * If you are looking to use WSL 2 on Windows 10 or to be on the bleeding edge of WSL 2 development, you may want to register for the [Windows Insider Program](https://insider.windows.com/en-us/getting-started/#register) and choose the appropriate [flighting channel](https://docs.microsoft.com/en-us/windows-insider/flighting#switching-between-channels) (previously fast rings) and get the latest build for your needs. * Learn more on [Releasing Windows 10 Build 19043.1263 (21H1) to Release Preview Channel](https://blogs.windows.com/windows-insider/2021/09/23/releasing-windows-10-build-19043-1263-21h1-to-release-preview-channel/) . * You can check your build version number by running `winver` via the Run command. 5.2. Troubleshooting[](#troubleshooting "Permalink to this headline") ----------------------------------------------------------------------- ### 5.2.1. Container Runtime Initialization Errors[](#container-runtime-initialization-errors "Permalink to this headline") In some cases, when running a Docker container, you may encounter `nvidia-container-cli : initialization error`: $ sudo docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark docker: Error response from daemon: OCI runtime create failed: container\_linux.go:349: starting container process caused "process\_linux.go:449: container init caused \\"process\_linux.go:432: running prestart hook 0 caused \\\\\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\\\\\n\\\\\\"\\"": unknown. ERRO\[0000\] error waiting for container: context canceled This usually indicates that the right Windows OS build or Microsoft Windows Insider Preview Builds (Windows 10 only), WSL 2, NVIDIA drivers and NVIDIA Container Toolkit may not be installed correctly. Review the known issues and changelog sections to ensure the right versions of the driver and container toolkit are installed. Ensure you have followed through the steps listed under Setup under Running CUDA containers; especially ensure that the `docker` daemon is still running. $ sudo service docker stop $ sudo service docker start Or start the daemon directly and see if that resolves the issue: $ sudo dockerd If you are still running into this issue, use the `dxdiag` tools from the Run dialog and provide the diagnostic logs to NVIDIA by posting in the [Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/cuda/cuda-on-windows-subsystem-for-linux/303) or by filing a [report](https://forums.developer.nvidia.com/t/how-to-report-a-bug/) . You can also use the CUDA on WSL 2 [Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/cuda/cuda-on-windows-subsystem-for-linux/303) to get in touch with NVIDIA product and engineering teams for help. ### 5.2.2. Checking WSL Kernel Version[](#checking-wsl-kernel-version "Permalink to this headline") 1. Ensure you have the latest kernel by running the following command in PowerShell: $ wsl cat /proc/version Linux version 5.10.16.3-microsoft-standard-WSL2 (x86\_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Fri Apr 2 22:23:49 UTC 2021 2. If you don’t have the latest WSL kernel, you will see the following blocking warning upon trying to launch a Linux distribution within the WSL 2 container: ![_images/wsl2-ubuntu-lts.png](_images/wsl2-ubuntu-lts.png) 5.3. Traditional Virtual Machines vs WSL 2[](#traditional-virtual-machines-vs-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- Whether to efficiently use hardware resources or to improve productivity, virtualization is a more widely used solution in both consumer and enterprise space. There are different types of virtualizations, and it is beyond the scope of this document to delve into the specifics. But traditional virtualization solutions require installation and setup of a virtualization management software to manage the guest virtual machines. Although WSL 2 is itself a Virtual Machine, unlike traditional VMs it is easy to setup as it is provided by the host operating system provider and is quite lightweight. Applications running within WSL see less overhead compared to traditional VMs especially if they require access to the hardware or perform privileged operations compared to when run directly on the system. This is especially important for GPU accelerated workload. While VMs allow applications to be run unmodified, due to constraints from setup and performance overhead, they are not the best option in many situations. 5.4. Containers vs WSL 2[](#containers-vs-wsl-2 "Permalink to this headline") ------------------------------------------------------------------------------- While a VM provides a secure self-contained, execution environment with a complete user space for the application, containers enable application composability without the overhead of VMs. Containers compose all the dependencies of the applications such as libraries, files etc., to be bundled together for development and easy and predictable deployment. Containers run on the operating system that is installed on the system directly and therefore do not provide full isolation from other containers like a VM does, but keeps overhead negligible as a result. To learn more about differences between VMs and containers, refer to [https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/containers-vs-vm](https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/containers-vs-vm) . 6\. Notices[](#notices "Permalink to this headline") ====================================================== 6.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 6.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 6.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # Installing Docker and The Docker Utility Engine for NVIDIA GPUs — NVIDIA AI Enterprise: VMware Deployment Guide [Skip to main content](#main-content) Back to top Ctrl+K [![NVIDIA AI Enterprise: VMware Deployment Guide - Home](_static/nvidia-logo-horiz-rgb-blk-for-screen.svg)\ \ NVIDIA AI Enterprise: VMware Deployment Guide](index.html) * [Documentation Home](https://docs.nvidia.com/ai-enterprise/) Installing Docker and The Docker Utility Engine for NVIDIA GPUs[#](#installing-docker-and-the-docker-utility-engine-for-nvidia-gpus "Link to this heading") ============================================================================================================================================================ The NVIDIA Container Toolkit allows users to build and run GPU accelerated Docker containers. The toolkit includes a container runtime [library](https://github.com/NVIDIA/libnvidia-container) and utilities to configure containers to leverage NVIDIA GPUs automatically. Complete documentation and frequently asked questions are available on the [repository wiki](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/overview.html) . ![_images/dg-docker-01.png](_images/dg-docker-01.png) Installing Docker[#](#installing-docker "Link to this heading") ---------------------------------------------------------------- Please refer to [Install Docker Engine on Ubuntu | Docker Documentation](https://docs.docker.com/engine/install/ubuntu/) for a current installation procedure for Ubuntu. Installing the NVIDIA Container Toolkit[#](#installing-the-nvidia-container-toolkit "Link to this heading") ------------------------------------------------------------------------------------------------------------ Please refer to [Installing the NVIDIA Container Toolkit | NVIDIA Documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-the-nvidia-container-toolkit) for a current installation procedure to enable the docker repository and install the NVIDIA Container Toolkit. Once the NVIDIA Container Toolkit is installed, to configure the docker container runtime, please refer to [Configuration | NVIDIA Documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuration) . Testing Docker and NVIDIA Container Runtime[#](#testing-docker-and-nvidia-container-runtime "Link to this heading") -------------------------------------------------------------------------------------------------------------------- Please refer to [Running a Sample Workload | NVIDIA Documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html#running-a-sample-workload) to run a sample CUDA container test on your GPU. On this page --- # Unknown CUDA on WSL Release 12.8 NVIDIA Corporation Feb 27, 2025 Contents 1 NVIDIA Compute Software Support on WSL 23 2 Getting Started with CUDA on WSL 25 2.1 Step 1: Install NVIDIA Driver for GPU Support. . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Step 2: Install WSL 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 Step 3: Set Up a Linux Development Environment. . . . . . . . . . . . . . . . . . . . . . . . 6 3 CUDA Support for WSL 27 4 WSL 2 Support Constraints9 4.1 Known Limitations for Linux CUDA Applications. . . . . . . . . . . . . . . . . . . . . . . . . 9 4.2 Features Not Yet Supported. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5 Appendix11 5.1 Windows Insider Preview and Windows 10 Support. . . . . . . . . . . . . . . . . . . . . . . 11 5.2 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2.1 Container Runtime Initialization Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2.2 Checking WSL Kernel Version. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.3 Traditional Virtual Machines vs WSL 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.4 Containers vs WSL 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 6 Notices15 6.1 Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 6.2 OpenCL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 6.3 Trademarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 i ii CUDA on WSL, Release 12.8 CUDA on WSL User Guide The guide for using NVIDIA CUDA on Windows Subsystem for Linux. WSL or Windows Subsystem for Linux is a Windows feature that enables users to run native Linux applications, containers and command-line tools directly on Windows 11 and later OS builds. CUDA support in this user guide is specifically for WSL 2, which is the second generation of WSL that offers the following benefits ▶Linux applications can run as is in WSL 2. WSL 2 is characteristically a VM with a Linux WSL Kernel in it that provides full compatibility with mainstream Linux kernel allowing support for native Linux applications including popular Linux distros. ▶Faster file system support and that’s more performant. ▶WSL 2 is tightly integrated with the Microsoft Windows operating system, which allows it to run Linux applications alongside and even interop with other Windows desktop and modern store apps. For the rest of this user guide, WSL and WSL 2 may be used interchangeably. Typically, developers working across both Linux and Windows environments have a very disruptive workflow. They either have to: ▶Use different systems for Linux and Windows, or ▶Dual Boot i.e. install Linux and Windows in separate partitions on the same or different hard disks on the system and boot to the OS of choice. In both cases, developers have to stop all the work and then switch the system or reboot. Also this has historically restricted the development of seamless, well integrated tools and software systems across two dominant ecosystems. WSL enables users to have a seamless transition across the two environments without the need for a resource intensive traditional virtual machine and to improve productivity and develop using tools and integrate their workflow. More importantly WSL 2 enables applications that were hitherto only available on Linux to be available on Windows. WSL 2 support for GPU allows for these applications to benefit from GPU accelerated computing and expands the domain of applications that can be devel- oped on WSL 2. With NVIDIA CUDA support for WSL 2, developers can leverage NVIDIA GPU accelerated computing technology for data science, machine learning and inference on Windows through WSL. GPU accel- eration also serves to bring down the performance overhead of running an application inside a WSL like environment close to near-native by being able to pipeline more parallel work on the GPU with less CPU intervention. NVIDIA driver support for WSL 2 includes not only CUDA but also DirectX and Direct ML sup- port. For some helpful examples, see https://docs.microsoft.com/en-us/windows/win32/direct3d12/ gpu-tensorflow-wsl. WSL 2 is a key enabler in making GPU acceleration to be seamlessly shared between Windows and Linux applications on the same system a reality. This offers flexibility and versatility while also serving to open up GPU accelerated computing by making it more accessible. This document describes a workflow for getting started with running CUDA applications or containers in a WSL 2 environment. Contents1 CUDA on WSL, Release 12.8 Fig. 1: Figure 1. Illustration of the possibilities with NVIDIA CUDA software stack on WSL 2 2Contents Chapter 1.NVIDIA Compute Software Support on WSL 2 This table captures the readiness and suggested software versions for NVIDIA software stack for WSL 2. 3 CUDA on WSL, Release 12.8 PackageSuggested VersionsInstallation NVIDIA Windows Driver x86Use the latest Windows x86 production driver. R495 and later windows will have CUDA support for WSL 2. NVIDIA-SMI will have a Lim- ited Feature Set on WSL 2. Legacy CUDA IPC APIs are support from R510. Windows x86 drivers can be directly downloaded fromhttps://www.nvidia. com/Download/index.aspxfor WSL 2 support on Pascal or later GPUs. Docker supportSupported. NVIDIA Container Toolkit - Minimum versions - v2.6.0 with libnvidia-container - 1.5.1+ CLI and Docker Desktop Supported. Refertohttps://docs. nvidia.com/ai-enterprise/ deployment-guide-vmware/0.1.0/ docker.html. CUDA Toolkit and CUDA De- veloper Tools Preview Support Compute Sanitizer - Pascal and later Nsight Systems CLI, and CUPTI (Trace) - Volta and later Developer tools - Debuggers - Pascal and later (Using driver r535+) Developer tools - Profilers - Volta and later (Using Win- dows 10 OS build 19044+ with driver r545+ or us- ing Windows 11 with driver r525+ ) Latest Linux CUDA toolkit package - WSL-Ubuntufrom12.xreleasescanbe downloaded fromhttps://developer. nvidia.com/cuda-downloads. RAPIDS22.04 or later 1.10 - Exper- imental Support for single GPU. https://docs.rapids.ai/notices/ rgn0024/ NCCL2.12 or later 1.4+Refer to theNCCL Installation guide for Linux x86. 4Chapter 1. NVIDIA Compute Software Support on WSL 2 Chapter 2.Getting Started with CUDA on WSL 2 To get started with running CUDA on WSL, complete these steps in order: 2.1.Step 1: Install NVIDIA Driver for GPU Support ▶Install NVIDIA GeForce Game Ready or NVIDIA RTX Quadro Windows 11 display driver on your system with a compatible GeForce or NVIDIA RTX/Quadro card fromhttps://www.nvidia.com/ Download/index.aspx. Refer to the system requirements in the Appendix.) Note: This is the only driver you need to install. Do not install any Linux display driver in WSL. 2.2.Step 2: Install WSL 2 1.Launch your preferred Windows Terminal / Command Prompt / Powershell and install WSL: wsl.exe --install 2.Ensure you have the latest WSL kernel: wsl.exe --update 5 CUDA on WSL, Release 12.8 2.3.Step 3: Set Up a Linux Development Environment From a Windows terminal, enter WSL: C:\\> wsl.exe The default distro is Ubuntu. To update the distro to your favorite distro from the command line and to review other WSL commands, refer to the following resources: ▶https://docs.microsoft.com/en-us/windows/wsl/install ▶https://docs.microsoft.com/en-us/windows/wsl/basic-commands From this point you should be able to run any existing Linux application which requires CUDA. Do not install any driver within the WSL environment. For building a CUDA application, you will need CUDA Toolkit. Read the next section for further information. 6Chapter 2. Getting Started with CUDA on WSL 2 Chapter 3.CUDA Support for WSL 2 Thelatest NVIDIA Windows GPU Driverwill fully support WSL 2. With CUDA support in the driver, exist- ing applications (compiled elsewhere on a Linux system for the same target GPU) can run unmodified within the WSL environment. To compile new CUDA applications, a CUDA Toolkit for Linux x86 is needed. CUDA Toolkit support for WSL is still in preview stage as developer tools such as profilers are not available yet. However, CUDA application development is fully supported in the WSL2 environment, as a result, users should be able to compile new CUDA Linux applications with the latest CUDA Toolkit for x86 Linux. Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2. The CUDA driver installed on Windows host will be stubbed inside the WSL 2 aslibcuda.so, therefore users must not install any NVIDIA GPU Linux driver within WSL 2. One has to be very careful here as the default CUDA Toolkit comes packaged with a driver, and it is easy to overwrite the WSL 2 NVIDIA driver with the default installation. We recommend developers to use a separate CUDA Toolkit for WSL 2 (Ubuntu) available from theCUDA Toolkit Downloadspage to avoid this overwriting. This WSL- Ubuntu CUDA toolkit installer will not overwrite the NVIDIA driver that was already mapped into the WSL 2 environment. To learn how to compile CUDA applications, please read the CUDA documentation for Linux. First, remove the old GPG key: sudo apt-key del 7fa2af80 Option 1: Installation of Linux x86 CUDA Toolkit using WSL-Ubuntu Package - Recommended The CUDA WSL-Ubuntu local installer does not contain the NVIDIA Linux GPU driver, so by following the steps on the CUDAdownload page for WSL-Ubuntu, you will be able to get just the CUDA toolkit installed on WSL. Option 2: Installation of Linux x86 CUDA Toolkit using Meta Package If you installed the toolkit using the WSL-Ubuntu package, please skip this section. Meta packages do not contain the driver, so by following the steps on the download page for Ubuntu, you will be able to get just the CUDA toolkit installed on WSL. The installation instructions for the CUDA Toolkit can be found in the CUDA Toolkit download page for each installer. But DO NOT choose the “cuda”, “cuda-12-x”, or “cuda-drivers” meta-packages under WSL 2 as these packages will result in an attempt to install the Linux NVIDIA driver under WSL 2. Install thecuda-toolkit-12-xmetapackage only. You can also install other components of the toolkit by choosing the right meta-package. 7 CUDA on WSL, Release 12.8 8Chapter 3. CUDA Support for WSL 2 Chapter 4.WSL 2 Support Constraints ▶WSL 2 GPU acceleration will be available on Pascal and later GPU architecture on both GeForce and Quadro product SKUs in WDDM mode. It will not be available on Quadro GPUs in TCC mode or Tesla GPUs yet. ▶Ensure you are on the latest WSL Kernel or at least 4.19.121+. We recommend 5.10.16.3 or later for better performance and functional fixes. ▶If you are on Windows 11, you no longer need to be on Windows Insider Program to use WSL. Refer to Windows11system requirements in the Microsoft Blog. ▶If you are continuing to use Windows 10, seeWindows Insider Preview and Windows 10 Support. 4.1.Known Limitations for Linux CUDA Applications The following table lists the known limitations on WSL 2 that may affect CUDA applications that use some of these features that are fully supported on Linux. 9 CUDA on WSL, Release 12.8 LimitationsImpact Maxwell GPU is not supported.Maxwell GPUs are not officially supported in WSL 2, but it may still work. Pascal and later GPU is recommended. Unified Memory - Full Managed Memory Support is not available on Windows native and therefore WSL 2 will not support it for the foreseeable fu- ture. UVM full features will not be available and there- fore applications relying on UVM full features may not work. If your application is using Managed Memory, your application could see reduced performance and high system memory usage. Concurrent CPU/GPU access is not supported. CUDA queries will say whether it is supported or not and applications are expected to check this. Pinned system memory (example: System mem- ory that an application makes resident for GPU accesses) availability for applications is limited. For example, some deep learning training work- loads, depending on the framework, model and dataset size used, can exceed this limit and may not work. Root user on bare metal (not containers) will not find nvidia-smi at the expected location. Use∕usr∕lib∕wsl∕lib∕nvidia-smior man- ually add∕usr∕lib∕wsl∕lib∕to the PATH). With the NVIDIA Container Toolkit for Docker 19.03, only--gpus allis supported. On multi-GPU systems it is not possible to filter for specific GPU devices by using specific index numbers to enumerate GPUs. 4.2.Features Not Yet Supported The following table lists the set of features that are currently not supported. LimitationsImpact NVML (nvidia-smi) does not support all the queries yet. GPU utilization, active compute process are some queries that are not yet supported. Mod- ifiable state features (ECC, Compute mode, Per- sistence mode) will not be supported. OpenGL-CUDA Interop is not yet supported.Applications relying on OpenGL will not work. 10Chapter 4. WSL 2 Support Constraints Chapter 5.Appendix 5.1.Windows Insider Preview and Windows 10 Support ▶If you are on Windows 11 please skip this section. Windows 11 is generally available to the public and therefore does not require special registration. All the instructions at the beginning of this user guide were mainly focused toward Windows 11 users. ▶If you are looking to use WSL 2 on Windows 10 or to be on the bleeding edge of WSL 2 devel- opment, you may want to register for theWindows Insider Programand choose the appropriate flighting channel(previously fast rings) and get the latest build for your needs. ▶Learn more onReleasing Windows 10 Build 19043.1263 (21H1) to Release Preview Channel. ▶You can check your build version number by runningwinvervia the Run command. 5.2.Troubleshooting 5.2.1.Container Runtime Initialization Errors In some cases, when running a Docker container, you may encounternvidia-container-cli : initialization error: $ sudo docker run --gpus all nvcr.io∕nvidia∕k8s∕cuda-sample:nbody nbody -gpu - ,→benchmark docker: Error response from daemon: OCI runtime create failed: container\_linux. ,→go:349: starting container process caused "process\_linux.go:449: container init￿ ,→caused \\"process\_linux.go:432: running prestart hook 0 caused \\\\\\"error running￿ ,→hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error:￿ ,→driver error: failed to process request\\\\\\\\n\\\\\\"\\"": unknown. ERRO\[0000\] error waiting for container: context canceled This usually indicates that the right Windows OS build or Microsoft Windows Insider Preview Builds (Windows 10 only), WSL 2, NVIDIA drivers and NVIDIA Container Toolkit may not be installed correctly. Review the known issues and changelog sections to ensure the right versions of the driver and con- tainer toolkit are installed. Ensure you have followed through the steps listed under Setup under Running CUDA containers; es- pecially ensure that thedockerdaemon is still running. 11 CUDA on WSL, Release 12.8 $ sudo service docker stop $ sudo service docker start Or start the daemon directly and see if that resolves the issue: $ sudo dockerd If you are still running into this issue, use thedxdiagtools from the Run dialog and provide the diag- nostic logs to NVIDIA by posting in theDeveloper Forumsor by filing areport. You can also use the CUDA on WSL 2Developer Forumsto get in touch with NVIDIA product and engineering teams for help. 5.2.2.Checking WSL Kernel Version 1.Ensure you have the latest kernel by running the following command in PowerShell: $ wsl cat ∕proc∕version Linux version 5.10.16.3-microsoft-standard-WSL2 (x86\_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP￿ ,→Fri Apr 2 22:23:49 UTC 2021 2.If you don’t have the latest WSL kernel, you will see the following blocking warning upon trying to launch a Linux distribution within the WSL 2 container: 12Chapter 5. Appendix CUDA on WSL, Release 12.8 5.3.Traditional Virtual Machines vs WSL 2 Whether to efficiently use hardware resources or to improve productivity, virtualization is a more widely used solution in both consumer and enterprise space. There are different types of virtualizations, and it is beyond the scope of this document to delve into the specifics. But traditional virtualization solutions require installation and setup of a virtualization management software to manage the guest virtual machines. Although WSL 2 is itself a Virtual Machine, unlike traditional VMs it is easy to setup as it is provided by the host operating system provider and is quite lightweight. Applications running within WSL see less overhead compared to traditional VMs especially if they require access to the hardware or perform privileged operations compared to when run directly on the system. This is especially important for GPU accelerated workload. While VMs allow applications to be run unmodified, due to constraints from setup and performance overhead, they are not the best option in many situations. 5.4.Containers vs WSL 2 While a VM provides a secure self-contained, execution environment with a complete user space for the application, containers enable application composability without the overhead of VMs. Containers compose all the dependencies of the applications such as libraries, files etc., to be bundled together for development and easy and predictable deployment. Containers run on the operating system that is installed on the system directly and therefore do not provide full isolation from other containers like a VM does, but keeps overhead negligible as a result. To learn more about differences between VMs and containers, refer to https://docs.microsoft.com/ en-us/virtualization/windowscontainers/about/containers-vs-vm. 5.3. Traditional Virtual Machines vs WSL 213 CUDA on WSL, Release 12.8 14Chapter 5. Appendix Chapter 6.Notices 6.1.Notice This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no repre- sentations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA productreferencedinthisdocument. Nocontractualobligationsareformedeitherdirectlyorindirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, orlifesupportequipment, norinapplicationswherefailureormalfunctionoftheNVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information con- tained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or prob- lem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or 15 CUDA on WSL, Release 12.8 services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WAR- RANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CON- SEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARIS- ING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatso- ever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 6.2.OpenCL OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 6.3.Trademarks NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. Copyright ©2020-2025, NVIDIA Corporation & affiliates. All rights reserved 16Chapter 6. Notices --- # 1. CUDA 12.8 Update 1 Release Notes — Release Notes 12.8 documentation * [](../index.html) » * 1\. CUDA 12.8 Update 1 Release Notes * v12.8 | [PDF](../pdf/CUDA_Toolkit_Release_Notes.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA CUDA Toolkit Release Notes The Release Notes for the CUDA Toolkit. 1\. CUDA 12.8 Update 1 Release Notes[](#cuda-12-8-update-1-release-notes "Permalink to this headline") ======================================================================================================== The release notes for the NVIDIA® CUDA® Toolkit can be found online at [https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) . Note The release notes have been reorganized into two major sections: the general CUDA release notes, and the CUDA libraries release notes including historical information for 12.x releases. 1.1. CUDA Toolkit Major Component Versions[](#cuda-toolkit-major-component-versions "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- CUDA Components Starting with CUDA 11, the various components in the toolkit are versioned independently. For CUDA 12.8, the table below indicates the versions: | | | | | | | --- | --- | --- | --- | --- |Table 1 CUDA 12.8 Update 1 Component Versions[](#id4 "Permalink to this table") | Component Name | | Version Information | Supported Architectures | Supported Platforms | | --- | --- | --- | --- | --- | | CUDA C++ Core Compute Libraries | Thrust | 2.7.0 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows | | CUB | 2.7.0 | | libcu++ | 2.7.0 | | Cooperative Groups | 12.8.90 | | CUDA Compatibility | | 12.8.39468522 | aarch64-jetson | Linux | | CUDA Runtime (cudart) | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | cuobjdump | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows | | CUPTI | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA cuxxfilt (demangler) | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows | | CUDA Demo Suite | | 12.8.90 | x86\_64 | Linux, Windows | | CUDA GDB | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, WSL | | CUDA Nsight Eclipse Plugin | | 12.8.90 | x86\_64 | Linux | | CUDA NVCC | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA nvdisasm | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows | | CUDA NVML Headers | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA nvprof | | 12.8.90 | x86\_64 | Linux, Windows | | CUDA nvprune | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA NVRTC | | 12.8.93 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | NVTX | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA NVVP | | 12.8.93 | x86\_64 | Linux, Windows | | CUDA OpenCL | | 12.8.90 | x86\_64 | Linux, Windows | | CUDA Profiler API | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA Compute Sanitizer API | | 12.8.93 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA cuBLAS | | 12.8.4.1 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | cuDLA | | 12.8.90 | aarch64-jetson | Linux | | CUDA cuFFT | | 11.3.3.76 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA cuFile | | 1.13.1.3 | x86\_64, arm64-sbsa, aarch64-jetson | Linux | | CUDA cuRAND | | 10.3.9.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA cuSOLVER | | 11.7.3.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA cuSPARSE | | 12.5.8.88 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA NPP | | 12.3.3.100 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA nvFatbin | | 12.8.90 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA nvJitLink | | 12.8.93 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | CUDA nvJPEG | | 12.3.5.92 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL | | Nsight Compute | | 2025.1.1.2 | x86\_64, arm64-sbsa, aarch64-jetson | Linux, Windows, WSL (Windows 11) | | Nsight Systems | | 2024.6.2.225 | x86\_64, arm64-sbsa | Linux, Windows, WSL | | Nsight Visual Studio Edition (VSE) | | 2025.1.0.25055 | x86\_64 (Windows) | Windows | | nvidia\_fs[1](#fn1) | | 2.24.3 | x86\_64, arm64-sbsa, aarch64-jetson | Linux | | Visual Studio Integration | | 12.8.90 | x86\_64 (Windows) | Windows | | NVIDIA Linux Driver | | 570.12 | x86\_64, arm64-sbsa | Linux | | NVIDIA Windows Driver | | 572.61 | x86\_64 (Windows) | Windows, WSL | CUDA Driver Running a CUDA application requires the system with at least one CUDA capable GPU and a driver that is compatible with the CUDA Toolkit. See [Table 3](index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions) . For more information various GPU products that are CUDA capable, visit [https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) . Each release of the CUDA Toolkit requires a minimum version of the CUDA driver. The CUDA driver is backward compatible, meaning that applications compiled against a particular version of the CUDA will continue to work on subsequent (later) driver releases. More information on compatibility can be found at [https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#cuda-compatibility-and-upgrades](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#cuda-compatibility-and-upgrades) . **Note**: Starting with CUDA 11.0, the toolkit components are individually versioned, and the toolkit itself is versioned as shown in the table below. The minimum required driver version for CUDA minor version compatibility is shown below. CUDA minor version compatibility is described in detail in [https://docs.nvidia.com/deploy/cuda-compatibility/index.html](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) | | | | | --- | --- | --- |Table 2 CUDA Toolkit and Minimum Required Driver Version for CUDA Minor Version Compatibility[](#id5 "Permalink to this table") | CUDA Toolkit | Minimum Required Driver Version for CUDA Minor Version Compatibility\* | | | --- | --- | --- | | | Linux x86\_64 Driver Version | Windows x86\_64 Driver Version | | CUDA 12.x | \>=525.60.13 | \>=528.33 | | CUDA 11.8.x CUDA 11.7.x CUDA 11.6.x CUDA 11.5.x CUDA 11.4.x CUDA 11.3.x CUDA 11.2.x CUDA 11.1.x | \>=450.80.02 | \>=452.39 | | CUDA 11.0 (11.0.3) | \>=450.36.06\*\* | \>=451.22\*\* | \* Using a Minimum Required Version that is **different** from Toolkit Driver Version could be allowed in compatibility mode – please read the CUDA Compatibility Guide for details. \*\* CUDA 11.0 was released with an earlier driver version, but by upgrading to Tesla Recommended Drivers 450.80.02 (Linux) / 452.39 (Windows), minor version compatibility is possible across the CUDA 11.x family of toolkits. The version of the development NVIDIA GPU Driver packaged in each CUDA Toolkit release is shown below. | | | | | --- | --- | --- |Table 3 CUDA Toolkit and Corresponding Driver Versions[](#id6 "Permalink to this table") | CUDA Toolkit | Toolkit Driver Version | | | --- | --- | --- | | | Linux x86\_64 Driver Version | Windows x86\_64 Driver Version | | CUDA 12.8 Update 1 | \>=570.124.06 | \>=572.61 | | CUDA 12.8 GA | \>=570.117 | \>=572.30 | | CUDA 12.6 Update 3 | \>=560.35.05 | \>=561.17 | | CUDA 12.6 Update 2 | \>=560.35.03 | \>=560.94 | | CUDA 12.6 Update 1 | \>=560.35.03 | \>=560.94 | | CUDA 12.6 GA | \>=560.28.03 | \>=560.76 | | CUDA 12.5 Update 1 | \>=555.42.06 | \>=555.85 | | CUDA 12.5 GA | \>=555.42.02 | \>=555.85 | | CUDA 12.4 Update 1 | \>=550.54.15 | \>=551.78 | | CUDA 12.4 GA | \>=550.54.14 | \>=551.61 | | CUDA 12.3 Update 1 | \>=545.23.08 | \>=546.12 | | CUDA 12.3 GA | \>=545.23.06 | \>=545.84 | | CUDA 12.2 Update 2 | \>=535.104.05 | \>=537.13 | | CUDA 12.2 Update 1 | \>=535.86.09 | \>=536.67 | | CUDA 12.2 GA | \>=535.54.03 | \>=536.25 | | CUDA 12.1 Update 1 | \>=530.30.02 | \>=531.14 | | CUDA 12.1 GA | \>=530.30.02 | \>=531.14 | | CUDA 12.0 Update 1 | \>=525.85.12 | \>=528.33 | | CUDA 12.0 GA | \>=525.60.13 | \>=527.41 | | CUDA 11.8 GA | \>=520.61.05 | \>=520.06 | | CUDA 11.7 Update 1 | \>=515.48.07 | \>=516.31 | | CUDA 11.7 GA | \>=515.43.04 | \>=516.01 | | CUDA 11.6 Update 2 | \>=510.47.03 | \>=511.65 | | CUDA 11.6 Update 1 | \>=510.47.03 | \>=511.65 | | CUDA 11.6 GA | \>=510.39.01 | \>=511.23 | | CUDA 11.5 Update 2 | \>=495.29.05 | \>=496.13 | | CUDA 11.5 Update 1 | \>=495.29.05 | \>=496.13 | | CUDA 11.5 GA | \>=495.29.05 | \>=496.04 | | CUDA 11.4 Update 4 | \>=470.82.01 | \>=472.50 | | CUDA 11.4 Update 3 | \>=470.82.01 | \>=472.50 | | CUDA 11.4 Update 2 | \>=470.57.02 | \>=471.41 | | CUDA 11.4 Update 1 | \>=470.57.02 | \>=471.41 | | CUDA 11.4.0 GA | \>=470.42.01 | \>=471.11 | | CUDA 11.3.1 Update 1 | \>=465.19.01 | \>=465.89 | | CUDA 11.3.0 GA | \>=465.19.01 | \>=465.89 | | CUDA 11.2.2 Update 2 | \>=460.32.03 | \>=461.33 | | CUDA 11.2.1 Update 1 | \>=460.32.03 | \>=461.09 | | CUDA 11.2.0 GA | \>=460.27.03 | \>=460.82 | | CUDA 11.1.1 Update 1 | \>=455.32 | \>=456.81 | | CUDA 11.1 GA | \>=455.23 | \>=456.38 | | CUDA 11.0.3 Update 1 | \>= 450.51.06 | \>= 451.82 | | CUDA 11.0.2 GA | \>= 450.51.05 | \>= 451.48 | | CUDA 11.0.1 RC | \>= 450.36.06 | \>= 451.22 | | CUDA 10.2.89 | \>= 440.33 | \>= 441.22 | | CUDA 10.1 (10.1.105 general release, and updates) | \>= 418.39 | \>= 418.96 | | CUDA 10.0.130 | \>= 410.48 | \>= 411.31 | | CUDA 9.2 (9.2.148 Update 1) | \>= 396.37 | \>= 398.26 | | CUDA 9.2 (9.2.88) | \>= 396.26 | \>= 397.44 | | CUDA 9.1 (9.1.85) | \>= 390.46 | \>= 391.29 | | CUDA 9.0 (9.0.76) | \>= 384.81 | \>= 385.54 | | CUDA 8.0 (8.0.61 GA2) | \>= 375.26 | \>= 376.51 | | CUDA 8.0 (8.0.44) | \>= 367.48 | \>= 369.30 | | CUDA 7.5 (7.5.16) | \>= 352.31 | \>= 353.66 | | CUDA 7.0 (7.0.28) | \>= 346.46 | \>= 347.62 | For convenience, the NVIDIA driver is installed as part of the CUDA Toolkit installation. Note that this driver is for development purposes and is not recommended for use in production with Tesla GPUs. For running CUDA applications in production with Tesla GPUs, it is recommended to download the latest driver for Tesla GPUs from the NVIDIA driver downloads site at [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . During the installation of the CUDA Toolkit, the installation of the NVIDIA driver may be skipped on Windows (when using the interactive or silent installation) or on Linux (by using meta packages). For more information on customizing the install process on Windows, see [https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#install-cuda-software](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#install-cuda-software) . For meta packages on Linux, see [https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-metas](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-metas) . 1.2. New Features[](#new-features "Permalink to this headline") ----------------------------------------------------------------- * This release adds compiler support for the following Nvidia Blackwell GPU architectures: * SM\_100 * SM\_101 * SM\_120 * Tegra-Specific: * Added MPS support for DRIVE OS QNX * Added support for GCC 13.2.0 * Added support for Unified Virtual Memory (UVM) with Extended GPU Memory (EGM) arrays * Hopper Confidential Computing: * Added multi-GPU support for protected PCIe mode * Added key rotation capability for single GPU passthrough mode * NVML Updates: * Fixed per-process memory usage reporting for Docker containers using Open GPU Kernel Module drivers * Added support for DRAM encryption query and control (Blackwell) * Added checkpoint/restore functionality for userspace applications * Added support for Blackwell reduced bandwidth mode (RBM) * CUDA Graphs: * Added conditional execution features for CUDA Graphs: > * ELSE graph support for IF nodes > > * SWITCH node support > * Introduced additional performance optimizations * CUDA Usermode Driver (UMD): * Added PCIe device ID to CUDA device properties * Added cudaStreamGetDevice and cuStreamGetDevice APIs to retrieve the device associated with a CUDA stream * Added CUDA support for INT101010 texture/surface format * Added batch CUDA asynchronous memory copy APIs (cuMemcpyBatchAsync and cuMemcpyBatch3DAsync) for variable-sized transfers between multiple source and destination buffers * Userspace Checkpoint and Restore: * Added new driver API for checkpoint/restore operations ### 1.2.1. CUDA Compiler[](#cuda-compiler "Permalink to this headline") * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-7](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-7) . * Added two new nvcc flags: * `static-global-template-stub {true|false}`: Controls host side linkage for global/device/constant/managed templates in whole program mode * `device-entity-has-hidden-visibility {true|false}`: Controls ELF visibility of global/device/constant/managed symbols The current default value for both flags is false. These defaults will change to true in our future release. For detailed information about these flags and their impact on existing programs, refer to the `nvcc --help` command or the online CUDA documentation. * **libNVVM** `libNVVM` now supports compilation for the Blackwell family of architectures. Compilation of compute capabilities `compute_100` and greater (Blackwell and future architectures) uses an updated NVVM IR dialect, based on LLVM 18.1.8 IR (the “modern” dialect) that differs from the older dialect used for pre-Blackwell architectures (a compute capability less than compute\_100). NVVM IR bitcode using the older dialect generated for pre-Blackwell architectures can be used to target Blackwell and later architectures, with the exception of debug metadata. * **nvdisasm** `Nvdisasm` now supports emitting JSON formatted SASS disassembly. ### 1.2.2. CUDA Developer Tools[](#cuda-developer-tools "Permalink to this headline") * For changes to nvprof and Visual Profiler, see the [changelog](https://docs.nvidia.com/cuda/profiler-users-guide/index.html#changelog) . * For new features, improvements, and bug fixes in Nsight Systems, see the [changelog](https://docs.nvidia.com/nsight-systems/ReleaseNotes/index.html) . * For new features, improvements, and bug fixes in Nsight Visual Studio Edition, see the [changelog](https://docs.nvidia.com/nsight-visual-studio-edition/release-notes/index.html) . * For new features, improvements, and bug fixes in CUPTI, see the [changelog](https://docs.nvidia.com/cupti//release-notes/release-notes.html#) . * For new features, improvements, and bug fixes in Nsight Compute, see the [changelog](https://docs.nvidia.com/nsight-compute/ReleaseNotes/index.html#whats-new) . * For new features, improvements, and bug fixes in Compute Sanitizer, see the [changelog](https://docs.nvidia.com/compute-sanitizer/ReleaseNotes/index.html) . * For new features, improvements, and bug fixes in CUDA-GDB, see the [changelog](https://docs.nvidia.com/cuda/cuda-gdb/index.html#release-notes) . 1.3. Resolved Issues[](#resolved-issues "Permalink to this headline") ----------------------------------------------------------------------- ### 1.3.1. CUDA Compiler[](#id2 "Permalink to this headline") * Resolved compilation issues where code that successfully built with GCC would fail to compile with NVCC on Ubuntu 24.04. This improves cross-compiler compatibility and ensures consistent behavior between GCC and NVIDIA’s CUDA compiler toolchain. \[_4893699_\] * Fixed incorrect handling of C++20 requires expressions, restoring proper functionality and standard compliance. This ensures that compile-time requirements on template parameters now evaluate correctly. \[_4843353_\] * Fixed an issue where NVCC (NVIDIA Compiler Driver) was ignoring the global namespace prefix of a type and thus incorrectly resolving it to a local type that shares the same name. \[_4804685_\] * Fixed a compilation error in NVCC that occurred when code contained three or more nested lambda expressions with variadic arguments. The compiler now properly handles deeply nested variadic lambdas. \[_4782817_\] * Fixed a limitation in NVRTC that caused compilation failures when kernel functions had long identifiers. The runtime compiler now properly handles kernel functions with extended name lengths. \[_4781023_\] * Resolved an issue where template alias resolution could produce incorrect template instances. Previously, when an alias template and its underlying type-id template had different default arguments, the compiler would sometimes incorrectly omit the differing default argument when substituting the alias with its underlying type. This resulted in references to incorrect template instances. The template argument resolution now properly preserves all necessary default arguments during alias substitution. \[_4721362_\] * Fixed invalid error reporting when using variables as template arguments from outside their visible scope. This resolves incorrect diagnostic messages particularly affecting cases involving braced initializers. The compiler now properly validates scope accessibility for template arguments. \[_4717351_\] * Added the ability to cancel ongoing NVRTC compilations through callback mechanisms. This new feature allows developers to safely interrupt and terminate compilation processes programmatically. \[_4082060_\] * The semantics of the `-expt-relaxed-constexpr` nvcc flag are now documented in the “C++ Language Support” section of the CUDA Programming Guide. \[_3288543_\] 1.4. Known Issues and Limitations[](#known-issues-and-limitations "Permalink to this headline") ------------------------------------------------------------------------------------------------- ### 1.4.1. CUDA[](#cuda "Permalink to this headline") * Certain Linux kernels with KASLR enabled have a known issue in HMM initialization, causing CUDA initialization to fail. This issue is indicated by the following debug message: > \[64689.125237\] nvidia-uvm: uvm\_pmm\_gpu.c:3176 devmem\_alloc\_pagemap\[pid:92821\] request\_free\_mem\_region() err \-34 > > Fixes to this issue are being handled in upstream kernels. In the meantime, you can use one of the following workarounds: > > * Option 1: Disable KASLR (Preferred option) > > If using GRUB, edit /etc/default/grub and add `nokaslr` to `GRUB_CMDLINE_LINUX_DEFAULT`: > > > GRUB\_CMDLINE\_LINUX\_DEFAULT="quiet splash nokaslr" > > Then, update GRUB and reboot: > > > sudo update-grub > > sudo reboot > > * Option 2: Disable HMM for UVM > > 1. Create or edit /etc/modprobe.d/uvm.conf. > > 2. Add or update the following line: > > options nvidia\_uvm uvm\_disable\_hmm=1 > > 3. Unload and reload the `nvidia_uvm` kernel module or reboot the system: > > > > sudo modprobe -r nvidia\_uvm > > sudo modprobe nvidia\_uvm > ### 1.4.2. CUDA Compiler[](#id3 "Permalink to this headline") * Some GPUs may experience higher-than-normal context creation times with driver version 570.xx.yyy. For many applications this will likely be unnoticeable, as context creation is usually done at initialization and amortized over the application lifetime. However, applications that create and destroy CUDA contexts frequently may see higher impact. NVIDIA will address this issue in an upcoming driver 570 release. \[_4886848_\] 1.5. Deprecated or Dropped Features[](#deprecated-or-dropped-features "Permalink to this headline") ----------------------------------------------------------------------------------------------------- Features deprecated in the current release of the CUDA software still work in the current release, but their documentation may have been removed, and they will become officially unsupported in a future release. We recommend that developers employ alternative solutions to these features in their software. ### 1.5.1. Deprecated Architectures[](#deprecated-architectures "Permalink to this headline") * Architecture support for Maxwell, Pascal, and Volta is considered feature-complete and will be frozen in an upcoming release. ### 1.5.2. Deprecated or Dropped Operating Systems[](#deprecated-or-dropped-operating-systems "Permalink to this headline") * Support for Microsoft Windows 10 21H2 has been dropped. * Support for Debian 11 has been dropped. * Support for versions prior to SLES 15 Service Pack 4 / OpenSUSE 15.4 has been dropped. * NVTX v2 is deprecated. To migrate to NVTX v3. Change your code from: `#include ` to `#include "nvtx3/nvtoolsext.h"`. This header is included in the toolkit. For the latest NVTX version and extensions, visit [NVIDIA NVTX](https://github.com/NVIDIA/NVTX) . ### 1.5.3. Deprecated CUDA Tools[](#deprecated-cuda-tools "Permalink to this headline") > * Profiling tools supporting pre-turing architectures, Visual Profiler and nvprof, are now deprecated will be dropped in an upcoming release. > > * The CUPTI Event API (from `header cupti_events.h`) and CUPTI Metric API (from `cupti_metrics.h`) are now deprecated and will be dropped in an upcoming release. > > * Nsight Eclipse plugins will no longer be included in Tegra (SOC) packages, such as DriveOS or Jetson. Users of these packages are encouraged to use Nsight Visual Studio Code, available in the VSCode Extension Gallery or from the Microsoft VSCode Marketplace. > > * Support for the macOS host client of CUDA-GDB has been dropped. > 2\. CUDA Libraries[](#cuda-libraries "Permalink to this headline") ==================================================================== This section covers CUDA Libraries release notes for 12.x releases. * CUDA Math Libraries toolchain uses C++11 features, and a C++11-compatible standard library (libstdc++ >= 20150422) is required on the host. 2.1. cuBLAS Library[](#cublas-library "Permalink to this headline") --------------------------------------------------------------------- ### 2.1.1. cuBLAS: Release 12.8 Update 1[](#cublas-release-12-8-update-1 "Permalink to this headline") * **New Features** * Performance Improvements on Nvidia Blackwell GPU Architecure: * Matrix Multiplication (Matmuls): Enhanced performance for FP8 (both block-scaled and tensor-wide scaled), FP4, and FP16/BF16. * BLAS Level 3: Optimized SSYRK, CSYRK, and CHERK operations, especially for unaligned problems. * Batched Operations: Improved efficiency for batched GEMMs and batched GEMVs. * Added support for block-scaled FP8 and FP4 datatypes on Blackwell GeForce-class GPUs. * Improved performance on Blackwell GeForce-class GPUs. * **Resolved Issues** * Using `cublasLtMatmul` with m or n equal to 1 and leading dimensions that cause the input or output matrices to exceed 2^31 elements may result in illegal memory access. \[_5113092, 4959900_\] * Using `cublasLtMatmul` with m or n equal to 1 and the `CUBLASLT_EPILOGUE_BIAS` epilogue may produce incorrect results. \[_5104822_\] * Under rare circumstances, `cublasLtMatmul` running FP8, FP16, or BF16 on a Blackwell GPU may result in a “CUDA Exception: Cluster target block not present” or a “CUDA Error 719: Unspecified launch failure”. \[_5124406_\] ### 2.1.2. cuBLAS: Release 12.8[](#cublas-release-12-8 "Permalink to this headline") * **New Features** * Added support for NVIDIA Blackwell GPU architecture. * Extended the cuBLASLt API to support micro-scaled 4-bit and 8-bit floating-point mixed-precision tensor core-accelerated matrix multiplication for compute capability 10.0 (Blackwell) and higher. Extensions include: * CUDA\_R\_4F\_E2M1: Integration with `CUDA_R_UE4M3` scales and 16-element scaling blocks. * CUDA\_R\_8F variants: Compatibility with `CUDA_R_UE8` scales and 32-element scaling blocks. * [FP8 Matmul Attribute extensions](https://docs.nvidia.com/cuda/cublas/index.html#d-block-scaling-for-fp8-and-fp4-data-types) * Support for block-scaled use cases with scaling factor tensors instead of scalars. * Ability to compute scaling factors dynamically for output tensors when the output is a 4-bit or 8-bit floating-point data type. * Introduced initial support for CUDA in Graphics (CIG) on Windows x64 for NVIDIA Ampere GPU architecture and Blackwell GeForce-class GPUs. CIG contexts are now auto-detected, and cuBLAS selects kernels that comply with CIG shared memory usage limits. * Performance improvement on all Hopper GPUs for non-aligned INT8 matmuls. * **Resolved Issues** * The use of `cublasLtMatmul` with `CUBLASLT_EPILOGUE_BGRAD{A,B}` epilogue allowed the output matrix to be in `CUBLASLT_ORDER_ROW` layout, which led to incorrectly computed bias gradients. This layout is now disallowed when using `CUBLASLT_EPILOGUE_BGRAD{A,B}` epilogue. \[_4910924_\] * **Deprecations** * The experimental feature for [Atomics Synchronization](https://docs.nvidia.com/cuda/cublas/#atomics-synchronization) along rows (`CUBLASLT_MATMUL_DESC_ATOMIC_SYNC_NUM_CHUNKS_D_ROWS`) or columns (`CUBLASLT_MATMUL_DESC_ATOMIC_SYNC_NUM_CHUNKS_D_COLS`) of the output matrix is now deprecated. The functional implementation is still available but not performant and will be removed in a future release. ### 2.1.3. cuBLAS: Release 12.6 Update 2[](#cublas-release-12-6-update-2 "Permalink to this headline") * **New Features** * Broad performance improvement on all Hopper GPUs for FP8, FP16 and BF16 matmuls. This improvement also includes the following fused epilogues `CUBLASLT_EPILOGUE_BIAS`, `CUBLASLT_EPILOGUE_RELU`, `CUBLASLT_EPILOGUE_RELU_BIAS`, `CUBLASLT_EPILOGUE_RELU_AUX`, `CUBLASLT_EPILOGUE_RELU_AUX_BIAS`, `CUBLASLT_EPILOGUE_GELU`, and `CUBLASLT_EPILOGUE_GELU_BIAS`. * **Known Issues** * cuBLAS in multi context scenarios may hang with R535 Driver for version below <535.91. \[_CUB-7024_\] * Users may observe suboptimal performance on Hopper GPUs for FP64 GEMMs. A potential workaround is to conditionally turn on swizzling. To do this, users can take the algo returned via `cublasLtMatmulAlgoGetHeuristic` and query if swizzling can be enabled by calling `cublasLtMatmulAlgoCapGetAttribute` with `CUBLASLT_ALGO_CAP_CTA_SWIZZLING_SUPPORT`. If swizzling is supported, you can enable swizzling by calling `cublasLtMatmulAlgoConfigSetAttribute` with `CUBLASLT_ALGO_CONFIG_CTA_SWIZZLING`. \[_4872420_\] * **Resolved Issues** * `cublasLtMatmul` could ignore the user specified Bias or Aux data types (`CUBLASLT_MATMUL_DESC_BIAS_DATA_TYPE` and `CUBLASLT_MATMUL_DESC_EPILOGUE_AUX_DATA_TYPE`) for FP8 matmul operations if these data types do not match the documented limitations in cublasLtMatmulDescAttributes\_t \_\_. \[_44750343, 4801528_\] * Setting `CUDA_MODULE_LOADING` to `EAGER` could lead to longer library load times on Hopper GPUs due to JIT compilation of PTX kernels. This can be mitigated by setting this environment variable to `LAZY`. \[_4720601_\] * `cublasLtMatmul` with INT8 inputs, INT32 accumulation, INT8 outputs, and FP32 scaling factors could have produced numerical inaccuracies when a `splitk` reduction was used. \[_4751576_\] ### 2.1.4. cuBLAS: Release 12.6 Update 1[](#cublas-release-12-6-update-1 "Permalink to this headline") * **Known Issues** * `cublasLtMatmul` could ignore the user specified Bias or Aux data types (`CUBLASLT_MATMUL_DESC_BIAS_DATA_TYPE` and `CUBLASLT_MATMUL_DESC_EPILOGUE_AUX_DATA_TYPE`) for FP8 matmul operations if these data types do not match the documented limitations in [cublasLtMatmulDescAttributes\_t](https://docs.nvidia.com/cuda/cublas/#cublasltmatmuldescattributes-t) . \[_4750343_\] * Setting `CUDA_MODULE_LOADING` to `EAGER` could lead to longer library load times on Hopper GPUs due to JIT compilation of PTX kernels. This can be mitigated by setting this environment variable to `LAZY`. \[_4720601_\] * `cublasLtMatmul` with INT8 inputs, INT32 accumulation, INT8 outputs, and FP32 scaling factors may produce accuracy issues when a `splitk` reduction is used. To workaround this issue, you can use `cublasLtMatmulAlgoConfigSetAttribute` to set the reduction scheme to none and set the `splitk` value to 1. \[_4751576_\] ### 2.1.5. cuBLAS: Release 12.6[](#cublas-release-12-6 "Permalink to this headline") * **Known Issues** * Computing matrix multiplication and an epilogue with INT8 inputs, INT8 outputs, and FP32 scaling factors can have numerical errors in cases when a second kernel is used to compute the epilogue. This happens because the first GEMM kernel converts the intermediate result from FP32 into INT8 and stores it for the subsequent epilogue kernel to use. If a value is outside of the range of INT8 before the epilogue and the epilogue would bring it into the range of INT8, there will be numerical errors. This issue has existed since before CUDA 12 and there is no known workaround. \[_CUB-6831_\] * `cublasLtMatmul` could ignore the user specified Bias or Aux data types (`CUBLASLT_MATMUL_DESC_BIAS_DATA_TYPE` and `CUBLASLT_MATMUL_DESC_EPILOGUE_AUX_DATA_TYPE`) for FP8 matmul operations if these data types do not match the documented limitations in [cublasLtMatmulDescAttributes\_t](https://docs.nvidia.com/cuda/cublas/#cublasltmatmuldescattributes-t) . \[_4750343_\] * **Resolved Issues** * `cublasLtMatmul` produced incorrect results when data types of matrices `A` and `B` were different FP8 (for example, `A` is `CUDA_R_8F_E4M3` and `B` is `CUDA_R_8F_E5M2`) and matrix `D` layout was `CUBLASLT_ORDER_ROW`. \[_4640468_\] * `cublasLt` may return not supported on Hopper GPUs in some cases when `A`, `B`, and `C` are of type `CUDA_R_8I` and the compute type is `CUBLAS_COMPUTE_32I`. \[_4381102_\] * cuBLAS could produce floating point exceptions when running GEMM with `K` equal to 0. \[_4614629_\] ### 2.1.6. cuBLAS: Release 12.5 Update 1[](#cublas-release-12-5-update-1 "Permalink to this headline") * **New Features** * Performance improvement to matrix multiplication targeting large language models, specifically for small batch sizes on Hopper GPUs. * **Known Issues** * The bias epilogue (without ReLU or GeLU) may be not supported on Hopper GPUs for strided batch cases. A workaround is to implement batching manually. This will be fixed in a future release. * `cublasGemmGroupedBatchedEx` and `cublasgemmGroupedBatched` have large CPU overheads. This will be addressed in an upcoming release. * **Resolved Issues** * Under rare circumstances, executing SYMM/HEMM concurrently with GEMM on Hopper GPUs might have caused race conditions in the host code, which could lead to an Illegal Memory Access CUDA error. \[_4403010_\] * `cublasLtMatmul` could produce an Illegal Instruction CUDA error on Pascal GPUs under the following conditions: batch is greater than 1, and beta is not equal to 0, and the computations are out-of-place (C != D). \[_4566993_\] ### 2.1.7. cuBLAS: Release 12.5[](#cublas-release-12-5 "Permalink to this headline") * **New Features** * cuBLAS adds an experimental API to support mixed precision grouped batched GEMMs. This enables grouped batched GEMMs with FP16 or BF16 inputs/outputs with the FP32 compute type. Refer to [cublasGemmGroupedBatchedEx](https://docs.nvidia.com/cuda/cublas/index.html#cublasgemmgroupedbatchedex) for more details. * **Known Issues** * `cublasLtMatmul` ignores inputs to `CUBLASLT_MATMUL_DESC_D_SCALE_POINTER` and `CUBLASLT_MATMUL_DESC_EPILOGUE_AUX_SCALE_POINTER` if the elements of the respective matrix are not of FP8 types. * **Resolved Issues** * `cublasLtMatmul` ignored the mismatch between the provided scale type and the implied by the documentation, assuming the latter. For instance, an unsupported configuration of `cublasLtMatmul` with the scale type being FP32 and all other types being FP16 would run with the implicit assumption that the scale type is FP16 and produce incorrect results. * cuBLAS SYMV failed for large n dimension: 131072 and above for ssymv, 92673 and above for csymv and dsymv, and 65536 and above for zsymv. ### 2.1.8. cuBLAS: Release 12.4 Update 1[](#cublas-release-12-4-update-1 "Permalink to this headline") * **Known Issues** * Setting a cuBLAS handle stream to `cudaStreamPerThread` and setting the workspace via `cublasSetWorkspace` will cause any subsequent `cublasSetWorkspace` calls to fail. This will be fixed in an upcoming release. * `cublasLtMatmul` ignores mismatches between the provided scale type and the scale type implied by the documentation and assumes the latter. For example, an unsupported configuration of `cublasLtMatmul` with the scale type being FP32 and all other types being FP16 would run with the implicit assumption that the scale type is FP16 which can produce incorrect results. This will be fixed in an upcoming release. * **Resolved Issues** * `cublasLtMatmul` ignored the `CUBLASLT_MATMUL_DESC_AMAX_D_POINTER` for unsupported configurations instead of returning an error. In particular, computing absolute maximum of D is currently supported only for FP8 Matmul when the output data type is also FP8 (`CUDA_R_8F_E4M3` or `CUDA_R_8F_E5M2`). * Reduced host-side overheads for some of the cuBLASLt APIs: `cublasLtMatmul()`, `cublasLtMatmulAlgoCheck()`, and `cublasLtMatmulAlgoGetHeuristic()`. The issue was introduced in CUDA Toolkit 12.4. * `cublasLtMatmul()` and `cublasLtMatmulAlgoGetHeuristic()` could have resulted in floating point exceptions (FPE) on some Hopper-based GPUs, including Multi-Instance GPU (MIG). The issue was introduced in cuBLAS 11.8. ### 2.1.9. cuBLAS: Release 12.4[](#cublas-release-12-4 "Permalink to this headline") * **New Features** * cuBLAS adds experimental APIs to support grouped batched GEMM for single precision and double precision. Single precision also supports the math mode, `CUBLAS_TF32_TENSOR_OP_MATH`. Grouped batch mode allows you to concurrently solve GEMMs of different dimensions (m, n, k), leading dimensions (lda, ldb, ldc), transpositions (transa, transb), and scaling factors (alpha, beta). Please see [gemmGroupedBatched](https://docs.nvidia.com/cuda/cublas/index.html#cublas-t-gemmgroupedbatched) for more details. * **Known Issues** * When the current context has been created using `cuGreenCtxCreate()`, cuBLAS does not properly detect the number of SMs available. The user may provide the corrected SM count to cuBLAS using an API such as `cublasSetSmCountTarget()`. * BLAS level 2 and 3 functions might not treat alpha in a BLAS compliant manner when alpha is zero and the pointer mode is set to `CUBLAS_POINTER_MODE_DEVICE`. This is the same known issue documented in cuBLAS 12.3 Update 1. * `cublasLtMatmul` with K equals 1 and epilogue `CUBLASLT_EPILOGUE_D{RELU,GELU}_BGRAD` could out-of-bound access the workspace. The issue exists since cuBLAS 11.3 Update 1. * `cublasLtMatmul` with K equals 1 and epilogue `CUBLASLT_EPILOGUE_D{RELU,GELU}` could produce illegal memory access if no workspace is provided. The issue exists since cuBLAS 11.6. * When captured in CUDA Graph stream capture, cuBLAS routines can create [memory nodes](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#graph-memory-nodes) through the use of stream-ordered allocation APIs, `cudaMallocAsync` and `cudaFreeAsync`. However, as there is currently no support for memory nodes in [child graphs](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#node-types) or graphs launched [from the device](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#device-graph-launch) , attempts to capture cuBLAS routines in such scenarios may fail. To avoid this issue, use the [cublasSetWorkspace()](https://docs.nvidia.com/cuda/cublas/index.html#cublassetworkspace) function to provide user-owned workspace memory. ### 2.1.10. cuBLAS: Release 12.3 Update 1[](#cublas-release-12-3-update-1 "Permalink to this headline") * **New Features** * Improved performance of heuristics cache for workloads that have a high eviction rate. * **Known Issues** * BLAS level 2 and 3 functions might not treat alpha in a BLAS compliant manner when alpha is zero and the pointer mode is set to `CUBLAS_POINTER_MODE_DEVICE`. The expected behavior is that the corresponding computations would be skipped. You may encounter the following issues: (1) HER{,2,X,K,2K} may zero the imaginary part on the diagonal elements of the output matrix; and (2) HER{,2,X,K,2K}, SYR{,2,X,K,2K} and others may produce NaN resulting from performing computation on matrices A and B which would otherwise be skipped. If strict compliance with BLAS is required, the user may manually check for alpha value before invoking the functions or switch to `CUBLAS_POINTER_MODE_HOST`. * **Resolved Issues** * cuBLASLt matmul operations might have computed the output incorrectly under the following conditions: the data type of matrices A and B is FP8, the data type of matrices C and D is FP32, FP16, or BF16, the beta value is 1.0, the C and D matrices are the same, the epilogue contains GELU activation function. * When an application compiled with cuBLASLt from CUDA Toolkit 12.2 update 1 or earlier runs with cuBLASLt from CUDA Toolkit 12.2 update 2 or CUDA Toolkit 12.3, matrix multiply descriptors initialized using `cublasLtMatmulDescInit()` sometimes did not respect attribute changes using `cublasLtMatmulDescSetAttribute()`. * Fixed creation of cuBLAS or cuBLASLt handles on Hopper GPUs under the Multi-Process Service (MPS). * `cublasLtMatmul` with K equals 1 and epilogue `CUBLASLT_EPILOGUE_BGRAD{A,B}` might have returned incorrect results for the bias gradient. ### 2.1.11. cuBLAS: Release 12.3[](#cublas-release-12-3 "Permalink to this headline") * **New Features** * Improved performance on NVIDIA L40S Ada GPUs. * **Known Issues** * cuBLASLt matmul operations may compute the output incorrectly under the following conditions: the data type of matrices A and B is FP8, the data type of matrices C and D is FP32, FP16, or BF16, the beta value is 1.0, the C and D matrices are the same, the epilogue contains GELU activation function. * When an application compiled with cuBLASLt from CUDA Toolkit 12.2 update 1 or earlier runs with cuBLASLt from CUDA Toolkit 12.2 update 2 or later, matrix multiply descriptors initialized using `cublasLtMatmulDescInit()` may not respect attribute changes using `cublasLtMatmulDescSetAttribute()`. To workaround this issue, create the matrix multiply descriptor using `cublasLtMatmulDescCreate()` instead of `cublasLtMatmulDescInit()`. This will be fixed in an upcoming release. ### 2.1.12. cuBLAS: Release 12.2 Update 2[](#cublas-release-12-2-update-2 "Permalink to this headline") * **New Features** * cuBLASLt will now attempt to decompose problems that cannot be run by a single gemm kernel. It does this by partitioning the problem into smaller chunks and executing the gemm kernel multiple times. This improves functional coverage for very large m, n, or batch size cases and makes the transition from the cuBLAS API to the cuBLASLt API more reliable. * **Known Issues** * cuBLASLt matmul operations may compute the output incorrectly under the following conditions: the data type of matrices A and B is FP8, the data type of matrices C and D is FP32, FP16, or BF16, the beta value is 1.0, the C and D matrices are the same, the epilogue contains GELU activation function. ### 2.1.13. cuBLAS: Release 12.2[](#cublas-release-12-2 "Permalink to this headline") * **Known Issues** * cuBLAS initialization fails on Hopper architecture GPUs when MPS is in use with `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` set to a value less than 100%. There is currently no workaround for this issue. * Some Hopper kernels produce incorrect results for batched matmuls with `CUBLASLT_EPILOGUE_RELU_BIAS` or `CUBLASLT_EPILOGUE_GELU_BIAS` and a non-zero `CUBLASLT_MATMUL_DESC_BIAS_BATCH_STRIDE`. The kernels apply the first batch’s bias vector to all batches. This will be fixed in a future release. ### 2.1.14. cuBLAS: Release 12.1 Update 1[](#cublas-release-12-1-update-1 "Permalink to this headline") * **New Features** * Support for FP8 on NVIDIA Ada GPUs. * Improved performance on NVIDIA L4 Ada GPUs. * Introduced an API that instructs the cuBLASLt library to not use some CPU instructions. This is useful in some rare cases where certain CPU instructions used by cuBLASLt heuristics negatively impact CPU performance. Refer to [https://docs.nvidia.com/cuda/cublas/index.html#disabling-cpu-instructions](https://docs.nvidia.com/cuda/cublas/index.html#disabling-cpu-instructions) . * **Known Issues** * When creating a matrix layout using the `cublasLtMatrixLayoutCreate()` function, the object pointed at by `cublasLtMatrixLayout_t` is smaller than `cublasLtMatrixLayoutOpaque_t` (but enough to hold the internal structure). As a result, the object should not be dereferenced or copied explicitly, as this might lead to out of bound accesses. If one needs to serialize the layout or copy it, it is recommended to manually allocate an object of size `sizeof(cublasLtMatrixLayoutOpaque_t)` bytes, and initialize it using `cublasLtMatrixLayoutInit()` function. The same applies to `cublasLtMatmulDesc_t` and `cublasLtMatrixTransformDesc_t`. The issue will be fixed in future releases by ensuring that `cublasLtMatrixLayoutCreate()` allocates at least `sizeof(cublasLtMatrixLayoutOpaque_t)` bytes. ### 2.1.15. cuBLAS: Release 12.0 Update 1[](#cublas-release-12-0-update-1 "Permalink to this headline") * **New Features** * Improved performance on NVIDIA H100 SXM and NVIDIA H100 PCIe GPUs. * **Known Issues** * For optimal performance on NVIDIA Hopper architecture, cuBLAS needs to allocate a bigger internal workspace (64 MiB) than on the previous architectures (8 MiB). In the current and previous releases, cuBLAS allocates 256 MiB. This will be addressed in a future release. A possible workaround is to set the `CUBLAS_WORKSPACE_CONFIG` environment variable to :32768:2 when running cuBLAS on NVIDIA Hopper architecture. * **Resolved Issues** * Reduced cuBLAS host-side overheads caused by not using the cublasLt heuristics cache. This began in the CUDA Toolkit 12.0 release. * Added forward compatible single precision complex GEMM that does not require workspace. ### 2.1.16. cuBLAS: Release 12.0[](#cublas-release-12-0 "Permalink to this headline") * **New Features** * `cublasLtMatmul` now supports FP8 with a non-zero beta. * Added `int64` APIs to enable larger problem sizes; refer to [64-bit integer interface](https://docs.nvidia.com/cuda/cublas/index.html#int64-interface) . * Added more Hopper-specific kernels for `cublasLtMatmul` with epilogues: * `CUBLASLT_EPILOGUE_BGRAD{A,B}` * `CUBLASLT_EPILOGUE_{RELU,GELU}_AUX` * `CUBLASLT_EPILOGUE_D{RELU,GELU}` * Improved Hopper performance on arm64-sbsa by adding Hopper kernels that were previously supported only on the x86\_64 architecture for Windows and Linux. * **Known Issues** * There are no forward compatible kernels for single precision complex gemms that do not require workspace. Support will be added in a later release. * **Resolved Issues** * Fixed an issue on NVIDIA Ampere architecture and newer GPUs where `cublasLtMatmul` with epilogue `CUBLASLT_EPILOGUE_BGRAD{A,B}` and a nontrivial reduction scheme (that is, not `CUBLASLT_REDUCTION_SCHEME_NONE`) could return incorrect results for the bias gradient. * `cublasLtMatmul` for gemv-like cases (that is, m or n equals 1) might ignore bias with the `CUBLASLT_EPILOGUE_RELU_BIAS` and `CUBLASLT_EPILOGUE_BIAS` epilogues. **Deprecations** * Disallow including `cublas.h` and `cublas_v2.h` in the same translation unit. * Removed: * `CUBLAS_MATMUL_STAGES_16x80` and `CUBLAS_MATMUL_STAGES_64x80` from `cublasLtMatmulStages_t`. No kernels utilize these stages anymore. * `cublasLt3mMode_t`, `CUBLASLT_MATMUL_PREF_MATH_MODE_MASK`, and `CUBLASLT_MATMUL_PREF_GAUSSIAN_MODE_MASK` from `cublasLtMatmulPreferenceAttributes_t`. Instead, use the corresponding flags from `cublasLtNumericalImplFlags_t`. * `CUBLASLT_MATMUL_PREF_POINTER_MODE_MASK`, `CUBLASLT_MATMUL_PREF_EPILOGUE_MASK`, and `CUBLASLT_MATMUL_PREF_SM_COUNT_TARGET` from `cublasLtMatmulPreferenceAttributes_t`. The corresponding parameters are taken directly from `cublasLtMatmulDesc_t`. * `CUBLASLT_POINTER_MODE_MASK_NO_FILTERING` from `cublasLtPointerModeMask_t`. This mask was only applicable to `CUBLASLT_MATMUL_PREF_MATH_MODE_MASK` which was removed. 2.2. cuFFT Library[](#cufft-library "Permalink to this headline") ------------------------------------------------------------------- ### 2.2.1. cuFFT: Release 12.8 Update 1[](#cufft-release-12-8-update-1 "Permalink to this headline") * **Resolved Issues** * Fixed an issue where SM120 was only supported via PTX JIT for legacy callback kernels. SASS support is now available. * Fixed an issue where large applications (over 2 GB in total binary size) linking against the static cuFFT libraries (`libcufft_static.a`, `libcufft_static_nocallback.a`) on x86\_64 systems without using the `-mcmodel=medium` flag would run into linking errors. ### 2.2.2. cuFFT: Release 12.8[](#cufft-release-12-8 "Permalink to this headline") * **New Features** * Added support for the NVIDIA Blackwell GPU architecture. * **Deprecations** * The static library `libcufft_static_nocallback.a` is deprecated and scheduled for removal in a future release. Users should migrate to `libcufft_static.a`, as both libraries provide equivalent functionality following the introduction of LTO callbacks in cuFFT with CUDA Toolkit 12.6 Update 2. * **Known Issues** * SM120 is only supported via PTX JIT for legacy callback kernels. As a result, non-LTO device callback code intended to be linked with `libcufft_static.a` must be compiled to PTX, not SASS. * Large applications (over 2 GB in total binary size) linking against the static cuFFT libraries (`libcufft_static.a`, `libcufft_static_nocallback.a`) in x86\_64 systems without using the `-mcmodel=medium` flag will run into linking errors (For example: `.gcc_except_table relocation R_X86_64_PC32 out of range; references DW.ref._ZTI13cufftResult_t`) This issue will be fixed in an upcoming release. Existing workarounds include: * Building or linking the application with `-mcmodel=medium flag` * Using `readelf` to analyze the `libcufft_static.a` symbols, it is possible to move the reference `ref._ZTI13cufftResult_t` from the large data section `.ldata.DW.ref._ZTI13cufftResult_t` to the non-large data section `.data.DW.ref._ZTI13cufftResult_t` ### 2.2.3. cuFFT: Release 12.6 Update 2[](#cufft-release-12-6-update-2 "Permalink to this headline") * **New Features** * Introduced LTO callbacks as a replacement for the deprecated legacy callbacks. LTO callbacks offer: * Additional performance vs. legacy callbacks * Support for callbacks on Windows and on dynamic (shared) libraries See the [cuFFT documentation](https://docs.nvidia.com/cuda/cufft/index.html) page for more information. * **Resolved Issues** * Several issues present in our [cuFFT LTO EA](https://docs.nvidia.com/cuda/cufft/ltoea/index.html) preview binary have been addressed. * **Deprecations** * [cuFFT LTO EA](https://docs.nvidia.com/cuda/cufft/ltoea/index.html) , our preview binary for LTO callback support, is deprecated and will be removed in the future. ### 2.2.4. cuFFT: Release 12.6[](#cufft-release-12-6 "Permalink to this headline") * **Known Issues** * FFT of size 1 with `istride/ostride > 1` is currently not supported for FP16. There is a known memory issue for this use case in CTK 12.1 or before. A `CUFFT_INVALID_SIZE` error is thrown in CTK 12.2 or after. \[_4662222_\] ### 2.2.5. cuFFT: Release 12.5[](#cufft-release-12-5 "Permalink to this headline") * **New Features** * Added [Just-In-Time Link-Time Optimized (JIT LTO) kernels for improved performance in R2C and C2R FFTs for many sizes](https://docs.nvidia.com/cuda/cufft/index.html#cufft-link-time-optimized-kernels) . * We recommend testing your R2C / C2R use cases with and without JIT LTO kernels and comparing the resulting performance. You can enable JIT LTO kernels using the [per-plan properties](https://docs.nvidia.com/cuda/cufft/index.html#cufft-plan-properties) cuFFT API. ### 2.2.6. cuFFT: Release 12.4 Update 1[](#cufft-release-12-4-update-1 "Permalink to this headline") * **Resolved Issues** * A routine from the [cuFFT LTO EA library](https://docs.nvidia.com/cuda/cufft/ltoea/api/index.html#associating-lto-callbacks-with-cufft-plan) was added by mistake to the cuFFT Advanced API header (`cufftXt.h`) in CUDA 12.4. This routine has now been removed from the header. ### 2.2.7. cuFFT: Release 12.4[](#cufft-release-12-4 "Permalink to this headline") * **New Features** * Added [Just-In-Time Link-Time Optimized (JIT LTO) kernels](https://docs.nvidia.com/cuda/cufft/index.html#cufft-link-time-optimized-kernels) for improved performance in FFTs with 64-bit indexing. * Added [per-plan properties](https://docs.nvidia.com/cuda/cufft/index.html#cufft-plan-properties) to the cuFFT API. These new routines can be leveraged to give users more control over the behavior of cuFFT. Currently they can be used to enable JIT LTO kernels for 64-bit FFTs. * Improved accuracy for certain single-precision (fp32) FFT cases, especially involving FFTs for larger sizes. * **Known Issues** * A routine from the cuFFT LTO EA library was added by mistake to the cuFFT Advanced API header (`cufftXt.h`). This routine is not supported by cuFFT, and will be removed from the header in a future release. * **Resolved Issues** * Fixed an issue that could cause overwriting of user data when performing out-of-place real-to-complex (R2C) transforms with user-specified output strides (i.e. using the `ostride` component of the [Advanced Data Layout API](https://docs.nvidia.com/cuda/cufft/index.html#advanced-data-layout) ). * Fixed inconsistent behavior between `libcufftw` and [FFTW](https://cluster.earlham.edu/bccd-ng/testing/mobeen/GALAXSEEHPC/fftw-3.3/doc/html/Advanced-Complex-DFTs.html) when both `inembed` and `onembed` are `nullptr / NULL`. From now on, as in FFTW, passing `nullptr / NULL` as `inembed/onembed` parameter is equivalent to passing n, that is, the logical size for that dimension. ### 2.2.8. cuFFT: Release 12.3 Update 1[](#cufft-release-12-3-update-1 "Permalink to this headline") * **Known Issues** * Executing a real-to-complex (R2C) or complex-to-real (C2R) plan in a context different to the one used to create the plan could cause undefined behavior. This issue will be fixed in an upcoming release of cuFFT. * **Resolved Issues** * Complex-to-complex (C2C) execution functions (`cufftExec` and similar) now properly error-out in case of error during kernel launch, for example due to a missing CUDA context. ### 2.2.9. cuFFT: Release 12.3[](#cufft-release-12-3 "Permalink to this headline") * **New Features** * Callback kernels are more relaxed in terms of resource usage, and will use fewer registers. * Improved accuracy for double precision prime and composite FFT sizes with factors larger than 127. * Slightly improved planning times for some FFT sizes. ### 2.2.10. cuFFT: Release 12.2[](#cufft-release-12-2 "Permalink to this headline") * **New Features** * `cufftSetStream` can be used in multi-GPU plans with a stream from any GPU context, instead of from the primary context of the first GPU listed in `cufftXtSetGPUs`. * Improved performance of 1000+ of FFTs of sizes ranging from 62 to 16380. The improved performance spans hundreds of single precision and double precision cases for FFTs with contiguous data layout, across multiple GPU architectures (from Maxwell to Hopper GPUs) via PTX JIT. * Reduced the size of the static libraries when compared to cuFFT in the 12.1 release. * **Resolved Issues** * cuFFT no longer exhibits a race condition when threads simultaneously create and access plans with more than 1023 plans alive. * cuFFT no longer exhibits a race condition when multiple threads call `cufftXtSetGPUs` concurrently. ### 2.2.11. cuFFT: Release 12.1 Update 1[](#cufft-release-12-1-update-1 "Permalink to this headline") * **Known Issues** * cuFFT exhibits a race condition when one thread calls `cufftCreate` (or `cufftDestroy`) and another thread calls any API (except `cufftCreate` or `cufftDestroy`), and when the total number of plans alive exceeds 1023. * cuFFT exhibits a race condition when multiple threads call `cufftXtSetGPUs` concurrently on different plans. ### 2.2.12. cuFFT: Release 12.1[](#cufft-release-12-1 "Permalink to this headline") * **New Features** * Improved performance on Hopper GPUs for hundreds of FFTs of sizes ranging from 14 to 28800. The improved performance spans over 542 cases across single and double precision for FFTs with contiguous data layout. * **Known Issues** * Starting from CUDA 11.8, CUDA Graphs are no longer supported for callback routines that load data in out-of-place mode transforms. An upcoming release will update the cuFFT callback implementation, removing this limitation. cuFFT deprecated callback functionality based on separate compiled device code in cuFFT 11.4. * **Resolved Issues** * cuFFT no longer produces errors with compute-sanitizer at program exit if the CUDA context used at plan creation was destroyed prior to program exit. ### 2.2.13. cuFFT: Release 12.0 Update 1[](#cufft-release-12-0-update-1 "Permalink to this headline") * **Resolved Issues** * Scratch space requirements for multi-GPU, single-batch, 1D FFTs were reduced. ### 2.2.14. cuFFT: Release 12.0[](#cufft-release-12-0 "Permalink to this headline") * **New Features** * PTX JIT kernel compilation allowed the addition of many new accelerated cases for Maxwell, Pascal, Volta and Turing architectures. * **Known Issues** * cuFFT plan generation time increases due to PTX JIT compiling. Refer to [Plan Initialization TIme](http://docs.nvidia.com/cuda/cufft/index.html#plan-initialization-time) . * **Resolved Issues** * cuFFT plans had an unintentional small memory overhead (of a few kB) per plan. This is resolved. 2.3. cuSOLVER Library[](#cusolver-library "Permalink to this headline") ------------------------------------------------------------------------- ### 2.3.1. cuSOLVER: Release 12.8[](#cusolver-release-12-8 "Permalink to this headline") * **New Features** * `cusolverDn{SDCZ}sytrf` and `cusolverDnXsytrs` now support symmetric factorization without pivoting when the input pivot array `devIpiv=NULL`, providing improved performance. * `cusolver{DZ}gesvdaStridedBatched` now offers improved accuracy and performance for a wide range of problems. * `cusolver{SDCZ}gesvdaStridedBatched` now returns the number of leading valid singular values and vectors in case of a convergence failure. * **Resolved Issues** * Fixed an issue with `cusolverDnXsyevBatched` when using `cuComplex` or `cuDoubleComplex` with a batch size of at least two, where an incorrect result could be returned if the workspace was not initialized to zero upon entry. * **Deprecations** * The following APIs in `cuSOLVERSp` and `cuSOLVERRf` include deprecation warning in 12.8 \[_4674686_\]: * `cusolverSp{SDCZ}csrlsvluHost` * `cusolverSp{SDCZ}csrlsvcholHost` * `cusolverSp{SDCZ}csrlsvchol` * `cusolverRfSetupHost` * `cusolverRfSetupDevice` * `cusolverRfResetValues` * `cusolverRfAnalyze` * `cusolverRfRefactor` * `cusolverRfAccessBundledFactorsDevice` * `cusolverRfExtractBundledFactorsHost` * `cusolverRfExtractSplitFactorsHost` * `cusolverRfSolve` The deprecation warning can be removed by adding a compiler flag `-DDISABLE_CUSOLVER_DEPRECATED`. Users are encouraged to use the [cuDSS library](https://developer.nvidia.com/cudss) for better performance and ongoing support. Refer to the [cuDSS samples](https://github.com/NVIDIA/CUDALibrarySamples/tree/master/cuDSS) for the transition. ### 2.3.2. cuSOLVER: Release 12.6 Update 2[](#cusolver-release-12-6-update-2 "Permalink to this headline") * **New Features** * New API `cusolverDnXgeev` to solve non-Hermitian eigenvalue problems. * New API `cusolverDnXsyevBatched` to solve uniform batched Hermitian eigenvalue problems. ### 2.3.3. cuSOLVER: Release 12.6[](#cusolver-release-12-6 "Permalink to this headline") * **New Features** * Performance improvements of `cusolverDnXgesvdp()`. ### 2.3.4. cuSOLVER: Release 12.5 Update 1[](#cusolver-release-12-5-update-1 "Permalink to this headline") * **Resolved Issues** * The potential out-of-bound accesses on `bufferOnDevice` by calls of `cusolverDnXlarft` have been resolved. ### 2.3.5. cuSOLVER: Release 12.5[](#cusolver-release-12-5 "Permalink to this headline") * **New Features** * Performance improvements of `cusolverDnXgesvd` and `cusolverDngesvd` if `jobu != 'N'` or `jobvt != 'N'`. * Performance improvements of `cusolverDnXgesvdp` if `jobz = CUSOLVER_EIG_MODE_NOVECTOR`. * Lower workspace requirement of `cusolverDnXgesvdp` for tall-and-skinny-matrices. * **Known Issues** * With CUDA Toolkit 12.4 Update 1, values `ldt > k` in calls of `cusolverDnXlarft` can result in out-of-bound memory accesses on `bufferOnDevice`. As a workaround it is possible to allocate a larger device workspace buffer of size `workspaceInBytesOnDevice=ALIGN_32((ldt*k + n*k)*sizeofCudaDataType(dataTypeT))`, with auto ALIGN\_32\=\[\](int64\_t val) { return ((val + 31)/32)\*32; }; and auto sizeofCudaDataType\=\[\](cudaDataType dt) { if (dt \== CUDA\_R\_32F) return sizeof(float); if (dt \== CUDA\_R\_64F) return sizeof(double); if (dt \== CUDA\_C\_32F) return sizeof(cuComplex); if (dt \== CUDA\_C\_64F) return sizeof(cuDoubleComplex); }; ### 2.3.6. cuSOLVER: Release 12.4 Update 1[](#cusolver-release-12-4-update-1 "Permalink to this headline") * **New Features** * The performance of `cusolverDnXlarft` has been improved. For large matrices, the speedup might exceed 100x. The performance on H100 is now consistently better than on A100. The change in `cusolverDnXlarft` also results in a modest speedup in `cusolverDnormqr`, `cusolverDnormtr`, and `cusolverDnXsyevd`. * The performance of `cusolverDnXgesvd` when singular vectors are sought has been improved. The job configuration that computes both left and right singular vectors is up to 1.5x faster. * **Resolved Issues** * `cusolverDnXtrtri_bufferSize` now returns the correct workspace size in bytes. * **Deprecations** * Using long-deprecated `cusolverDnPotrf`, `cusolverDnPotrs`, `cusolverDnGeqrf`, `cusolverDnGetrf`, `cusolverDnGetrs`, `cusolverDnSyevd`, `cusolverDnSyevdx`, `cusolverDnGesvd`, and their accompanying `bufferSize` functions will result in a deprecation warning. The warning can be turned off by using the `-DDISABLE_CUSOLVER_DEPRECATED` flag while compiling; however, users should use `cusolverDnXpotrf`, `cusolverDnXpotrs`, `cusolverDnXgeqrf`, `cusolverDnXgetrf`, `cusolverDnXgetrs`, `cusolverDnXsyevd`, `cusolverDnXsyevdx`, `cusolverDnXgesvd`, and the corresponding `bufferSize` functions instead. ### 2.3.7. cuSOLVER: Release 12.4[](#cusolver-release-12-4 "Permalink to this headline") * **New Features** * `cusolverDnXlarft` and `cusolverDnXlarft_bufferSize` APIs were introduced. `cusolverDnXlarft` forms the triangular factor of a real block reflector, while `cusolverDnXlarft_bufferSize` returns its required workspace sizes in bytes. * **Known Issues** * `cusolverDnXtrtri_bufferSize` returns an incorrect required device workspace size. As a workaround the returned size can be multiplied by the size of the data type (for example, 8 bytes if matrix A is of type double) to obtain the correct workspace size. ### 2.3.8. cuSOLVER: Release 12.2 Update 2[](#cusolver-release-12-2-update-2 "Permalink to this headline") * **Resolved Issues** * Fixed an issue with `cusolverDngesvd()`, `cusolverDnGesvd()`, and `cusolverDnXgesvd()`, which could cause wrong results for matrices larger than 18918 if `jobu` or `jobvt` was unequal to ‘`N`’. ### 2.3.9. cuSOLVER: Release 12.2[](#cusolver-release-12-2 "Permalink to this headline") * **New Features** * A new API to ensure deterministic results or allow non-deterministic results for improved performance. See `cusolverDnSetDeterministicMode()` and `cusolverDnGetDeterministicMode()`. Affected functions are: `cusolverDngeqrf()`, `cusolverDnsyevd()`, `cusolverDnsyevdx()`, `cusolverDngesvdj()`, `cusolverDnXgeqrf()`, `cusolverDnXsyevd()`, `cusolverDnXsyevdx()`, `cusolverDnXgesvdr()`, and `cusolverDnXgesvdp()`. * **Known Issues** * Concurrent executions of `cusolverDngetrf()` or `cusolverDnXgetrf()` in different non-blocking CUDA streams on the same device might result in a deadlock. 2.4. cuSPARSE Library[](#cusparse-library "Permalink to this headline") ------------------------------------------------------------------------- ### 2.4.1. cuSPARSE: Release 12.8 Update 1[](#cusparse-release-12-8-update-1 "Permalink to this headline") * **Resolved Issues** - `cusparseSpMM` and `cusparseSDDMM` previously produced incorrect results if the output matrix had multiple batches with `batchStride = 0`. This case now returns an error code instead. \[_CUSPARSE-2141_\] * **Known Issues** * Many cuSPARSE routines may not function correctly with large matrices when nnz approaches the signed 32-bit integer limit (e.g., `nnz = 2^31 - 1`), even when using 64-bit indices. \[\[_4966852_\]\ \ * Some cuSPARSE routines may not function correctly with small matrices, particularly those with very few elements or a zero dimension. This issue affects at least `cusparseDenseToSparse` and `cusparseSpMV` with CSR matrices. \[_CUSPARSE-2263_\]\ \ * Many cuSPARSE routines require 16-byte alignment for data arrays to function correctly. This applies to matrix values, indices, offsets, and the temporary buffer. \[_5053391_\]\ \ * `CUSPARSE_SPMM_CSR_ALG1` may return incorrect results when the dense matrix has more than 2^20 - 16 columns.\ \ \ ### 2.4.2. cuSPARSE: Release 12.8[](#cusparse-release-12-8 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for NVIDIA Blackwell GPUs with significant performance improvements in sparse matrix operations:\ \ * SpMV (Sparse Matrix-Vector multiplication): Up to 2.3x faster than Hopper\ \ * SpMM (Sparse Matrix-Matrix multiplication): Up to 2.4x faster than Hopper\ \ * **Resolved Issues**\ \ * Fixed an issue in cusparseSpMM that caused “misaligned address” errors when using the CUSPARSE\_SPMM\_CSR\_ALG3 algorithm with CUDA\_R\_64F data type and mismatched memory layouts between two dense matrices - op(B) and C. \[_CUSPARSE-2081_\]\ \ * Fixed an issue where subsequent calls to SpMV preprocess on the same matrix would fail after the first call. \[_CUSPARSE-1897_\]\ \ * Fixed an issue where SpMV preprocess would not execute when alpha=0. \[_CUSPARSE-1897_\]\ \ * Fixed issues to enable preprocessing operations (SpMV, SpMM, SDDMM) with different memory buffers. \[_CUSPARSE-1962_\]\ \ * Addressed an issue in SpSV where incorrect results occurred when the matrix was in SlicedELL format with lower triangular structure and diagonal elements. \[_CUSPARSE-1996_\]\ \ * **Known Issues**\ \ * SpMM and certain other routines are currently limited when processing matrices approaching 2^31 non-zero elements. \[_CUSPARSE-2133_\]\ \ * **Deprecations**\ \ * The following cuSPARSE functions are deprecated and planned for removal in a future major release \[_4687069_\]:\ \ \ > * `cusparseSpVV()`\ > \ > * `cusparseAxpby()`\ > \ > * `cusparseXgemvi()`\ > \ > * `cusparseSbsr2csr()`\ > \ > * `cusparseSgebsr2csr()`\ > \ > * `cusparseSgebsr2gebsr()`\ > \ > * `cusparseXbsrmm()` (use `cusparseSpMM` instead)\ > \ > \ > Contact [Math-Libs-Feedback@nvidia.com](mailto:Math-Libs-Feedback%40nvidia.com)\ > or visit [https://forums.developer.nvidia.com/](https://forums.developer.nvidia.com/)\ > with any concerns.\ \ * Support for 16-bit complex floating-point (CUDA\_C\_16F) and 16-bit complex bfloat floating-point (CUDA\_C\_16BF) data types will be removed from cuSPARSE in a future release. These data types have been marked as deprecated since CUDA 12.2. \[_CUSPARSE-2225_\]\ \ \ ### 2.4.3. cuSPARSE: Release 12.6 Update 2[](#cusparse-release-12-6-update-2 "Permalink to this headline")\ \ * **Resolved Issues**\ \ * Re-wrote the documentation for `cusparseSpMV_preprocess()`, `cusparseSpMM_preprocess()`, and `cusparseSDDMM_preprocess()`. The documentation now explains the additional constraints that code must satisfy when using these functions. \[_CUSPARSE-1962_\]\ \ * `cusparseSpMV()` would expect the values in the external buffer to be maintained from one call to the next. If this was not true, it could compute the incorrect result or crash. \[_CUSPARSE-1897_\]\ \ * `cusparseSpMV_preprocess()` wouldn’t run correctly if `cusparseSpMM_preprocess()` was executed on the same matrix, and vice versa. \[_CUSPARSE-1897_\]\ \ * `cusparseSpMV_preprocess()` runs SpMV computation if it’s called two or more times on the same matrix. \[_CUSPARSE-1897_\]\ \ * `cusparseSpMV()` could cause subsequent calls to `cusparseSpMM()` with the same matrix to produce incorrect results or crash. \[_CUSPARSE-1897_\]\ \ * With a single sparse matrix `A` and a dense matrix `X` that has only a single column, calling both `cusparseSpMM_preprocess(A,X,...)` could cause subsequent calls to `cusparseSpMV()` to crash or produce incorrect results. The same is true with the roles of SpMV and SpMM swapped. \[_CUSPARSE-1921_\]\ \ \ ### 2.4.4. cuSPARSE: Release 12.6[](#cusparse-release-12-6 "Permalink to this headline")\ \ * **Known Issues**\ \ * `cusparseSpMV_preprocess()` runs SpMV computation if it is called two or more times on the same matrix. \[_CUSPARSE-1897_\]\ \ * `cusparseSpMV_preprocess()` will not run if `cusparseSpMM_preprocess()` was executed on the same matrix, and vice versa. \[_CUSPARSE-1897_\]\ \ * The same external\_buffer must be used for all `cusparseSpMV` calls. \[_CUSPARSE-1897_\]\ \ \ ### 2.4.5. cuSPARSE: Release 12.5 Update 1[](#cusparse-release-12-5-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for BSR format in `cusparseSpMM`.\ \ * **Resolved Issues**\ \ * `cusparseSpMM()` would sometimes get incorrect results when `alpha=0`, `num_batches>1`, `batch_stride` indicates that there is padding between batches.\ \ * `cusparseSpMM_bufferSize()` would return the wrong size when the sparse matrix is Blocked Ellpack and the dense matrices have only a single column (n=1).\ \ * `cusparseSpMM` returned the wrong result when `k=0` (for example when A has zero columns). The correct behavior is doing `C \*= beta`. The bug behavior was not modifying `C` at all.\ \ * `cusparseCreateSlicedEll` would return an error when the slice size is greater than the matrix number of rows.\ \ * Sliced-ELLPACK `cusparseSpSV` produced wrong results for diagonal matrices.\ \ * Sliced-ELLPACK `cusparseSpSV_analysis()` failed due to insufficient resources for some matrices and some slice sizes.\ \ \ ### 2.4.6. cuSPARSE: Release 12.5[](#cusparse-release-12-5 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for mixed input types in SpMV: single precision input matrix, double precision input vector, double precision output vector.\ \ * **Resolved Issues**\ \ * `cusparseSpMV()` introduces invalid memory accesses when the output vector is not aligned to 16 bytes.\ \ \ ### 2.4.7. cuSPARSE: Release 12.4[](#cusparse-release-12-4 "Permalink to this headline")\ \ * **New Features**\ \ * Added the preprocessing step for sparse matrix-vector multiplication `cusparseSpMV_preprocess()`.\ \ * Added support for mixed real and complex types for `cusparseSpMM()`.\ \ * Added a new API `cusparseSpSM_updateMatrix()` to update the sparse matrix between the analysis and solving phase of `cusparseSpSM()`.\ \ * **Known Issues**\ \ * `cusparseSpMV()` introduces invalid memory accesses when the output vector is not aligned to 16 bytes.\ \ * **Resolved Issues**\ \ * `cusparseSpVV()` provided incorrect results when the sparse vector has many non-zeros.\ \ \ ### 2.4.8. cuSPARSE: Release 12.3 Update 1[](#cusparse-release-12-3-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for block sizes of 64 and 128 in `cusparseSDDMM()`.\ \ * Added a preprocessing step `cusparseSDDMM_preprocess()` for BSR `cusparseSDDMM()` that helps improve performance of the main computing stage.\ \ \ ### 2.4.9. cuSPARSE: Release 12.3[](#cusparse-release-12-3 "Permalink to this headline")\ \ * **New Features**\ \ * The `cusparseSpSV_bufferSize()` and `cusparseSpSV_analysis()` routines now accept NULL pointers for the dense vector.\ \ * The `cusparseSpSM_bufferSize()` and `cusparseSpSM_analysis()` routines now accept dense matrix descriptors with NULL pointer for values.\ \ * **Known Issues**\ \ * The `cusparseSpSV_analysis()` and `cusparseSpSM_analysis()` routines are blocking calls/not asynchronous.\ \ * Wrong results can occur for `cusparseSpSV()` using sliced ELLPACK format and transpose/transpose conjugate operation on matrix A.\ \ * **Resolved Issues**\ \ * `cusparseSpSV()` provided indeterministic results in some cases.\ \ * Fixed an issue that caused `cusparseSpSV_analysis()` to hang sometimes in a multi-thread environment.\ \ * Fixed an issue with `cusparseSpSV()` and `cusparseSpSV()` that sometimes yielded wrong output when the output vector/matrix or input matrix contained NaN.\ \ \ ### 2.4.10. cuSPARSE: Release 12.2 Update 1[](#cusparse-release-12-2-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * The library now provides the opportunity to dump sparse matrices to files during the creation of the descriptor for debugging purposes. See logging API [https://docs.nvidia.com/cuda/cusparse/index.html#cusparse-logging-api](https://docs.nvidia.com/cuda/cusparse/index.html#cusparse-logging-api)\ .\ \ * **Resolved Issues**\ \ * Removed `CUSPARSE_SPMM_CSR_ALG3` fallback to avoid confusion in the algorithm selection process.\ \ * Clarified the supported operations for `cusparseSDDMM()`.\ \ * `cusparseCreateConstSlicedEll()` now uses `const` pointers.\ \ * Fixed wrong results in rare edge cases of `cusparseCsr2CscEx2()` with base 1 indexing.\ \ * `cusparseSpSM_bufferSize()` could ask slightly less memory than needed.\ \ * `cusparseSpMV()` now checks the validity of the buffer pointer only when it is strictly needed.\ \ * **Deprecations**\ \ * Several legacy APIs have been officially deprecated. A compile-time warning has been added to all of them.\ \ \ ### 2.4.11. cuSPARSE: Release 12.1 Update 1[](#cusparse-release-12-1-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * Introduced Block Sparse Row (BSR) sparse matrix storage for the Generic APIs with support for SDDMM routine (`cusparseSDDMM`).\ \ * Introduced Sliced Ellpack (SELL) sparse matrix storage format for the Generic APIs with support for sparse matrix-vector multiplication (`cusparseSpMV`) and triangular solver with a single right-hand side (`cusparseSpSV`).\ \ * Added a new API call (`cusparseSpSV_updateMatrix`) to update matrix values and/or the matrix diagonal in the sparse triangular solver with a single right-hand side after the analysis step.\ \ \ ### 2.4.12. cuSPARSE: Release 12.0 Update 1[](#cusparse-release-12-0-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * `cusparseSDDMM()` now supports mixed precision computation.\ \ * Improved `cusparseSpMM()` alg2 mixed-precision performance on some matrices on NVIDIA Ampere architecture GPUs.\ \ * Improved `cusparseSpMV()` performance with a new load balancing algorithm.\ \ * `cusparseSpSV()` and `cusparseSpSM()` now support in-place computation, namely the output and input vectors/matrices have the same memory address.\ \ * **Resolved Issues**\ \ * `cusparseSpSM()` could produce wrong results if the leading dimension (ld) of the RHS matrix is greater than the number of columns/rows.\ \ \ ### 2.4.13. cuSPARSE: Release 12.0[](#cusparse-release-12-0 "Permalink to this headline")\ \ * **New Features**\ \ * JIT LTO functionalities (`cusparseSpMMOp()`) switched from driver to nvJitLto library. Starting from CUDA 12.0 the user needs to link to `libnvJitLto.so`, see [cuSPARSE documentation](https://docs.nvidia.com/cuda/cusparse/index.html)\ . JIT LTO performance has also been improved for `cusparseSpMMOpPlan()`.\ \ * Introduced const descriptors for the Generic APIs, for example, `cusparseConstSpVecGet()`. Now the Generic APIs interface clearly declares when a descriptor and its data are modified by the cuSPARSE functions.\ \ * Added two new algorithms to `cusparseSpGEMM()` with lower memory utilization. The first algorithm computes a strict bound on the number of intermediate product, while the second one allows partitioning the computation in chunks.\ \ * Added `int8_t` support to `cusparseGather()`, `cusparseScatter()`, and `cusparseCsr2cscEx2()`.\ \ * Improved `cusparseSpSV()` performance for both the analysis and the solving phases.\ \ * Improved `cusparseSpSM()` performance for both the analysis and the solving phases.\ \ * Improved `cusparseSDDMM()` performance and added support for batch computation.\ \ * Improved `cusparseCsr2cscEx2()` performance.\ \ * **Resolved Issues**\ \ * `cusparseSpSV()` and `cusparseSpSM()` could produce wrong results.\ \ * `cusparseDnMatGetStridedBatch()` did not accept `batchStride == 0`.\ \ * **Deprecations**\ \ * Removed deprecated CUDA 11.x APIs, enumerators, and descriptors.\ \ \ 2.5. Math Library[](#math-library "Permalink to this headline")\ \ -----------------------------------------------------------------\ \ ### 2.5.1. CUDA Math: Release 12.8 Update 1[](#cuda-math-release-12-8-update-1 "Permalink to this headline")\ \ > * Users of the E8M0 (`__nv_fp8_e0m8`) types defined in `cuda_fp8.h` should be aware of a change in the rounding behavior for the C++ converting constructors when converting from other floating-point and integer types. The constructors now take the absolute value of the input and apply round-toward-positive-infinity rounding with saturation to convert to the E8M0 representation. Previously, the constructors used absolute value with round-toward-zero rounding and saturation. This previous behavior can now be accessed through specific conversion functions, such as `__nv_cvt_bfloat16raw_to_e8m0`. \[_5066830_\]\ > \ \ ### 2.5.2. CUDA Math: Release 12.8[](#cuda-math-release-12-8 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for several new floating point datatypes:\ \ * E2M1 (2-bit exponent, 1-bit mantissa)\ \ * E2M3 (2-bit exponent, 3-bit mantissa)\ \ * E3M2 (3-bit exponent, 2-bit mantissa)\ \ * E8M0 (8-bit exponent, 0-bit mantissa)\ \ \ For detailed information about FP4, FP6, and FP8 types, including conversion operators and intrinsics, refer to the CUDA Math API documentation. \[_CUMATH-1385_\]\ \ * Conversion operations for these types are natively supported by specific devices (e.g. devices of compute capability 10.0a), other devices use emulation path.\ \ * Optimized standard single precision hyperbolic tangent (`tanhf()`) function, achieving 30-40% faster performance. \[_4557267_\]\ \ * Added several new tanh implementations:\ \ * `__tanhf(float x)`: New fast reduced-accuracy math intrinsic\ \ * `htanh()` and `h2tanh()`: tanh functions for half and bfloat16 types in scalar and packed formats\ \ * `htanh_approx()` and `h2tanh_approx()`: Fast reduced-accuracy versions\ \ \ Refer to CUDA Math API documentation for detailed usage information. \[_CUMATH-6821_\]\ \ * Added support for quad-precision `__float128` data type and select math library operations in device computations on GPUs with compute capability 10.0 and above. Refer to CUDA Math API documentation for details. \[_CUMATH-5463_\]\ \ * **Known Issues**\ \ * When converting to MXFP4/MXFP6/MXFP8 formats developers should not use the C++ converting constructors, which currently implement only round-toward-zero behavior. Conversions to MXFP formats should use round-toward-positive-infinity, which is implemented as an option in conversion functions like `__nv_cvt_bfloat16raw_to_e8m0`. C++ converting constructors behavior will change in a future update.\ \ \ ### 2.5.3. CUDA Math: Release 12.6 Update 1[](#cuda-math-release-12-6-update-1 "Permalink to this headline")\ \ * **Resolved Issues**\ \ * Issue 4731352 from release 12.6 is resolved.\ \ \ ### 2.5.4. CUDA Math: Release 12.6[](#cuda-math-release-12-6 "Permalink to this headline")\ \ * **Known Issues**\ \ > * As a result of ongoing compatibility testing NVIDIA identified that a number of CUDA Math Integer SIMD APIs silently produced wrong results if used on the CPU in programs compiled with MSVC 17.10. The root cause is found to be the coding error in the header-based implementation of the APIs exposed to the undefined behavior during narrowing integer conversion when doing a host-based emulation of the GPU functionality. The issue will be fixed in a future release of CUDA. Applications affected are those calling `__vimax3_s16x2`, `__vimin3_s16x2`, `__vibmax_s16x2`, and `__vibmin_s16x2` on the CPU and not in CUDA kernels. \[_4731352_\]\ > \ \ \ ### 2.5.5. CUDA Math: Release 12.5[](#cuda-math-release-12-5 "Permalink to this headline")\ \ * **Known Issues**\ \ * As a result of ongoing testing we updated the interval bounds in which double precision `lgamma()` function may experience greater than the documented 4 ulp accuracy loss. New interval shall read (-23.0001; -2.2637). This finding is applicable to CUDA 12.5 and all previous versions. \[_4662420_\]\ \ \ ### 2.5.6. CUDA Math: Release 12.4[](#cuda-math-release-12-4 "Permalink to this headline")\ \ * **Resolved Issues**\ \ * Host-specific code in `cuda_fp16/bf16` headers is now free from type-punning and shall work correctly in the presence of optimizations based on strict-aliasing rules. \[_4311216_\]\ \ \ ### 2.5.7. CUDA Math: Release 12.3[](#cuda-math-release-12-3 "Permalink to this headline")\ \ * **New Features**\ \ * Performance of SIMD Integer CUDA Math APIs was improved.\ \ * **Resolved Issues**\ \ * The `__hisinf()` Math APIs from `cuda_fp16.h` and `cuda_bf16.h` headers were silently producing wrong results if compiled with the `-std=c++20` compiler option because of an underlying nvcc compiler issue, resolved in version 12.3.\ \ * **Known Issues**\ \ * Users of `cuda_fp16.h` and `cuda_bf16.h` headers are advised to disable host compilers strict aliasing rules based optimizations (e.g. pass `-fno-strict-aliasing` to host GCC compiler) as these may interfere with the type-punning idioms used in the `__half`, `__half2`, `__nv_bfloat16`, `__nv_bfloat162` types implementations and expose the user program to undefined behavior. Note, the headers suppress GCC diagnostics through: #pragma GCC diagnostic ignored `-Wstrict-aliasing`. This behavior may improve in future versions of the headers.\ \ \ ### 2.5.8. CUDA Math: Release 12.2[](#cuda-math-release-12-2 "Permalink to this headline")\ \ * **New Features**\ \ * CUDA Math APIs for `__half` and `__nv_bfloat16` types received usability improvements, including host side support for many of the arithmetic operations and conversions.\ \ * `__half` and `__nv_bfloat16` types have implicit conversions to/from integral types, which are now available with host compilers by default. These may cause build issues due to ambiguous overloads resolution. Users are advised to update their code to select proper overloads. To opt-out user may want to define the following macros (these macros will be removed in the future CUDA release):\ \ * `__CUDA_FP16_DISABLE_IMPLICIT_INTEGER_CONVERTS_FOR_HOST_COMPILERS__`\ \ * `__CUDA_BF16_DISABLE_IMPLICIT_INTEGER_CONVERTS_FOR_HOST_COMPILERS__`\ \ * **Resolved Issues**\ \ * During ongoing testing, NVIDIA identified that due to an algorithm error the results of 64-bit floating-point division in default round-to-nearest-even mode could produce spurious overflow to infinity. NVIDIA recommends that all developers requiring strict IEEE754 compliance update to CUDA Toolkit 12.2 or newer. The affected algorithm was present in both offline compilation as well as just-in-time (JIT) compilation. As JIT compilation is handled by the driver, NVIDIA recommends updating to driver version greater than or equal to R535 (R536 on Windows) when IEEE754 compliance is required and when using JIT. This is a software algorithm fix and is not tied to specific hardware.\ \ * Updated the observed worst case error bounds for single precision intrinsic functions `__expf()`, `__exp10f()` and double precision functions `asinh()`, `acosh()`.\ \ \ ### 2.5.9. CUDA Math: Release 12.1[](#cuda-math-release-12-1 "Permalink to this headline")\ \ * **New Features**\ \ * Performance and accuracy improvements in `atanf`, `acosf`, `asinf`, `sinpif`, `cospif`, `powf`, `erff`, and `tgammaf`.\ \ \ ### 2.5.10. CUDA Math: Release 12.0[](#cuda-math-release-12-0 "Permalink to this headline")\ \ * **New Features**\ \ * Introduced new integer/fp16/bf16 CUDA Math APIs to help expose performance benefits of new DPX instructions. Refer to [https://docs.nvidia.com/cuda/cuda-math-api/index.html](https://docs.nvidia.com/cuda/cuda-math-api/index.html)\ .\ \ * **Known Issues**\ \ * Double precision inputs that cause the double precision division algorithm in the default ‘round to nearest even mode’ produce spurious overflow: an infinite result is delivered where `DBL_MAX 0x7FEF_FFFF_FFFF_FFFF` is expected. Affected CUDA Math APIs: `__ddiv_rn()`. Affected CUDA language operation: double precision / operation in the device code.\ \ * **Deprecations**\ \ * All previously deprecated undocumented APIs are removed from CUDA 12.0.\ \ \ 2.6. NVIDIA Performance Primitives (NPP)[](#nvidia-performance-primitives-npp "Permalink to this headline")\ \ -------------------------------------------------------------------------------------------------------------\ \ ### 2.6.1. NPP: Release 12.4[](#npp-release-12-4 "Permalink to this headline")\ \ * **New Features**\ \ * Enhanced large file support with `size_t`.\ \ \ ### 2.6.2. NPP: Release 12.0[](#npp-release-12-0 "Permalink to this headline")\ \ * **Deprecations**\ \ * Deprecating non-CTX API support from next release.\ \ * **Resolved Issues**\ \ * A performance issue with the NPP `ResizeSqrPixel` API is now fixed and shows improved performance.\ \ \ 2.7. nvJPEG Library[](#nvjpeg-library "Permalink to this headline")\ \ ---------------------------------------------------------------------\ \ ### 2.7.1. nvJPEG: Release 12.8[](#nvjpeg-release-12-8 "Permalink to this headline")\ \ * **New Features**\ \ * Added hardware-accelerated JPEG decoding support in nvJPEG for NVIDIA Blackwell architecture GPUs.\ \ * The nvJPEG library now uses significantly less GPU memory during encoding, achieving memory savings of 30% to 50%, depending on image size and chroma subsampling mode. For images larger than 5 MB (approximately 2K x 1K pixels) and popular subsampling modes such as 4:2:2 and 4:2:0, memory savings are around 50%. Additionally, nvJPEG no longer artificially runs out of memory when processing large or complex images, enhancing its reliability and performance.\ \ * **Resolved Issues**\ \ * Resolved an issue in nvJPEG that prevented the correct encoding of very small images with dimensions less than 25 pixels. \[_4655922_\]\ \ * Fixed an issue that caused out-of-bound reads when decoding a truncated JPEG file using `nvjpegDecodeJpegHost` with the `NVJPEG_BACKEND_GPU_HYBRID backend`. \[_4663831_\]\ \ \ ### 2.7.2. nvJPEG: Release 12.4[](#nvjpeg-release-12-4 "Permalink to this headline")\ \ * **New Features**\ \ * IDCT performance optimizations for single image CUDA decode.\ \ * Zero Copy behavior has been changed: Setting `NVJPEG_FLAGS_REDUCED_MEMORY_DECODE_ZERO_COPY` flag will no longer enable `NVJPEG_FLAGS_REDUCED_MEMORY_DECODE`.\ \ \ ### 2.7.3. nvJPEG: Release 12.3 Update 1[](#nvjpeg-release-12-3-update-1 "Permalink to this headline")\ \ * **New Features**\ \ * New APIs: `nvjpegBufferPinnedResize` and `nvjpegBufferDeviceResize` which can be used to resize pinned and device buffers before using them.\ \ \ ### 2.7.4. nvJPEG: Release 12.2[](#nvjpeg-release-12-2 "Permalink to this headline")\ \ * **New Features**\ \ * Added support for JPEG Lossless decode (process 14, FO prediction).\ \ * nvJPEG is now supported on L4T.\ \ \ ### 2.7.5. nvJPEG: Release 12.0[](#nvjpeg-release-12-0 "Permalink to this headline")\ \ * **New Features**\ \ * Immproved the GPU Memory optimisation for the nvJPEG codec.\ \ * **Resolved Issues**\ \ * An issue that causes runtime failures when `nvJPEGDecMultipleInstances` was tested with a large number of threads is resolved.\ \ * An issue with CMYK four component color conversion is now resolved.\ \ * **Known Issues**\ \ * Backend `NVJPEG_BACKEND_GPU_HYBRID` - Unable to handle bistreams with extra scans lengths.\ \ * **Deprecations**\ \ * The reuse of Huffman table in Encoder (`nvjpegEncoderParamsCopyHuffmanTables`).\ \ \ [1](#id1)\ \ Only available on select Linux distros\ \ 3\. Notices[](#notices "Permalink to this headline")\ \ ======================================================\ \ 3.1. Notice[](#notice "Permalink to this headline")\ \ -----------------------------------------------------\ \ This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.\ \ NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.\ \ Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.\ \ NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.\ \ NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.\ \ NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.\ \ No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.\ \ Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.\ \ THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.\ \ 3.2. OpenCL[](#opencl "Permalink to this headline")\ \ -----------------------------------------------------\ \ OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc.\ \ 3.3. Trademarks[](#trademarks "Permalink to this headline")\ \ -------------------------------------------------------------\ \ NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Introduction — Installation Guide for Linux 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/CUDA_Installation_Guide_Linux.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA CUDA Installation Guide for Linux The installation instructions for the CUDA Toolkit on Linux. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ CUDA® is a parallel computing platform and programming model invented by NVIDIA®. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU). CUDA was developed with several design goals in mind: * Provide a small set of extensions to standard programming languages, like C, that enable a straightforward implementation of parallel algorithms. With CUDA C/C++, programmers can focus on the task of parallelization of the algorithms rather than spending time on their implementation. * Support heterogeneous computation where applications use both the CPU and GPU. Serial portions of applications are run on the CPU, and parallel portions are offloaded to the GPU. As such, CUDA can be incrementally applied to existing applications. The CPU and GPU are treated as separate devices that have their own memory spaces. This configuration also allows simultaneous computation on the CPU and GPU without contention for memory resources. CUDA-capable GPUs have hundreds of cores that can collectively run thousands of computing threads. These cores have shared resources including a register file and a shared memory. The on-chip shared memory allows parallel tasks running on these cores to share data without sending it over the system memory bus. This guide will show you how to install and check the correct operation of the CUDA development tools. Note Instructions for installing NVIDIA Drivers are now in [https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html) . 1.1. System Requirements[](#system-requirements "Permalink to this headline") ------------------------------------------------------------------------------- To use NVIDIA CUDA on your system, you will need the following installed: * CUDA-capable GPU * A supported version of Linux with a gcc compiler and toolchain * CUDA Toolkit (available at [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads) ) The CUDA development environment relies on tight integration with the host development environment, including the host compiler and C runtime libraries, and is therefore only supported on distribution versions that have been qualified for this CUDA Toolkit release. The following table lists the supported Linux distributions. Please review the footnotes associated with the table. | | | | | | --- | --- | --- | --- |Table 1 Native Linux Distribution Support in CUDA 12.8 Update 1[](#id47 "Permalink to this table") | Distribution | Kernel1 | Default GCC | GLIBC | | --- | --- | --- | --- | | **x86\_64** | | | | | RHEL 9.y (y <= 5) | 5.14.0-503 | 11.5.0 | 2.34 | | RHEL 8.y (y <= 10) | 4.18.0-553 | 8.5.0 | 2.28 | | OpenSUSE Leap 15.y (y == 6) | 6.4.0-150600.21 | 7.5.0 | 2.38 | | Rocky Linux 8.y (y<=10) | 4.18.0-553 | 8.5.0 | 2.28 | | Rocky Linux 9.y (y<=5) | 5.14.0-503 | 11.5.0 | 2.34 | | Ubuntu 24.04.z (z <= 1) LTS | 6.8.0-41 | 13.2.0 | 2.39 | | Ubuntu 22.04.z (z <= 5) LTS | 6.5.0-45 | 12.3.0 | 2.35 | | Ubuntu 20.04.z (z <= 6) LTS | 5.15.0-67 | 9.4.0 | 2.31 | | Debian 12.9 | 6.1.123-1 | 12.2.0 | 2.36 | | Fedora 41 | 6.11.4-301 | 14.2.1 | 2.40 | | KylinOS V10 SP3 2403 | 4.19.90-89.11.v2401 | 10.x | 2.28 | | MSFT Azure Linux 2.0 | 5.15.158.2-1 | 11.2.0 | 2.35 | | Amazon Linux 2023 | 6.1.82-99.168 | 11.4.1 | 2.34 | | Oracle Linux 8 | 4.18.0-553 | 8.5.0 | 2.28 | | Oracle Linux 9 | 5.14.0-427 | 11.4.1 | 2.34 | | **Arm64 sbsa** | | | | | RHEL 9.y (y <= 5) | 5.14.0-503 | 11.5.0 | 2.34 | | RHEL 8.y (y <= 10) | 4.18.0-553 | 8.5.0 | 2.28 | | SUSE SLES 15.y (y == 6) | 6.4.0-150600.21 | 7.5.0 | 2.38 | | Kylin V10 SP3 2403 | 4.19.90-89 | 10.x | 2.28 | | Ubuntu 24.04.z (z <= 1) LTS | 6.8.0-31 | 13.2.0 | 2.39 | | Ubuntu 22.04 LTS (z <= 5) LTS | 6.5.0-1019 | 11.4.0 | 2.35 | | Ubuntu 20.04.z (z <= 5) LTS | 5.4.0-174 | 11.4.0 | 2.31 | | **Arm64 sbsa Jetson (dGPU)** | | | | | 20.04.06 LTS Rel35 JP 5.x | 5.10.65-tegra | 9.4.0 | 2.31 | | 22.04.4 LTS Rel36 - JP6.x | 5.15.136-tegra | 11.4.0 | 2.35 | | **AArch64 Jetson (iGPU)** | | | | | L4T Ubuntu 22.04 Rel36 - JP6.x | 6.1.80-tegra | 11.4.0 | 2.35 | 1. The following notes apply to the kernel versions supported by CUDA: > * For specific kernel versions supported on Red Hat Enterprise Linux (RHEL), visit [https://access.redhat.com/articles/3078](https://access.redhat.com/articles/3078) > . > > * A list of kernel versions including the release dates for SUSE Linux Enterprise Server (SLES) is available at [https://www.suse.com/support/kb/doc/?id=000019587](https://www.suse.com/support/kb/doc/?id=000019587) > . > 2. Support for Debian 11.x is deprecated. 1.2. OS Support Policy[](#os-support-policy "Permalink to this headline") --------------------------------------------------------------------------- * CUDA support for Ubuntu 20.04.x, Ubuntu 22.04.x, Ubuntu 24.04.x, RHEL 8.x, RHEL 9.x, Rocky Linux 8.x, Rocky Linux 9.x, SUSE SLES 15.x, OpenSUSE Leap 15.x, Amazon linux 2023, and Azure Linux 2.0 will be until the standard EOSS as defined for each OS. Please refer to the support lifecycle for these OSes to know their support timelines. * CUDA supports the latest Fedora release version. The version supported might require a specific GCC compatibility package. For Fedora release timelines, visit [https://docs.fedoraproject.org/en-US/releases/](https://docs.fedoraproject.org/en-US/releases/) . * CUDA supports a single KylinOS release version. For details, visit [https://www.kylinos.cn/](https://www.kylinos.cn/) . Refer to the support lifecycle for these supported OSes to know their support timelines and plan to move to newer releases accordingly. 1.3. Host Compiler Support Policy[](#host-compiler-support-policy "Permalink to this headline") ------------------------------------------------------------------------------------------------- In order to compile the CPU “Host” code in the CUDA source, the CUDA compiler NVCC requires a compatible host compiler to be installed on the system. The version of the host compiler supported on Linux platforms is tabulated as below. NVCC performs a version check on the host compiler’s major version and so newer minor versions of the compilers listed below will be supported, but major versions falling outside the range will not be supported. | | | | | | | | | --- | --- | --- | --- | --- | --- | --- |Table 2 Supported Compilers[](#id48 "Permalink to this table") | Distribution | GCC | Clang | NVHPC | XLC | ArmC/C++ | ICC | | --- | --- | --- | --- | --- | --- | --- | | x86\_64 | 6.x - 14.x | 7.x - 19.x | 24.9 | No | No | 2021.7 | | Arm64 sbsa | 6.x - 14.x | 7.x - 19.x | 24.9 | No | 24.04 | No | For GCC and Clang, the preceding table indicates the minimum version and the latest version supported. If you are on a Linux distribution that may use an older version of GCC toolchain as default than what is listed above, it is recommended to upgrade to a newer toolchain CUDA 11.0 or later toolkit. Newer GCC toolchains are available with the Red Hat Developer Toolset for example. For platforms that ship a compiler version older than GCC 6 by default, linking to static or dynamic libraries that are shipped with the CUDA Toolkit is not supported. We only support libstdc++ (GCC’s implementation) for all the supported host compilers for the platforms listed above. ### 1.3.1. Host Compiler Compatibility Packages[](#host-compiler-compatibility-packages "Permalink to this headline") Really up to date distributions might ship with a newer compiler than what is covered by the Supported Compilers table above. Usually, those distribution also provide a GCC compatibility package that can be used instead of the default one. Depending on the distribution, the package that needs to be installed is different, but the logic for configuring it is the same. If required, configuration steps are described in the relevant section for the specific Linux distribution, but they always end up with configuring the `NVCC_BIN` environment variable as described in the [NVCC documentation](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#nvcc-environment-variables) . ### 1.3.2. Supported C++ Dialects[](#supported-c-dialects "Permalink to this headline") NVCC and NVRTC (CUDA Runtime Compiler) support the following C++ dialect: C++11, C++14, C++17, C++20 on supported host compilers. The default C++ dialect of NVCC is determined by the default dialect of the host compiler used for compilation. Refer to host compiler documentation and the _CUDA Programming Guide_ for more details on language support. C++20 is supported with the following flavors of host compiler in both host and device code. | GCC | Clang | NVHPC | Arm C/C++ | | --- | --- | --- | --- | | \>=10.x | \>=11.x | \>=22.x | \>=22.x | 1.4. About This Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This document is intended for readers familiar with the Linux environment and the compilation of C programs from the command line. You do not need previous experience with CUDA or experience with parallel computation. Note: This guide covers installation only on systems with X Windows installed. Note Many commands in this document might require _superuser_ privileges. On most distributions of Linux, this will require you to log in as root. For systems that have enabled the sudo package, use the sudo prefix for all necessary commands. 2\. Pre-installation Actions[](#pre-installation-actions "Permalink to this headline") ======================================================================================== Some actions must be taken before the CUDA Toolkit can be installed on Linux: * Verify the system has a CUDA-capable GPU. * Verify the system is running a supported version of Linux. * Verify the system has gcc installed. * Download the NVIDIA CUDA Toolkit. * Handle conflicting installation methods. Note You can override the install-time prerequisite checks by running the installer with the `-override` flag. Remember that the prerequisites will still be required to use the NVIDIA CUDA Toolkit. 2.1. Verify You Have a CUDA-Capable GPU[](#verify-you-have-a-cuda-capable-gpu "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- To verify that your GPU is CUDA-capable, go to your distribution’s equivalent of System Properties, or, from the command line, enter: lspci | grep -i nvidia If you do not see any settings, update the PCI hardware database that Linux maintains by entering `update-pciids` (generally found in `/sbin`) at the command line and rerun the previous `lspci` command. If your graphics card is from NVIDIA and it is listed in [https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) , your GPU is CUDA-capable. The Release Notes for the CUDA Toolkit also contain a list of supported products. 2.2. Verify You Have a Supported Version of Linux[](#verify-you-have-a-supported-version-of-linux "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------- The CUDA Development Tools are only supported on some specific distributions of Linux. These are listed in the CUDA Toolkit release notes. To determine which distribution and release number you’re running, type the following at the command line: uname -m && cat /etc/\*release You should see output similar to the following, modified for your particular system: x86\_64 Red Hat Enterprise Linux Workstation release 6.0 (Santiago) The `x86_64` line indicates you are running on a 64-bit system. The remainder gives information about your distribution. 2.3. Verify the System Has gcc Installed[](#verify-the-system-has-gcc-installed "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- The `gcc` compiler is required for development using the CUDA Toolkit. It is not required for running CUDA applications. It is generally installed as part of the Linux installation, and in most cases the version of gcc installed with a supported version of Linux will work correctly. To verify the version of gcc installed on your system, type the following on the command line: gcc --version If an error message displays, you need to install the development tools from your Linux distribution or obtain a version of `gcc` and its accompanying toolchain from the Web. 2.4. Choose an Installation Method[](#choose-an-installation-method "Permalink to this headline") --------------------------------------------------------------------------------------------------- The CUDA Toolkit can be installed using either of two different installation mechanisms: distribution-specific packages (RPM and Deb packages), or a distribution-independent package (runfile packages). The distribution-independent package has the advantage of working across a wider set of Linux distributions, but does not update the distribution’s native package management system. The distribution-specific packages interface with the distribution’s native package management system. It is recommended to use the distribution-specific packages, where possible. Note For both native as well as cross development, the toolkit must be installed using the distribution-specific installer. See the [CUDA Cross-Platform Installation](#cuda-cross-platform-installation) section for more details. 2.5. Download the NVIDIA CUDA Toolkit[](#download-the-nvidia-cuda-toolkit "Permalink to this headline") --------------------------------------------------------------------------------------------------------- The NVIDIA CUDA Toolkit is available at [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads) . Choose the platform you are using and download the NVIDIA CUDA Toolkit. The CUDA Toolkit contains the tools needed to create, build and run a CUDA application as well as libraries, header files, and other resources. **Download Verification** The download can be verified by comparing the MD5 checksum posted at [https://developer.download.nvidia.com/compute/cuda/12.6.2/docs/sidebar/md5sum.txt](https://developer.download.nvidia.com/compute/cuda/12.6.2/docs/sidebar/md5sum.txt) with that of the downloaded file. If either of the checksums differ, the downloaded file is corrupt and needs to be downloaded again. To calculate the MD5 checksum of the downloaded file, run the following: md5sum 2.6. Handle Conflicting Installation Methods[](#handle-conflicting-installation-methods "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------- Before installing CUDA, any previous installations that could conflict should be uninstalled. This will not affect systems which have not had CUDA installed previously, or systems where the installation method has been preserved (RPM/Deb vs. Runfile). See the following charts for specifics. | | | | | | | | --- | --- | --- | --- | --- | --- |Table 3 CUDA Toolkit Installation Compatibility Matrix[](#id49 "Permalink to this table") | | | Installed Toolkit Version == X.Y | | Installed Toolkit Version != X.Y | | | RPM/Deb | run | RPM/Deb | run | | Installing Toolkit Version X.Y | RPM/Deb | No Action | Uninstall Run | No Action | No Action | | run | Uninstall RPM/Deb | Uninstall Run | No Action | No Action | Use the following command to uninstall a Toolkit runfile installation: sudo /usr/local/cuda-X.Y/bin/cuda-uninstaller Use the following commands to uninstall an RPM/Deb installation: sudo dnf remove # RHEL 8 / Rocky Linux 8 / RHEL 9 / Rocky Linux 9 / Fedora / KylinOS 10 / Amazon Linux 2023 sudo tdnf remove # Azure Linux sudo zypper remove # OpenSUSE / SLES sudo apt-get --purge remove # Debian / Ubuntu 3\. Package Manager Installation[](#package-manager-installation "Permalink to this headline") ================================================================================================ Basic instructions can be found in the [Quick Start Guide](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#linux) . Read on for more detailed instructions. 3.1. Overview[](#overview "Permalink to this headline") --------------------------------------------------------- Installation using RPM or Debian packages interfaces with your system’s package management system. When using RPM or Debian local repo installers, the downloaded package contains a repository snapshot stored on the local filesystem in /var/. Such a package only informs the package manager where to find the actual installation packages, but will not install them. If the online network repository is enabled, RPM or Debian packages will be automatically downloaded at installation time using the package manager: apt-get, dnf, tdnf, or zypper. Distribution-specific instructions detail how to install CUDA: * [RHEL / Rocky](#rhel-rocky-installation) * [KylinOS](#kylinos-installation) * [Fedora](#fedora-installation) * [SLES](#sles-installation) * [OpenSUSE](#opensuse-installation) * [WSL](#wsl-installation) * [Ubuntu](#ubuntu-installation) * [Debian](#debian-installation) * [Amazon Linux](#amazon-installation) * [Azure Linux CM2](#azure-installation) Finally, some helpful [package manager capabilities](#additional-package-manager-capabilities) are detailed. These instructions are for native development only. For cross-platform development, see the [CUDA Cross-Platform Environment](#cross-platform) section. Note Optional components such as `nvidia-fs`, `libnvidia_nscq`, and `fabricmanager` are not installed by default and will have to be installed separately as needed. 3.2. RHEL / Rocky[](#rhel-rocky "Permalink to this headline") --------------------------------------------------------------- ### 3.2.1. Prepare RHEL / Rocky[](#prepare-rhel-rocky "Permalink to this headline") 1. Perform the [Pre-installation Actions](#pre-installation-actions) . 2. **Satisfy third-party package dependency:** * **Enable optional repos:** On **RHEL 9 Linux** only, execute the following steps to enable optional repositories. * **On x86\_64 systems:** subscription-manager repos --enable=rhel-9-for-x86\_64-appstream-rpms subscription-manager repos --enable=rhel-9-for-x86\_64-baseos-rpms subscription-manager repos --enable=codeready-builder-for-rhel-9-x86\_64-rpms On **RHEL 8 Linux** only, execute the following steps to enable optional repositories. * **On x86\_64 systems:** subscription-manager repos --enable=rhel-8-for-x86\_64-appstream-rpms subscription-manager repos --enable=rhel-8-for-x86\_64-baseos-rpms subscription-manager repos --enable=codeready-builder-for-rhel-8-x86\_64-rpms 3. **Remove Outdated Signing Key:** sudo rpm --erase gpg-pubkey-7fa2af80\* 4. Choose an installation method: [Local Repo Installation for RHEL / Rocky](#local-repo-installation-for-rhel-rocky) or [Network Repo Installation for RHEL / Rocky](#network-repo-installation-for-rhel-rocky) . ### 3.2.2. Local Repo Installation for RHEL / Rocky[](#local-repo-installation-for-rhel-rocky "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*..rpm where `` should be replaced by one of the following: * `rhel8` * `rhel9` and `` should be replaced by one of the following: * `x86_64` * `aarch64` ### 3.2.3. Network Repo Installation for RHEL / Rocky[](#network-repo-installation-for-rhel-rocky "Permalink to this headline") 1. **Enable the network repo:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos///cuda-.repo where `/` should be replaced by one of the following: * `rhel8/sbsa` * `rhel8/x86_64` * `rhel9/sbsa` * `rhel9/x86_64` 2. **Install the new CUDA public GPG key:** The new GPG public key for the CUDA repository (RPM-based distros) is [d42d0685](https://developer.download.nvidia.com/compute/cuda/repos/fedora39/x86_64/D42D0685.pub) . On a fresh installation of RHEL, the dnf package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted. For upgrades, you must also also fetch an updated .repo entry: sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos///cuda-.repo 3. **Clean DNF repository:** sudo dnf clean all ### 3.2.4. Common Instructions for RHEL / Rocky[](#common-instructions-for-rhel-rocky "Permalink to this headline") These instructions apply to both local and network installation. 1. **Install CUDA SDK:** sudo dnf install cuda-toolkit 2. **Install GPUDirect Filesystem:** sudo dnf install nvidia-gds 3. **Add libcuda.so symbolic link, if necessary** The `libcuda.so` library is installed in the `/usr/lib{,64}/nvidia` directory. For pre-existing projects which use `libcuda.so`, it may be useful to add a symbolic link from `libcuda.so` in the `/usr/lib{,64}` directory. 4. **Reboot the system:** sudo reboot 5. Perform the [post-installation actions](#post-installation-actions) . 3.3. KylinOS[](#kylinos "Permalink to this headline") ------------------------------------------------------- ### 3.3.1. Prepare KylinOS[](#prepare-kylinos "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. Choose an installation method: [local repo](#local-repo-installation-for-kylinos) or [network repo](#network-repo-installation-for-kylinos) . ### 3.3.2. Local Repo Installation for KylinOS[](#local-repo-installation-for-kylinos "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*..rpm where `` should be replaced by one of the following: * `kylin10` and `` should be replaced by one of the following: * `x86_64` * `aarch64` ### 3.3.3. Network Repo Installation for KylinOS[](#network-repo-installation-for-kylinos "Permalink to this headline") 1. **Enable the network repo:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos///cuda-.repo where `/` should be replaced by one of the following: * `kylin10/sbsa` * `kylin10/x86_64` 2. **Install the new CUDA public GPG key:** The new GPG public key for the CUDA repository (RPM-based distros) is [d42d0685](https://developer.download.nvidia.com/compute/cuda/repos/fedora39/x86_64/D42D0685.pub) . On a fresh installation of KylinOS, the dnf package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted. 3. **Clean DNF repository:** sudo dnf clean all ### 3.3.4. Common Instructions for KylinOS[](#common-instructions-for-kylinos "Permalink to this headline") These instructions apply to both local and network installation. 1. **Install CUDA SDK:** sudo dnf install cuda-toolkit 2. **Install GPUDirect Filesystem:** sudo dnf install nvidia-gds 3. **Add libcuda.so symbolic link, if necessary** The `libcuda.so` library is installed in the `/usr/lib{,64}/nvidia` directory. For pre-existing projects which use `libcuda.so`, it may be useful to add a symbolic link from `libcuda.so` in the `/usr/lib{,64}` directory. 4. **Reboot the system:** sudo reboot 5. Perform the [post-installation actions](#post-installation-actions) . 3.4. Fedora[](#fedora "Permalink to this headline") ----------------------------------------------------- ### 3.4.1. Prepare Fedora[](#prepare-fedora "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **Remove Outdated Signing Key:** sudo rpm --erase gpg-pubkey-7fa2af80\* 3. Choose an installation method: [local repo](#local-repo-installation-for-fedora) or [network repo](#network-repo-installation-for-fedora) . ### 3.4.2. Local Repo Installation for Fedora[](#local-repo-installation-for-fedora "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*.x86\_64.rpm where `` should be replaced by one of the following: * `fedora39` ### 3.4.3. Network Repo Installation for Fedora[](#network-repo-installation-for-fedora "Permalink to this headline") 1. **Enable the network repo:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo where `` should be replaced by one of the following: * `fedora39` 2. **Install the new CUDA public GPG key:** The new GPG public key for the CUDA repository (RPM-based distros) is [d42d0685](https://developer.download.nvidia.com/compute/cuda/repos/fedora39/x86_64/D42D0685.pub) . On a fresh installation of Fedora, the dnf package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted. For upgrades, you must also fetch an updated `.repo` entry: sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo 3. **Clean DNF repository:** sudo dnf clean all ### 3.4.4. Common Installation Instructions for Fedora[](#common-installation-instructions-for-fedora "Permalink to this headline") These instructions apply to both local and network installation for Fedora. 1. **Install CUDA SDK:** sudo dnf install cuda-toolkit 2. **Reboot the system:** sudo reboot 3. **Add libcuda.so symbolic link, if necessary:** The `libcuda.so` library is installed in the `/usr/lib{,64}/nvidia` directory. For pre-existing projects which use `libcuda.so`, it may be useful to add a symbolic link from `libcuda.so` in the `/usr/lib{,64}` directory. 4. Perform the [post-installation actions](#post-installation-actions) . ### 3.4.5. GCC Compatibility Package for Fedora[](#gcc-compatibility-package-for-fedora "Permalink to this headline") The Fedora version supported might ship with a newer compiler than what is actually supported by NVCC. This can be overcome by installing the GCC compatibility package and setting a few environment variables. As an example, Fedora 41 ships with GCC 14 and also with a compatible GCC 13 version, which can be used for NVCC. To install and configure the local NVCC binary to use that version, proceed as follows. 1. Install the packages required: sudo dnf install gcc13-c++ The binaries then appear on the system in the following way: /usr/bin/gcc-13 /usr/bin/g++-13 2. Override the default `g++` compiler. Refer to the [documentation for NVCC regarding the environment variables](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#nvcc-environment-variables) . For example: export NVCC\_CCBIN='g++-13' 3.5. SLES[](#sles "Permalink to this headline") ------------------------------------------------- ### 3.5.1. Prepare SLES[](#prepare-sles "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **On SLES12 SP4, install the Mesa-libgl-devel Linux packages before proceeding.** See [Mesa-libGL-devel.](https://pkgs.org/download/Mesa-libGL-devel) 3. **Add the user to the video group:** sudo usermod -a -G video 4. **Remove Outdated Signing Key:** sudo rpm --erase gpg-pubkey-7fa2af80\* 5. Choose an installation method: [local repo](#local-repo-installation-for-sles) or [network repo](#network-repo-installation-for-sles) . ### 3.5.2. Local Repo Installation for SLES[](#local-repo-installation-for-sles "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*..rpm where `` should be replaced by one of the following: * `sles15` and `` should be replaced by one of the following: * `x86_64` * `aarch64` ### 3.5.3. Network Repo Installation for SLES[](#network-repo-installation-for-sles "Permalink to this headline") 1. **Enable the network repo:** sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos///cuda-.repo where `/` should be replaced by one of the following: * `sles15/sbsa` * `sles15/x86_64` 2. **Install the new CUDA public GPG key:** The new GPG public key for the CUDA repository (RPM-based distros) is [d42d0685](https://developer.download.nvidia.com/compute/cuda/repos/fedora39/x86_64/D42D0685.pub) . On a fresh installation of SLES, the zypper package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted. For upgrades, you must also also fetch an updated .repo entry: sudo zypper removerepo cuda-- sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos///cuda-.repo 3. **Refresh Zypper repository cache:** sudo SUSEConnect --product PackageHub// sudo zypper refresh ### 3.5.4. Common Installation Instructions for SLES[](#common-installation-instructions-for-sles "Permalink to this headline") These instructions apply to both local and network installation for SLES. 1. **Install CUDA SDK:** sudo zypper install cuda-toolkit 2. **Reboot the system:** sudo reboot 3. Perform the [post-installation actions](#post-installation-actions) . 3.6. OpenSUSE[](#opensuse "Permalink to this headline") --------------------------------------------------------- ### 3.6.1. Prepare OpenSUSE[](#prepare-opensuse "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **Add the user to the video group:** sudo usermod -a -G video 3. **Remove Outdated Signing Key:** sudo rpm --erase gpg-pubkey-7fa2af80\* 4. Choose an installation method: [local repo](#local-repo-installation-for-opensuse) or [network repo](#network-repo-installation-for-opensuse) . ### 3.6.2. Local Repo Installation for OpenSUSE[](#local-repo-installation-for-opensuse "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*.x86\_64.rpm where `` should be replaced by one of the following: * `opensuse15` ### 3.6.3. Network Repo Installation for OpenSUSE[](#network-repo-installation-for-opensuse "Permalink to this headline") 1. **Enable the network repo:** sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo where `` should be replaced by one of the following: * `opensuse15` 2. **Install the new CUDA public GPG key:** The new GPG public key for the CUDA repository (RPM-based distros) is [d42d0685](https://developer.download.nvidia.com/compute/cuda/repos/fedora39/x86_64/D42D0685.pub) . On fresh installation of openSUSE, the zypper package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted. For upgrades, you must also also fetch an updated .repo entry: sudo zypper removerepo cuda--x86\_64 sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo 3. **Refresh Zypper repository cache:** sudo zypper refresh ### 3.6.4. Common Installation Instructions for OpenSUSE[](#common-installation-instructions-for-opensuse "Permalink to this headline") These instructions apply to both local and network installation for OpenSUSE. 1. **Install CUDA SDK:** sudo zypper install cuda-toolkit 2. **Reboot the system:** sudo reboot 3. Perform the [post-installation actions](#post-installation-actions) . 3.7. WSL[](#wsl "Permalink to this headline") ----------------------------------------------- These instructions must be used if you are installing in a WSL environment. ### 3.7.1. Prepare WSL[](#prepare-wsl "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **Remove Outdated Signing Key:** sudo apt-key del 7fa2af80 3. Choose an installation method: [local repo](#local-repo-installation-for-wsl) or [network repo](#network-repo-installation-for-wsl) . ### 3.7.2. Local Repo Installation for WSL[](#local-repo-installation-for-wsl "Permalink to this headline") 1. **Install local repository on file system:** sudo dpkg -i cuda-repo--X-Y-local\_\*\_amd64.deb where `` should be replaced by one of the following: * `wsl-ubuntu` 2. **Enroll ephemeral public GPG key:** sudo cp /var/cuda-repo--X-Y-local/cuda-\*-keyring.gpg /usr/share/keyrings/ 3. **Add pin file to prioritize CUDA repository:** wget https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.pin sudo mv cuda-.pin /etc/apt/preferences.d/cuda-repository-pin-600 ### 3.7.3. Network Repo Installation for WSL[](#network-repo-installation-for-wsl "Permalink to this headline") The new GPG public key for the CUDA repository (Debian-based distros) is [3bf863cc](https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub) . This must be enrolled on the system, either using the `cuda-keyring` package or manually; the `apt-key` command is deprecated and not recommended. 1. **Install the new cuda-keyring package:** wget https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb where `` should be replaced by one of the following: * `wsl-ubuntu` ### 3.7.4. Common Installation Instructions for WSL[](#common-installation-instructions-for-wsl "Permalink to this headline") These instructions apply to both local and network installation for WSL. 1. **Update the Apt repository cache:** sudo apt-get update 2. **Install CUDA SDK:** sudo apt-get install cuda-toolkit 3. Perform the [post-installation actions](#post-installation-actions) . 3.8. Ubuntu[](#ubuntu "Permalink to this headline") ----------------------------------------------------- ### 3.8.1. Prepare Ubuntu[](#prepare-ubuntu "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **Remove Outdated Signing Key:** sudo apt-key del 7fa2af80 3. Choose an installation method: [local repo](#local-repo-installation-for-ubuntu) or [network repo](#network-repo-installation-for-ubuntu) . ### 3.8.2. Local Repo Installation for Ubuntu[](#local-repo-installation-for-ubuntu "Permalink to this headline") 1. **Install local repository on file system:** sudo dpkg -i cuda-repo--X-Y-local\_\*\_.deb where `` should be replaced by one of the following: * `ubuntu2004` * `ubuntu2204` * `ubuntu2404` and `` should be replaced by one of the following: * `amd64` * `arm64` 2. **Enroll ephemeral public GPG key:** sudo cp /var/cuda-repo--X-Y-local/cuda-\*-keyring.gpg /usr/share/keyrings/ 3. **Add pin file to prioritize CUDA repository:** wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-.pin sudo mv cuda-.pin /etc/apt/preferences.d/cuda-repository-pin-600 ### 3.8.3. Network Repo Installation for Ubuntu[](#network-repo-installation-for-ubuntu "Permalink to this headline") The new GPG public key for the CUDA repository is [3bf863cc](https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub) . This must be enrolled on the system, either using the `cuda-keyring` package or manually; the `apt-key` command is deprecated and not recommended. 1. **Install the new cuda-keyring package:** wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb where `/` should be replaced by one of the following: * `ubuntu2004/arm64` * `ubuntu2004/sbsa` * `ubuntu2004/x86_64` * `ubuntu2204/sbsa` * `ubuntu2204/x86_64` * `ubuntu2404/sbsa` * `ubuntu2404/x86_64` Note arm64-Jetson repo: * native: `/arm64` sudo dpkg -i cuda-keyring\_1.1-1\_all.deb ### 3.8.4. Common Installation Instructions for Ubuntu[](#common-installation-instructions-for-ubuntu "Permalink to this headline") These instructions apply to both local and network installation for Ubuntu. 1. **Update the Apt repository cache:** sudo apt-get update 2. **Install CUDA SDK:** Note These two commands must be executed separately. sudo apt-get install cuda-toolkit To include all GDS packages: sudo apt-get install nvidia-gds 1. **For native arm64-Jetson repos, install the additional packages:** sudo apt-get install cuda-compat 3. **Reboot the system** sudo reboot 4. Perform the [Post-installation Actions](#post-installation-actions) . 3.9. Debian[](#debian "Permalink to this headline") ----------------------------------------------------- ### 3.9.1. Prepare Debian[](#prepare-debian "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. **Enable the contrib repository:** sudo add-apt-repository contrib 3. **Remove Outdated Signing Key:** sudo apt-key del 7fa2af80 4. Choose an installation method: [local repo](#local-repo-installation-for-debian) or [network repo](#network-repo-installation-for-debian) . ### 3.9.2. Local Repo Installation for Debian[](#local-repo-installation-for-debian "Permalink to this headline") 1. **Install local repository on file system:** sudo dpkg -i cuda-repo--X-Y-local\_\*\_amd64.deb where `` should be replaced by one of the following: * `debian11` * `debian12` 2. **Enroll ephemeral public GPG key:** sudo cp /var/cuda-repo--X-Y-local/cuda-\*-keyring.gpg /usr/share/keyrings/ ### 3.9.3. Network Repo Installation for Debian[](#network-repo-installation-for-debian "Permalink to this headline") The new GPG public key for the CUDA repository (Debian-based distros) is [3bf863cc](https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub) . This must be enrolled on the system, either using the cuda-keyring package or manually; the `apt-key` command is deprecated and not recommended. 1. **Install the new cuda-keyring package:** wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb where `/` should be replaced by one of the following: * `debian11/x86_64` * `debian12/x86_64` sudo dpkg -i cuda-keyring\_1.1-1\_all.deb ### 3.9.4. Common Installation Instructions for Debian[](#common-installation-instructions-for-debian "Permalink to this headline") These instructions apply to both local and network installation for Debian. 1. **Update the Apt repository cache:** sudo apt-get update Note If you are using Debian 11, you may instead need to run: sudo apt-get --allow-releaseinfo-change update 2. **Install CUDA SDK:** sudo apt-get install cuda-toolkit 3. **Reboot the system:** sudo reboot 4. Perform the [post-installation actions](#post-installation-actions) . 3.10. Amazon Linux[](#amazon-linux "Permalink to this headline") ------------------------------------------------------------------ ### 3.10.1. Prepare Amazon Linux[](#prepare-amazon-linux "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. Choose an installation method: [local repo](#local-repo-installation-for-amazon-linux) or [network repo](#network-repo-installation-for-amazon-linux) . ### 3.10.2. Local Repo Installation for Amazon Linux[](#local-repo-installation-for-amazon-linux "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*.x86\_64.rpm where `` should be replaced by one of the following: * `amzn2023` ### 3.10.3. Network Repo Installation for Amazon Linux[](#network-repo-installation-for-amazon-linux "Permalink to this headline") 1. **Enable the network repository:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo 2. **Clean DNF repository:** sudo dnf clean all ### 3.10.4. Common Installation Instructions for Amazon Linux[](#common-installation-instructions-for-amazon-linux "Permalink to this headline") These instructions apply to both local and network installation for Amazon Linux. 1. **Install CUDA SDK:** sudo dnf install cuda-toolkit 2. **Install GPUDirect Filesystem:** sudo dnf install nvidia-gds 3. **Add libcuda.so symbolic link, if necessary:** The `libcuda.so` library is installed in the `/usr/lib{,64}/nvidia` directory. For pre-existing projects which use `libcuda.so`, it may be useful to add a symbolic link from `libcuda.so` in the `/usr/lib{,64}` directory. 4. **Reboot the system:** sudo reboot 5. Perform the [post-installation actions](#post-installation-actions) . 3.11. Azure Linux CM2[](#azure-linux-cm2 "Permalink to this headline") ------------------------------------------------------------------------ ### 3.11.1. Prepare Azure Linux CM2[](#prepare-azure-linux-cm2 "Permalink to this headline") 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. Choose an installation method: [local repo](#local-repo-installation-for-azure-linux) or [network repo](#network-repo-installation-for-azure-linux) . ### 3.11.2. Local Repo Installation for Azure Linux[](#local-repo-installation-for-azure-linux "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo--X-Y-local-\*.x86\_64.rpm where `` should be replaced by one of the following: * `cm2` ### 3.11.3. Network Repo Installation for Azure Linux[](#network-repo-installation-for-azure-linux "Permalink to this headline") 1. **Enable the network repository:** curl https://developer.download.nvidia.com/compute/cuda/repos//x86\_64/cuda-.repo | sudo tee /etc/yum.repos.d/cuda-.repo 2. **Clean TDNF repository cache:** sudo tdnf clean expire-cache ### 3.11.4. Common Installation Instructions for Azure Linux[](#common-installation-instructions-for-azure-linux "Permalink to this headline") These instructions apply to both local and network installation for Azure Linux. 1. **Enable Mariner extended repo:** sudo tdnf install mariner-repos-extended 2. **Install Cuda SDK:** sudo tdnf install cuda-toolkit 3. **Install GPUDirect Filesystem:** sudo tdnf install nvidia-gds 4. **Reboot the system:** sudo reboot 5. Perform the post-installation actions . 3.12. Additional Package Manager Capabilities[](#additional-package-manager-capabilities "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------ Below are some additional capabilities of the package manager that users can take advantage of. ### 3.12.1. Available Packages[](#available-packages "Permalink to this headline") The recommended installation package is the `cuda` package. This package will install the full set of other CUDA packages required for native development and should cover most scenarios. The `cuda` package installs all the available packages for native developments. That includes the compiler, the debugger, the profiler, the math libraries, and so on. For x86\_64 platforms, this also includes Nsight Eclipse Edition and the visual profilers. On supported platforms, the `cuda-cross-aarch64` and `cuda-cross-sbsa` packages install all the packages required for cross-platform development to arm64-Jetson and arm64-Server, respectively. Note 32-bit compilation native and cross-compilation is removed from CUDA 12.0 and later Toolkit. Use the CUDA Toolkit from earlier releases for 32-bit compilation. Hopper does not support 32-bit applications. The packages installed by the packages above can also be installed individually by specifying their names explicitly. The list of available packages be can obtained with: dnf --disablerepo="\*" --enablerepo="cuda\*" list available # Amazon Linux / Fedora / KylinOS / RHEL / Rocky Linux tdnf --disablerepo="\*" --enablerepo="cuda-cm2--local" list available # Azure Linux zypper packages -r cuda # OpenSUSE / SLES cat /var/lib/apt/lists/\*cuda\*Packages | grep "Package:" # Debian / Ubuntu ### 3.12.2. Meta Packages[](#meta-packages "Permalink to this headline") Meta packages are RPM/Deb/Conda packages which contain no (or few) files but have multiple dependencies. They are used to install many CUDA packages when you may not know the details of the packages you want. The following table lists the meta packages. | | | | --- | --- |Table 4 Meta Packages Available for CUDA 12.8[](#id50 "Permalink to this table") | Meta Package | Purpose | | --- | --- | | cuda | Installs all CUDA Toolkit and Driver packages. Handles upgrading to the next version of the `cuda` package when it’s released. | | cuda-12-8 | Installs all CUDA Toolkit and Driver packages. Remains at version 12.5 until an additional version of CUDA is installed. | | cuda-toolkit-12-8 | Installs all CUDA Toolkit packages required to develop CUDA applications. Does not include the driver. | | cuda-toolkit-12 | Installs all CUDA Toolkit packages required to develop applications. Will not upgrade beyond the 12.x series toolkits. Does not include the driver. | | cuda-toolkit | Installs all CUDA Toolkit packages required to develop applications. Handles upgrading to the next 12.x version of CUDA when it’s released. Does not include the driver. | | cuda-tools-12-8 | Installs all CUDA command line and visual tools. | | cuda-runtime-12-8 | Installs all CUDA Toolkit packages required to run CUDA applications, as well as the Driver packages. | | cuda-compiler-12-8 | Installs all CUDA compiler packages. | | cuda-libraries-12-8 | Installs all runtime CUDA Library packages. | | cuda-libraries-dev-12-8 | Installs all development CUDA Library packages. | ### 3.12.3. Package Upgrades[](#package-upgrades "Permalink to this headline") The `cuda` package points to the latest stable release of the CUDA Toolkit. When a new version is available, use the following commands to upgrade the toolkit: #### 3.12.3.1. Amazon Linux[](#id26 "Permalink to this headline") sudo dnf upgrade cuda-toolkit #### 3.12.3.2. Fedora[](#id27 "Permalink to this headline") When upgrading the toolkit to a new **major** branch: sudo dnf install cuda-toolkit When upgrading the toolkit to a new **minor** branch: sudo dnf upgrade cuda-toolkit #### 3.12.3.3. KylinOS / RHEL / Rocky Linux[](#kylinos-rhel-rocky-linux "Permalink to this headline") sudo dnf install cuda-toolkit #### 3.12.3.4. Azure Linux[](#azure-linux "Permalink to this headline") sudo tdnf install cuda-toolkit #### 3.12.3.5. OpenSUSE / SLES[](#opensuse-sles "Permalink to this headline") sudo zypper install cuda-toolkit #### 3.12.3.6. Debian / Ubuntu[](#debian-ubuntu "Permalink to this headline") sudo apt-get install cuda-toolkit #### 3.12.3.7. Other Package Notes[](#other-package-notes "Permalink to this headline") The `cuda-cross-` packages can also be upgraded in the same manner. Some desktop environments, such as GNOME or KDE, will display a notification alert when new packages are available. To avoid any automatic upgrade, and lock down the toolkit installation to the X.Y release, install the `cuda-toolkit-X-Y` or `cuda-cross--X-Y` package. Side-by-side installations are supported. For instance, to install both the X.Y CUDA Toolkit and the X.Y+1 CUDA Toolkit, install the `cuda-toolkit-X.Y` and `cuda-toolkit-X.Y+1` packages. 4\. Driver Installation[](#driver-installation "Permalink to this headline") ============================================================================== More information about driver installation can be found in the [Driver Installation Guide for Linux](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html) 5\. Runfile Installation[](#runfile-installation "Permalink to this headline") ================================================================================ Basic instructions can be found in the [Quick Start Guide](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#linux) . Read on for more detailed instructions. This section describes the installation and configuration of CUDA when using the standalone installer. The standalone installer is a `.run` file and is completely self-contained. 5.1. Runfile Overview[](#runfile-overview "Permalink to this headline") ------------------------------------------------------------------------- The Runfile installation installs the CUDA Toolkit via an interactive ncurses-based interface. The [installation steps](#id29) are listed below. Finally, [advanced options](#runfile-advanced) for the installer and [uninstallation steps](#runfile-uninstallation) are detailed below. The Runfile installation does not include support for cross-platform development. For cross-platform development, see the [CUDA Cross-Platform Environment](#cross-platform) section. 5.2. Installation[](#installation "Permalink to this headline") ----------------------------------------------------------------- 1. Perform the [pre-installation actions](#pre-installation-actions) . 2. Reboot into text mode (runlevel 3). This can usually be accomplished by adding the number “3” to the end of the system’s kernel boot parameters. Since the NVIDIA drivers are not yet installed, the text terminals may not display correctly. Temporarily adding “nomodeset” to the system’s kernel boot parameters may fix this issue. Consult your system’s bootloader documentation for information on how to make the above boot parameter changes. 3. Run the installer and follow the on-screen prompts: sudo sh cuda\_\_linux.run The installer will prompt for the following: * EULA Acceptance * CUDA Toolkit installation, location, and `/usr/local/cuda` symbolic link The default installation location for the toolkit is `/usr/local/cuda-12.6`: The `/usr/local/cuda` symbolic link points to the location where the CUDA Toolkit was installed. This link allows projects to use the latest CUDA Toolkit without any configuration file update. The installer must be executed with sufficient privileges to perform some actions. When the current privileges are insufficient to perform an action, the installer will ask for the user’s password to attempt to install with root privileges. Actions that cause the installer to attempt to install with root privileges are: * installing the CUDA Toolkit to a location the user does not have permission to write to * creating the `/usr/local/cuda` symbolic link Running the installer with **sudo**, as shown above, will give permission to install to directories that require root permissions. Directories and files created while running the installer with **sudo** will have root ownership. 4. Reboot the system to reload the graphical interface: sudo reboot 5. Perform the [post-installation actions](#post-installation-actions) . 5.3. Advanced Options[](#advanced-options "Permalink to this headline") ------------------------------------------------------------------------- | Action | Options Used | Explanation | | --- | --- | --- | | Silent Installation | `--silent` | Required for any silent installation. Performs an installation with no further user-input and minimal command-line output based on the options provided below. Silent installations are useful for scripting the installation of CUDA. Using this option implies acceptance of the EULA. The following flags can be used to customize the actions taken during installation. At least one of `--driver`, `--uninstall`, and `--toolkit` must be passed if running with non-root permissions. | | `--driver` | Install the CUDA Driver. | | `--toolkit` | Install the CUDA Toolkit. | | `--toolkitpath=` | Install the CUDA Toolkit to the directory. If not provided, the default path of `/usr/local/cuda-12.6` is used. | | `--defaultroot=` | Install libraries to the directory. If the is not provided, then the default path of your distribution is used. _This only applies to the libraries installed outside of the CUDA Toolkit path._ | | Extraction | `--extract=` | Extracts to the the following: the driver runfile, the raw files of the toolkit to .

This is especially useful when one wants to install the driver using one or more of the command-line options provided by the driver installer which are not exposed in this installer. | | Overriding Installation Checks | `--override` | Ignores compiler, third-party library, and toolkit detection checks which would prevent the CUDA Toolkit from installing. | | No OpenGL Libraries | `--no-opengl-libs` | Prevents the driver installation from installing NVIDIA’s GL libraries. Useful for systems where the display is driven by a non-NVIDIA GPU. In such systems, NVIDIA’s GL libraries could prevent X from loading properly. | | No man pages | `--no-man-page` | Do not install the man pages under `/usr/share/man`. | | Overriding Kernel Source | `--kernel-source-path=` | Tells the driver installation to use as the kernel source directory when building the NVIDIA kernel module. Required for systems where the kernel source is installed to a non-standard location. | | Running nvidia-xconfig | `--run-nvidia-xconfig` | Tells the driver installation to run nvidia-xconfig to update the system X configuration file so that the NVIDIA X driver is used. The pre-existing X configuration file will be backed up. | | No nvidia-drm kernel module | `--no-drm` | Do not install the nvidia-drm kernel module. This option should only be used to work around failures to build or install the nvidia-drm kernel module on systems that do not need the provided features. | | Custom Temporary Directory Selection | `--tmpdir=` | Performs any temporary actions within instead of `/tmp`. Useful in cases where `/tmp` cannot be used (doesn’t exist, is full, is mounted with ‘noexec’, etc.). | | Kernel Module Build Directory | `--kernel-module-build-directory=` | Tells the driver installation to use legacy or open flavor of kernel source when building the NVIDIA kernel module. The kernel-open flavor is only supported on Turing GPUs and newer. | | `-m=kernel` | Tells the driver installation to use legacy flavor of kernel source when building the NVIDIA kernel module. Shorthand for `--kernel-module-build-directory=kernel` | | `m=kernel-open` | Tells the driver installation to use open flavor of kernel source when building the NVIDIA kernel module. The kernel-open flavor is only supported on Turing GPUs and newer. Shorthand for `--kernel-module-build-directory=kernel-open` | | Show Installer Options | `--help` | Prints the list of command-line options to stdout. | 5.4. Uninstallation[](#uninstallation "Permalink to this headline") --------------------------------------------------------------------- To uninstall the CUDA Toolkit, run the uninstallation script provided in the bin directory of the toolkit. By default, it is located in `/usr/local/cuda-12.6/bin`: sudo /usr/local/cuda-12.6/bin/cuda-uninstaller 6\. Conda Installation[](#conda-installation "Permalink to this headline") ============================================================================ This section describes the installation and configuration of CUDA when using the Conda installer. The Conda packages are available at [https://anaconda.org/nvidia](https://anaconda.org/nvidia) . 6.1. Conda Overview[](#conda-overview "Permalink to this headline") --------------------------------------------------------------------- The Conda installation installs the CUDA Toolkit. The installation steps are listed below. 6.2. Installing CUDA Using Conda[](#installing-cuda-using-conda "Permalink to this headline") ----------------------------------------------------------------------------------------------- To perform a basic install of all CUDA Toolkit components using Conda, run the following command: conda install cuda -c nvidia Note Install CUDA in a dedicated Conda environment instead of the base environment to avoid installation issues. 6.3. Uninstalling CUDA Using Conda[](#uninstalling-cuda-using-conda "Permalink to this headline") --------------------------------------------------------------------------------------------------- To uninstall the CUDA Toolkit using Conda, run the following command: conda remove cuda 6.4. Installing Previous CUDA Releases[](#installing-previous-cuda-releases "Permalink to this headline") ----------------------------------------------------------------------------------------------------------- All Conda packages released under a specific CUDA version are labeled with that release version. To install a previous version, include that label in the `install` command such as: conda install cuda -c nvidia/label/cuda-12.4.0 6.5. Upgrading from cudatoolkit Package[](#upgrading-from-cudatoolkit-package "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- If you had previously installed CUDA using the `cudatoolkit` package and want to maintain a similar install footprint, you can limit your installation to the following packages: * `cuda-libraries-dev` * `cuda-nvcc` * `cuda-nvtx` * `cuda-cupti` Note Some extra files, such as headers, will be included in this installation which were not included in the `cudatoolkit` package. If you need to reduce your installation further, replace `cuda-libraries-dev` with the specific libraries you need. 7\. Pip Wheels[](#pip-wheels "Permalink to this headline") ============================================================ NVIDIA provides Python Wheels for installing CUDA through pip, primarily for using CUDA with Python. These packages are intended for runtime use and do not currently include developer tools (these can be installed separately). Please note that with this installation method, CUDA installation environment is managed via pip and additional care must be taken to set up your host environment to use CUDA outside the pip environment. **Prerequisites** To install Wheels, you must first install the `nvidia-pyindex` package, which is required in order to set up your pip installation to fetch additional Python modules from the NVIDIA NGC PyPI repo. If your pip and setuptools Python modules are not up-to-date, then use the following command to upgrade these Python modules. If these Python modules are out-of-date then the commands which follow later in this section may fail. python3 -m pip install --upgrade setuptools pip wheel You should now be able to install the `nvidia-pyindex` module. python3 -m pip install nvidia-pyindex If your project is using a `requirements.txt` file, then you can add the following line to your `requirements.txt` file as an alternative to installing the `nvidia-pyindex` package: \--extra-index-url https://pypi.org/simple **Procedure** Install the CUDA runtime package: python3 -m pip install nvidia-cuda-runtime-cu12 Optionally, install additional packages as listed below using the following command: python3 -m pip install nvidia- **Metapackages** The following metapackages will install the latest version of the named component on Linux for the indicated CUDA version. “cu12” should be read as “cuda12”. * nvidia-cuda-runtime-cu12 * nvidia-cuda-cccl-cu12 * nvidia-cuda-cupti-cu12 * nvidia-cuda-nvcc-cu12 * nvidia-cuda-opencl-cu12 * nvidia-cuda-nvrtc-cu12 * nvidia-cublas-cu12 * nvidia-cuda-sanitizer-api-cu12 * nvidia-cufft-cu12 * nvidia-curand-cu12 * nvidia-cusolver-cu12 * nvidia-cusparse-cu12 * nvidia-npp-cu12 * nvidia-nvfatbin-cu12 * nvidia-nvjitlink-cu12 * nvidia-nvjpeg-cu12 * nvidia-nvml-dev-cu12 * nvidia-nvtx-cu12 These metapackages install the following packages: * nvidia-cuda-runtime-cu128 * nvidia-cuda-cccl-cu128 * nvidia-cuda-cupti-cu128 * nvidia-cuda-nvcc-cu128 * nvidia-cuda-opencl-cu128 * nvidia-cublas-cu126 * nvidia-cuda-sanitizer-api-cu128 * nvidia-cuda-nvrtc-cu128 * nvidia-cufft-cu128 * nvidia-curand-cu128 * nvidia-cusolver-cu128 * nvidia-cusparse-cu128 * nvidia-npp-cu128 * nvidia-nvfatbin-cu128 * nvidia-nvjitlink-cu128 * nvidia-nvjpeg-cu128 * nvidia-nvml-dev-cu128 * nvidia-nvtx-cu128 8\. CUDA Cross-Platform Environment[](#cuda-cross-platform-environment "Permalink to this headline") ====================================================================================================== Cross development for arm64-sbsa is supported on Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04, KylinOS 10, RHEL 8, RHEL 9, and SLES 15. Cross development for arm64-Jetson is only supported on Ubuntu 22.04 We recommend selecting a host development environment that matches the supported cross-target environment. This selection helps prevent possible host/target incompatibilities, such as GCC or GLIBC version mismatches. 8.1. CUDA Cross-Platform Installation[](#cuda-cross-platform-installation "Permalink to this headline") --------------------------------------------------------------------------------------------------------- Some of the following steps may have already been performed as part of the [native installation sections](#package-manager-installation) . Such steps can safely be skipped. These steps should be performed on the x86\_64 host system, rather than the target system. To install the native CUDA Toolkit on the target system, refer to the native installation sections in [Package Manager Installation](#package-manager-installation) . ### 8.1.1. Ubuntu[](#id33 "Permalink to this headline") 1. Perform the [pre-installation actions.](#pre-installation-actions) 2. Choose an installation method: [local repo](#local-cross-repo-installation-for-ubuntu) or [network repo](#network-cross-repo-installation-for-ubuntu) . #### 8.1.1.1. Local Cross Repo Installation for Ubuntu[](#local-cross-repo-installation-for-ubuntu "Permalink to this headline") 1. Install repository meta-data package with: sudo dpkg -i cuda-repo-cross---X-Y-local-\*\_all.deb where `-` should be replaced by one of the following: * `aarch64-ubuntu2204` * `sbsa-ubuntu2004` * `sbsa-ubuntu2204` * `sbsa-ubuntu2404` #### 8.1.1.2. Network Cross Repo Installation for Ubuntu[](#network-cross-repo-installation-for-ubuntu "Permalink to this headline") The new GPG public key for the CUDA repository is [3bf863cc](https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub) . This must be enrolled on the system, either using the `cuda-keyring` package or manually; the `apt-key` command is deprecated and not recommended. 1. Install the new cuda-keyring package: wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb where `/` should be replaced by one of the following: * `ubuntu2004/cross-linux-sbsa` * `ubuntu2204/cross-linux-aarch64` * `ubuntu2204/cross-linux-sbsa` * `ubuntu2404/cross-linux-sbsa` #### 8.1.1.3. Common Installation Instructions for Ubuntu[](#id36 "Permalink to this headline") 1. Update the Apt repository cache: sudo apt-get update 2. Install the appropriate cross-platform CUDA Toolkit: 1. For arm64-sbsa: sudo apt-get install cuda-cross-sbsa 2. For arm64-Jetson sudo apt-get install cuda-cross-aarch64 3. For QNX: sudo apt-get install cuda-cross-qnx 3. Perform the [post-installation actions.](#post-installation-actions) ### 8.1.2. KylinOS / RHEL / Rocky Linux[](#id37 "Permalink to this headline") 1. Perform the [pre-installation actions.](#pre-installation-actions) 2. Choose an installation method: [local repo](#local-cross-repo-installation-for-kylinos-rhel-rocky-linux) or [network repo](#network-cross-repo-installation-for-kylinos-rhel-rocky-linux) . #### 8.1.2.1. Local Cross Repo Installation for KylinOS / RHEL / Rocky Linux[](#local-cross-repo-installation-for-kylinos-rhel-rocky-linux "Permalink to this headline") 1. Install repository meta-data package with: sudo rpm -i cuda-repo-cross---X-Y-local-\*.noarch.rpm where `-` should be replaced by one of the following: * `sbsa-kylin10` * `sbsa-rhel8` * `sbsa-rhel9` #### 8.1.2.2. Network Cross Repo Installation for KylinOS / RHEL / Rocky Linux[](#network-cross-repo-installation-for-kylinos-rhel-rocky-linux "Permalink to this headline") 1. **Enable the network repo:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos///cuda--cross-linux-sbsa.repo where `/` should be replaced by one of the following: * `kylin10/cross-linux-sbsa` * `rhel8/cross-linux-sbsa` * `rhel9/cross-linux-sbsa` #### 8.1.2.3. Common Installation Instructions for KylinOS / RHEL / Rocky Linux[](#common-installation-instructions-for-kylinos-rhel-rocky-linux "Permalink to this headline") 1. **Clean DNF repository:** sudo dnf clean all 2. **Install CUDA tool:** sudo dnf install cuda-cross-sbsa ### 8.1.3. SLES[](#id40 "Permalink to this headline") 1. Perform the [pre-installation actions.](#pre-installation-actions) 2. Choose an installation method: [local repo](#local-cross-repo-installation-for-sles) or [network repo](#network-cross-repo-installation-for-sles) . #### 8.1.3.1. Local Cross Repo Installation for SLES[](#local-cross-repo-installation-for-sles "Permalink to this headline") 1. Install repository meta-data package with: sudo rpm -i cuda-repo-cross---X-Y-local-\*.noarch.rpm where `-` should be replaced by one of the following: * `sbsa-sles15` #### 8.1.3.2. Network Cross Repo Installation for SLES[](#network-cross-repo-installation-for-sles "Permalink to this headline") 1. **Enable the network repo:** sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos///cuda--cross-linux-sbsa.repo where `/` should be replaced by one of the following: * `sles15/cross-linux-sbsa` #### 8.1.3.3. Common Installation Instructions for SLES[](#id43 "Permalink to this headline") 1. **Refresh Zypper repository cache:** sudo zypper refresh 2. **Install CUDA tool:** sudo zypper install cuda-cross-sbsa 9\. Tarball and Zip Archive Deliverables[](#tarball-and-zip-archive-deliverables "Permalink to this headline") ================================================================================================================ In an effort to meet the needs of a growing customer base requiring alternative installer packaging formats, as well as a means of input into community CI/CD systems, tarball and zip archives are available for each component. These tarball and zip archives, known as binary archives, are provided at [https://developer.download.nvidia.com/compute/cuda/redist/](https://developer.download.nvidia.com/compute/cuda/redist/) . [![_images/tarball-archives.png](_images/tarball-archives.png)](_images/tarball-archives.png) These component .tar.xz and .zip binary archives do not replace existing packages such as .deb, .rpm, runfile, conda, etc. and are not meant for general consumption, as they are not installers. However this standardized approach will replace existing .txz archives. For each release, a JSON manifest is provided such as **redistrib\_11.4.2.json**, which corresponds to the CUDA 11.4.2 release label (CUDA 11.4 update 2) which includes the release date, the name of each component, license name, relative URL for each platform and checksums. Package maintainers are advised to check the provided LICENSE for each component prior to redistribution. Instructions for developers using CMake and Bazel build systems are provided in the next sections. 9.1. Parsing Redistrib JSON[](#parsing-redistrib-json "Permalink to this headline") ------------------------------------------------------------------------------------- The following example of a JSON manifest contains keys for each component: name, license, version, and a platform array which includes relative\_path, sha256, md5, and size (bytes) for each archive. { "release\_date": "2021-09-07", "cuda\_cudart": { "name": "CUDA Runtime (cudart)", "license": "CUDA Toolkit", "version": "11.4.108", "linux-x86\_64": { "relative\_path": "cuda\_cudart/linux-x86\_64/cuda\_cudart-linux-x86\_64-11.4.108-archive.tar.xz", "sha256": "d08a1b731e5175aa3ae06a6d1c6b3059dd9ea13836d947018ea5e3ec2ca3d62b", "md5": "da198656b27a3559004c3b7f20e5d074", "size": "828300" }, "linux-ppc64le": { "relative\_path": "cuda\_cudart/linux-ppc64le/cuda\_cudart-linux-ppc64le-11.4.108-archive.tar.xz", "sha256": "831dffe062ae3ebda3d3c4010d0ee4e40a01fd5e6358098a87bb318ea7c79e0c", "md5": "ca73328e3f8e2bb5b1f2184c98c3a510", "size": "776840" }, "linux-sbsa": { "relative\_path": "cuda\_cudart/linux-sbsa/cuda\_cudart-linux-sbsa-11.4.108-archive.tar.xz", "sha256": "2ab9599bbaebdcf59add73d1f1a352ae619f8cb5ccec254093c98efd4c14553c", "md5": "aeb5c19661f06b6398741015ba368102", "size": "782372" }, "windows-x86\_64": { "relative\_path": "cuda\_cudart/windows-x86\_64/cuda\_cudart-windows-x86\_64-11.4.108-archive.zip", "sha256": "b59756c27658d1ea87a17c06d064d1336576431cd64da5d1790d909e455d06d3", "md5": "7f6837a46b78198402429a3760ab28fc", "size": "2897751" } } } A JSON schema is provided at [https://developer.download.nvidia.com/compute/redist/redistrib-v2.schema.json](https://developer.download.nvidia.com/compute/redist/redistrib-v2.schema.json) . A sample script that parses these JSON manifests is available on [GitHub](https://github.com/NVIDIA/build-system-archive-import-examples/blob/main/parse_redist.py) : * Downloads each archive * Validates SHA256 checksums * Extracts archives * Flattens into a collapsed directory structure | | | | --- | --- |Table 5 Available Tarball and Zip Archives[](#id51 "Permalink to this table") | Product | Example | | --- | --- | | [CUDA Toolkit](https://developer.download.nvidia.com/compute/cuda/redist) | `./parse_redist.py --product cuda --label 12.6.0` | | [cuBLASMp](https://developer.download.nvidia.com/compute/cublasmp/redist/) | `./parse_redist.py --product cublasmp --label 0.2.1` | | [cuDNN](https://developer.download.nvidia.com/compute/cudnn/redist) | `./parse_redist.py --product cudnn --label 9.2.1` | | [cuDSS](https://developer.download.nvidia.com/compute/cudss/redist) | `./parse_redist.py --product cudss --label 0.3.0` | | [cuQuantum](https://developer.download.nvidia.com/compute/cuquantum/redist) | `./parse_redist.py --product cuquantum --label 24.03.0` | | [cuSPARSELt](https://developer.download.nvidia.com/compute/cusparselt/redist) | `./parse_redist.py --product cusparselt --label 0.6.2` | | [cuTENSOR](https://developer.download.nvidia.com/compute/cutensor/redist) | `./parse_redist.py --product cutensor --label 2.0.2.1` | | [NVIDIA driver](https://developer.download.nvidia.com/compute/nvidia-driver/redist) | `./parse_redist.py --product nvidia-driver --label 550.90.07` | | [nvJPEG2000](https://developer.download.nvidia.com/compute/nvjpeg2000/redist) | `./parse_redist.py --product nvjpeg2000 --label 0.7.5` | | [NVPL](https://developer.download.nvidia.com/compute/nvpl/redist) | `./parse_redist.py --product nvpl --label 24.7` | | [nvTIFF](https://developer.download.nvidia.com/compute/nvtiff/redist) | `./parse_redist.py --product nvtiff --label 0.3.0` | 9.2. Importing Tarballs into CMake[](#importing-tarballs-into-cmake "Permalink to this headline") --------------------------------------------------------------------------------------------------- The recommended module for importing these tarballs into the CMake build system is via [FindCUDAToolkit](https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html) (3.17 and newer). Note The FindCUDA module is deprecated. The path to the extraction location can be specified with the `CUDAToolkit_ROOT` environmental variable. For example `CMakeLists.txt` and commands, see [cmake/1\_FindCUDAToolkit/](https://github.com/NVIDIA/build-system-archive-import-examples/blob/main/cmake/1_FindCUDAToolkit) . For older versions of CMake, the [ExternalProject\_Add](https://cmake.org/cmake/help/latest/module/ExternalProject.html) module is an alternative method. For example `CMakeLists.txt` file and commands, see [cmake/2\_ExternalProject/](https://github.com/NVIDIA/build-system-archive-import-examples/tree/main/cmake/2_ExternalProject) . 9.3. Importing Tarballs into Bazel[](#importing-tarballs-into-bazel "Permalink to this headline") --------------------------------------------------------------------------------------------------- The recommended method of importing these tarballs into the Bazel build system is using [http\_archive](https://docs.bazel.build/versions/main/repo/http.html) and [pkg\_tar](https://docs.bazel.build/versions/main/be/pkg.html#pkg_tar) . For an example, see [bazel/1\_pkg\_tar/](https://github.com/NVIDIA/build-system-archive-import-examples/blob/main/bazel/1_pkg_tar) . 10\. Post-installation Actions[](#post-installation-actions "Permalink to this headline") =========================================================================================== The post-installation actions must be manually performed. These actions are split into mandatory, recommended, and optional sections. 10.1. Mandatory Actions[](#mandatory-actions "Permalink to this headline") ---------------------------------------------------------------------------- Some actions must be taken after the installation before the CUDA Toolkit can be used. ### 10.1.1. Environment Setup[](#environment-setup "Permalink to this headline") The `PATH` variable needs to include `export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}}`. Nsight Compute has moved to `/opt/nvidia/nsight-compute/` only in rpm/deb installation method. When using `.run` installer it is still located under `/usr/local/cuda-12.8/`. To add this path to the `PATH` variable: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} In addition, when using the runfile installation method, the `LD_LIBRARY_PATH` variable needs to contain `/usr/local/cuda-12.8/lib64` on a 64-bit system, or `/usr/local/cuda-12.8/lib` on a 32-bit system * To change the environment variables for 64-bit operating systems: export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} * To change the environment variables for 32-bit operating systems: export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} Note that the above paths change when using a custom install path with the runfile installation method. 10.2. Recommended Actions[](#recommended-actions "Permalink to this headline") -------------------------------------------------------------------------------- Other actions are recommended to verify the integrity of the installation. ### 10.2.1. Install Writable Samples[](#install-writable-samples "Permalink to this headline") CUDA Samples are now located in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , which includes instructions for obtaining, building, and running the samples. ### 10.2.2. Verify the Installation[](#verify-the-installation "Permalink to this headline") Before continuing, it is important to verify that the CUDA toolkit can find and communicate correctly with the CUDA-capable hardware. To do this, you need to compile and run some of the sample programs, located in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) . Note Ensure the PATH and, if using the runfile installation method, `LD_LIBRARY_PATH` variables are [set correctly](index.html#environment-setup) . #### 10.2.2.1. Running the Binaries[](#running-the-binaries "Permalink to this headline") After compilation, find and run `deviceQuery`from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) . If the CUDA software is installed and configured correctly, the output for `deviceQuery` should look similar to that shown in [Figure 1](index.html#running-binaries-valid-results-from-sample-cuda-devicequery-program) . [![Valid Results from deviceQuery CUDA Sample](_images/valid-results-from-sample-cuda-devicequery-program.png)](_images/valid-results-from-sample-cuda-devicequery-program.png) Figure 1 Valid Results from deviceQuery CUDA Sample[](#running-binaries-valid-results-from-sample-cuda-devicequery-program "Permalink to this image") The exact appearance and the output lines might be different on your system. The important outcomes are that a device was found (the first highlighted line), that the device matches the one on your system (the second highlighted line), and that the test passed (the final highlighted line). If a CUDA-capable device is installed but `deviceQuery` reports that no CUDA-capable devices are present, this likely means that the `/dev/nvidia*` files are missing or have the wrong permissions. On systems where `SELinux` is enabled, you might need to temporarily disable this security feature to run `deviceQuery`. To do this, type: setenforce 0 from the command line as the superuser. Running the `bandwidthTest` program ensures that the system and the CUDA-capable device are able to communicate correctly. Its output is shown in [Figure 2](#running-binaries-valid-results-from-sample-cuda-bandwidthtest-program) . [![Valid Results from bandwidthTest CUDA Sample](_images/valid-results-from-sample-cuda-bandwidthtest-program.png)](_images/valid-results-from-sample-cuda-bandwidthtest-program.png) Figure 2 Valid Results from bandwidthTest CUDA Sample[](#running-binaries-valid-results-from-sample-cuda-bandwidthtest-program "Permalink to this image") Note that the measurements for your CUDA-capable device description will vary from system to system. The important point is that you obtain measurements, and that the second-to-last line (in [Figure 2](#running-binaries-valid-results-from-sample-cuda-bandwidthtest-program) ) confirms that all necessary tests passed. Should the tests not pass, make sure you have a CUDA-capable NVIDIA GPU on your system and make sure it is properly installed. If you run into difficulties with the link step (such as libraries not being found), consult the Linux Release Notes found in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) . ### 10.2.3. Install Nsight Eclipse Plugins[](#install-nsight-eclipse-plugins "Permalink to this headline") To install Nsight Eclipse plugins, an installation script is provided: /usr/local/cuda-12.8/bin/nsight\_ee\_plugins\_manage.sh install Refer to [Nsight Eclipse Plugins Installation Guide](https://docs.nvidia.com/cuda/nsightee-plugins-install-guide/index.html) for more details. ### 10.2.4. Local Repo Removal[](#local-repo-removal "Permalink to this headline") Removal of the local repo installer is recommended after installation of **CUDA SDK**. **Debian / Ubuntu** sudo apt-get remove --purge "cuda-repo--X-Y-local\*" **Amazon Linux / Fedora / KylinOS / RHEL / Rocky Linux** sudo dnf remove "cuda-repo--X-Y-local\*" **Azure Linux** sudo tdnf remove "cuda-repo--X-Y-local\*" **OpenSUSE / SLES** sudo zypper remove "cuda-repo--X-Y-local\*" 10.3. Optional Actions[](#optional-actions "Permalink to this headline") -------------------------------------------------------------------------- Other options are not necessary to use the CUDA Toolkit, but are available to provide additional features. ### 10.3.1. Install Third-party Libraries[](#install-third-party-libraries "Permalink to this headline") Some CUDA samples use third-party libraries which may not be installed by default on your system. These samples attempt to detect any required libraries when building. If a library is not detected, it waives itself and warns you which library is missing. To build and run these samples, you must install the missing libraries. In cases where these dependencies are not installed, follow the instructions below. **Amazon Linux / Fedora / KylinOS / RHEL / Rocky Linux** sudo dnf install freeglut-devel libX11-devel libXi-devel libXmu-devel \\ make mesa-libGLU-devel freeimage-devel libglfw3-devel **SLES** sudo zypper install libglut3 libX11-devel libXi6 libXmu6 libGLU1 make **OpenSUSE** sudo zypper install freeglut-devel libX11-devel libXi-devel libXmu-devel \\ make Mesa-libGL-devel freeimage-devel **Debian / Ubuntu** sudo apt-get install g++ freeglut3-dev build-essential libx11-dev \\ libxmu-dev libxi-dev libglu1-mesa-dev libfreeimage-dev libglfw3-dev ### 10.3.2. Install the Source Code for cuda-gdb[](#install-the-source-code-for-cuda-gdb "Permalink to this headline") The `cuda-gdb` source must be explicitly selected for installation with the runfile installation method. During the installation, in the component selection page, expand the component “CUDA Tools 12.8” and select `cuda-gdb-src` for installation. It is unchecked by default. To obtain a copy of the source code for `cuda-gdb` using the RPM and Debian installation methods, the `cuda-gdb-src` package must be installed. The source code is installed as a tarball in the `/usr/local/cuda-12.8/extras` directory. ### 10.3.3. Select the Active Version of CUDA[](#select-the-active-version-of-cuda "Permalink to this headline") For applications that rely on the symlinks `/usr/local/cuda` and `/usr/local/cuda-MAJOR`, you may wish to change to a different installed version of CUDA using the provided alternatives. To show the active version of CUDA and all available versions: update-alternatives --display cuda To show the active minor version of a given major CUDA release: update-alternatives --display cuda-12 To update the active version of CUDA: sudo update-alternatives --config cuda 11\. Removing CUDA Toolkit[](#removing-cuda-toolkit "Permalink to this headline") =================================================================================== Follow the below steps to properly uninstall the CUDA Toolkit from your system. These steps will ensure that the uninstallation will be clean. **Amazon Linux / Fedora / Kylin OS / RHEL / Rocky Linux** To remove CUDA Toolkit: sudo dnf remove "cuda\*" "\*cublas\*" "\*cufft\*" "\*cufile\*" "\*curand\*" \\ "\*cusolver\*" "\*cusparse\*" "\*gds-tools\*" "\*npp\*" "\*nvjpeg\*" "nsight\*" "\*nvvm\*" **Azure Linux** To remove CUDA Toolkit: sudo tdnf remove "cuda\*" "\*cublas\*" "\*cufft\*" "\*cufile\*" "\*curand\*" "\*cusolver\*" "\*cusparse\*" "\*gds-tools\*" "\*npp\*" "\*nvjpeg\*" "nsight\*" "\*nvvm\*" To clean up the uninstall: sudo tdnf autoremove **OpenSUSE / SLES** To remove CUDA Toolkit: sudo zypper remove "cuda\*" "\*cublas\*" "\*cufft\*" "\*cufile\*" "\*curand\*" \\ "\*cusolver\*" "\*cusparse\*" "\*gds-tools\*" "\*npp\*" "\*nvjpeg\*" "nsight\*" "\*nvvm\*" **Debian / Ubuntu** To remove CUDA Toolkit: sudo apt-get --purge remove "\*cuda\*" "\*cublas\*" "\*cufft\*" "\*cufile\*" "\*curand\*" \\ "\*cusolver\*" "\*cusparse\*" "\*gds-tools\*" "\*npp\*" "\*nvjpeg\*" "nsight\*" "\*nvvm\*" To clean up the uninstall: sudo apt-get autoremove --purge -V 12\. Advanced Setup[](#advanced-setup "Permalink to this headline") ===================================================================== Below is information on some advanced setup scenarios which are not covered in the basic instructions above. | | | | --- | --- |Table 6 Advanced Setup Scenarios when Installing CUDA[](#id52 "Permalink to this table") | Scenario | Instructions | | Install GPUDirect Storage | Refer to [Installing GPUDirect Storage](https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html)
.

GDS is supported in two different modes:

> * GDS (default/full perf mode)
>
> * Compatibility mode.
>

Installation instructions for them differ slightly. Compatibility mode is the only mode that is supported on certain distributions due to software dependency limitations.

Full GDS support is restricted to the following Linux distros:

> * Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04
>
> * RHEL 8.y (y <= 10), RHEL 9.y (y <= 5)
> | | Install CUDA to a specific directory using the Package Manager installation method. | **RPM**

The RPM packages don’t support custom install locations through the package managers (Yum and Zypper), but it is possible to install the RPM packages to a custom location using rpm’s `--relocate` parameter:

sudo rpm --install --relocate /usr/local/cuda-12.6=/new/toolkit package.rpm

You will need to install the packages in the correct dependency order; this task is normally taken care of by the package managers. For example, if package “foo” has a dependency on package “bar”, you should install package “bar” first, and package “foo” second. You can check the dependencies of a RPM package as follows:

rpm -qRp package.rpm

Note that the driver packages cannot be relocated.

**Deb**

The Deb packages do not support custom install locations. It is however possible to extract the contents of the Deb packages and move the files to the desired install location. See the next scenario for more details one xtracting Deb packages. | | Extract the contents of the installers. | **Runfile**

The Runfile can be extracted into the standalone Toolkit Runfiles by using the `--extract` parameter. The Toolkit standalone Runfiles can be further extracted by running:

./runfile.run --tar mxvf

./runfile.run -x

**RPM**

The RPM packages can be extracted by running:

rpm2cpio package.rpm \| cpio -idmv

**Deb**

The Deb packages can be extracted by running:

dpkg-deb -x package.deb output\_dir | | Modify Ubuntu’s apt package manager to query specific architectures for specific repositories.

This is useful when a foreign architecture has been added, causing “404 Not Found” errors to appear when the repository meta-data is updated. | Each repository you wish to restrict to specific architectures must have its `sources.list` entry modified. This is done by modifying the `/etc/apt/sources.list` file and any files containing repositories you wish to restrict under the `/etc/apt/sources.list.d/` directory. Normally, it is sufficient to modify only the entries in `/etc/apt/sources.list`

An architecture-restricted repository entry looks like:

deb \[arch=,\]

For example, if you wanted to restrict a repository to only the amd64 and i386 architectures, it would look like:

deb \[arch=amd64,i386\]

It is not necessary to restrict the `deb-src` repositories, as these repositories don’t provide architecture-specific packages.

For more details, see the `sources.list` manpage. | | The runfile installer fails to extract due to limited space in the TMP directory. | This can occur on systems with limited storage in the TMP directory (usually `/tmp`), or on systems which use a tmpfs in memory to handle temporary storage. In this case, the `--tmpdir` command-line option should be used to instruct the runfile to use a directory with sufficient space to extract into. More information on this option can be found in [Advanced Options](#runfile-advanced)
. | | In case of the error: `E: Failed to fetch file:/var/cuda-repo File not found` | **Debian and Ubuntu**

This can occur when installing CUDA after uninstalling a different version. Use the following command before installation:

sudo rm -v /var/lib/apt/lists/\*cuda\* /var/lib/apt/lists/\*nvidia\* | | Verbose installation on Debian and Ubuntu | Use the `--verbose-versions` flag, for example:

sudo apt-get install --verbose-versions cuda | 13\. Additional Considerations[](#additional-considerations "Permalink to this headline") =========================================================================================== Now that you have CUDA-capable hardware and the NVIDIA CUDA Toolkit installed, you can examine and enjoy the numerous included programs. To begin using CUDA to accelerate the performance of your own applications, consult the CUDA C++ Programming Guide, located in `/usr/local/cuda-12.8/doc`. A number of helpful development tools are included in the CUDA Toolkit to assist you as you develop your CUDA programs, such as NVIDIA® Nsight™ Eclipse Edition, NVIDIA Visual Profiler, CUDA-GDB, and CUDA-MEMCHECK. For technical support on programming questions, consult and participate in the developer forums at [https://forums.developer.nvidia.com/c/accelerated-computing/cuda/206](https://forums.developer.nvidia.com/c/accelerated-computing/cuda/206) . 14\. Frequently Asked Questions[](#frequently-asked-questions "Permalink to this headline") ============================================================================================= 14.1. How do I install the Toolkit in a different location?[](#how-do-i-install-the-toolkit-in-a-different-location "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------- The Runfile installation asks where you wish to install the Toolkit during an interactive install. If installing using a non-interactive install, you can use the `--toolkitpath` parameter to change the install location: ./runfile.run --silent \\ --toolkit --toolkitpath=/my/new/toolkit The RPM and Deb packages cannot be installed to a custom install location directly using the package managers. See the “Install CUDA to a specific directory using the Package Manager installation method” scenario in the [Advanced Setup](#advanced-setup) section for more information. 14.2. Why do I see “nvcc: No such file or directory” when I try to build a CUDA application?[](#why-do-i-see-nvcc-no-such-file-or-directory-when-i-try-to-build-a-cuda-application "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Your PATH environment variable is not set up correctly. Ensure that your PATH includes the bin directory where you installed the Toolkit, usually `/usr/local/cuda-12.8/bin`. export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}} 14.3. [Why do I see “error while loading shared libraries: : cannot open shared object file: No such file or directory” when I try to run a CUDA application that uses a CUDA library?](#faq3) [](#why-do-i-see-error-while-loading-shared-libraries-lib-name-cannot-open-shared-object-file-no-such-file-or-directory-when-i-try-to-run-a-cuda-application-that-uses-a-cuda-library "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Your LD\_LIBRARY\_PATH environment variable is not set up correctly. Ensure that your LD\_LIBRARY\_PATH includes the lib and/or lib64 directory where you installed the Toolkit, usually `/usr/local/cuda-12.8/lib{,64}`: export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 14.4. Why do I see multiple “404 Not Found” errors when updating my repository meta-data on Ubuntu?[](#why-do-i-see-multiple-404-not-found-errors-when-updating-my-repository-meta-data-on-ubuntu "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These errors occur after adding a foreign architecture because apt is attempting to query for each architecture within each repository listed in the system’s sources.list file. Repositories that do not host packages for the newly added architecture will present this error. While noisy, the error itself does no harm. Please see the [Advanced Setup](#advanced-setup) section for details on how to modify your `sources.list` file to prevent these errors. 14.5. How can I tell X to ignore a GPU for compute-only use?[](#how-can-i-tell-x-to-ignore-a-gpu-for-compute-only-use "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------- To make sure X doesn’t use a certain GPU for display, you need to specify which **other** GPU to use for display. For more information, please refer to the “Use a specific GPU for rendering the display” scenario in the [Advanced Setup](index.html#advanced-setup) section. 14.6. Why doesn’t the cuda-repo package install the CUDA Toolkit?[](#why-doesn-t-the-cuda-repo-package-install-the-cuda-toolkit "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------------- When using RPM or Deb, the downloaded package is a repository package. Such a package only informs the package manager where to find the actual installation packages, but will not install them. See the [Package Manager Installation](#package-manager-installation) section for more details. 14.7. How do I install an older CUDA version using a network repo?[](#how-do-i-install-an-older-cuda-version-using-a-network-repo "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Depending on your system configuration, you may not be able to install old versions of CUDA using the cuda metapackage. In order to install a specific version of CUDA, you may need to specify all of the packages that would normally be installed by the cuda metapackage at the version you want to install. If you are using yum to install certain packages at an older version, the dependencies may not resolve as expected. In this case you may need to pass “`--setopt=obsoletes=0`” to yum to allow an install of packages which are obsoleted at a later version than you are trying to install. 14.8. How do I handle “Errors were encountered while processing: glx-diversions”?[](#how-do-i-handle-errors-were-encountered-while-processing-glx-diversions "Permalink to this headline") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This sometimes occurs when trying to uninstall CUDA after a clean .deb installation. Run the following commands: sudo apt-get install glx-diversions --reinstall sudo apt-get remove nvidia-alternative Then re-run the commands from [Removing CUDA Toolkit](#removing-cuda-tk) . 15\. Notices[](#notices "Permalink to this headline") ======================================================= 15.1. Notice[](#notice "Permalink to this headline") ------------------------------------------------------ This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 15.2. OpenCL[](#opencl "Permalink to this headline") ------------------------------------------------------ OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 15.3. Trademarks[](#trademarks "Permalink to this headline") -------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. 16\. Copyright[](#copyright "Permalink to this headline") =========================================================== © 2009-2024 NVIDIA Corporation & affiliates. All rights reserved. This product includes software developed by the Syncro Soft SRL ([http://www.sync.ro/](http://www.sync.ro/) ). --- # 1. Introduction — Installation Guide Windows 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/CUDA_Installation_Guide_Windows.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA Installation Guide for Microsoft Windows The installation instructions for the CUDA Toolkit on Microsoft Windows systems. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ CUDA® is a parallel computing platform and programming model invented by NVIDIA. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU). CUDA was developed with several design goals in mind: * Provide a small set of extensions to standard programming languages, like C, that enable a straightforward implementation of parallel algorithms. With CUDA C/C++, programmers can focus on the task of parallelization of the algorithms rather than spending time on their implementation. * Support heterogeneous computation where applications use both the CPU and GPU. Serial portions of applications are run on the CPU, and parallel portions are offloaded to the GPU. As such, CUDA can be incrementally applied to existing applications. The CPU and GPU are treated as separate devices that have their own memory spaces. This configuration also allows simultaneous computation on the CPU and GPU without contention for memory resources. CUDA-capable GPUs have hundreds of cores that can collectively run thousands of computing threads. These cores have shared resources including a register file and a shared memory. The on-chip shared memory allows parallel tasks running on these cores to share data without sending it over the system memory bus. This guide will show you how to install and check the correct operation of the CUDA development tools. 1.1. System Requirements[](#system-requirements "Permalink to this headline") ------------------------------------------------------------------------------- To use CUDA on your system, you will need the following installed: * A CUDA-capable GPU * A supported version of Linux with a gcc compiler and toolchain * NVIDIA CUDA Toolkit (available at [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads) ) Supported Microsoft Windows® operating systems: * Microsoft Windows 11 24H2 * Microsoft Windows 11 22H2-SV2 * Microsoft Windows 11 23H2 * Microsoft Windows 10 22H2 * Microsoft Windows Server 2022 * Microsoft Windows Server 2025 | | | | | | | --- | --- | --- | --- | --- |Table 1 Windows Compiler Support in CUDA 12.8[](#id2 "Permalink to this table") | Compiler\* | IDE | Native x86\_64 | Cross-compilation (32-bit on 64-bit) | C++ Dialect | | --- | --- | --- | --- | --- | | MSVC Version 193x | Visual Studio 2022 17.x | YES | Not supported | C++14 (default), C++17, C++20 | | MSVC Version 192x | Visual Studio 2019 16.x | YES | C++14 (default), C++17 | \* Support for Visual Studio 2017 has been deprecated since release 12.5 and will be removed in a future release. Use Visual Studio 2019 or later to avoid link errors when using `nvrtc_static.lib` or `nvJitLink_static.lib`. 32-bit compilation native and cross-compilation is removed from CUDA 12.0 and later Toolkit. Use the CUDA Toolkit from earlier releases for 32-bit compilation. CUDA Driver will continue to support running 32-bit application binaries on GeForce GPUs until Ada. Ada will be the last architecture with driver support for 32-bit applications. Hopper does not support 32-bit applications. Support for running x86 32-bit applications on x86\_64 Windows is limited to use with: * CUDA Driver * CUDA Runtime (cudart) * CUDA Math Library (math.h) 1.2. About This Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This document is intended for readers familiar with Microsoft Windows operating systems and the Microsoft Visual Studio environment. You do not need previous experience with CUDA or experience with parallel computation. 2\. Installing CUDA Development Tools[](#installing-cuda-development-tools "Permalink to this headline") ========================================================================================================== Basic instructions can be found in the [Quick Start Guide](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#windows) . Read on for more detailed instructions. The setup of CUDA development tools on a system running the appropriate version of Windows consists of a few simple steps: * Verify the system has a CUDA-capable GPU. * Download the NVIDIA CUDA Toolkit. * Install the NVIDIA CUDA Toolkit. * Test that the installed software runs correctly and communicates with the hardware. 2.1. Verify You Have a CUDA-capable GPU[](#verify-you-have-a-cuda-capable-gpu "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- You can verify that you have a CUDA-capable GPU through the **Display Adapters** section in the **Windows Device Manager**. Here you will find the vendor name and model of your graphics card(s). If you have an NVIDIA card that is listed in [https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) , that GPU is CUDA-capable. The Release Notes for the CUDA Toolkit also contain a list of supported products. The **Windows Device Manager** can be opened via the following steps: 1. Open a run window from the Start Menu 2. Run: control /name Microsoft.DeviceManager 2.2. Download the NVIDIA CUDA Toolkit[](#download-the-nvidia-cuda-toolkit "Permalink to this headline") --------------------------------------------------------------------------------------------------------- The NVIDIA CUDA Toolkit is available at [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads) . Choose the platform you are using and one of the following installer formats: 1. Network Installer: A minimal installer which later downloads packages required for installation. Only the packages selected during the selection phase of the installer are downloaded. This installer is useful for users who want to minimize download time. 2. Full Installer: An installer which contains all the components of the CUDA Toolkit and does not require any further download. This installer is useful for systems which lack network access and for enterprise deployment. The CUDA Toolkit installs the CUDA driver and tools needed to create, build and run a CUDA application as well as libraries, header files, and other resources. **Download Verification** The download can be verified by comparing the MD5 checksum posted at [https://developer.download.nvidia.com/compute/cuda/12.8.0/docs/sidebar/md5sum.txt](https://developer.download.nvidia.com/compute/cuda/12.8.0/docs/sidebar/md5sum.txt) with that of the downloaded file. If either of the checksums differ, the downloaded file is corrupt and needs to be downloaded again. 2.3. Install the CUDA Software[](#install-the-cuda-software "Permalink to this headline") ------------------------------------------------------------------------------------------- Before installing the toolkit, you should read the Release Notes, as they provide details on installation and software functionality. Note The driver and toolkit must be installed for CUDA to function. If you have not installed a stand-alone driver, install the driver from the NVIDIA CUDA Toolkit. Note The installation may fail if Windows Update starts after the installation has begun. Wait until Windows Update is complete and then try the installation again. **Graphical Installation** Install the CUDA Software by executing the CUDA installer and following the on-screen prompts. **Silent Installation** The installer can be executed in silent mode by executing the package with the `-s` flag. Additional parameters can be passed which will install specific subpackages instead of all packages. See the table below for a list of all the subpackage names. | | | | --- | --- |Table 2 Possible Subpackage Names[](#id3 "Permalink to this table") | Subpackage Name | Subpackage Description | | --- | --- | | Toolkit Subpackages (defaults to C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.8) | | | cuda\_profiler\_api\_12.8 | CUDA Profiler API. | | cudart\_12.8 | CUDA Runtime libraries. | | cuobjdump\_12.8 | Extracts information from cubin files. | | cupti\_12.8 | The CUDA Profiling Tools Interface for creating profiling and tracing tools that target CUDA applications. | | cuxxfilt\_12.8 | The CUDA cu++ filt demangler tool. | | demo\_suite\_12.8 | Prebuilt demo applications using CUDA. | | documentation\_12.8 | CUDA HTML and PDF documentation files including the CUDA C++ Programming Guide, CUDA C++ Best Practices Guide, CUDA library documentation, etc. | | nvcc\_12.8 | CUDA compiler. | | nvdisasm\_12.8 | Extracts information from standalone cubin files. | | nvfatbin\_12.8 | Library for creating fatbinaries at runtime. | | nvjitlink\_12.8 | nvJitLink library. | | nvml\_dev\_12.8 | NVML development libraries and headers. | | nvprof\_12.8 | Tool for collecting and viewing CUDA application profiling data from the command-line. | | nvprune\_12.8 | Prunes host object files and libraries to only contain device code for the specified targets. | | nvrtc\_12.8

nvrtc\_dev\_12.8 | NVRTC runtime libraries. | | nvtx\_12.8 | NVTX on Windows. | | opencl\_12.8 | OpenCL library. | | visual\_profiler\_12.8 | Visual Profiler. | | sanitizer\_12.8 | Compute Sanitizer API. | | thrust\_12.8 | CUDA Thrust. | | cublas\_12.8

cublas\_dev\_12.8 | cuBLAS runtime libraries. | | cufft\_12.8

cufft\_dev\_12.8 | cuFFT runtime libraries. | | curand\_12.8

curand\_dev\_12.8 | cuRAND runtime libraries. | | cusolver\_12.8

cusolver\_dev\_12.8 | cuSOLVER runtime libraries. | | cusparse\_12.8

cusparse\_dev\_12.8 | cuSPARSE runtime libraries. | | npp\_12.8

npp\_dev\_12.8 | NPP runtime libraries. | | nvjpeg\_12.8

nvjpeg\_dev\_12.8 | nvJPEG libraries. | | nsight\_compute\_12.8 | Nsight Compute. | | nsight\_systems\_12.8 | Nsight Systems. | | nsight\_vse\_12.8 | Installs the Nsight Visual Studio Edition plugin in all VS. | | occupancy\_calculator\_12.8 | Installs the CUDA\_Occupancy\_Calculator.xls tool. | | visual\_studio\_integration\_12.8 | Installs CUDA project wizard and builds customization files in VS. | | Driver Subpackages | | | Display.Driver | The NVIDIA Display Driver. Required to run CUDA applications. | For example, to install only the compiler and driver components: .exe -s nvcc\_12.1 Display.Driver Use the `-n` option if you do not want to reboot automatically after install or uninstall, even if reboot is required. **Extracting and Inspecting the Files Manually** Sometimes it may be desirable to extract or inspect the installable files directly, such as in enterprise deployment, or to browse the files before installation. The full installation package can be extracted using a decompression tool which supports the LZMA compression method, such as [7-zip](http://www.7-zip.org/) or [WinZip](http://www.winzip.com/) . Once extracted, the CUDA Toolkit files will be in the `CUDAToolkit` folder, and similarly for CUDA Visual Studio Integration. Within each directory is a .dll and .nvi file that can be ignored as they are not part of the installable files. Note Accessing the files in this manner does not set up any environment settings, such as variables or Visual Studio integration. This is intended for enterprise-level deployment. ### 2.3.1. Uninstalling the CUDA Software[](#uninstalling-the-cuda-software "Permalink to this headline") All subpackages can be uninstalled through the Windows Control Panel by using the Programs and Features widget. 2.4. Using Conda to Install the CUDA Software[](#using-conda-to-install-the-cuda-software "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- This section describes the installation and configuration of CUDA when using the Conda installer. The Conda packages are available at [https://anaconda.org/nvidia](https://anaconda.org/nvidia) . ### 2.4.1. Conda Overview[](#conda-overview "Permalink to this headline") The Conda installation installs the CUDA Toolkit. The installation steps are listed below. ### 2.4.2. Installation[](#installation "Permalink to this headline") To perform a basic install of all CUDA Toolkit components using Conda, run the following command: conda install cuda \-c nvidia Note Install CUDA in a dedicated Conda environment instead of the base environment to avoid installation issues. ### 2.4.3. Uninstallation[](#uninstallation "Permalink to this headline") To uninstall the CUDA Toolkit using Conda, run the following command: conda remove cuda ### 2.4.4. Installing Previous CUDA Releases[](#installing-previous-cuda-releases "Permalink to this headline") All Conda packages released under a specific CUDA version are labeled with that release version. To install a previous version, include that label in the `install` command such as: conda install cuda \-c nvidia/label/cuda\-11.3.0 Note Some CUDA releases do not move to new versions of all installable components. When this is the case these components will be moved to the new label, and you may need to modify the install command to include both labels such as: conda install cuda \-c nvidia/label/cuda\-11.3.0 \-c nvidia/label/cuda\-11.3.1 This example will install all packages released as part of CUDA 11.3.1. 2.5. Use a Suitable Driver Model[](#use-a-suitable-driver-model "Permalink to this headline") ----------------------------------------------------------------------------------------------- On Windows 10 and later, the operating system provides two driver models under which the NVIDIA Driver may operate: * The WDDM driver model is used for display devices. * The [Tesla Compute Cluster (TCC)](https://www.nvidia.com/object/software-for-tesla-products.html) mode of the NVIDIA Driver is available for non-display devices such as NVIDIA Tesla GPUs and the GeForce GTX Titan GPUs; it uses the Windows WDM driver model. TCC is enabled by default on most recent NVIDIA Tesla GPUs. To check which driver mode is in use and/or to switch driver modes, use the `nvidia-smi` tool that is included with the NVIDIA Driver installation (see `nvidia-smi -h` for details). Note Keep in mind that when TCC mode is enabled for a particular GPU, that GPU _cannot_ be used as a display device. Note NVIDIA GeForce GPUs (excluding GeForce GTX Titan GPUs) do not support TCC mode. 2.6. Verify the Installation[](#verify-the-installation "Permalink to this headline") --------------------------------------------------------------------------------------- Before continuing, it is important to verify that the CUDA toolkit can find and communicate correctly with the CUDA-capable hardware. To do this, you need to compile and run some of the included sample programs. ### 2.6.1. Running the Compiled Examples[](#running-the-compiled-examples "Permalink to this headline") The version of the CUDA Toolkit can be checked by running `nvcc -V` in a Command Prompt window. You can display a Command Prompt window by going to: **Start > All Programs > Accessories > Command Prompt** CUDA Samples are located in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) . To use the samples, clone the project, build the samples, and run them using the instructions on the Github page. To verify a correct configuration of the hardware and software, it is highly recommended that you build and run the `deviceQuery` sample program. The sample can be built using the provided VS solution files in the `deviceQuery` folder. This assumes that you used the default installation directory structure. If CUDA is installed and configured correctly, the output should look similar to [Figure 1](index.html#compiling-examples__valid-results-from-sample-cuda-devicequery-program) . [![Valid Results from deviceQuery CUDA Sample](_images/valid-results-from-sample-cuda-devicequery-program.png)](_images/valid-results-from-sample-cuda-devicequery-program.png) Figure 1 Valid Results from deviceQuery CUDA Sample[](#compiling-examples-valid-results-from-sample-cuda-devicequery-program "Permalink to this image") The exact appearance and the output lines might be different on your system. The important outcomes are that a device was found, that the device(s) match what is installed in your system, and that the test passed. If a CUDA-capable device and the CUDA Driver are installed but `deviceQuery` reports that no CUDA-capable devices are present, ensure the device and driver are properly installed. Running the `bandwidthTest` program, located in the same directory as `deviceQuery` above, ensures that the system and the CUDA-capable device are able to communicate correctly. The output should resemble [Figure 2](index.html#compiling-examples__valid-results-from-sample-cuda-bandwidthtest-program) . [![Valid Results from bandwidthTest CUDA Sample](_images/valid-results-from-sample-cuda-bandwidthtest-program.png)](_images/valid-results-from-sample-cuda-bandwidthtest-program.png) Figure 2 Valid Results from bandwidthTest CUDA Sample[](#compiling-examples-valid-results-from-sample-cuda-bandwidthtest-program "Permalink to this image") The device name (second line) and the bandwidth numbers vary from system to system. The important items are the second line, which confirms a CUDA device was found, and the second-to-last line, which confirms that all necessary tests passed. If the tests do not pass, make sure you do have a CUDA-capable NVIDIA GPU on your system and make sure it is properly installed. To see a graphical representation of what CUDA can do, run the `particles` sample at https://github.com/NVIDIA/cuda-samples/tree/master/Samples/2\_Concepts\_and\_Techniques/particles 3\. Pip Wheels[](#pip-wheels "Permalink to this headline") ============================================================ NVIDIA provides Python Wheels for installing CUDA through pip, primarily for using CUDA with Python. These packages are intended for runtime use and do not currently include developer tools (these can be installed separately). Please note that with this installation method, CUDA installation environment is managed via pip and additional care must be taken to set up your host environment to use CUDA outside the pip environment. **Prerequisites** To install Wheels, you must first install the `nvidia-pyindex` package, which is required in order to set up your pip installation to fetch additional Python modules from the NVIDIA NGC PyPI repo. If your pip and setuptools Python modules are not up-to-date, then use the following command to upgrade these Python modules. If these Python modules are out-of-date then the commands which follow later in this section may fail. py -m pip install --upgrade setuptools pip wheel You should now be able to install the `nvidia-pyindex` module. py -m pip install nvidia-pyindex If your project is using a `requirements.txt` file, then you can add the following line to your `requirements.txt` file as an alternative to installing the `nvidia-pyindex` package: \--extra-index-url https://pypi.ngc.nvidia.com **Procedure** Install the CUDA runtime package: py -m pip install nvidia-cuda-runtime-cu12 Optionally, install additional packages as listed below using the following command: py -m pip install nvidia- **Metapackages** The following metapackages will install the latest version of the named component on Windows for the indicated CUDA version. “cu12” should be read as “cuda12”. * nvidia-cublas-cu12 * nvidia-cuda-runtime-cu12 * nvidia-cuda-cupti-cu12 * nvidia-cuda-nvcc-cu12 * nvidia-cuda-nvrtc-cu12 * nvidia-cuda-sanitizer-api-cu12 * nvidia-cufft-cu12 * nvidia-curand-cu12 * nvidia-cusolver-cu12 * nvidia-cusparse-cu12 * nvidia-npp-cu12 * nvidia-nvfatbin-cu12 * nvidia-nvjitlink-cu12 * nvidia-nvjpeg-cu12 * nvidia-nvml-dev-cu12 * nvidia-nvtx-cu12 * nvidia-opencl-cu12 These metapackages install the following packages: * nvidia-cublas-cu128 * nvidia-cuda-runtime-cu128 * nvidia-cuda-cupti-cu128 * nvidia-cuda-nvcc-cu128 * nvidia-cuda-nvrtc-cu128 * nvidia-cuda-sanitizer-api-cu128 * nvidia-cufft-cu128 * nvidia-curand-cu128 * nvidia-cusolver-cu128 * nvidia-cusparse-cu128 * nvidia-npp-cu128 * nvidia-nvfatbin-cu128 * nvidia-nvjitlink-cu128 * nvidia-nvjpeg-cu128 * nvidia-nvml-dev-cu128 * nvidia-nvtx-cu128 * nvidia-opencl-cu128 4\. Compiling CUDA Programs[](#compiling-cuda-programs "Permalink to this headline") ====================================================================================== The project files in the CUDA Samples have been designed to provide simple, one-click builds of the programs that include all source code. To build the Windows projects (for release or debug mode), use the provided `*.sln` solution files for Microsoft Visual Studio 2015 (deprecated in CUDA 11.1), 2017, 2019, or 2022. You can use either the solution files located in each of the examples directories in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) 4.1. Compiling Sample Projects[](#compiling-sample-projects "Permalink to this headline") ------------------------------------------------------------------------------------------- The `bandwidthTest` project is a good sample project to build and run. It is located in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1\_Utilities/bandwidthTest](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest) . If you elected to use the default installation location, the output is placed in `CUDA Samples\v12.8\bin\win64\Release`. Build the program using the appropriate solution file and run the executable. If all works correctly, the output should be similar to [Figure 2](index.html#compiling-examples__valid-results-from-sample-cuda-bandwidthtest-program) . 4.2. Sample Projects[](#sample-projects "Permalink to this headline") ----------------------------------------------------------------------- The sample projects come in two configurations: debug and release (where release contains no debugging information) and different Visual Studio projects. A few of the example projects require some additional setup. These sample projects also make use of the `$CUDA_PATH` environment variable to locate where the CUDA Toolkit and the associated `.props` files are. The environment variable is set automatically using the Build Customization `CUDA 12.8.props` file, and is installed automatically as part of the CUDA Toolkit installation process. | | | | --- | --- |Table 3 CUDA Visual Studio .props locations[](#id4 "Permalink to this table") | Visual Studio | CUDA 12.8 .props file Install Directory | | --- | --- | | Visual Studio 2017 | \\Common7\\IDE\\VC\\VCTargets\\BuildCustomizations | | Visual Studio 2019 | C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Professional\\MSBuild\\Microsoft\\VC\\v160\\BuildCustomizations | | Visual Studio 2022 | C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\MSBuild\\Microsoft\\VC\\v170\\BuildCustomizations | You can reference this `CUDA 12.8.props` file when building your own CUDA applications. 4.3. Build Customizations for New Projects[](#build-customizations-for-new-projects "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- When creating a new CUDA application, the Visual Studio project file must be configured to include CUDA build customizations. To accomplish this, click File-> New | Project… NVIDIA-> CUDA->, then select a template for your CUDA Toolkit version. For example, selecting the “CUDA 12.8 Runtime” template will configure your project for use with the CUDA 12.8 Toolkit. The new project is technically a C++ project (.vcxproj) that is preconfigured to use NVIDIA’s Build Customizations. All standard capabilities of Visual Studio C++ projects will be available. To specify a custom CUDA Toolkit location, under **CUDA C/C++**, select **Common**, and set the **CUDA Toolkit Custom Dir** field as desired. Note that the selected toolkit must match the version of the Build Customizations. Note A supported version of MSVC must be installed to use this feature. 4.4. Build Customizations for Existing Projects[](#build-customizations-for-existing-projects "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------- When adding CUDA acceleration to existing applications, the relevant Visual Studio project files must be updated to include CUDA build customizations. This can be done using one of the following two methods: 1. Open the Visual Studio project, right click on the project name, and select **Build Dependencies > Build Customizations…**, then select the CUDA Toolkit version you would like to target. 2. Alternatively, you can configure your project always to build with the most recently installed version of the CUDA Toolkit. First add a CUDA build customization to your project as above. Then, right click on the project name and select **Properties**. Under **CUDA C/C++**, select **Common**, and set the **CUDA Toolkit Custom Dir** field to `$(CUDA_PATH)` . Note that the `$(CUDA_PATH)` environment variable is set by the installer. While Option 2 will allow your project to automatically use any new CUDA Toolkit version you may install in the future, selecting the toolkit version explicitly as in Option 1 is often better in practice, because if there are new CUDA configuration options added to the build customization rules accompanying the newer toolkit, you would not see those new options using Option 2. If you use the `$(CUDA_PATH)` environment variable to target a version of the CUDA Toolkit for building, and you perform an installation or uninstallation of any version of the CUDA Toolkit, you should validate that the `$(CUDA_PATH)` environment variable points to the correct installation directory of the CUDA Toolkit for your purposes. You can access the value of the `$(CUDA_PATH)` environment variable via the following steps: 1. Open a run window from the Start Menu. 2. Run: control sysdm.cpl 3. Select the **Advanced** tab at the top of the window. 4. Click **Environment Variables** at the bottom of the window. Files which contain CUDA code must be marked as a `CUDA C/C++` file. This can done when adding the file by right clicking the project you wish to add the file to, selecting **Add New Item**, selecting **NVIDIA CUDA 12.8\\CodeCUDA C/C++ File**, and then selecting the file you wish to add. For advanced users, if you wish to try building your project against a newer CUDA Toolkit without making changes to any of your project files, go to the Visual Studio command prompt, change the current directory to the location of your project, and execute a command such as the following: msbuild /t:Rebuild /p:CudaToolkitDir="drive:/path/to/new/toolkit/" 5\. Additional Considerations[](#additional-considerations "Permalink to this headline") ========================================================================================== Now that you have CUDA-capable hardware and the NVIDIA CUDA Toolkit installed, you can examine and enjoy the numerous included programs. To begin using CUDA to accelerate the performance of your own applications, consult the CUDA C Programming Guide, located in the CUDA Toolkit documentation directory. A number of helpful development tools are included in the CUDA Toolkit or are available for download from the NVIDIA Developer Zone to assist you as you develop your CUDA programs, such as NVIDIA® Nsight™ Visual Studio Edition, and NVIDIA Visual Profiler. For technical support on programming questions, consult and participate in the developer forums by clicking [here](https://developer.nvidia.com/cuda/) . 6\. Notices[](#notices "Permalink to this headline") ====================================================== 6.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 6.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 6.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Introduction — Quick Start Guide 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/CUDA_Quick_Start_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA Quick Start Guide Minimal first-steps instructions to get CUDA running on a standard system. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ This guide covers the basic instructions needed to install CUDA and verify that a CUDA application can run on each supported platform. These instructions are intended to be used on a clean installation of a supported platform. For questions which are not answered in this document, please refer to the [Windows Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/) and [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . The CUDA installation packages can be found on the [CUDA Downloads Page](https://developer.nvidia.com/cuda-downloads/) . 2\. Windows[](#windows "Permalink to this headline") ====================================================== When installing CUDA on Windows, you can choose between the Network Installer and the Local Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. For more details, refer to the [Windows Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/) . 2.1. Network Installer[](#network-installer "Permalink to this headline") --------------------------------------------------------------------------- Perform the following steps to install CUDA and verify the installation. 1. Launch the downloaded installer package. 2. Read and accept the EULA. 3. Select **next** to download and install all components. 4. Once the download completes, the installation will begin automatically. 5. Once the installation completes, click “next” to acknowledge the Nsight Visual Studio Edition installation summary. 6. Click **close** to close the installer. 7. Navigate to the Samples’ `nbody` directory in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . 8. Open the `nbody` Visual Studio solution file for the version of Visual Studio you have installed, for example, `nbody_vs2019.sln`. ![_images/navigate_nbody.png](_images/navigate_nbody.png) 9. Open the **Build** menu within Visual Studio and click **Build Solution**. ![_images/nbody_build.png](_images/nbody_build.png) 10. Navigate to the CUDA Samples build directory and run the nbody sample. Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. 2.2. Local Installer[](#local-installer "Permalink to this headline") ----------------------------------------------------------------------- Perform the following steps to install CUDA and verify the installation. 1. Launch the downloaded installer package. 2. Read and accept the EULA. 3. Select **next** to install all components. 4. Once the installation completes, click **next** to acknowledge the Nsight Visual Studio Edition installation summary. 5. Click **close** to close the installer. 6. Navigate to the Samples’ `nbody` directory in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . 7. Open the nbody Visual Studio solution file for the version of Visual Studio you have installed. ![_images/navigate_nbody.png](_images/navigate_nbody.png) 8. Open the **Build** menu within Visual Studio and click **Build Solution**. ![_images/nbody_build.png](_images/nbody_build.png) 9. Navigate to the CUDA Samples build directory and run the nbody sample. Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. 2.3. Pip Wheels - Windows[](#pip-wheels-windows "Permalink to this headline") ------------------------------------------------------------------------------- NVIDIA provides Python Wheels for installing CUDA through pip, primarily for using CUDA with Python. These packages are intended for runtime use and do not currently include developer tools (these can be installed separately). Please note that with this installation method, CUDA installation environment is managed via pip and additional care must be taken to set up your host environment to use CUDA outside the pip environment. **Prerequisites** To install Wheels, you must first install the `nvidia-pyindex` package, which is required in order to set up your pip installation to fetch additional Python modules from the NVIDIA NGC PyPI repo. If your pip and setuptools Python modules are not up-to-date, then use the following command to upgrade these Python modules. If these Python modules are out-of-date then the commands which follow later in this section may fail. py -m pip install --upgrade setuptools pip wheel You should now be able to install the `nvidia-pyindex` module. py -m pip install nvidia-pyindex If your project is using a `requirements.txt` file, then you can add the following line to your `requirements.txt` file as an alternative to installing the `nvidia-pyindex` package: \--extra-index-url https://pypi.ngc.nvidia.com **Procedure** Install the CUDA runtime package: py -m pip install nvidia-cuda-runtime-cu12 Optionally, install additional packages as listed below using the following command: py -m pip install nvidia- **Metapackages** The following metapackages will install the latest version of the named component on Windows for the indicated CUDA version. “cu12” should be read as “cuda12”. * nvidia-cuda-runtime-cu12 * nvidia-cuda-cupti-cu12 * nvidia-cuda-nvcc-cu12 * nvidia-nvml-dev-cu12 * nvidia-cuda-nvrtc-cu12 * nvidia-nvtx-cu12 * nvidia-cuda-sanitizer-api-cu12 * nvidia-cublas-cu12 * nvidia-cufft-cu12 * nvidia-curand-cu12 * nvidia-cusolver-cu12 * nvidia-cusparse-cu12 * nvidia-npp-cu12 * nvidia-nvjpeg-cu12 These metapackages install the following packages: * nvidia-nvml-dev-cu126 * nvidia-cuda-nvcc-cu126 * nvidia-cuda-runtime-cu126 * nvidia-cuda-cupti-cu126 * nvidia-cublas-cu126 * nvidia-cuda-sanitizer-api-cu126 * nvidia-nvtx-cu126 * nvidia-cuda-nvrtc-cu126 * nvidia-npp-cu126 * nvidia-cusparse-cu126 * nvidia-cusolver-cu126 * nvidia-curand-cu126 * nvidia-cufft-cu126 * nvidia-nvjpeg-cu126 2.4. Conda[](#conda "Permalink to this headline") --------------------------------------------------- The Conda packages are available at [https://anaconda.org/nvidia](https://anaconda.org/nvidia) . **Installation** To perform a basic install of all CUDA Toolkit components using Conda, run the following command: conda install cuda -c nvidia **Uninstallation** To uninstall the CUDA Toolkit using Conda, run the following command: conda remove cuda 3\. Linux[](#linux "Permalink to this headline") ================================================== CUDA on Linux can be installed using an RPM, Debian, Runfile, or Conda package, depending on the platform being installed on. 3.1. Linux x86\_64[](#linux-x86-64 "Permalink to this headline") ------------------------------------------------------------------ For development on the x86\_64 architecture. In some cases, x86\_64 systems may act as host platforms targeting other architectures. See the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) for more details. ### 3.1.1. Redhat / CentOS[](#redhat-centos "Permalink to this headline") When installing CUDA on Redhat or CentOS, you can choose between the Runfile Installer and the RPM Installer. The Runfile Installer is only available as a Local Installer. The RPM Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the RPM installers, the instructions for the Local and Network variants are the same. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.1.1. RPM Installer[](#rpm-installer "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install EPEL to satisfy the DKMS dependency by following the instructions at [EPEL’s website](https://fedoraproject.org/wiki/EPEL) . 2. **Enable optional repos**: On **RHEL 8 Linux** only, execute the following steps to enable optional repositories. * **On x86\_64 workstation:** subscription-manager repos --enable=rhel-8-for-x86\_64-appstream-rpms subscription-manager repos --enable=rhel-8-for-x86\_64-baseos-rpms subscription-manager repos --enable=codeready-builder-for-rhel-8-x86\_64-rpms 3. Install the repository meta-data, clean the yum cache, and install CUDA: sudo rpm --install cuda-repo--..rpm sudo rpm --erase gpg-pubkey-7fa2af80\* sudo yum clean expire-cache sudo yum install cuda 4. Reboot the system to load the NVIDIA drivers: sudo reboot 5. Set up the development environment by modifying the `PATH` and `LD_LIBRARY_PATH` variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 6. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.1.2. Runfile Installer[](#runfile-installer "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Disable the Nouveau drivers: 1. Create a file at `/etc/modprobe.d/blacklist-nouveau.conf` with the following contents: blacklist nouveau options nouveau modeset=0 2. Regenerate the kernel initramfs: sudo dracut --force 2. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 3. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 4. Create an xorg.conf file to use the NVIDIA GPU for display: sudo nvidia-xconfig 5. Reboot the system to load the graphical interface: sudo reboot 6. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 7. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. ### 3.1.2. Fedora[](#fedora "Permalink to this headline") When installing CUDA on Fedora, you can choose between the Runfile Installer and the RPM Installer. The Runfile Installer is only available as a Local Installer. The RPM Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the RPM installers, the instructions for the Local and Network variants are the same. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.2.1. RPM Installer[](#fedora-x86-64-rpm "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install the RPMFusion free repository to satisfy the Akmods dependency: su -c 'dnf install --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm' 2. Install the repository meta-data, clean the dnf cache, and install CUDA: sudo rpm --install cuda-repo--..rpm sudo rpm --erase gpg-pubkey-7fa2af80\* sudo dnf clean expire-cache sudo dnf install cuda 3. Reboot the system to load the NVIDIA drivers: sudo reboot 4. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 5. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.2.2. Runfile Installer[](#fedora-x86-64-run "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Disable the Nouveau drivers: 1. Create a file at `/usr/lib/modprobe.d/blacklist-nouveau.conf` with the following contents: blacklist nouveau options nouveau modeset=0 2. Regenerate the kernel initramfs: sudo dracut --force 3. Run the below command: sudo grub2-mkconfig -o /boot/grub2/grub.cfg 4. Reboot the system: sudo reboot 2. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 3. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 4. Create an xorg.conf file to use the NVIDIA GPU for display: sudo nvidia-xconfig 5. Reboot the system to load the graphical interface. 6. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 7. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. ### 3.1.3. SUSE Linux Enterprise Server[](#suse-linux-enterprise-server "Permalink to this headline") When installing CUDA on SUSE Linux Enterprise Server, you can choose between the Runfile Installer and the RPM Installer. The Runfile Installer is only available as a Local Installer. The RPM Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the RPM installers, the instructions for the Local and Network variants are the same. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.3.1. RPM Installer[](#sles-x86-64-rpm "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install the repository meta-data, refresh the Zypper cache, update the GPG key, and install CUDA: sudo rpm --install cuda-repo--..rpm sudo SUSEConnect --product PackageHub/15/x86\_64 sudo zypper refresh sudo rpm --erase gpg-pubkey-7fa2af80\* sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo sudo zypper install cuda 2. Add the user to the video group: sudo usermod -a -G video 3. Reboot the system to load the NVIDIA drivers: sudo reboot 4. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 5. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the vectorAdd sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/0\_Introduction/vectorAdd](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/0_Introduction/vectorAdd) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.3.2. Runfile Installer[](#sles-x86-64-run "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 2. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 3. Create an xorg.conf file to use the NVIDIA GPU for display: sudo nvidia-xconfig 4. Reboot the system to load the graphical interface: sudo reboot 5. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 6. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the vectorAdd sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/0\_Introduction/vectorAdd](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/0_Introduction/vectorAdd) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. ### 3.1.4. OpenSUSE[](#opensuse "Permalink to this headline") When installing CUDA on OpenSUSE, you can choose between the Runfile Installer and the RPM Installer. The Runfile Installer is only available as a Local Installer. The RPM Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the RPM installers, the instructions for the Local and Network variants are the same. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.4.1. RPM Installer[](#suse-x86-64-rpm "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install the repository meta-data, refresh the Zypper cache, and install CUDA: sudo rpm --install cuda-repo--..rpm sudo rpm --erase gpg-pubkey-7fa2af80\* sudo zypper refresh sudo zypper install cuda 2. Add the user to the video group: sudo usermod -a -G video 3. Reboot the system to load the NVIDIA drivers: sudo reboot 4. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 5. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.4.2. Runfile Installer[](#suse-x86-64-run "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Disable the Nouveau drivers: 1. Create a file at `/etc/modprobe.d/blacklist-nouveau.conf` with the following contents: blacklist nouveau options nouveau modeset=0 2. Regenerate the kernel initrd: sudo /sbin/mkinitrd 2. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 3. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 4. Create an xorg.conf file to use the NVIDIA GPU for display: sudo nvidia-xconfig 5. Reboot the system to load the graphical interface: sudo reboot 6. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 7. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. ### 3.1.5. Amazon Linux 2023[](#amazon-linux-2023 "Permalink to this headline") #### 3.1.5.1. Prepare Amazon Linux 2023[](#prepare-amazon-linux-2023 "Permalink to this headline") 1. Perform the [pre-installation actions.](index.html#pre-installation-actions) 2. The kernel headers and development packages for the currently running kernel can be installed with: sudo dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r) kernel-modules-extra-$(uname -r) 3. Choose an installation method: [local repo](index.html#local-repo-installation-for-amazon) or [network repo](index.html#network-repo-installation-for-amazon) . #### 3.1.5.2. Local Repo Installation for Amazon Linux[](#local-repo-installation-for-amazon-linux "Permalink to this headline") 1. **Install local repository on file system:** sudo rpm --install cuda-repo-amzn2023-X-Y-local-\*.x86\_64.rpm #### 3.1.5.3. Network Repo Installation for Amazon Linux[](#network-repo-installation-for-amazon-linux "Permalink to this headline") 1. **Enable the network repository and clean the DN cache:** sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86\_64/cuda-amzn2023.repo sudo dnf clean expire-cache #### 3.1.5.4. Common Installation Instructions for Amazon Linux[](#common-installation-instructions-for-amazon-linux "Permalink to this headline") These instructions apply to both local and network installation for Amazon Linux. 1. **Install CUDA SDK:** sudo dnf module install nvidia-driver:latest-dkms sudo dnf install cuda-toolkit 2. **Install GPUDirect Filesystem:** sudo dnf install nvidia-gds 3. **Add libcuda.so symbolic link, if necessary:** The `libcuda.so` library is installed in the `/usr/lib{,64}/nvidia` directory. For pre-existing projects which use `libcuda.so`, it may be useful to add a symbolic link from `libcuda.so` in the `/usr/lib{,64}` directory. 4. **Reboot the system:** sudo reboot 5. Perform the [post-installation actions.](index.html#post-installation-actions) ### 3.1.6. Pip Wheels - Linux[](#pip-wheels-linux "Permalink to this headline") NVIDIA provides Python Wheels for installing CUDA through pip, primarily for using CUDA with Python. These packages are intended for runtime use and do not currently include developer tools (these can be installed separately). Please note that with this installation method, CUDA installation environment is managed via pip and additional care must be taken to set up your host environment to use CUDA outside the pip environment. **Prerequisites** To install Wheels, you must first install the `nvidia-pyindex` package, which is required in order to set up your pip installation to fetch additional Python modules from the NVIDIA NGC PyPI repo. If your pip and setuptools Python modules are not up-to-date, then use the following command to upgrade these Python modules. If these Python modules are out-of-date then the commands which follow later in this section may fail. python3 -m pip install --upgrade setuptools pip wheel You should now be able to install the `nvidia-pyindex` module. python3 -m pip install nvidia-pyindex If your project is using a `requirements.txt` file, then you can add the following line to your `requirements.txt` file as an alternative to installing the `nvidia-pyindex` package: \--extra-index-url https://pypi.ngc.nvidia.com **Procedure** Install the CUDA runtime package: python3 -m pip install nvidia-cuda-runtime-cu12 Optionally, install additional packages as listed below using the following command: python3 -m pip install nvidia- **Metapackages** The following metapackages will install the latest version of the named component on Linux for the indicated CUDA version. “cu12” should be read as “cuda12”. * nvidia-cuda-runtime-cu12 * nvidia-cuda-cupti-cu12 * nvidia-cuda-nvcc-cu12 * nvidia-nvml-dev-cu12 * nvidia-cuda-nvrtc-cu12 * nvidia-nvtx-cu12 * nvidia-cuda-sanitizer-api-cu12 * nvidia-cublas-cu12 * nvidia-cufft-cu12 * nvidia-curand-cu12 * nvidia-cusolver-cu12 * nvidia-cusparse-cu12 * nvidia-npp-cu12 * nvidia-nvjpeg-cu12 * nvidia-opencl-cu12 * nvidia-nvjitlink-cu12 These metapackages install the following packages: * nvidia-nvml-dev-cu126 * nvidia-cuda-nvcc-cu126 * nvidia-cuda-runtime-cu126 * nvidia-cuda-cupti-cu126 * nvidia-cublas-cu126 * nvidia-cuda-sanitizer-api-cu126 * nvidia-nvtx-cu126 * nvidia-cuda-nvrtc-cu126 * nvidia-npp-cu126 * nvidia-cusparse-cu126 * nvidia-cusolver-cu126 * nvidia-curand-cu126 * nvidia-cufft-cu126 * nvidia-nvjpeg-cu126 * nvidia-opencl-cu126 * nvidia-nvjitlink-cu126 ### 3.1.7. Conda[](#x86-64-conda "Permalink to this headline") The Conda packages are available at [https://anaconda.org/nvidia](https://anaconda.org/nvidia) . **Installation** To perform a basic install of all CUDA Toolkit components using Conda, run the following command: conda install cuda -c nvidia **Uninstallation** To uninstall the CUDA Toolkit using Conda, run the following command: conda remove cuda ### 3.1.8. WSL[](#wsl "Permalink to this headline") These instructions must be used if you are installing in a WSL environment. Do not use the Ubuntu instructions in this case. 1. **Install repository meta-data** sudo dpkg -i cuda-repo-\_\_.deb 1. **Update the CUDA public GPG key** sudo apt-key del 7fa2af80 When installing using the local repo: sudo cp /var/cuda-repo-ubuntu2004-12-0-local/cuda-\*-keyring.gpg /usr/share/keyrings/ When installing using the network repo: wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb Pin file to prioritize CUDA repository: wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-.pin sudo mv cuda-.pin /etc/apt/preferences.d/cuda-repository-pin-600 2. **Update the Apt repository cache and install CUDA** sudo apt-get update sudo apt-get install cuda ### 3.1.9. Ubuntu[](#ubuntu "Permalink to this headline") When installing CUDA on Ubuntu, you can choose between the Runfile Installer and the Debian Installer. The Runfile Installer is only available as a Local Installer. The Debian Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the Debian installers, the instructions for the Local and Network variants are the same. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.9.1. Debian Installer[](#debian-installer "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install the repository meta-data, update the GPG key, update the apt-get cache, and install CUDA: sudo dpkg --install cuda-repo--..deb sudo apt-key del 7fa2af80 wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb sudo add-apt-repository contrib sudo apt-get update sudo apt-get -y install cuda 2. Reboot the system to load the NVIDIA drivers: sudo reboot 3. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 4. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.9.2. Runfile Installer[](#ubuntu-x86-64-run "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Disable the Nouveau drivers: 1. Create a file at `/etc/modprobe.d/blacklist-nouveau.conf` with the following contents: blacklist nouveau options nouveau modeset=0 2. Regenerate the kernel initramfs: sudo update-initramfs -u 2. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 3. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 4. Create an `xorg.conf` file to use the NVIDIA GPU for display: sudo nvidia-xconfig 5. Reboot the system to load the graphical interface: sudo reboot 6. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 7. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. ### 3.1.10. Debian[](#debian "Permalink to this headline") When installing CUDA on Debian 10, you can choose between the Runfile Installer and the Debian Installer. The Runfile Installer is only available as a Local Installer. The Debian Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. For more details, refer to the [Linux Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) . #### 3.1.10.1. Debian Installer[](#debian-x86-64-deb "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Install the repository meta-data, remove old GPG key, install GPG key, update the apt-get cache, and install CUDA: sudo dpkg -i cuda-repo-\_\_.deb sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/debian10/x86\_64/7fa2af80.pub sudo apt-key del 7fa2af80 wget https://developer.download.nvidia.com/compute/cuda/repos///cuda-keyring\_1.1-1\_all.deb sudo dpkg -i cuda-keyring\_1.1-1\_all.deb sudo add-apt-repository contrib sudo apt-get update sudo apt-get -y install cuda 2. Reboot the system to load the NVIDIA drivers: sudo reboot 3. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 4. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. #### 3.1.10.2. Runfile Installer[](#debian-x86-64-run "Permalink to this headline") Perform the following steps to install CUDA and verify the installation. 1. Disable the Nouveau drivers: 1. Create a file at `/etc/modprobe.d/blacklist-nouveau.conf` with the following contents: blacklist nouveau options nouveau modeset=0 2. Regenerate the kernel initramfs: sudo update-initramfs -u 2. Reboot into runlevel 3 by temporarily adding the number “3” and the word “nomodeset” to the end of the system’s kernel boot parameters. 3. Run the installer silently to install with the default selections (implies acceptance of the EULA): sudo sh cuda\_\_linux.run --silent 4. Create an xorg.conf file to use the NVIDIA GPU for display: sudo nvidia-xconfig 5. Reboot the system to load the graphical interface: sudo reboot 6. Set up the development environment by modifying the PATH and LD\_LIBRARY\_PATH variables: export PATH=/usr/local/cuda-12.8/bin${PATH:+:${PATH}} export LD\_LIBRARY\_PATH=/usr/local/cuda-12.8/lib64\\ ${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}} 7. Install a writable copy of the samples from [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , then build and run the nbody sample using the Linux instructions in [https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5\_Domain\_Specific/nbody](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/nbody) . Note Run samples by navigating to the executable’s location, otherwise it will fail to locate dependent resources. 4\. Notices[](#notices "Permalink to this headline") ====================================================== 4.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 4.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 4.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. CUDA 12.8 Features — CUDA Features Archive 12.8 documentation * [](../index.html) » * 1\. CUDA 12.8 Features * v12.8 | [PDF](../pdf/CUDA_Features_Archive.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA CUDA Features Archive The list of CUDA features by release. 1\. CUDA 12.8 Features[](#cuda-12-8-features "Permalink to this headline") ============================================================================ 1.1. General CUDA[](#general-cuda "Permalink to this headline") ----------------------------------------------------------------- * This release adds compiler support for the following Nvidia Blackwell GPU architectures: * SM\_100 * SM\_101 * SM\_120 * Tegra-Specific: * Added MPS support for DRIVE OS QNX * Added support for GCC 13.2.0 * Added support for Unified Virtual Memory (UVM) with Extended GPU Memory (EGM) arrays * Hopper Confidential Computing: * Added multi-GPU support for protected PCIe mode * Added key rotation capability for single GPU passthrough mode * NVML Updates: * Fixed per-process memory usage reporting for Docker containers using Open GPU Kernel Module drivers * Added support for DRAM encryption query and control (Blackwell) * Added checkpoint/restore functionality for userspace applications * Added support for Blackwell reduced bandwidth mode (RBM) * CUDA Graphs: * Added conditional execution features for CUDA Graphs: > * ELSE graph support for IF nodes > > * SWITCH node support > * Introduced additional performance optimizations * CUDA Usermode Driver (UMD): * Added PCIe device ID to CUDA device properties * Added cudaStreamGetDevice and cuStreamGetDevice APIs to retrieve the device associated with a CUDA stream * Added CUDA support for INT101010 texture/surface format * Added batch CUDA asynchronous memory copy APIs (cuMemcpyBatchAsync and cuMemcpyBatch3DAsync) for variable-sized transfers between multiple source and destination buffers * Userspace Checkpoint and Restore: * Added new driver API for checkpoint/restore operations 1.2. CUDA Compiler[](#cuda-compiler "Permalink to this headline") ------------------------------------------------------------------- * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-7](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-7) . * Added two new nvcc flags: * `static-global-template-stub {true|false}`: Controls host side linkage for global/device/constant/managed templates in whole program mode * `device-entity-has-hidden-visibility {true|false}`: Controls ELF visibility of global/device/constant/managed symbols The current default value for both flags is false. These defaults will change to true in our future release. For detailed information about these flags and their impact on existing programs, refer to the `nvcc --help` command or the online CUDA documentation. * **libNVVM** `libNVVM` now supports compilation for the Blackwell family of architectures. Compilation of compute capabilities `compute_100` and greater (Blackwell and future architectures) uses an updated NVVM IR dialect, based on LLVM 18.1.8 IR (the “modern” dialect) that differs from the older dialect used for pre-Blackwell architectures (a compute capability less than compute\_100). NVVM IR bitcode using the older dialect generated for pre-Blackwell architectures can be used to target Blackwell and later architectures, with the exception of debug metadata. * **nvdisasm** `Nvdisasm` now supports emitting JSON formatted SASS disassembly. 2\. CUDA 12.6 Features[](#cuda-12-6-features "Permalink to this headline") ============================================================================ 2.1. General CUDA[](#id1 "Permalink to this headline") -------------------------------------------------------- * The default Linux driver installation changes in this release, preferring NVIDIA GPU Open Kernel Modules to proprietary drivers. The open source drivers are now the default and recommended installation option. **Important:** The GPU Open Kernel Modules drivers are only compatible with Turing and newer GPUs. If your GPU is from an older family (Maxwell, Pascal, or Volta) you must continue to use the proprietary drivers. For additional information, refer to this blog post: [https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/](https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/) . And, for full details, the CUDA Installation Guide for Linux: [https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) * New `nvidia-open` meta-packages are available to improve driver installation of NVIDIA Open GPU kernel modules. \[_4752203_\] 2.2. CUDA Compiler[](#id2 "Permalink to this headline") --------------------------------------------------------- * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-5](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-5) . * Latest host compiler Clang-18 support. * Support for Stack Canaries in device code. CUDA compilers can now insert stack canaries in device code. The NVCC flag `--device-stack-protector=true` enables this feature. Stack canaries make it more difficult to exploit certain types of memory safety bugs involving stack-local variables. The compiler uses heuristics to assess the risk of such a bug in each function. Only those functions which are deemed high-risk make use of a stack canary. * Added a new compiler option `-forward-slash-prefix-opts` (Windows only). If this flag is specified, and forwarding unknown options to host toolchain is enabled (`-forward-unknown-opts` or `-forward-unknown-to-host-linker` or `-forward-unknown-to-host-compiler`), then a command line argument beginning with ‘/’ is forwarded to the host toolchain. For example: `nvcc -forward-slash-prefix-opts -forward-unknown-opts /T foo.cu` will forward the flag `/T` to the host compiler and linker. When this flag is not specified, a command line argument beginning with `/` is treated as an input file. For example, `nvcc /T foo.cu` will treat `/T` as an input file, and the Windows API function `GetFullPathName()` is used to determine the full path name. Note: This flag is only supported on Windows. For more details, refer to `nvcc-help`. * An environment variable `NVCC_CCBIN` is introduced for NVCC: Users can set `NVCC_CCBIN` to specify the host compiler, but it has lower priority than command-line option `-ccbin`. If `NVCC_CCBIN` and `-ccbin` are both set, NVCC uses the host compiler specified by `-ccbin`. 3\. CUDA 12.5 Features[](#cuda-12-5-features "Permalink to this headline") ============================================================================ 3.1. General CUDA[](#id3 "Permalink to this headline") -------------------------------------------------------- * In an upcoming CUDA release the NVIDIA Open GPU kernel module flavor will be the default and recommended installation option. End-users with Maxwell, Pascal, or Volta GPUs may need to take action to install the NVIDIA proprietary kernel modules. * MPS (Multi-process service) is now supported on L4T and embedded-Linux Tegra platforms. More details can be found [here](https://docs.nvidia.com/deploy/mps/index.html) . 4\. CUDA 12.4 Features[](#cuda-12-4-features "Permalink to this headline") ============================================================================ 4.1. General CUDA[](#id4 "Permalink to this headline") -------------------------------------------------------- * Green contexts are a lightweight alternative to traditional contexts, with the ability to pass in a set of resources that they should be initialized with. This allows the developer to represent distinct spatial partitions of the GPU, provision resources for them, and target them via the same programming model that CUDA exposes (streams, kernel launches, etc.). For detail, refer to [https://docs.nvidia.com/cuda/cuda-driver-api/group\_\_CUDA\_\_GREEN\_\_CONTEXTS.html](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html) . * Access-counter-based memory migration for Grace Hopper systems is now enabled by default. As this is the first release with the capability enabled, developers may find that applications that had been optimized for earlier memory migration algorithms may see a performance regression if optimized for the earlier behaviors. Should this occur, we introduce a supported but temporary flag to opt out of this behavior. You can control the enablement of this feature by unloading and reloading the NVIDIA UVM driver, as follows: \# modprobe -r nvidia\_uvm \# modprobe nvidia\_uvm uvm\_perf\_access\_counter\_mimc\_migration\_enable=0 * This release introduces support for the following new features in CUDA graphs: * Graph conditional nodes (enhanced from 12.3) * Device-side node parameter update for device graphs * Updatable graph node priorities without recompilation * Enhanced monitoring capabilities through NVML and nvidia-smi: * NVJPG and NVOFA utilization percentage * PCIe class and subclass reporting * dmon reports are now available in CSV format * More descriptive error codes returned from NVML * dmon now reports gpm-metrics for MIG (that is, `nvidia-smi dmon --gpm-metrics` runs in MIG mode) * NVML running against older drivers will report `FUNCTION_NOT_FOUND` in some cases, failing gracefully if NVML is newer than the driver * NVML APIs to query protected memory information for Hopper Confidential Computing * This release introduces nvFatbin, a new library to create CUDA fat binary files at runtime. For more details, please visit [https://docs.nvidia.com/cuda/nvfatbin/index.html](https://docs.nvidia.com/cuda/nvfatbin/index.html) . 4.2. CUDA Compilers[](#cuda-compilers "Permalink to this headline") --------------------------------------------------------------------- * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-4](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-4) . * Added the `__maxnreg__` kernel function qualifier to allow users to directly specify the maximum number of registers to be allocated to a single thread in a thread block in CUDA C++. * Added a new flag `-fdevice-syntax-only` that ends device compilation after front-end syntax checking. This option can provide rapid feedback (warnings and errors) of source code changes as it will not invoke the optimizer. Note: this option will not generate valid object code. * Add a new flag `-minimal` for NVRTC compilation. The `-minimal` flag omits certain language features to reduce compile time for small programs. In particular, the following are omitted: * Texture and surface functions and associated types (for example, `cudaTextureObject_t`). * CUDA Runtime Functions that are provided by the cudadevrt device code library, typically named with prefix “cuda”, for example, `cudaMalloc`. * Kernel launch from device code. * Types and macros associated with CUDA Runtime and Driver APIs, provided by `cuda/tools/cudart/driver_types.h`, typically named with the prefix “cuda” for example, `cudaError_t`. * Starting in CUDA 12.4, PTXAS enables position independent code (`-pic`) as default when the compilation mode is whole program compilation. Users can opt out by specifying the `-pic=false` option to PTXAS. Debug compilation and separate compilation continue to have position independent code disabled by default. In future, position independent code will allow the CUDA Driver to share a single copy of text section across contexts and reduce resident memory usage. 5\. CUDA 12.3 Features[](#cuda-12-3-features "Permalink to this headline") ============================================================================ 5.1. General CUDA[](#id5 "Permalink to this headline") -------------------------------------------------------- * CUDA User Mode Driver, CUDA Runtime libraries and CUBLAS now come with obfuscated symbol names and with frame pointers enabled. * Frame Pointers are enabled for other libraries in the CUDA Toolkit: NVIDIA Management Library, CUDA Profiling Tools Interface, cuBLAS, Compiler libraries – NVRTC, PTXJIT compiler, nvJitLink, and libnvvm. * Allows better runtime visibility and traceability, and allows easier exchange of runtime information with NVIDIA when needed for debugging purposes. * See [https://developer.nvidia.com/blog/cuda-toolkit-symbol-server/](https://developer.nvidia.com/blog/cuda-toolkit-symbol-server/) for information on how to use obfuscated symbols. * Symbol server address is: [https://cudatoolkit-symbols.nvidia.com/](https://cudatoolkit-symbols.nvidia.com/) . * Lazy loading default enablement for Windows: * Brings the significant memory savings and load-time reductions of lazy loading to Windows by default. Additionally, makes the behavior equivalent between Linux and Windows. * Single-step CUDA uninstall for Windows: * It is no longer necessary to uninstall multiple components of the CUDA Toolkit individually to upgrade or uninstall CUDA. This can now be done in a single step. * CUDA Graphs: * Graph edge data, allowing modified dependencies between nodes. Programmatic Dependent Launch may now be described natively in CUDA Graphs. * Launch completion events: * Allows a dependency on scheduling, but not completion, of all blocks in a kernel, enabling tighter control of scheduling. * MPS: * Added a CUDA API to query whether or not MPS is running. * Added a driver API to return the name of a kernel function. * Added an API to libnvJitLink to return the nvJitLink version. * Added support for reading kernel parameters in device functions. * Enable querying the return type of \_\_device\_\_ lambdas with trailing return type. Fixes uncommon failures when using device-side lambdas. * NVML / nvidia-smi: * Metric for front-end context switch utilization (FECS) * Added metrics for Ada Lovelace AV1 codec utilization * Support GPU monitoring on Tegra * Added an NVML API to expose H100 PCIe counters and corresponding PCIe section in nvidia-smi 5.2. CUDA Compilers[](#id6 "Permalink to this headline") ---------------------------------------------------------- * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-3](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-3) . * Enhanced thread support when using the libNVVM API. Clients can take advantage of improved compilation speeds by spawning multiple compilation threads concurrently. * Improved compile time in some common scenarios: * Extended split compilation to cubin for LTO. * Turned on concurrent NVVM processing by default, with documented fallback to serialized compilation. * Reduced NVRTC compile time for small programs via moving CUDA C++ builtin function declarations into compiler. * Moved `cuda_fp16.h` and `cuda_bf16.h` into compiler bitcode. * Added new keyword\`\`\_\_inline\_hint\_\_\`\` to specify device functions in a different `.cu` file to be inlined during LTO. * Enabled querying return type of \_\_device\_\_ lambdas with trailing return type. * Provided information about unused bytes to compute-sanitizer for better diagnostics. 6\. CUDA 12.2 Features[](#cuda-12-2-features "Permalink to this headline") ============================================================================ 6.1. General CUDA[](#id7 "Permalink to this headline") -------------------------------------------------------- * This release introduces Heterogeneous Memory Management (HMM), allowing seamless sharing of data between host memory and accelerator devices. HMM is supported on Linux only and requires a recent kernel (6.1.24+ or 6.2.11+). HMM requires the use of NVIDIA’s GPU Open Kernel Modules driver. As this is the first release of HMM, some limitations exist: * GPU atomic operations on file-backed memory are not yet supported. * Arm CPUs are not yet supported. * HugeTLBfs pages are not yet supported on HMM (this is an uncommon scenario). * The `fork()` system call is not fully supported yet when attempting to share GPU-accessible memory between parent and child processes. * HMM is not yet fully optimized, and may perform slower than programs using `cudaMalloc()`, `cudaMallocManaged()`, or other existing CUDA memory management APIs. The performance of programs not using HMM will not be affected. * The Lazy Loading feature (introduced in CUDA 11.7) is now enabled by default on Linux with the 535 driver. To disable this feature on Linux, set the environment variable `CUDA_MODULE_LOADING=EAGER` before launch. Default enablement for Windows will happen in a future CUDA driver release. To enable this feature on Windows, set the environment variable `CUDA_MODULE_LOADING=LAZY` before launch. * Host NUMA memory allocation: Allocate a CPU memory targeting a specific NUMA node using either the CUDA virtual memory management APIs or the CUDA stream-ordered memory allocator. Applications must ensure device accesses to pointer backed by HOST allocations from these APIs are performed only after they have explicitly requested accessibility for the memory on the accessing device. It is undefined behavior to access these host allocations from a device without accessibility for the address range, regardless of whether the device supports pageable memory access or not. * Added per-client priority mapping at runtime for CUDA Multi-Process Service (MPS). This allows multiple processes running under MPS to arbitrate priority at a coarse-grained level between multiple processes without changing the application code. We introduce a new environment variable `CUDA_MPS_CLIENT_PRIORITY`, which accepts two values: NORMAL priority, 0, and BELOW\_NORMAL priority, 1. For example, given two clients, a potential configuration is as follows: | | | | --- | --- | | `// Client 1’s Environment`

`export CUDA_MPS_CLIENT_PRIORITY=0`

`// NORMAL` | `// Client 2’s Environment`

`export CUDA_MPS_CLIENT_PRIORITY=1`

`// BELOW NORMAL` | 6.2. CUDA Compilers[](#id8 "Permalink to this headline") ---------------------------------------------------------- * libNVVM samples have been moved out of the toolkit and made publicly available on GitHub as part of the NVIDIA/cuda-samples project. Similarly, the nvvmir-samples have been moved from the nvidia-compiler-sdk project on GitHub to the new location of the libNVVM samples in the NVIDIA/cuda-samples project. * For changes to PTX, refer to [https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-2](https://docs.nvidia.com/cuda/parallel-thread-execution/#ptx-isa-version-8-2) . 7\. CUDA 12.1 Features[](#cuda-12-1-features "Permalink to this headline") ============================================================================ 7.1. General CUDA[](#id9 "Permalink to this headline") -------------------------------------------------------- > * New meta-packages for Linux installation. > > > * `cuda-toolkit` > > > > * Installs all CUDA Toolkit packages required to develop CUDA applications. > > > > * Handles upgrading to the latest version of CUDA when it’s released. > > > > * Does not include the driver. > > > > * `cuda-toolkit-12` > > > > * Installs all CUDA Toolkit packages required to develop CUDA applications. > > > > * Handles upgrading to the next 12.x version of CUDA when it’s released. > > > > * Does not include the driver. > > > > * New CUDA API to enable mini core dump programmatically is now available. Refer to [https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support](https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support) > and [https://docs.nvidia.com/cuda/cuda-driver-api/group\_\_CUDA\_\_COREDUMP.html#group\_\_CUDA\_\_COREDUMP](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__COREDUMP.html#group__CUDA__COREDUMP) > for more information. > 7.2. CUDA Compilers[](#id10 "Permalink to this headline") ----------------------------------------------------------- * NVCC has added support for host compiler: GCC 12.2, NVC++ 22.11, Clang 15.0, VS2022 17.4 * Breakpoint and single stepping behavior for a multi-line statement in device code has been improved, when code is compiled with nvcc using gcc/clang host compiler compiler or when compiled with NVRTC on non-Windows platforms. The debugger will now correctly breakpoint and single-step on each source line of the multiline source code statement. * PTX has exposed a new special register in the public ISA, which can be used to query total size of shared memory which includes user shared memory and SW reserved shared memory. * NVCC and NVRTC now show preprocessed source line and column info in a diagnostic to help users to understand the message and identify the issue causing the diagnostic. The source line and column info can be turned off with `--brief-diagnostics=true`. 8\. CUDA 12.0 Features[](#cuda-12-0-features "Permalink to this headline") ============================================================================ 8.1. General CUDA[](#id11 "Permalink to this headline") --------------------------------------------------------- * CUDA 12.0 exposes programmable functionality for many features of the Hopper and Ada Lovelace architectures: * Many tensor operations now available via public PTX: * TMA operations * TMA bulk operations * 32x Ultra xMMA (including FP8/FP16) * Membar domains in Hopper, controlled via launch parameters * Support Hopper asynchronous transaction barrier in C++ and PTX * Introduced C intrinsics for Cooperative Grid Array (CGA) relaxed barrier support * Programmatic L2 Cache to SM multicast (Hopper-only) * Public PTX for SIMT collectives - elect\_one * Genomics/DPX instructions now available for Hopper GPUs to provide faster combined-math arithmetic operations (three-way max, fused add+max, etc.) * Enhancements to the CUDA graphs API: * You can now schedule graph launches from GPU device-side kernels by calling built-in functions. With this ability, user code in kernels can dynamically schedule graph launches, greatly increasing the flexibility of CUDA graphs. * The `cudaGraphInstantiate()` API has been refactored to remove unused parameters. * Added the ability to use virtual memory management (VMM) APIs such as `cuMemCreate()` with GPUs masked by `CUDA_VISIBLE_DEVICES`. * Application and library developers can now programmatically update the priority of CUDA streams. * CUDA 12.0 adds support for revamped CUDA Dynamic Parallelism APIs, offering substantial performance improvements vs. the legacy CUDA Dynamic Parallelism APIs. * Added new APIs to obtain unique stream and context IDs from user-provided objects: * `cuStreamGetId(CUstream hStream, unsigned long long *streamId)` * `cuCtxGetId(CUcontext ctx, unsigned long long *ctxId)` * Added support for read-only `cuMemSetAccess()` flag `CU_MEM_ACCESS_FLAGS_PROT_READ`. 8.2. CUDA Compilers[](#id12 "Permalink to this headline") ----------------------------------------------------------- * JIT LTO support is now officially part of the CUDA Toolkit through a separate nvJitLink library. A technical deep dive blog will go into more details. Note that the earlier implementation of this feature has been deprecated. Refer to the Deprecation/Dropped Features section below for details. * New host compiler support: * GCC 12.1 (Official) and 12.2.1 ( Experimental) * VS 2022 17.4 Preview 3 fixes compiler errors mentioning an internal function `std::_Bit_cast` by using CUDA’s support for `__builtin_bit_cast`. * NVCC and NVRTC now support the c++20 dialect. Most of the language features are available in host and device code; some such as coroutines are not supported in device code. Modules are not supported for both host and device code. Host Compiler Minimum Versions: GCC 10, Clang 11, VS2022, Arm C/C++ 22.x. Refer to the individual Host Compiler documentation for other feature limitations. Note that a compilation issue in C++20 mode with header mentioning an internal function `std::_Bit_cast` is resolved in VS2022 17.4. * NVRTC default C++ dialect changed from C++14 to C++17. Refer to the ISO C++ standard for reference on the feature set and compatibility between the dialects. * NVVM IR Update: with CUDA 12.0 we are releasing NVVM IR 2.0 which is incompatible with NVVM IR 1.x accepted by the libNVVM compiler in prior CUDA toolkit releases. Linking of NVVM IR Version 1.11 with 2.0 will result in a compiler error. Users of the libNVVM compiler in CUDA 12.0 toolkit must generate [NVVM IR 2.0](https://docs.nvidia.com/cuda/nvvm-ir-spec/index.html) . 9\. CUDA 11.8 Features[](#cuda-11-8-features "Permalink to this headline") ============================================================================ 9.1. General CUDA[](#id13 "Permalink to this headline") --------------------------------------------------------- > * This release introduces support for both the Hopper and Ada Lovelace GPU families. > > * Added support for Rocky Linux 9. > > * Added support for Kylin OS. > > * Package upgradable CUDA is now available starting CUDA 11.8 for Jetson devices. Refer to [https://docs.nvidia.com/cuda/cuda-for-tegra-appnote/index.html#upgradable-package-for-jetson](https://docs.nvidia.com/cuda/cuda-for-tegra-appnote/index.html#upgradable-package-for-jetson) > for details on how to upgrade to the latest CUDA version on Jetson and the supported JetPack versions. > 10\. CUDA 11.7 Features[](#cuda-11-7-features "Permalink to this headline") ============================================================================= 10.1. General CUDA[](#id14 "Permalink to this headline") ---------------------------------------------------------- * To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install packages from CUDA repositories. To ensure continued access to the latest NVIDIA software, please follow the instructions here: [https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/](https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/) . * NVIDIA Open GPU Kernel Modules: With CUDA 11.7 and R515 driver, NVIDIA is open sourcing the GPU kernel mode driver under dual GPL/MIT license. Refer to [https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#open-gpu-kernel-modules](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#open-gpu-kernel-modules) for more information. * Lazy Loading: Delay kernel loading from host to GPU to the point where the kernel is called. This also only loads used kernels, which may result in a significant device-side memory savings. This also defers load latency from the beginning of the application to the point where a kernel is first called—overall binary load latency is usually significantly reduced, but is also shifted to later points in the application. To enable this feature, set the environment variable CUDA\_MODULE\_LOADING=LAZY before launching your process. Note that this feature is only compatible with libraries compiled with CUDA versions >= 11.7. 10.2. CUDA Compilers[](#id15 "Permalink to this headline") ------------------------------------------------------------ * Grid private constants * NVCC host compiler support for clang13 11\. CUDA 11.6 Features[](#cuda-11-6-features "Permalink to this headline") ============================================================================= 11.1. Compiler[](#compiler "Permalink to this headline") ---------------------------------------------------------- * VS2022 Support: CUDA 11.6 officially supports the latest VS2022 as host compiler. A separate Nsight Visual Studio installer 2022.1.1 must be downloaded from [here](https://developer.nvidia.com/nsight-visual-studio-edition) . A future CUDA release will have the Nsight Visual Studio installer with VS2022 support integrated into it. * New instructions in public PTX: New instructions for bit mask creation—BMSK, and sign extension—SZEXT, are added to the public PTX ISA. You can find documentation for these instructions in the PTX ISA guide: [BMSK](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bmsk) and [SZEXT](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-szext) . * Unused Kernel Optimization: In CUDA 11.5, unused kernel pruning was introduced with the potential benefits of reducing binary size and improving performance through more efficient optimizations. This was an opt-in feature but in 11.6, this feature is enabled by default. As mentioned in the 11.5 blog, there is an opt-out flag that can be used in case it becomes necessary for debug purposes or for other special situations. > $ nvcc -rdc=true user.cu testlib.a -o user -Xnvlink -ignore-host-info * New -arch=native option: In addition to the `-arch=all` and `-arch=all-major` options added in CUDA 11.5, NVCC introduced `-arch= native` in CUDA 11.5 update 1. This -arch=native option is a convenient way for users to let NVCC determine the right target architecture to compile the CUDA device code to based on the GPU installed on the system. This can be particularly helpful for testing when applications are run on the same system they are compiled in. * Generate PTX from nvlink: Using the following command line, device linker, nvlink will produce PTX as an output in addition to CUBIN: > nvcc -dlto -dlink -ptx * Device linking by nvlink is the final stage in the CUDA compilation process. Applications that have multiple source translation units have to be compiled in separate compilation mode. LTO (introduced in CUDA 11.4) allowed nvlink to perform optimizations at device link time instead of at compile time so that separately compiled applications with several translation units can be optimized to the same level as whole program compilations with a single translation unit. However, without the option to output PTX, applications that cared about forward compatibility of device code could not benefit from Link Time Optimization or had to constrain the device code to a single source file. * With the option for nvlink that performs LTO to generate the output in PTX, customer applications that require forward compatibility across GPU architectures can span across multiple files and can also take advantage of Link Time Optimization. * Bullseye support: NVCC compiled source code now works with the code coverage tool Bullseye. The code coverage is only for the CPU or the host functions. Code coverage for device function is not supported through bullseye. * INT128 developer tool support: In 11.5, CUDA C++ support for 128 bit was added. In 11.6, developer tools support the datatype as well. With the latest version of libcu++, int 128 data datype is supported by math functions. 12\. Notices[](#notices "Permalink to this headline") ======================================================= 12.1. Notice[](#notice "Permalink to this headline") ------------------------------------------------------ This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 12.2. OpenCL[](#opencl "Permalink to this headline") ------------------------------------------------------ OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 12.3. Trademarks[](#trademarks "Permalink to this headline") -------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. License Agreement for NVIDIA Software Development Kits — EULA * [](../index.html) » * 1\. License Agreement for NVIDIA Software Development Kits * v12.8 | [PDF](../pdf/EULA.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * End User License Agreement NVIDIA Software License Agreement and CUDA Supplement to Software License Agreement. The CUDA Toolkit End User License Agreement applies to the NVIDIA CUDA Toolkit, the NVIDIA CUDA Samples, the NVIDIA Display Driver, NVIDIA Nsight tools (Visual Studio Edition), and the associated documentation on CUDA APIs, programming model and development tools. If you do not agree with the terms and conditions of the license agreement, then do not download or use the software. Last updated: January 7, 2025 Preface The Software License Agreement in [Chapter 1](index.html#nvidia-driver-license) and the Supplement in [Chapter 2](index.html#cuda-toolkit-supplement-license-agreement) contain license terms and conditions that govern the use of NVIDIA CUDA toolkit. By accepting this agreement, you agree to comply with all the terms and conditions applicable to the product(s) included herein. NVIDIA Driver **Description** This package contains the operating system driver and fundamental system software components for NVIDIA GPUs. NVIDIA CUDA Toolkit **Description** The NVIDIA CUDA Toolkit provides command-line and graphical tools for building, debugging and optimizing the performance of applications accelerated by NVIDIA GPUs, runtime and math libraries, and documentation including programming guides, user manuals, and API references. **Default Install Location of CUDA Toolkit** Windows platform: %ProgramFiles%\\NVIDIA GPU Computing Toolkit\\CUDA\\v#.# Linux platform: /usr/local/cuda-#.# Mac platform: /Developer/NVIDIA/CUDA-#.# NVIDIA CUDA Samples **Description** CUDA Samples are now located in [https://github.com/nvidia/cuda-samples](https://github.com/nvidia/cuda-samples) , which includes instructions for obtaining, building, and running the samples. They are no longer included in the CUDA toolkit. NVIDIA Nsight Visual Studio Edition (Windows only) **Description** NVIDIA Nsight Development Platform, Visual Studio Edition is a development environment integrated into Microsoft Visual Studio that provides tools for debugging, profiling, analyzing and optimizing your GPU computing and graphics applications. **Default Install Location of Nsight Visual Studio Edition** Windows platform: %ProgramFiles(x86)%\\NVIDIA Corporation\\Nsight Visual Studio Edition #.# 1\. License Agreement for NVIDIA Software Development Kits[](#license-agreement-for-nvidia-software-development-kits "Permalink to this headline") ==================================================================================================================================================== **Important Notice—Read before downloading, installing, copying or using the licensed software:** This license agreement, including exhibits attached (“Agreement”) is a legal agreement between you and NVIDIA Corporation (“NVIDIA”) and governs your use of a NVIDIA software development kit (“SDK”). Each SDK has its own set of software and materials, but here is a description of the types of items that may be included in a SDK: source code, header files, APIs, data sets and assets (examples include images, textures, models, scenes, videos, native API input/output files), binary software, sample code, libraries, utility programs, programming code and documentation. This Agreement can be accepted only by an adult of legal age of majority in the country in which the SDK is used. If you are entering into this Agreement on behalf of a company or other legal entity, you represent that you have the legal authority to bind the entity to this Agreement, in which case “you” will mean the entity you represent. If you don’t have the required age or authority to accept this Agreement, or if you don’t accept all the terms and conditions of this Agreement, do not download, install or use the SDK. You agree to use the SDK only for purposes that are permitted by (a) this Agreement, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. 1.1. License[](#license "Permalink to this headline") ------------------------------------------------------- ### 1.1.1. License Grant[](#license-grant "Permalink to this headline") Subject to the terms of this Agreement, NVIDIA hereby grants you a non-exclusive, non-transferable license, without the right to sublicense (except as expressly provided in this Agreement) to: 1. Install and use the SDK, 2. Modify and create derivative works of sample source code delivered in the SDK, and 3. Distribute those portions of the SDK that are identified in this Agreement as distributable, as incorporated in object code format into a software application that meets the distribution requirements indicated in this Agreement. ### 1.1.2. Distribution Requirements[](#distribution-requirements "Permalink to this headline") These are the distribution requirements for you to exercise the distribution grant: 1. Your application must have material additional functionality, beyond the included portions of the SDK. 2. The distributable portions of the SDK shall only be accessed by your application. 3. The following notice shall be included in modifications and derivative works of sample source code distributed: “This software contains source code provided by NVIDIA Corporation.” 4. Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. 5. The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA’s intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. 6. You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SDK not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed SDK. ### 1.1.3. Authorized Users[](#authorized-users "Permalink to this headline") You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the SDK from your secure network to perform work on your behalf. If you are an academic institution you may allow users enrolled or employed by the academic institution to access and use the SDK from your secure network. You are responsible for the compliance with the terms of this Agreement by your authorized users. If you become aware that your authorized users didn’t follow the terms of this Agreement, you agree to take reasonable steps to resolve the non-compliance and prevent new occurrences. ### 1.1.4. Pre-Release SDK[](#pre-release-sdk "Permalink to this headline") The SDK versions identified as alpha, beta, preview or otherwise as pre-release, may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercial versions of NVIDIA software and materials. Use of a pre-release SDK may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. You may use a pre-release SDK at your own risk, understanding that pre-release SDKs are not intended for use in production or business-critical systems. NVIDIA may choose not to make available a commercial version of any pre-release SDK. NVIDIA may also choose to abandon development and terminate the availability of a pre-release SDK at any time without liability. ### 1.1.5. Updates[](#updates "Permalink to this headline") NVIDIA may, at its option, make available patches, workarounds or other updates to this SDK. Unless the updates are provided with their separate governing terms, they are deemed part of the SDK licensed to you as provided in this Agreement. You agree that the form and content of the SDK that NVIDIA provides may change without prior notice to you. While NVIDIA generally maintains compatibility between versions, NVIDIA may in some cases make changes that introduce incompatibilities in future versions of the SDK. ### 1.1.6. Components Under Other Licenses[](#components-under-other-licenses "Permalink to this headline") The SDK may come bundled with, or otherwise include or be distributed with, NVIDIA or third-party components with separate legal notices or terms as may be described in proprietary notices accompanying the SDK. If and to the extent there is a conflict between the terms in this Agreement and the license terms associated with the component, the license terms associated with the components control only to the extent necessary to resolve the conflict. Subject to the other terms of this Agreement, you may use the SDK to develop and test applications released under Open Source Initiative (OSI) approved open source software licenses. ### 1.1.7. Reservation of Rights[](#reservation-of-rights "Permalink to this headline") NVIDIA reserves all rights, title, and interest in and to the SDK, not expressly granted to you under this Agreement. 1.2. Limitations[](#limitations "Permalink to this headline") --------------------------------------------------------------- The following license limitations apply to your use of the SDK: 1. You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SDK or copies of the SDK. 2. Except as expressly provided in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify, or create derivative works of any portion of the SDK. For clarity, you may not distribute or sublicense the SDK as a stand-alone product. 3. Unless you have an agreement with NVIDIA for this purpose, you may not indicate that an application created with the SDK is sponsored or endorsed by NVIDIA. 4. You may not bypass, disable, or circumvent any encryption, security, digital rights management or authentication mechanism in the SDK. 5. You may not use the SDK in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SDK be: 1. Disclosed or distributed in source code form; 2. Licensed for the purpose of making derivative works; or 3. Redistributable at no charge. 6. You acknowledge that the SDK as delivered is not tested or certified by NVIDIA for use in connection with the design, construction, maintenance, and/or operation of any system where the use or failure of such system could result in a situation that threatens the safety of human life or results in catastrophic damages (each, a “Critical Application”). Examples of Critical Applications include use in avionics, navigation, autonomous vehicle applications, ai solutions for automotive products, military, medical, life support or other life critical applications. NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. You are solely responsible for ensuring that any product or service developed with the SDK as a whole includes sufficient features to comply with all applicable legal and regulatory standards and requirements. 7. You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of or related to products or services that use the SDK in or for Critical Applications, and for use of the SDK outside of the scope of this Agreement or not in compliance with its terms. 8. You may not reverse engineer, decompile or disassemble any portion of the output generated using SDK elements for the purpose of translating such output artifacts to target a non-NVIDIA platform. 1.3. Ownership[](#ownership "Permalink to this headline") ----------------------------------------------------------- 1. NVIDIA or its licensors hold all rights, title and interest in and to the SDK and its modifications and derivative works, including their respective intellectual property rights, subject to your rights under [Section 1.3.2](index.html#ownership-driver-your-rights) . This SDK may include software and materials from NVIDIA’s licensors, and these licensors are intended third party beneficiaries that may enforce this Agreement with respect to their intellectual property rights. 1. You hold all rights, title and interest in and to your applications and your derivative works of the sample source code delivered in the SDK, including their respective intellectual property rights, subject to NVIDIA’s rights under [Section 1.3.1](index.html#ownership-driver-nvidia-rights) . 2. You may, but don’t have to, provide to NVIDIA suggestions, feature requests or other feedback regarding the SDK, including possible enhancements or modifications to the SDK. For any feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) it without the payment of any royalties or fees to you. NVIDIA will use feedback at its choice. NVIDIA is constantly looking for ways to improve its products, so you may send feedback to NVIDIA through the developer portal at [https://developer.nvidia.com](https://developer.nvidia.com.) . 1.4. No Warranties[](#no-warranties "Permalink to this headline") ------------------------------------------------------------------- THE SDK IS PROVIDED BY NVIDIA “AS IS” AND “WITH ALL FAULTS.” TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. 1.5. Limitation of Liability[](#limitation-of-liability "Permalink to this headline") --------------------------------------------------------------------------------------- TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY (I) SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR (II) DAMAGES FOR (A) ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR (B) THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SDK, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA’S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS AGREEMENT EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. These exclusions and limitations of liability shall apply regardless if NVIDIA or its affiliates have been advised of the possibility of such damages, and regardless of whether a remedy fails its essential purpose. These exclusions and limitations of liability form an essential basis of the bargain between the parties, and, absent any of these exclusions or limitations of liability, the provisions of this Agreement, including, without limitation, the economic terms, would be substantially different. 1.6. Termination[](#termination "Permalink to this headline") --------------------------------------------------------------- 1. This Agreement will continue to apply until terminated by either you or NVIDIA as described below. 2. If you want to terminate this Agreement, you may do so by stopping to use the SDK. 3. NVIDIA may, at any time, terminate this Agreement if: 1. (i) you fail to comply with any term of this Agreement and the non-compliance is not fixed within thirty (30) days following notice from NVIDIA (or immediately if you violate NVIDIA’s intellectual property rights); 2. (ii) you commence or participate in any legal proceeding against NVIDIA with respect to the SDK; or 3. (iii) NVIDIA decides to no longer provide the SDK in a country or, in NVIDIA’s sole discretion, the continued use of it is no longer commercially viable. 4. Upon any termination of this Agreement, you agree to promptly discontinue use of the SDK and destroy all copies in your possession or control. Your prior distributions in accordance with this Agreement are not affected by the termination of this Agreement. Upon written request, you will certify in writing that you have complied with your commitments under this section. Upon any termination of this Agreement all provisions survive except for the license grant provisions. 1.7. General[](#general "Permalink to this headline") ------------------------------------------------------- If you wish to assign this Agreement or your rights and obligations, including by merger, consolidation, dissolution or operation of law, contact NVIDIA to ask for permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. NVIDIA may assign, delegate or transfer this Agreement and its rights and obligations, and if to a non-affiliate you will be notified. You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. This Agreement will be governed in all respects by the laws of the United States and of the State of Delaware, without regard to the conflicts of laws principles. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, such provision will be construed as limited to the extent necessary to be consistent with and fully enforceable under the law and the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. Each party acknowledges and agrees that the other is an independent contractor in the performance of this Agreement. The SDK has been developed entirely at private expense and is “commercial items” consisting of “commercial computer software” and “commercial computer software documentation” provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions in this Agreement pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (c)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. The SDK is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SDK into any country, or use the SDK in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury’s Office of Foreign Assets Control (OFAC), or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this Agreement, you confirm that you are not located in a country currently embargoed by the U.S. or otherwise prohibited from receiving the SDK under U.S. law. Any notice delivered by NVIDIA to you under this Agreement will be delivered via mail, email or fax. You agree that any notices that NVIDIA sends you electronically will satisfy any legal communication requirements. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. This Agreement and any exhibits incorporated into this Agreement constitute the entire agreement of the parties with respect to the subject matter of this Agreement and supersede all prior negotiations or documentation exchanged between the parties relating to this SDK license. Any additional and/or conflicting terms on documents issued by you are null, void, and invalid. Any amendment or waiver under this Agreement shall be in writing and signed by representatives of both parties. 2\. CUDA Toolkit Supplement to Software License Agreement for NVIDIA Software Development Kits[](#cuda-toolkit-supplement-to-software-license-agreement-for-nvidia-software-development-kits "Permalink to this headline") ============================================================================================================================================================================================================================ The terms in this supplement govern your use of the NVIDIA CUDA Toolkit SDK under the terms of your license agreement (“Agreement”) as modified by this supplement. Capitalized terms used but not defined below have the meaning assigned to them in the Agreement. This supplement is an exhibit to the Agreement and is incorporated as an integral part of the Agreement. In the event of conflict between the terms in this supplement and the terms in the Agreement, the terms in this supplement govern. 2.1. License Scope[](#license-scope "Permalink to this headline") ------------------------------------------------------------------- The SDK is licensed for you to develop applications only for use in systems with NVIDIA GPUs. 2.2. Distribution[](#distribution "Permalink to this headline") ----------------------------------------------------------------- The portions of the SDK that are distributable under the Agreement are listed in [Attachment A.](index.html#attachment-a) 2.3. Operating Systems[](#operating-systems "Permalink to this headline") --------------------------------------------------------------------------- Those portions of the SDK designed exclusively for use on the Linux or FreeBSD operating systems, or other operating systems derived from the source code to these operating systems, may be copied and redistributed for use in accordance with this Agreement, provided that the object code files are not modified in any way (except for unzipping of compressed files). 2.4. Audio and Video Encoders and Decoders[](#audio-and-video-encoders-and-decoders "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- You acknowledge and agree that it is your sole responsibility to obtain any additional third-party licenses required to make, have made, use, have used, sell, import, and offer for sale your products or services that include or incorporate any third-party software and content relating to audio and/or video encoders and decoders from, including but not limited to, Microsoft, Thomson, Fraunhofer IIS, Sisvel S.p.A., MPEG-LA, and Coding Technologies. NVIDIA does not grant to you under this Agreement any necessary patent or other rights with respect to any audio and/or video encoders and decoders. 2.5. Licensing[](#licensing "Permalink to this headline") ----------------------------------------------------------- If the distribution terms in this Agreement are not suitable for your organization, or for any questions regarding this Agreement, please contact NVIDIA at [nvidia-compute-license-questions@nvidia.com](mailto:nvidia-compute-license-questions%40nvidia.com) . 2.6. Attachment A[](#attachment-a "Permalink to this headline") ----------------------------------------------------------------- The following CUDA Toolkit files may be distributed with applications developed by you, including certain variations of these files that have version number or architecture specific information embedded in the file name - as an example only, for release version 9.0 of the 64-bit Windows software, the file cudart64\_90.dll is redistributable. | | | | --- | --- | | **Component** | **CUDA Runtime** | | Windows | cudart.dll, cudart\_static.lib, cudadevrt.lib | | Mac OSX | libcudart.dylib, libcudart\_static.a, libcudadevrt.a | | Linux | libcudart.so, libcudart\_static.a, libcudadevrt.a | | Android | libcudart.so, libcudart\_static.a, libcudadevrt.a | | **Component** | **CUDA FFT Library** | | Windows | cufft.dll, cufftw.dll, cufft.lib, cufftw.lib | | Mac OSX | libcufft.dylib, libcufft\_static.a, libcufftw.dylib, libcufftw\_static.a | | Linux | libcufft.so, libcufft\_static.a, libcufftw.so, libcufftw\_static.a | | Android | libcufft.so, libcufft\_static.a, libcufftw.so, libcufftw\_static.a | | **Component** | **CUDA BLAS Library** | | Windows | cublas.dll, cublasLt.dll | | Mac OSX | libcublas.dylib, libcublasLt.dylib, libcublas\_static.a, libcublasLt\_static.a | | Linux | libcublas.so, libcublasLt.so, libcublas\_static.a, libcublasLt\_static.a | | Android | libcublas.so, libcublasLt.so, libcublas\_static.a, libcublasLt\_static.a | | **Component** | **NVIDIA “Drop-in” BLAS Library** | | Windows | nvblas.dll | | Mac OSX | libnvblas.dylib | | Linux | libnvblas.so | | **Component** | **CUDA Sparse Matrix Library** | | Windows | cusparse.dll, cusparse.lib | | Mac OSX | libcusparse.dylib, libcusparse\_static.a | | Linux | libcusparse.so, libcusparse\_static.a | | Android | libcusparse.so, libcusparse\_static.a | | **Component** | **CUDA Linear Solver Library** | | Windows | cusolver.dll, cusolver.lib | | Mac OSX | libcusolver.dylib, libcusolver\_static.a | | Linux | libcusolver.so, libcusolver\_static.a | | Android | libcusolver.so, libcusolver\_static.a | | **Component** | **CUDA Random Number Generation Library** | | Windows | curand.dll, curand.lib | | Mac OSX | libcurand.dylib, libcurand\_static.a | | Linux | libcurand.so, libcurand\_static.a | | Android | libcurand.so, libcurand\_static.a | | **Component** | **NVIDIA Performance Primitives Library** | | Windows | nppc.dll, nppc.lib, nppial.dll, nppial.lib, nppicc.dll, nppicc.lib, nppicom.dll, nppicom.lib, nppidei.dll, nppidei.lib, nppif.dll, nppif.lib, nppig.dll, nppig.lib, nppim.dll, nppim.lib, nppist.dll, nppist.lib, nppisu.dll, nppisu.lib, nppitc.dll, nppitc.lib, npps.dll, npps.lib | | Mac OSX | libnppc.dylib, libnppc\_static.a, libnppial.dylib, libnppial\_static.a, libnppicc.dylib, libnppicc\_static.a, libnppicom.dylib, libnppicom\_static.a, libnppidei.dylib, libnppidei\_static.a, libnppif.dylib, libnppif\_static.a, libnppig.dylib, libnppig\_static.a, libnppim.dylib, libnppisu\_static.a, libnppitc.dylib, libnppitc\_static.a, libnpps.dylib, libnpps\_static.a | | Linux | libnppc.so, libnppc\_static.a, libnppial.so, libnppial\_static.a, libnppicc.so, libnppicc\_static.a, libnppicom.so, libnppicom\_static.a, libnppidei.so, libnppidei\_static.a, libnppif.so, libnppif\_static.a libnppig.so, libnppig\_static.a, libnppim.so, libnppim\_static.a, libnppist.so, libnppist\_static.a, libnppisu.so, libnppisu\_static.a, libnppitc.so libnppitc\_static.a, libnpps.so, libnpps\_static.a | | Android | libnppc.so, libnppc\_static.a, libnppial.so, libnppial\_static.a, libnppicc.so, libnppicc\_static.a, libnppicom.so, libnppicom\_static.a, libnppidei.so, libnppidei\_static.a, libnppif.so, libnppif\_static.a libnppig.so, libnppig\_static.a, libnppim.so, libnppim\_static.a, libnppist.so, libnppist\_static.a, libnppisu.so, libnppisu\_static.a, libnppitc.so libnppitc\_static.a, libnpps.so, libnpps\_static.a | | **Component** | **NVIDIA JPEG Library** | | Windows | nvjpeg.lib, nvjpeg.dll | | Linux | libnvjpeg.so, libnvjpeg\_static.a | | **Component** | **Internal common library required for statically linking to cuBLAS, cuSPARSE, cuFFT, cuRAND, nvJPEG and NPP** | | Mac OSX | libculibos.a | | Linux | libculibos.a | | **Component** | **NVIDIA Runtime Compilation Library and Header** | | All | nvrtc.h | | Windows | nvrtc.dll, nvrtc-builtins.dll | | Mac OSX | libnvrtc.dylib, libnvrtc-builtins.dylib | | Linux | libnvrtc.so, libnvrtc-builtins.so, libnvrtc\_static.a, libnvrtc-builtins\_static.a | | **Component** | **NVIDIA Optimizing Compiler Library** | | Windows | nvvm.dll | | Mac OSX | libnvvm.dylib | | Linux | libnvvm.so | | **Component** | **NVIDIA JIT Linking Library** | | Windows | libnvJitLink.dll, libnvJitLink.lib | | Linux | libnvJitLink.so, libnvJitLink\_static.a | | **Component** | **NVIDIA Common Device Math Functions Library** | | Windows | libdevice.10.bc | | Mac OSX | libdevice.10.bc | | Linux | libdevice.10.bc | | **Component** | **CUDA Occupancy Calculation Header Library** | | All | cuda\_occupancy.h | | **Component** | **CUDA Floating Point Type Headers** | | All | cuda\_fp16.h, cuda\_fp16.hpp, cuda\_bf16.h, cuda\_bf16.hpp, cuda\_fp8.h, cuda\_fp8.hpp, cuda\_fp6.h, cuda\_fp6.hpp, cuda\_fp4.h, cuda\_fp4.hpp | | **Component** | **CUDA Headers for Runtime Compilation** | | All | crt/host\_defines.h, cuComplex.h, cuda\_awbarrier\_helpers.h, cuda\_awbarrier\_primitives.h, cuda\_awbarrier.h, cuda\_pipeline\_helpers.h, ccuda\_pipeline\_primitives.h, ccuda\_pipeline.h, cuda\_runtime\_api.h, cuda.h, cuda/std/tuple, cuda/std/type\_traits, cuda/std/type\_traits, cuda/std/utility, device\_types.h, vector\_functions.h, vector\_types.h | | **Component** | **CUDA Profiling Tools Interface (CUPTI) Library** | | Windows | cupti.dll | | Mac OSX | libcupti.dylib | | Linux | libcupti.so | | **Component** | **NVIDIA Tools Extension Library** | | Windows | nvToolsExt.dll, nvToolsExt.lib | | Mac OSX | libnvToolsExt.dylib | | Linux | libnvToolsExt.so | | **Component** | **NVIDIA CUDA Driver Libraries** | | Linux | libcuda.so, libnvidia-ptxjitcompiler.so, libnvptxcompiler\_static.a | | **Component** | **NVIDIA CUDA File IO Libraries and Header** | | All | cufile.h | | Linux | libcufile.so, libcufile\_rdma.so, libcufile\_static.a, libcufile\_rdma\_static.a | In addition to the rights above, for parties that are developing software intended solely for use on Jetson development kits or Jetson modules, and running Linux for Tegra software, the following shall apply: * The SDK may be distributed in its entirety, as provided by NVIDIA, and without separation of its components, for you and/or your licensees to create software development kits for use only on the Jetson platform and running Linux for Tegra software. 2.7. Attachment B[](#attachment-b "Permalink to this headline") ----------------------------------------------------------------- **Additional Licensing Obligations** The following third party components included in the SOFTWARE are licensed to Licensee pursuant to the following terms and conditions: 1. Licensee’s use of the GDB third party component is subject to the terms and conditions of GNU GPL v3: This product includes copyrighted third-party software licensed under the terms of the GNU General Public License v3 ("GPL v3"). All third-party software packages are copyright by their respective authors. GPL v3 terms and conditions are hereby incorporated into the Agreement by this reference: http://www.gnu.org/licenses/gpl.txt Consistent with these licensing requirements, the software listed below is provided under the terms of the specified open source software licenses. To obtain source code for software provided under licenses that require redistribution of source code, including the GNU General Public License (GPL) and GNU Lesser General Public License (LGPL), contact [oss-requests@nvidia.com](mailto:oss-requests%40nvidia.com) . This offer is valid for a period of three (3) years from the date of the distribution of this product by NVIDIA CORPORATION. Component License CUDA-GDB GPL v3 2. Licensee represents and warrants that any and all third party licensing and/or royalty payment obligations in connection with Licensee’s use of the H.264 video codecs are solely the responsibility of Licensee. 3. Licensee’s use of the Thrust library is subject to the terms and conditions of the Apache License Version 2.0. All third-party software packages are copyright by their respective authors. Apache License Version 2.0 terms and conditions are hereby incorporated into the Agreement by this reference. [http://www.apache.org/licenses/LICENSE-2.0.html](http://www.apache.org/licenses/LICENSE-2.0.html) In addition, Licensee acknowledges the following notice: Thrust includes source code from the Boost Iterator, Tuple, System, and Random Number libraries. Boost Software License - Version 1.0 - August 17th, 2003 . . . . Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following: The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 4. Licensee’s use of the LLVM third party component is subject to the following terms and conditions: ======================================================================================================== All LLVM after 8.0 are distributed under Apache-2.0 with LLVM-exception license, an OSI-approved license ======================================================================================================== Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "\[\]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright \[yyyy\] \[name of copyright owner\] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. \----------------------------------------- LLVM Exceptions to the Apache 2.0 License ----------------------------------------- As an exception, if, as a result of your compiling your source code, portions of this Software are embedded into an Object form of such source code, you may redistribute such embedded portions in such Object form without complying with the conditions of Sections 4(a), 4(b) and 4(d) of the License. In addition, if you combine or link compiled forms of this Software with software that is licensed under the GPLv2 ("Combined Software") and if a court of competent jurisdiction determines that the patent provision (Section 3), the indemnity provision (Section 9) or other Section of the License conflicts with the conditions of the GPLv2, you may retroactively and prospectively choose to deem waived or otherwise exclude such Section(s) of the License, but only in their entirety and only with respect to the Combined Software. \======================================================== Software from third parties included in the LLVM Project ======================================================== The LLVM Project contains third party software which is under different license terms. All such code will be identified clearly using at least one of two mechanisms: 1) It will be in a separate directory tree with its own \`LICENSE.txt\` or \`LICENSE\` file at the top containing the specific license and restrictions which apply to that software, or 2) It will contain specific license and restriction terms at the top of every file. \================================================================================================== LLVM releases prior to LLVM 8.0 was licensed under this University of Illinois Open Source License ================================================================================================== University of Illinois/NCSA Open Source License Copyright (c) 2003-2019 University of Illinois at Urbana-Champaign. All rights reserved. Developed by: LLVM Team University of Illinois at Urbana-Champaign http://llvm.org Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal with the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided with the distribution. \* Neither the names of the LLVM Team, University of Illinois at Urbana-Champaign, nor the names of its contributors may be used to endorse or promote products derived from this Software without specific prior written permission. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE SOFTWARE. 5. Licensee’s use of the PCRE third party component is subject to the following terms and conditions: \------------ PCRE LICENCE ------------ PCRE is a library of functions to support regular expressions whose syntax and semantics are as close as possible to those of the Perl 5 language. Release 8 of PCRE is distributed under the terms of the "BSD" licence, as specified below. The documentation for PCRE, supplied in the "doc" directory, is distributed under the same terms as the software itself. The basic library functions are written in C and are freestanding. Also included in the distribution is a set of C++ wrapper functions, and a just- in-time compiler that can be used to optimize pattern matching. These are both optional features that can be omitted when the library is built. THE BASIC LIBRARY FUNCTIONS --------------------------- Written by: Philip Hazel Email local part: ph10 Email domain: cam.ac.uk University of Cambridge Computing Service, Cambridge, England. Copyright (c) 1997-2012 University of Cambridge All rights reserved. PCRE JUST-IN-TIME COMPILATION SUPPORT ------------------------------------- Written by: Zoltan Herczeg Email local part: hzmester Emain domain: freemail.hu Copyright(c) 2010-2012 Zoltan Herczeg All rights reserved. STACK-LESS JUST-IN-TIME COMPILER -------------------------------- Written by: Zoltan Herczeg Email local part: hzmester Emain domain: freemail.hu Copyright(c) 2009-2012 Zoltan Herczeg All rights reserved. THE C++ WRAPPER FUNCTIONS ------------------------- Contributed by: Google Inc. Copyright (c) 2007-2012, Google Inc. All rights reserved. THE "BSD" LICENCE ----------------- Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of the University of Cambridge nor the name of Google Inc. nor the names of their contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 6. Some of the cuBLAS library routines were written by or derived from code written by Vasily Volkov and are subject to the Modified Berkeley Software Distribution License as follows: Copyright (c) 2007-2009, Regents of the University of California All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of the University of California, Berkeley nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 7. Some of the cuBLAS library routines were written by or derived from code written by Davide Barbieri and are subject to the Modified Berkeley Software Distribution License as follows: Copyright (c) 2008-2009 Davide Barbieri @ University of Rome Tor Vergata. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 8. Some of the cuBLAS library routines were derived from code developed by the University of Tennessee and are subject to the Modified Berkeley Software Distribution License as follows: Copyright (c) 2010 The University of Tennessee. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer listed in this license in the documentation and/or other materials provided with the distribution. \* Neither the name of the copyright holders nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 9. Some of the cuBLAS library routines were written by or derived from code written by Jonathan Hogg and are subject to the Modified Berkeley Software Distribution License as follows: Copyright (c) 2012, The Science and Technology Facilities Council (STFC). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of the STFC nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE STFC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 10. Some of the cuBLAS library routines were written by or derived from code written by Ahmad M. Abdelfattah, David Keyes, and Hatem Ltaief, and are subject to the Apache License, Version 2.0, as follows: \-- (C) Copyright 2013 King Abdullah University of Science and Technology Authors: Ahmad Abdelfattah (ahmad.ahmad@kaust.edu.sa) David Keyes (david.keyes@kaust.edu.sa) Hatem Ltaief (hatem.ltaief@kaust.edu.sa) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of the King Abdullah University of Science and Technology nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \`\`AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE 11. Some of the cuSPARSE library routines were written by or derived from code written by Li-Wen Chang and are subject to the NCSA Open Source License as follows: Copyright (c) 2012, University of Illinois. All rights reserved. Developed by: IMPACT Group, University of Illinois, http://impact.crhc.illinois.edu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal with the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided with the distribution. \* Neither the names of IMPACT Group, University of Illinois, nor the names of its contributors may be used to endorse or promote products derived from this Software without specific prior written permission. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE SOFTWARE. 12. Some of the cuRAND library routines were written by or derived from code written by Mutsuo Saito and Makoto Matsumoto and are subject to the following license: Copyright (c) 2009, 2010 Mutsuo Saito, Makoto Matsumoto and Hiroshima University. All rights reserved. Copyright (c) 2011 Mutsuo Saito, Makoto Matsumoto, Hiroshima University and University of Tokyo. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of the Hiroshima University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 13. Some of the cuRAND library routines were derived from code developed by D. E. Shaw Research and are subject to the following license: Copyright 2010-2011, D. E. Shaw Research. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions, and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name of D. E. Shaw Research nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 14. Some of the Math library routines were written by or derived from code developed by Norbert Juffa and are subject to the following license: Copyright (c) 2015-2017, Norbert Juffa All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 15. Licensee’s use of the lz4 third party component is subject to the following terms and conditions: Copyright (C) 2011-2013, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 16. The NPP library uses code from the Boost Math Toolkit, and is subject to the following license: Boost Software License - Version 1.0 - August 17th, 2003 . . . . Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following: The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17. Portions of the Nsight Eclipse Edition is subject to the following license: The Eclipse Foundation makes available all content in this plug-in ("Content"). Unless otherwise indicated below, the Content is provided to you under the terms and conditions of the Eclipse Public License Version 1.0 ("EPL"). A copy of the EPL is available at http:// www.eclipse.org/legal/epl-v10.html. For purposes of the EPL, "Program" will mean the Content. If you did not receive this Content directly from the Eclipse Foundation, the Content is being redistributed by another party ("Redistributor") and different terms and conditions may apply to your use of any object code in the Content. Check the Redistributor's license that was provided with the Content. If no such license exists, contact the Redistributor. Unless otherwise indicated below, the terms and conditions of the EPL still apply to any source code in the Content and such source code may be obtained at http://www.eclipse.org. 18. Some of the cuBLAS library routines uses code from OpenAI, which is subject to the following license: License URL https://github.com/openai/openai-gemm/blob/master/LICENSE License Text The MIT License Copyright (c) 2016 OpenAI (http://openai.com), 2016 Google Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 19. Licensee’s use of the Visual Studio Setup Configuration Samples is subject to the following license: The MIT License (MIT) Copyright (C) Microsoft Corporation. All rights reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20. Licensee’s use of `linmath.h` header for CPU functions for GL vector/matrix operations from [lunarG](https://www.lunarg.com/vulkan-sdk/) is subject to the [Apache License Version 2.0.](http://www.apache.org/licenses/) 21. The DX12-CUDA sample uses the `d3dx12.h` header, which is subject to the MIT [license .](https://opensource.org/licenses/MIT) 22. Components of the driver and compiler used for binary management, including nvFatBin, nvcc, and cuobjdump, use the Zstandard library which is subject to the following license: BSD License For Zstandard software Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: \* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. \* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. \* Neither the name Facebook, nor Meta, nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. --- # 1. Maxwell Compatibility — Maxwell Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Maxwell Compatibility * v12.8 | [PDF](../pdf/Maxwell_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Maxwell Compatibility Guide for CUDA Applications The guide to building CUDA applications for GPUs based on the NVIDIA Maxwell Architecture. 1\. Maxwell Compatibility[](#maxwell-compatibility "Permalink to this headline") ================================================================================== 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Maxwell Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on GPUs based on the NVIDIA® Maxwell Architecture. This document provides guidance to developers who are already familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Maxwell. 1.2. Application Compatibility on Maxwell[](#application-compatibility-on-maxwell "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------- The NVIDIA CUDA C++ compiler, `nvcc`, can be used to generate both architecture-specific _cubin_ files and forward-compatible _PTX_ versions of each kernel. Each cubin file targets a specific compute-capability version and is forward-compatible _only with GPU architectures of the same major version number_. For example, cubin files that target compute capability 3.0 are supported on all compute-capability 3.x (Kepler) devices but are _not_ supported on compute-capability 5.x (Maxwell) devices. For this reason, to ensure forward compatibility with GPU architectures introduced after the application has been released, it is recommended that all applications include PTX versions of their kernels. Note CUDA Runtime applications containing both cubin and PTX code for a given architecture will automatically use the cubin by default, keeping the PTX path strictly for forward-compatibility purposes. Applications that already include PTX versions of their kernels should work as-is on Maxwell-based GPUs. Applications that only support specific GPU architectures via cubin files, however, will need to be updated to provide Maxwell-compatible PTX or cubins. 1.3. Verifying Maxwell Compatibility for Existing Applications[](#verifying-maxwell-compatibility-for-existing-applications "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------- The first step is to check that Maxwell-compatible device code (at least PTX) is compiled in to the application. The following sections show how to accomplish this for applications built with different CUDA Toolkit versions. ### 1.3.1. Applications Using CUDA Toolkit 5.5 or Earlier[](#applications-using-cuda-toolkit-5-5-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 5.5 are compatible with Maxwell as long as they are built to include PTX versions of their kernels. To test that PTX JIT is working for your application, you can do the following: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch your application. When starting a CUDA application for the first time with the above environment flag, the CUDA driver will JIT-compile the PTX for each CUDA kernel that is used into native cubin code. If you set the environment variable above and then launch your program and it works properly, then you have successfully verified Maxwell compatibility. Note Be sure to unset the CUDA\_FORCE\_PTX\_JIT environment variable when you are done testing. ### 1.3.2. Applications Using CUDA Toolkit 6.0 or Later[](#applications-using-cuda-toolkit-6-0-or-later "Permalink to this headline") CUDA applications built using CUDA Toolkit 6.0 or Later[1](#fn1) are compatible with Maxwell as long as they are built to include kernels in either Maxwell-native cubin format (see [Building Applications with Maxwell Support](#building-applications-with-maxwell-support) ) or PTX format (see [Applications Using CUDA Toolkit 5.5 or Earlier](#verifying-maxwell-compatibility-using-cuda-5-5) ) or both. 1.4. Building Applications with Maxwell Support[](#building-applications-with-maxwell-support "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------- When a CUDA application launches a kernel, the CUDA Runtime determines the compute capability of each GPU in the system and uses this information to automatically find the best matching cubin or PTX version of the kernel that is available. If a cubin file supporting the architecture of the target GPU is available, it is used; otherwise, the CUDA Runtime will load the PTX and JIT-compile that PTX to the GPU’s native cubin format before launching it. If neither is available, then the kernel launch will fail. The method used to build your application with either native cubin or at least PTX support for Maxwell depend on the version of the CUDA Toolkit used. The main advantages of providing native cubins are as follows: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels compiled into the application must have native binaries at load time or else they will be built just-in-time from PTX, including kernels from all libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver will cache the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a given user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled code may be faster or of greater accuracy. ### 1.4.1. Applications Using CUDA Toolkit 5.5 or Earlier[](#building-maxwell-compatible-apps-using-cuda-5-5 "Permalink to this headline") The compilers included in CUDA Toolkit 5.5 or earlier generate cubin files native to earlier NVIDIA architectures such as Fermi and Kepler, but they _cannot_ generate cubin files native to the Maxwell architecture. To allow support for Maxwell and future architectures when using version 5.5 or earlier of the CUDA Toolkit, the compiler must generate a PTX version of each kernel. Below are compiler settings that could be used to build `mykernel.cu` to run on Fermi or Kepler devices natively and on Maxwell devices via PTX JIT. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one must be PTX to provide Maxwell compatibility. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_20,code=sm\_20 -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_35,code=compute\_35 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_20,code=sm\_20 -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_35,code=compute\_35 -O2 -o mykernel.o -c mykernel.cu Alternatively, you may be familiar with the simplified `nvcc` command-line option `-arch=sm_XX`, which is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. ### 1.4.2. Applications Using CUDA Toolkit 6.0 or Later[](#building-maxwell-compatible-apps-using-cuda-6-0 "Permalink to this headline") With version 6.0 of the CUDA Toolkit, `nvcc` can generate cubin files native to the first-generation Maxwell architecture (compute capability 5.0); CUDA Toolkit 6.5 and later further add native support for second-generation Maxwell devices (compute capability 5.2). When using CUDA Toolkit 6.x or Later, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_20,code=sm\_20 -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_52,code=compute\_52 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_20,code=sm\_20 -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_52,code=compute\_52 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. **Version 1.1** * Updated for second-generation Maxwell (compute capability 5.2). **Version 1.2** * Use CUDA C++ instead of CUDA C/C++. * Updated CUDA Toolkit reference to 6.0 and Later. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id4) Future CUDA Toolkit version might deprecate support for the Maxwell Architecture. --- # 1. Turing Compatibility — Turing Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Turing Compatibility * v12.8 | [PDF](../pdf/Turing_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Turing Compatibility Guide for CUDA Applications The guide to building CUDA applications for NVIDIA Turing GPUs. 1\. Turing Compatibility[](#turing-compatibility "Permalink to this headline") ================================================================================ 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Turing Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on GPUs based on the NVIDIA® Turing Architecture. This document provides guidance to developers who are already familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Turing. 1.2. Application Compatibility on Turing[](#application-compatibility-on-turing "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- The NVIDIA CUDA C++ compiler, `nvcc`, can be used to generate both architecture-specific _cubin_ files and forward-compatible _PTX_ versions of each kernel. Each cubin file targets a specific compute-capability version and is forward-compatible _only with GPU architectures of the same major version number_. For example, cubin files that target compute capability 3.0 are supported on all compute-capability 3.x (Kepler) devices but are _not_ supported on compute-capability 5.x (Maxwell) or 6.x (Pascal) devices. For this reason, to ensure forward compatibility with GPU architectures introduced after the application has been released, it is recommended that all applications include PTX versions of their kernels. Note CUDA Runtime applications containing both cubin and PTX code for a given architecture will automatically use the cubin by default, keeping the PTX path strictly for forward-compatibility purposes. Applications that already include PTX versions of their kernels should work as-is on Turing-based GPUs. Applications that only support specific GPU architectures via cubin files, however, will need to be updated to provide Turing-compatible PTX or cubins. 1.3. Compatibility between Volta and Turing[](#compatibility-between-volta-and-turing "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------- The Turing architecture is based on Volta’s Instruction Set Architecture _ISA_ 7.0, extending it with new instructions. As a consequence, any binary that runs on Volta will be able to run on Turing (forward compatibility), but a Turing binary will not be able to run on Volta. Please note that Volta kernels using more than 64KB of shared memory (via the explicit opt-in, see _CUDA C++ Programming Guide_) will not be able to launch on Turing, as they would exceed Turing’s shared memory capacity. Most applications compiled for Volta should run efficiently on Turing, except if the application uses heavily the Tensor Cores, or if recompiling would allow use of new Turing-specific instructions. Volta’s Tensor Core instructions can only reach half of the peak performance on Turing. Recompiling explicitly for Turing is thus recommended. 1.4. Verifying Turing Compatibility for Existing Applications[](#verifying-turing-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------- The first step is to check that Turing-compatible device code (at least PTX) is compiled into the application. The following sections show how to accomplish this for applications built with different CUDA Toolkit versions. ### 1.4.1. Applications Using CUDA Toolkit 8.0 or Earlier[](#applications-using-cuda-toolkit-8-0-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 8.0 are compatible with Turing as long as they are built to include PTX versions of their kernels. To test that PTX JIT is working for your application, you can do the following: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch your application. When starting a CUDA application for the first time with the above environment flag, the CUDA driver will JIT-compile the PTX for each CUDA kernel that is used into native cubin code. If you set the environment variable above and then launch your program and it works properly, then you have successfully verified Turing compatibility. Note Be sure to unset the CUDA\_FORCE\_PTX\_JIT environment variable when you are done testing. ### 1.4.2. Applications Using CUDA Toolkit 9.x[](#applications-using-cuda-toolkit-9-x "Permalink to this headline") CUDA applications built using CUDA Toolkit 9.x are compatible with Turing as long as they are built to include kernels in either Volta-native cubin format (see [Compatibility between Volta and Turing](#turing-volta-compatibility) ) or PTX format (see [Applications Using CUDA Toolkit 8.0 or Earlier](#verifying-turing-compatibility-using-cuda-8-0) ) or both. ### 1.4.3. Applications Using CUDA Toolkit 10.0[](#applications-using-cuda-toolkit-10-0 "Permalink to this headline") CUDA applications built using CUDA Toolkit 10.0 are compatible with Turing as long as they are built to include kernels in Volta-native or Turing-native cubin format (see [Compatibility between Volta and Turing](#turing-volta-compatibility) ), or PTX format (see [Applications Using CUDA Toolkit 8.0 or Earlier](#verifying-turing-compatibility-using-cuda-8-0) ), or both. 1.5. Building Applications with Turing Support[](#building-applications-with-turing-support "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------- When a CUDA application launches a kernel, the CUDA Runtime determines the compute capability of each GPU in the system and uses this information to automatically find the best matching cubin or PTX version of the kernel that is available. If a cubin file supporting the architecture of the target GPU is available, it is used; otherwise, the CUDA Runtime will load the PTX and JIT-compile that PTX to the GPU’s native cubin format before launching it. If neither is available, then the kernel launch will fail. The method used to build your application with either native cubin or at least PTX support for Turing depend on the version of the CUDA Toolkit used. The main advantages of providing native cubins are as follows: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels compiled into the application must have native binaries at load time or else they will be built just-in-time from PTX, including kernels from all libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver will cache the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a given user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled code may be faster or of greater accuracy. ### 1.5.1. Applications Using CUDA Toolkit 8.0 or Earlier[](#building-turing-compatible-apps-using-cuda-8-0 "Permalink to this headline") The compilers included in CUDA Toolkit 8.0 or earlier generate cubin files native to earlier NVIDIA architectures such as Maxwell and Pascal, but they _cannot_ generate cubin files native to Volta or Turing architecture. To allow support for Volta, Turing and future architectures when using version 8.0 or earlier of the CUDA Toolkit, the compiler must generate a PTX version of each kernel. Below are compiler settings that could be used to build `mykernel.cu` to run on Maxwell or Pascal devices natively and on Turing devices via PTX JIT. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one must be PTX to provide Turing compatibility. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 -O2 -o mykernel.o -c mykernel.cu Alternatively, you may be familiar with the simplified `nvcc` command-line option `-arch=sm_XX`, which is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. ### 1.5.2. Applications Using CUDA Toolkit 9.x[](#building-turing-compatible-apps-using-cuda-9-0 "Permalink to this headline") With versions 9.x of the CUDA Toolkit, `nvcc` can generate cubin files native to the Volta architecture (compute capability 7.0). When using CUDA Toolkit 9.x, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. Also, note that CUDA 9.0 removes support for compute capability 2.x (Fermi) devices. Any compute\_2x and sm\_2x flags need to be removed from your compiler commands. ### 1.5.3. Applications Using CUDA Toolkit 10.0[](#building-turing-compatible-apps-using-cuda-10-0 "Permalink to this headline") With version 10.0 of the CUDA Toolkit, `nvcc` can generate cubin files native to the Turing architecture (compute capability 7.5). When using CUDA Toolkit 10.0, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.5.4. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") The Volta and Turing architectures feature Independent Thread Scheduling among threads in a warp. If the developer made assumptions about warp-synchronicity, [1](#fn1) this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see Compute Capability 7.0 in the _CUDA C++ Programming Guide_ for details and corrective actions. To aid migration Volta and Turing developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -arch=compute\_60 -code=sm\_70 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. **Version 1.1** * Use CUDA C++ instead of CUDA C/C++ * Updated references to the CUDA C++ Programming Guide and CUDA C++ Best Practices Guide. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id9) _Warp-synchronous_ refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. --- # 1. Volta Compatibility — Volta Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Volta Compatibility * v12.8 | [PDF](../pdf/Volta_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Volta Compatibility Guide for CUDA Applications The guide to building CUDA applications for GPUs based on the NVIDIA Volta Architecture. 1\. Volta Compatibility[](#volta-compatibility "Permalink to this headline") ============================================================================== 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Volta Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on GPUs based on the NVIDIA® Volta Architecture. This document provides guidance to developers who are already familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Volta. 1.2. Application Compatibility on Volta[](#application-compatibility-on-volta "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- The NVIDIA CUDA C++ compiler, `nvcc`, can be used to generate both architecture-specific _cubin_ files and forward-compatible _PTX_ versions of each kernel. Each cubin file targets a specific compute-capability version and is forward-compatible _only with GPU architectures of the same major version number_. For example, cubin files that target compute capability 3.0 are supported on all compute-capability 3.x (Kepler) devices but are _not_ supported on compute-capability 5.x (Maxwell) or 6.x (Pascal) devices. For this reason, to ensure forward compatibility with GPU architectures introduced after the application has been released, it is recommended that all applications include PTX versions of their kernels. Note CUDA Runtime applications containing both cubin and PTX code for a given architecture will automatically use the cubin by default, keeping the PTX path strictly for forward-compatibility purposes. Applications that already include PTX versions of their kernels should work as-is on Volta-based GPUs. Applications that only support specific GPU architectures via cubin files, however, will need to be updated to provide Volta-compatible PTX or cubins. 1.3. Verifying Volta Compatibility for Existing Applications[](#verifying-volta-compatibility-for-existing-applications "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------------------------- The first step is to check that Volta-compatible device code (at least PTX) is compiled into the application. The following sections show how to accomplish this for applications built with different CUDA Toolkit versions. ### 1.3.1. Applications Using CUDA Toolkit 8.0 or Earlier[](#applications-using-cuda-toolkit-8-0-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 8.0 are compatible with Volta as long as they are built to include PTX versions of their kernels. To test that PTX JIT is working for your application, you can do the following: * Download and install the latest driver from [http://www.nvidia.com/drivers](http://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch your application. When starting a CUDA application for the first time with the above environment flag, the CUDA driver will JIT-compile the PTX for each CUDA kernel that is used into native cubin code. If you set the environment variable above and then launch your program and it works properly, then you have successfully verified Volta compatibility. Note Be sure to unset the CUDA\_FORCE\_PTX\_JIT environment variable when you are done testing. ### 1.3.2. Applications Using CUDA Toolkit 9.0[](#applications-using-cuda-toolkit-9-0 "Permalink to this headline") CUDA applications built using CUDA Toolkit 9.0 are compatible with Volta as long as they are built to include kernels in either Volta-native cubin format (see [Building Applications with Volta Support](#building-applications-with-volta-support) ) or PTX format (see [Applications Using CUDA Toolkit 8.0 or Earlier](#verifying-volta-compatibility-using-cuda-8-0) ) or both. 1.4. Building Applications with Volta Support[](#building-applications-with-volta-support "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- When a CUDA application launches a kernel, the CUDA Runtime determines the compute capability of each GPU in the system and uses this information to automatically find the best matching cubin or PTX version of the kernel that is available. If a cubin file supporting the architecture of the target GPU is available, it is used; otherwise, the CUDA Runtime will load the PTX and JIT-compile that PTX to the GPU’s native cubin format before launching it. If neither is available, then the kernel launch will fail. The method used to build your application with either native cubin or at least PTX support for Volta depend on the version of the CUDA Toolkit used. The main advantages of providing native cubins are as follows: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels compiled into the application must have native binaries at load time or else they will be built just-in-time from PTX, including kernels from all libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver will cache the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a given user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled code may be faster or of greater accuracy. ### 1.4.1. Applications Using CUDA Toolkit 8.0 or Earlier[](#building-volta-compatible-apps-using-cuda-8-0 "Permalink to this headline") The compilers included in CUDA Toolkit 8.0 or earlier generate cubin files native to earlier NVIDIA architectures such as Maxwell and Pascal, but they _cannot_ generate cubin files native to the Volta architecture. To allow support for Volta and future architectures when using version 8.0 or earlier of the CUDA Toolkit, the compiler must generate a PTX version of each kernel. Below are compiler settings that could be used to build `mykernel.cu` to run on Maxwell or Pascal devices natively and on Volta devices via PTX JIT. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one must be PTX to provide Volta compatibility. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 -O2 -o mykernel.o -c mykernel.cu Alternatively, you may be familiar with the simplified `nvcc` command-line option `-arch=sm_XX`, which is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. ### 1.4.2. Applications Using CUDA Toolkit 9.0[](#building-volta-compatible-apps-using-cuda-9-0 "Permalink to this headline") With version 9.0 of the CUDA Toolkit, `nvcc` can generate cubin files native to the Volta architecture (compute capability 7.0). When using CUDA Toolkit 9.0, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. Also, note that CUDA 9.0 removes support for compute capability 2.x (Fermi) devices. Any compute\_2x and sm\_2x flags need to be removed from your compiler commands. ### 1.4.3. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") The Volta architecture introduces Independent Thread Scheduling among threads in a warp. If the developer made assumptions about warp-synchronicity,[1](#fn1) this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see Compute Capability 7.0 in the CUDA C++ Programming Guide for details and corrective actions. To aid migration Volta developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -arch=compute\_60 -code=sm\_70 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. **Version 1.1** * Use CUDA C++ instead of CUDA C/C++ * Updated references to the CUDA C++ Programming Guide and CUDA C++ Best Practices Guide. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id8) _Warp-synchronous_ refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. --- # 1. NVIDIA Ampere GPU Architecture Compatibility — NVIDIA Ampere Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Ampere GPU Architecture Compatibility * v12.8 | [PDF](../pdf/NVIDIA_Ampere_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA Ampere GPU Architecture Compatibility Guide for CUDA Applications The guide to building CUDA applications for GPUs based on the NVIDIA Ampere GPU Architecture. 1\. NVIDIA Ampere GPU Architecture Compatibility[](#nvidia-ampere-gpu-architecture-compatibility "Permalink to this headline") ================================================================================================================================ 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, NVIDIA Ampere GPU Architecture Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on the NVIDIA® Ampere Architecture based GPUs. This document provides guidance to developers who are familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with the NVIDIA Ampere GPU architecture. 1.2. Application Compatibility on the NVIDIA Ampere GPU Architecture[](#application-compatibility-on-the-nvidia-ampere-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- A CUDA application binary (with one or more GPU kernels) can contain the compiled GPU code in two forms, binary _cubin_ objects and forward-compatible _PTX_ assembly for each kernel. Both cubin and PTX are generated for a certain target compute capability. A cubin generated for a certain compute capability is supported to run on any GPU with the same major revision and same or higher minor revision of compute capability. For example, a cubin generated for compute capability 7.0 is supported to run on a GPU with compute capability 7.5, however a cubin generated for compute capability 7.5 is _not_ supported to run on a GPU with compute capability 7.0, and a cubin generated with compute capability 7.x is _not_ supported to run on a GPU with compute capability 8.x. Kernel can also be compiled to a PTX form. At the application load time, PTX is compiled to cubin and the cubin is used for kernel execution. Unlike cubin, PTX is forward-compatible. Meaning PTX is supported to run on any GPU with compute capability higher than the compute capability assumed for generation of that PTX. For example, PTX code generated for compute capability 7.x is supported to run on compute capability 7.x or any higher revision (major or minor), including compute capability 8.x. Therefore although it is optional, **it is recommended that all applications should include PTX of the kernels to ensure forward-compatibility.** To read more about cubin and PTX compatibilities see Compilation with NVCC from the Programming Guide. When a CUDA application launches a kernel on a GPU, the CUDA Runtime determines the compute capability of the GPU in the system and uses this information to find the best matching cubin or PTX version of the kernel. If a cubin compatible with that GPU is present in the binary, the cubin is used as-is for execution. Otherwise, the CUDA Runtime first generates compatible cubin by JIT-compiling [1](#fn1) the PTX and then the cubin is used for the execution. If neither compatible cubin nor PTX is available, kernel launch results in a failure. Application binaries that include PTX version of kernels, should work as-is on the NVIDIA Ampere architecture based GPUs. In such cases, rebuilding the application is not required. However application binaries which do not include PTX (only include cubins), need to be rebuilt to run on the NVIDIA Ampere architecture based GPUs. To know more about building compatible applications read [Building Applications with the NVIDIA Ampere GPU Architecture Support](#building-applications-with-ampere-support) . 1.3. Verifying Ampere Compatibility for Existing Applications[](#verifying-ampere-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------- The first step towards making a CUDA application compatible with the NVIDIA Ampere GPU architecture is to check if the application binary already contains compatible GPU code (at least the PTX). The following sections explain how to accomplish this for an already built CUDA application. ### 1.3.1. Applications Built Using CUDA Toolkit 10.2 or Earlier[](#applications-built-using-cuda-toolkit-10-2-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ampere architecture based GPUs as long as they are built to include PTX versions of their kernels. This can be tested by forcing the PTX to JIT-compile at application load time with following the steps: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch the application. With `CUDA_FORCE_PTX_JIT=1`, GPU binary code embedded in an application binary is ignored. Instead PTX code for each kernel is JIT-compiled to produce GPU binary code. An application fails to execute if it does not include PTX. This means the application is not compatible with the NVIDIA Ampere GPU architecture and needs to be rebuilt for compatibility. On the other hand, if the application works properly with this environment variable set, then the application is compatible with the NVIDIA Ampere GPU architecture. Note Be sure to unset the `CUDA_FORCE_PTX_JIT` environment variable after testing is done. ### 1.3.2. Applications Built Using CUDA Toolkit 11.0[](#applications-built-using-cuda-toolkit-11-0 "Permalink to this headline") CUDA applications built using CUDA Toolkit 11.0 are compatible with the NVIDIA Ampere GPU architecture as long as they are built to include kernels in native cubin (compute capability 8.0) or PTX form or both. 1.4. Building Applications with the NVIDIA Ampere GPU Architecture Support[](#building-applications-with-the-nvidia-ampere-gpu-architecture-support "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Depending on the version of the CUDA Toolkit used for building the application, it can be built to include PTX and/or native cubin for the NVIDIA Ampere GPU architecture. Although it is enough to just include PTX, including native cubin also has the following advantages: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels which do not have native cubins are JIT-compiled from PTX, including kernels from all the libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver caches the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled cubins may be faster or of greater accuracy. ### 1.4.1. Building Applications Using CUDA Toolkit 10.x or Earlier[](#building-applications-using-cuda-toolkit-10-x-or-earlier "Permalink to this headline") The `nvcc` compiler included with versions 10.x (10.0, 10.1 and 10.2) of the CUDA Toolkit can generate cubins native to the Volta and Turing architectures (compute capability 7.x). When using CUDA Toolkit 10.x, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 -O2 -o mykernel.o -c mykernel.cu Alternatively, the simplified `nvcc` command-line option `-arch=sm_XX` can be used. It is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target binary by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. For CUDA toolkits prior to 10.0, one or more of the `-gencode` options will need to be removed according to the architectures supported by the specific toolkit version (for example, CUDA toolkit 9.x supports architectures up to \_60 and \_61). The final `-gencode` to generate PTX would also need to be update – for further information and examples see the documentation for the specific CUDA toolkit version. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.4.2. Building Applications Using CUDA Toolkit 11.0[](#building-applications-using-cuda-toolkit-11-0 "Permalink to this headline") With versions 11.0 of the CUDA Toolkit, `nvcc` can generate cubin native to the NVIDIA Ampere GPU architecture (compute capability 8.0). When using CUDA Toolkit 11.0, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_80,code=compute\_80 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_80,code=compute\_80 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.4.3. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") NVIDIA GPUs since Volta architecture have _Independent Thread Scheduling_ among threads in a warp. If the developer made assumptions about warp-synchronicity[2](#fn2) , this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see Compute Capability 7.0 in the Programming Guide for details and corrective actions. To aid migration to the NVIDIA Ampere GPU architecture, developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -gencode=arch=compute\_60,code=sm\_80 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id2) Just-in-time compilation [2](#id4) Warp-synchronous refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. --- # 1. NVIDIA Ada GPU Architecture Compatibility — Ada Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Ada GPU Architecture Compatibility * v12.8 | [PDF](../pdf/Ada_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA Ada GPU Architecture Compatibility Guide for CUDA Applications The guide to building CUDA applications for NVIDIA Ada GPUs. 1\. NVIDIA Ada GPU Architecture Compatibility[](#nvidia-ada-gpu-architecture-compatibility "Permalink to this headline") ========================================================================================================================== 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, _NVIDIA Ada GPU Architecture Compatibility Guide for CUDA Applications_, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on the NVIDIA® Ada Architecture based GPUs. This document provides guidance to developers who are familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with the NVIDIA Ada GPU architecture. 1.2. Application Compatibility on the NVIDIA Ada GPU Architecture[](#application-compatibility-on-the-nvidia-ada-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- A CUDA application binary (with one or more GPU kernels) can contain the compiled GPU code in two forms, binary _cubin_ objects and forward-compatible _PTX_ assembly for each kernel. Both cubin and PTX are generated for a certain target compute capability. A cubin generated for a certain compute capability is supported to run on any GPU with the same major revision and same or higher minor revision of compute capability. For example, a cubin generated for compute capability 8.6 is supported to run on a GPU with compute capability 8.9; however, a cubin generated for compute capability 8.9 is _not_ supported to run on a GPU with compute capability 8.6, and a cubin generated with compute capability 8.x is _not_ supported to run on a GPU with compute capability 9.0. Kernels can also be compiled to a PTX form. At the application load time, PTX is compiled to cubin and the cubin is used for kernel execution. Unlike cubin, PTX is forward-compatible. Meaning PTX is supported to run on any GPU with compute capability higher than the compute capability assumed for generation of that PTX. For example, PTX code generated for compute capability 8.x is supported to run on compute capability 8.x or any higher revision (major or minor), including compute capability 9.x. Therefore, although it is optional, **it is recommended that all applications should include PTX of the kernels to ensure forward-compatibility.** To read more about cubin and PTX compatibilities see [Compilation with NVCC](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-with-nvcc) from the _CUDA C++ Programming Guide_. When a CUDA application launches a kernel on a GPU, the CUDA Runtime determines the compute capability of the GPU in the system and uses this information to find the best matching cubin or PTX version of the kernel. If a cubin compatible with that GPU is present in the binary, the cubin is used as-is for execution. Otherwise, the CUDA Runtime first generates compatible cubin by JIT-compiling [1](#fn1) the PTX and then the cubin is used for the execution. If neither compatible cubin nor PTX is available, kernel launch results in a failure. Application binaries that include PTX version of kernels should work as-is on the NVIDIA Ada architecture based GPUs. In such cases, rebuilding the application is not required. However, application binaries that do not include PTX (only include cubins) need to be rebuilt to run on the NVIDIA Ada architecture based GPUs. To know more about building compatible applications, read [Building Applications with the NVIDIA Ada GPU Architecture Support](#building-applications-with-the-nvidia-ada-gpu-architecture-support) . 1.3. Compatibility between Ampere and Ada[](#compatibility-between-ampere-and-ada "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------- The NVIDIA Ada architecture is based on Ampere’s Instruction Set Architecture _ISA_ 8.0, extending it with new instructions. As a consequence, any binary that runs on Ampere will be able to run on Ada (forward compatibility), but an Ada binary will not be able to run on Ampere. 1.4. Verifying Ada Compatibility for Existing Applications[](#verifying-ada-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------- The first step towards making a CUDA application compatible with the NVIDIA Ada GPU architecture is to check if the application binary already contains compatible GPU code (at least the PTX). The following sections explain how to accomplish this for an already built CUDA application. ### 1.4.1. Applications Built Using CUDA Toolkit 10.2 or Earlier[](#applications-built-using-cuda-toolkit-10-2-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ada architecture based GPUs as long as they are built to include PTX versions of their kernels. This can be tested by forcing the PTX to JIT-compile at application load time with following the steps: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch the application. With `CUDA_FORCE_PTX_JIT=1`, GPU binary code embedded in an application binary is ignored. Instead PTX code for each kernel is JIT-compiled to produce GPU binary code. An application fails to execute if it does not include PTX. This means the application is not compatible with the NVIDIA Ada GPU architecture and needs to be rebuilt for compatibility. On the other hand, if the application works properly with this environment variable set, then the application is compatible with the NVIDIA Ada GPU architecture. Note Be sure to unset the `CUDA_FORCE_PTX_JIT` environment variable after testing is done. ### 1.4.2. Applications Built Using CUDA Toolkit 11.0 through 11.7[](#applications-built-using-cuda-toolkit-11-0-through-11-7 "Permalink to this headline") CUDA applications built using CUDA Toolkit 11.0 through 11.7 are compatible with the NVIDIA Ada GPU architecture as long as they are built to include kernels in Ampere-native cubin (see [Compatibility between Ampere and Ada](#compatibility-between-ampere-and-ada) ) or PTX format (see [Applications Built Using CUDA Toolkit 10.2 or Earlier](#applications-built-using-cuda-toolkit-10-2-or-earlier) ), or both. ### 1.4.3. Applications Built Using CUDA Toolkit 11.8[](#applications-built-using-cuda-toolkit-11-8 "Permalink to this headline") CUDA applications built using CUDA Toolkit 11.8 are compatible with the NVIDIA Ada GPU architecture as long as they are built to include kernels in Ampere-native or Ada-native cubin (see [Compatibility between Ampere and Ada](#compatibility-between-ampere-and-ada) ), or PTX format (see [Applications Built Using CUDA Toolkit 10.2 or Earlier](#applications-built-using-cuda-toolkit-10-2-or-earlier) ), or both. 1.5. Building Applications with the NVIDIA Ada GPU Architecture Support[](#building-applications-with-the-nvidia-ada-gpu-architecture-support "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Depending on the version of the CUDA Toolkit used for building the application, it can be built to include PTX and/or native cubin for the NVIDIA Ada GPU architecture. Although it is sufficient to just include PTX, including native cubin also has the following advantages: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels that do not have native cubins are JIT-compiled from PTX, including kernels from all the libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver caches the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled cubins may be faster or of greater accuracy. ### 1.5.1. Building Applications Using CUDA Toolkit 10.x or Earlier[](#building-applications-using-cuda-toolkit-10-x-or-earlier "Permalink to this headline") The `nvcc` compiler included with versions 10.x (10.0, 10.1 and 10.2) of the CUDA Toolkit can generate cubins native to the Volta and Turing architectures (compute capability 7.x). When using CUDA Toolkit 10.x, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=compute\_75 -O2 -o mykernel.o -c mykernel.cu Alternatively, the simplified `nvcc` command-line option `-arch=sm_XX` can be used. It is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target binary by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. For CUDA toolkits prior to 10.0, one or more of the `-gencode` options will need to be removed according to the architectures supported by the specific toolkit version (for example, CUDA toolkit 9.x supports architectures up to \_60 and \_61). The final `-gencode` to generate PTX would also need to be updated. For further information and examples, see the documentation for the specific CUDA toolkit version. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX, or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.5.2. Building Applications Using CUDA Toolkit 11.0 through 11.7[](#building-applications-using-cuda-toolkit-11-0-through-11-7 "Permalink to this headline") The `nvcc` compiler included with versions 11.0 through 11.7 of the CUDA Toolkit can generate cubins native to the Ampere architecture (compute capability 8.0 and 8.6). When using CUDA Toolkit 11.0 through 11.7, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_86,code=sm\_86 -gencode=arch=compute\_86,code=compute\_86 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_86,code=sm\_86 -gencode=arch=compute\_86,code=compute\_86 -O2 -o mykernel.o -c mykernel.cu Alternatively, the simplified `nvcc` command-line option `-arch=sm_XX` can be used. It is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target binary by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. For CUDA toolkits prior to 11.0, one or more of the `-gencode` options need to be removed according to the architectures supported by the specific toolkit version (for example, CUDA toolkit 10.x supports architectures up to \_72 and \_75). The final `-gencode` to generate PTX also needs to be updated. For further information and examples, see the documentation for the specific CUDA toolkit version. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX, or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.5.3. Building Applications Using CUDA Toolkit 11.8[](#building-applications-using-cuda-toolkit-11-8 "Permalink to this headline") With version 11.8 of the CUDA Toolkit, `nvcc` can generate cubin native to the NVIDIA Ada GPU architecture (compute capability 8.9). When using CUDA Toolkit 11.8, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_86,code=sm\_86 -gencode=arch=compute\_89,code=sm\_89 -gencode=arch=compute\_89,code=compute\_89 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_86,code=sm\_86 -gencode=arch=compute\_89,code=sm\_89 -gencode=arch=compute\_89,code=compute\_89 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX, or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. ### 1.5.4. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") NVIDIA GPUs since Volta architecture have _Independent Thread Scheduling_ among threads in a warp. If the developer made assumptions about warp-synchronicity[2](#fn2) , this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see [Compute Capability 7.x](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability-7-x) in the _CUDA C++ Programming Guide_ for details and corrective actions. To aid migration to the NVIDIA Ada GPU architecture, developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -gencode=arch=compute\_60,code=sm\_89 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. [1](#id2) Just-in-time compilation [2](#id7) Warp-synchronous refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Pascal Tuning Guide — Pascal Tuning Guide 12.8 documentation * [](../index.html) » * 1\. Pascal Tuning Guide * v12.8 | [PDF](../pdf/Pascal_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for Pascal The programming guide to tuning CUDA Applications for GPUs based on the NVIDIA Pascal Architecture. 1\. Pascal Tuning Guide[](#pascal-tuning-guide "Permalink to this headline") ============================================================================== 1.1. NVIDIA Pascal Compute Architecture[](#nvidia-pascal-compute-architecture "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- Pascal retains and extends the same CUDA programming model provided by previous NVIDIA architectures such as Maxwell, and applications that follow the best practices for those architectures should typically see speedups on the Pascal architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging Pascal architectural features.[1](#fn1) Pascal architecture comprises two major variants: GP100 and GP104.[2](#fn2) A detailed overview of the major improvements in GP100 and GP104 over earlier NVIDIA architectures are described in a pair of white papers entitled [NVIDIA Tesla P100: The Most Advanced Datacenter Accelerator Ever Built](http://images.nvidia.com/content/pdf/tesla/whitepaper/pascal-architecture-whitepaper.pdf) for GP100 and [NVIDIA GeForce GTX 1080: Gaming Perfected](http://international.download.nvidia.com/geforce-com/international/pdfs/GeForce_GTX_1080_Whitepaper_FINAL.pdf) for GP104. For further details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) . Some of the Pascal features described in this guide are specific to either GP100 or GP104, as noted; if not specified, features apply to both Pascal variants. 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) and the [CUDA C++ Best Practices Guide](http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code, * Minimize data transfers between the host and the device, * Adjust kernel launch configuration to maximize device utilization, * Ensure global memory accesses are coalesced, * Minimize redundant accesses to global memory whenever possible, * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Pascal Compatibility Guide for CUDA Applications](http://docs.nvidia.com/cuda/pascal-compatibility-guide/index.html) to ensure that your application is compiled in a way that is compatible with Pascal. 1.4. Pascal Tuning[](#pascal-tuning "Permalink to this headline") ------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The Pascal Streaming Multiprocessor (SM) is in many respects similar to that of Maxwell. Pascal further improves the already excellent power efficiency provided by the Maxwell architecture through both an improved 16-nm FinFET manufacturing process and various architectural modifications. #### 1.4.1.1. Instruction Scheduling[](#instruction-scheduling "Permalink to this headline") Like Maxwell, Pascal employs a power-of-two number of CUDA Cores per partition. This simplifies scheduling, since each of the SM’s warp schedulers issue to a dedicated set of CUDA Cores equal to the warp width (32). Each warp scheduler still has the flexibility to dual-issue (such as issuing a math operation to a CUDA Core in the same cycle as a memory operation to a load/store unit), but single-issue is now sufficient to fully utilize all CUDA Cores. GP100 and GP104 designs incorporate different numbers of CUDA Cores per SM. Like Maxwell, each GP104 SM provides four warp schedulers managing a total of 128 single-precision (FP32) and four double-precision (FP64) cores. A GP104 processor provides up to 20 SMs, and the similar GP102 design provides up to 30 SMs. By contrast GP100 provides smaller but more numerous SMs. Each GP100 provides up to 60 SMs.[3](#fn3) Each SM contains two warp schedulers managing a total of 64 FP32 and 32 FP64 cores. The resulting 2:1 ratio of FP32 to FP64 cores aligns well with GP100’s new datapath configuration, allowing Pascal to process FP64 workloads more efficiently than Kepler GK210, the previous NVIDIA architecture to emphasize FP64 performance. #### 1.4.1.2. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM remains the same as in Maxwell (i.e., 64), and other [factors influencing warp occupancy](http://developer.download.nvidia.com/compute/cuda/CUDA_Occupancy_calculator.xls) remain similar as well: * The register file size (64k 32-bit registers) is the same as that of Maxwell. * The maximum registers per thread, 255, matches that of Maxwell. As with previous architectures, experimentation should be used to determine the optimum balance of register spilling vs. occupancy, however. * The maximum number of thread blocks per SM is 32, the same as Maxwell. * Shared memory capacity per SM is 64KB for GP100 and 96KB for GP104. For comparison, Maxwell provided 96KB and up to 112KB of shared memory, respectively. But each GP100 SM contains fewer CUDA Cores, so the shared memory available per core actually increases on GP100. The maximum shared memory per block remains limited at 48KB as with prior architectures (see [Shared Memory Capacity](#shared-memory-capacity) ). As such, developers can expect similar occupancy as on Maxwell without changes to their application. As a result of scheduling improvements relative to Kepler, warp occupancy requirements (i.e., available parallelism) needed for maximum device utilization are generally reduced. ### 1.4.2. New Arithmetic Primitives[](#new-arithmetic-primitives "Permalink to this headline") #### 1.4.2.1. FP16 Arithmetic Support[](#fp16-arithmetic-support "Permalink to this headline") Pascal provides improved FP16 support for applications, like deep learning, that are tolerant of low floating-point precision. The `half` type is used to represent FP16 values on the device. As with Maxwell, FP16 storage can be used to reduce the required memory footprint and bandwidth compared to FP32 or FP64 storage. Pascal also adds support for native FP16 instructions. Peak FP16 throughput is attained by using a paired operation to perform two FP16 instructions per core simultaneously. To be eligible for the paired operation the operands must be stored in a `half2` vector type. GP100 and GP104 provide different FP16 throughputs. GP100, designed with training deep neural networks in mind, provides FP16 throughput up to 2x that of FP32 arithmetic. On GP104, FP16 throughput is lower, 1/64th that of FP32. However, compensating for reduced FP16 throughput, GP104 provides additional high-throughput INT8 support not available in GP100. #### 1.4.2.2. INT8 Dot Product[](#int8-dot-product "Permalink to this headline") GP104 provides specialized instructions for two-way and four-way integer dot products. These are well suited for accelerating Deep Learning inference workloads. The `__dp4a` intrinsic computes a dot product of four 8-bit integers with accumulation into a 32-bit integer. Similarly, `__dp2a` performs a two-element dot product between two 16-bit integers in one vector, and two 8-bit integers in another with accumulation into a 32-bit integer. Both instructions offer a throughput equal to that of FP32 arithmetic. ### 1.4.3. Memory Throughput[](#memory-throughput "Permalink to this headline") #### 1.4.3.1. High Bandwidth Memory 2 DRAM[](#high-bandwidth-memory-2-dram "Permalink to this headline") GP100 uses High Bandwidth Memory 2 (HBM2) for its DRAM. HBM2 memories are stacked on a single silicon package along with the GPU die. This allows much wider interfaces at similar power compared to traditional GDDR technology. GP100 is linked to up to four stacks of HBM2 and uses two 512-bit memory controllers for each stack. The effective width of the memory bus is then 4096 bits, a significant increase over the 384 bits in GM200. This allows a tremendous boost in peak bandwidth even at reduced memory clocks. Thus, the GP100 equipped Tesla P100 has a peak bandwidth of 732 GB/s with a modest 715 MHz memory clock. DRAM access latencies remain similar to those observed on Maxwell. In order to hide DRAM latencies at full HBM2 bandwidth, more memory accesses must be kept in flight compared to GPUs equipped with traditional GDDR5. Helpfully, the large complement of SMs in GP100 will typically boost the number of concurrent threads (and thus reads-in-flight) compared to previous architectures. Resource constrained kernels that are limited to low occupancy may benefit from increasing the number of concurrent memory accesses per thread. The GP100 GPU’s register files, shared memories, L1 and L2 caches, and DRAM are all protected by Single-Error Correct Double-Error Detect (SECDED) ECC code. When enabling ECC support on a Kepler GK210, the available DRAM would be reduced by 6.25% to allow for the storage of ECC bits. Fetching ECC bits for each memory transaction also reduced the effective bandwidth by approximately 20% compared to the same GPU with ECC disabled. HBM2 memories, on the other hand, provide dedicated ECC resources, allowing overhead-free ECC protection.[4](#fn4) #### 1.4.3.2. Unified L1/Texture Cache[](#unified-l1-texture-cache "Permalink to this headline") Like Maxwell, Pascal combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. By default, GP100 caches global loads in the L1/Texture cache. In contrast, GP104 follows Maxwell in caching global loads in L2 only, unless using the _LDG_ read-only data cache mechanism. As with previous architectures, GP104 allows the developer to opt-in to caching all global loads in the unified L1/Texture cache by passing the `-Xptxas -dlcm=ca` flag to `nvcc` at compile time. Kepler serviced loads at a granularity of 128B when L1 caching of global loads was enabled and 32B otherwise. On Pascal the data access unit is 32B regardless of whether global loads are cached in L1. So it is no longer necessary to turn off L1 caching in order to reduce wasted global memory transactions associated with uncoalesced accesses. Unlike Maxwell, Pascal caches thread-local memory in the L1 cache. This can mitigate the cost of register spills compared to Maxwell. The balance of occupancy versus spilling should therefore be re-evaluated to ensure best performance. Two new device attributes were added in CUDA Toolkit 6.0: `globalL1CacheSupported` and `localL1CacheSupported`. Developers who wish to have separately-tuned paths for various architecture generations can use these fields to simplify the path selection process. Note Enabling caching of globals in GP104 can affect occupancy. If per-thread-block SM resource usage would result in zero occupancy with caching enabled, the CUDA driver will override the caching selection to allow the kernel launch to succeed. This situation is reported by the profiler. ### 1.4.4. Atomic Memory Operations[](#atomic-memory-operations "Permalink to this headline") Like Maxwell, Pascal provides native _shared_ memory atomic operations for 32-bit integer arithmetic, along with native 32 or 64-bit compare-and-swap (CAS). Developers coming from Kepler, where shared memory atomics were implemented in software using a lock/update/unlock sequence, should see a large performance improvement particularly for heavily contended shared-memory atomics. Pascal also extends atomic addition in global memory to function on FP64 data. The `atomicAdd()` function in CUDA has thus been generalized to support 32 and 64-bit integer and floating-point types. The rounding mode for all floating-point atomic operations is round-to-nearest-even in Pascal. As in previous generations FP32 `atomicAdd()` flushes denormalized values to zero. For GP100 atomic operations may target the memories of peer GPUs connected through NVLink. Peer-to-peer atomics over NVLink use the same API as atomics targeting global memory. GPUs connected via PCIE do not support this feature. Pascal GPUs provide support system-wide atomic operations targeting _migratable allocations_[5](#fn5) If system-wide atomic visibility is desired, operations targeting migratable memory must specify a system scope by using the `atomic[Op]_system()` intrinsics[6](#fn6) . Using the device-scope atomics (e.g. `atomicAdd()`) on migratable memory remains valid, but enforces atomic visibility only within the local GPU. Note Given the potential for incorrect usage of atomic scopes, it is recommended that applications use compute-sanitizer to detect and eliminate errors. As implemented for Pascal, system-wide atomics are intended to allow developers to experiment with enhanced memory models. They are implemented in software and some care is required to achieve good performance. When an atomic targets a migratable address backed by a remote memory space, the local processor page-faults so that the kernel can migrate the appropriate memory page to local memory. Then the usual hardware instructions are used to execute the atomic. Since the page is now locally resident, subsequent atomics from the same processor will not result in additional page-faults. However, atomic updates from different processors can incur frequent page-faults. ### 1.4.5. Shared Memory[](#shared-memory "Permalink to this headline") #### 1.4.5.1. Shared Memory Capacity[](#shared-memory-capacity "Permalink to this headline") For Kepler, shared memory and the L1 cache shared the same on-chip storage. Maxwell and Pascal, by contrast, provide dedicated space to the shared memory of each SM, since the functionality of the L1 and texture caches have been merged. This increases the shared memory space available per SM as compared to Kepler: GP100 offers 64 KB shared memory per SM, and GP104 provides 96 KB per SM. This presents several benefits to application developers: * Algorithms with significant shared memory capacity requirements (e.g., radix sort) see an automatic 33% to 100% boost in capacity per SM on top of the aggregate boost from higher SM count. * Applications no longer need to select a preference of the L1/shared split for optimal performance. Note Thread-blocks remain limited to 48 KB of shared memory. For maximum flexibility, NVIDIA recommends that applications use at most 32 KB of shared memory in any one thread block. This would, for example, allow at least two thread blocks to fit per GP100 SM, or 3 thread blocks per GP104 SM. #### 1.4.5.2. Shared Memory Bandwidth[](#shared-memory-bandwidth "Permalink to this headline") Kepler provided an optional 8-byte shared memory banking mode, which had the potential to increase shared memory bandwidth per SM for shared memory accesses of 8 or 16 bytes. However, applications could only benefit from this when storing these larger elements in shared memory (i.e., integers and fp32 values saw no benefit), and only when the developer explicitly opted in to the 8-byte bank mode via the API. To simplify this, Pascal follows Maxwell in returning to fixed four-byte banks. This allows all applications using shared memory to benefit from the higher bandwidth, without specifying any particular preference via the API. ### 1.4.6. Inter-GPU Communication[](#inter-gpu-communication "Permalink to this headline") #### 1.4.6.1. NVLink Interconnect[](#nvlink-interconnect "Permalink to this headline") NVLink is NVIDIA’s new high-speed data interconnect. NVLink can be used to significantly increase performance for both GPU-to-GPU communication and for GPU access to system memory. GP100 supports up to four NVLink connections with each connection carrying up to 40 GB/s of bi-directional bandwidth. NVLink operates transparently within the existing CUDA model. Transfers between NVLink-connected endpoints are automatically routed through NVLink, rather than PCIe. The `cudaDeviceEnablePeerAccess()` API call remains necessary to enable direct transfers (over either PCIe or NVLink) between GPUs. The `cudaDeviceCanAccessPeer()` can be used to determine if peer access is possible between any pair of GPUs. #### 1.4.6.2. GPUDirect RDMA Bandwidth[](#gpudirect-rdma-bandwidth "Permalink to this headline") GPUDirect RDMA allows third party devices such as network interface cards (NICs) to directly access GPU memory. This eliminates unnecessary copy buffers, lowers CPU overhead, and significantly decreases the latency of MPI send/receive messages from/to GPU memory. Pascal doubles the delivered RDMA bandwidth when reading data from the source GPU memory and writing to the target NIC memory over PCIe. ### 1.4.7. Compute Preemption[](#compute-preemption "Permalink to this headline") Compute Preemption is a new feature specific to GP100. Compute Preemption allows compute tasks running on the GPU to be interrupted at instruction-level granularity. The execution context (registers, shared memory, etc.) are swapped to GPU DRAM so that another application can be swapped in and run. Compute preemption offers two key advantages for developers: * Long-running kernels no longer need to be broken up into small timeslices to avoid an unresponsive graphical user interface or kernel timeouts when a GPU is used simultaneously for compute and graphics. * Interactive kernel debugging on a single-GPU system is now possible. ### 1.4.8. Unified Memory Improvements[](#unified-memory-improvements "Permalink to this headline") Pascal offers new hardware capabilities to extend Unified Memory (UM) support. An extended 49-bit virtual addressing space allows Pascal GPUs to address the full 48-bit virtual address space of modern CPUs as well as the memories of all GPUs in the system through a single virtual address space, not limited by the physical memory sizes of any one processor. Pascal GPUs also support memory page faulting. Page faulting allows applications to access the same managed memory allocations from both host and device without explicit synchronization. It also removes the need for the CUDA runtime to pre-synchronize _all_ managed memory allocations before each kernel launch. Instead, when a kernel accesses a non-resident memory page, it faults, and the page can be migrated to the GPU memory on-demand, or mapped into the GPU address space for access over PCIe/NVLink interfaces. These features boost performance on Pascal for many typical UM workloads. In cases where the UM heuristics prove suboptimal, further tuning is possible through a set of migration hints that can be added to the source code. On supporting operating system platforms, any memory allocated with the default OS allocator (for example, malloc or new) can be accessed from both GPU and CPU code using the same pointer. In fact, all system virtual memory can be accessed from the GPU. On such systems, there is no need to explicitly allocate managed memory using `cudaMallocManaged()`. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release **Version 1.1** * Updated references to the CUDA C++ Programming Guide and CUDA C++ Best Practices Guide. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id2) Throughout this guide, _Kepler_ refers to devices of compute capability 3.x, _Maxwell_ refers to devices of compute capability 5.x, and _Pascal_ refers to device of compute capability 6.x. [2](#id3) The specific compute capabilities of GP100 and GP104 are 6.0 and 6.1, respectively. The GP102 architecture is similar to GP104. [3](#id7) The Tesla P100 has 56 SMs enabled. [4](#id9) As an exception, scattered writes to HBM2 see some overhead from ECC but much less than the overhead with similar access patterns on ECC-protected GDDR5 memory. [5](#id10) Migratable, or _Unified Memory (UM)_, allocations are made with `cudaMallocManaged()` or, for systems with Heterogeneous Memory Management (HMM) support, `malloc()`. [6](#id11) Here \[Op\] would be one of `Add`, `CAS`, etc. --- # 1. Hopper Architecture Compatibility — Hopper Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Hopper Architecture Compatibility * v12.8 | [PDF](../pdf/Hopper_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Hopper Compatibility Guide for CUDA Applications The guide to building CUDA applications for Hopper GPUs 1\. Hopper Architecture Compatibility[](#hopper-architecture-compatibility "Permalink to this headline") ========================================================================================================== 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Hopper Architecture Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on the NVIDIA® Hopper architecture based GPUs. This document provides guidance to developers who are familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Hopper architecture. 1.2. Application Compatibility on Hopper Architecture[](#application-compatibility-on-hopper-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------- A CUDA application binary (with one or more GPU kernels) can contain the compiled GPU code in two forms, binary cubin objects and forward-compatible PTX assembly for each kernel. Both cubin and PTX are generated for a certain target compute capability. A cubin generated for a certain compute capability is supported to run on any GPU with the same major revision and same or higher minor revision of compute capability. For example, a cubin generated for compute capability 8.0 is supported to run on a GPU with compute capability 8.6, however a cubin generated for compute capability 8.6 is _not_ supported to run on a GPU with compute capability 8.0, and a cubin generated with compute capability 8.x is _not_ supported to run on a GPU with compute capability 9.0. Kernel can also be compiled to a PTX form. At the application load time, PTX is compiled to cubin and the cubin is used for kernel execution. Unlike cubin, PTX is forward-compatible. Meaning PTX is supported to run on any GPU with compute capability higher than the compute capability assumed for generation of that PTX. For example, PTX code generated for compute capability 8.x is supported to run on compute capability 8.x or any higher revision (major or minor), including compute capability 9.0. Therefore although it is optional, **it is recommended that all applications should include PTX of the kernels to ensure forward-compatibility.** To read more about cubin and PTX compatibilities see [Compilation with NVCC](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-with-nvcc) from the _CUDA C++ Programming Guide_. When a CUDA application launches a kernel on a GPU, the CUDA Runtime determines the compute capability of the GPU in the system and uses this information to find the best matching cubin or PTX version of the kernel. If a cubin compatible with that GPU is present in the binary, the cubin is used as-is for execution. Otherwise, the CUDA Runtime first generates compatible cubin by JIT-compiling [1](#fn1) the PTX and then the cubin is used for the execution. If neither compatible cubin nor PTX is available, kernel launch results in a failure. Application binaries that include PTX version of kernels, should work as-is on the Hopper GPUs. In such cases, rebuilding the application is not required. However application binaries which do not include PTX (only include cubins), need to be rebuilt to run on the Hopper GPUs. To know more about building compatible applications read [Building Applications with Hopper Architecture Support](#building-applications-with-hopper-support) . Application binaries that include PTX version of kernels with architecture conditional features using `sm_90a` or `compute_90a` in order to take full advantage of Hopper GPU architecture, are not forward or backward compatible. 1.3. Verifying Hopper Compatibility for Existing Applications[](#verifying-hopper-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------- The first step towards making a CUDA application compatible with Hopper architecture is to check if the application binary already contains compatible GPU code (at least the PTX). The following sections explain how to accomplish this for an already built CUDA application. ### 1.3.1. Applications Built Using CUDA Toolkit 11.7 or Earlier[](#applications-built-using-cuda-toolkit-11-7-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 11.7 are compatible with Hopper GPUs as long as they are built to include PTX versions of their kernels. This can be tested by forcing the PTX to JIT-compile at application load time with following the steps: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch the application. With `CUDA_FORCE_PTX_JIT=1`, GPU binary code embedded in an application binary is ignored. Instead PTX code for each kernel is JIT-compiled to produce GPU binary code. An application fails to execute if it does not include PTX. This means the application is not Hopper architecture compatible and needs to be rebuilt for compatibility. On the other hand, if the application works properly with this environment variable set, then the application is Hopper compatible. Note Be sure to unset the `CUDA_FORCE_PTX_JIT` environment variable after testing is done. ### 1.3.2. Applications Built Using CUDA Toolkit 11.8[](#applications-built-using-cuda-toolkit-11-8 "Permalink to this headline") CUDA applications built using CUDA Toolkit 11.8 are compatible with Hopper architecture as long as they are built to include kernels in native cubin (compute capability 9.0) or PTX form or both. 1.4. Building Applications with Hopper Architecture Support[](#building-applications-with-hopper-architecture-support "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------- Depending on the version of the CUDA Toolkit used for building the application, it can be built to include PTX and/or native cubin for the Hopper architecture. Although it is enough to just include PTX, including native cubin also has the following advantages: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels which do not have native cubins are JIT-compiled from PTX, including kernels from all the libraries linked to the application, even if those kernels are never launched by the application[2](#fn2) . Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver caches the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled cubins may be faster or of greater accuracy. * PTX code compiled to target architecture conditional features using `sm_90a` or `compute_90a` only runs on devices with compute capability 9.0 and is not backward or forward compatible. ### 1.4.1. Building Applications Using CUDA Toolkit 11.7 or Earlier[](#building-applications-using-cuda-toolkit-11-7-or-earlier "Permalink to this headline") The `nvcc` compiler included with version 11.7 or earlier (11.0-11.7) of the CUDA Toolkit can generate cubins native to the NVIDIA Ampere GPU architectures (compute capability 8.x). When using CUDA Toolkit 11.7 or earlier, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_80,code=compute\_80 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_80,code=compute\_80 -O2 -o mykernel.o -c mykernel.cu Alternatively, the simplified `nvcc` command-line option `-arch=sm_XX` can be used. It is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target binary by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. For CUDA toolkits prior to 11.0, one or more of the `-gencode` options need to be removed according to the architectures supported by the specific toolkit version (for example, CUDA toolkit 10.x supports architectures up to sm\_72 and sm\_75). The final `-gencode` to generate PTX also needs to be updated. For further information and examples see the documentation for the specific CUDA toolkit version. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. **Only the back-end target version(s) specified by the code= clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures.** ### 1.4.2. Building Applications Using CUDA Toolkit 11.8[](#building-applications-using-cuda-toolkit-11-8 "Permalink to this headline") With versions 11.8 of the CUDA Toolkit, `nvcc` can generate cubin native to the Hopper architecture (compute capability 9.0). When using CUDA Toolkit 11.8, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_90,code=compute\_90 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_90,code=compute\_90 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. **Only the back-end target version(s) specified by the code= clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures.** ### 1.4.3. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") NVIDIA GPUs since Volta architecture have Independent Thread Scheduling among threads in a warp. If the developer made assumptions about warp-synchronicity[3](#fn3) , this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see [Compute Capability 7.x](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability-7-x) in the _CUDA C++ Programming Guide_ for details and corrective actions. To aid migration to the Hopper architecture, developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -gencode=arch=compute\_60,code=sm\_90 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id1) Just-in-time compilation. [2](#id2) Starting with CUDA toolkit 11.8, this default behavior can be changed with environment variable CUDA\_MODULE\_LOADING. See [Environment Variables](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars) in the _CUDA C++ Programming Guide_ for details. [3](#id3) Warp-synchronous refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. --- # 1. Turing Tuning Guide — Turing Tuning Guide 12.8 documentation * [](../index.html) » * 1\. Turing Tuning Guide * v12.8 | [PDF](../pdf/Turing_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for Turing The programming guide to tuning CUDA Applications for GPUs based on the NVIDIA Turing Architecture. 1\. Turing Tuning Guide[](#turing-tuning-guide "Permalink to this headline") ============================================================================== 1.1. NVIDIA Turing Compute Architecture[](#nvidia-turing-compute-architecture "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- Turing is NVIDIA’s latest architecture for CUDA compute applications. Turing retains and extends the same CUDA programming model provided by previous NVIDIA architectures such as Pascal and Volta, and applications that follow the best practices for those architectures should typically see speedups on the Turing architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging Turing architectural features.[1](#fn1) For further details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guideindex.html/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) and the [CUDA C++ Best Practices Guide](http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code, * Minimize data transfers between the host and the device, * Adjust kernel launch configuration to maximize device utilization, * Ensure global memory accesses are coalesced, * Minimize redundant accesses to global memory whenever possible, * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Turing Compatibility Guide for CUDA Applications](http://docs.nvidia.com/cuda/turing-compatibility-guide/index.html) to ensure that your application is compiled in a way that is compatible with Turing. 1.4. Turing Tuning[](#turing-tuning "Permalink to this headline") ------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The Turing Streaming Multiprocessor (SM) is based on the same major architecture (7.x) as Volta, and provides similar improvements over Pascal. #### 1.4.1.1. Instruction Scheduling[](#instruction-scheduling "Permalink to this headline") Each Turing SM includes 4 warp-scheduler units. Each scheduler handles a static set of warps and issues to a dedicated set of arithmetic instruction units. Instructions are performed over two cycles, and the schedulers can issue independent instructions every cycle. Dependent instruction issue latency for core FMA math operations is four clock cycles, like Volta, compared to six cycles on Pascal. As a result, execution latencies of core math operations can be hidden by as few as 4 warps per SM, assuming 4-way instruction-level parallelism _ILP_ per warp, or by 16 warps per SM without any instuction-level parallelism. Like Volta, the Turing SM provides 64 FP32 cores, 64 INT32 cores and 8 improved mixed-precision Tensor Cores. Turing has a lower double precision throughput than Volta with only 2 FP64 cores. #### 1.4.1.2. Independent Thread Scheduling[](#independent-thread-scheduling "Permalink to this headline") The Turing architecture features the same _Independent Thread Scheduling_ introduced with Volta. This enables intra-warp synchronization patterns previously unavailable and simplifies code changes when porting CPU code. However, Independent Thread Scheduling can also lead to a rather different set of threads participating in the executed code than intended if the developer made assumptions about warp-synchronicity[2](#fn2) of previous hardware architectures. When porting existing codes to Volta or Turing, the following three code patterns need careful attention. For more details see the _CUDA C++ Programming Guide_. * To avoid data corruption, applications using warp intrinsics (`__shfl*`, `__any`, `__all`, and `__ballot`) should transition to the new, safe, synchronizing counterparts, with the `*_sync` suffix. The new warp intrinsics take in a mask of threads that explicitly define which lanes (threads of a warp) must participate in the warp intrinsic. * Applications that assume reads and writes are implicitly visible to other threads in the same warp need to insert the new `__syncwarp()` warp-wide barrier synchronization instruction between steps where data is exchanged between threads via global or shared memory. Assumptions that code is executed in lockstep or that reads/writes from separate threads are visible across a warp without synchronization are invalid. * Applications using `__syncthreads()` or the PTX `bar.sync` (and their derivatives) in such a way that a barrier will not be reached by some non-exited thread in the thread block must be modified to ensure that all non-exited threads reach the barrier. The `racecheck` and `synccheck` tools provided by `compute-sanitizer` can help with locating violations. #### 1.4.1.3. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM is 32 on Turing (versus 64 on Volta). Other [factors influencing warp occupancy](http://developer.download.nvidia.com/compute/cuda/CUDA_Occupancy_calculator.xls) remain otherwise similar: * The register file size is 64k 32-bit registers per SM. * The maximum registers per thread is 255. * The maximum number of thread blocks per SM is 16. * Shared memory capacity per SM is 64KB. Overall, developers can expect similar occupancy as on Pascal or Volta without changes to their application. #### 1.4.1.4. Integer Arithmetic[](#integer-arithmetic "Permalink to this headline") Similar to Volta, the Turing SM includes dedicated FP32 and INT32 cores. This enables simultaneous execution of FP32 and INT32 operations. Applications can interleave pointer arithmetic with floating-point computations. For example, each iteration of a pipelined loop could update addresses and load data for the next iteration while simultaneously processing the current iteration at full FP32 throughput. ### 1.4.2. Tensor Core Operations[](#tensor-core-operations "Permalink to this headline") Volta introduced Tensor Cores to accelerate matrix multiply operations on mixed precision floating point data. Turing adds acceleration for integer matrix multiply operations. The tensor cores are exposed as Warp-Level Matrix Operations in the CUDA 10 C++ API. The API provides specialized matrix load, matrix multiply and accumulate, and matrix store operations, where each warp processes a small matrix fragment, allowing to efficiently use Tensor Cores from a CUDA-C++ program. In practice, Tensor Cores are used to perform much larger 2D or higher dimensional matrix operations, built up from these smaller matrix fragments. Each Tensor Core performs the matrix multiply-accumulate: D = A x B + C. The Tensor Cores support half precision matrix multiplication, where the matrix multiply inputs A and B are FP16 matrices, while the accumulation matrices C and D may be either FP16 or FP32 matrices. When accumulating in FP32, the FP16 multiply results in a full precision product that is then accumulated using FP32 addition. CUDA 10 supports several fragment sizes, 16x16x16, 32x8x16, and 8x32x16 to use the Tensor Cores on Volta or Turing with FP16 inputs. Any binary compiled for Volta will run on Turing, but Volta binaries using Tensor Cores will only be able to reach half of Turing’s Tensor Core peak performance. Recompiling the binary specifically for Turing would allow it to reach the peak performance. See the Turing Compatibility Guide for more information. Turing’s Tensor Core supports integer matrix multiply operations, which can operate on 8-bit, 4-bit and 1-bit integer inputs, with 32-bit integer accumulation. When operating on 8-bit inputs, CUDA exposes fragment sizes of 16x16x16, 32x8x16, and 8x32x16. For sub-byte operations the fragment sizes available are 8x8x32 for 4-bit inputs, or 8x8x128 for 1-bit inputs. See the _CUDA C++ Programming Guide_ for more information. ### 1.4.3. Memory Throughput[](#memory-throughput "Permalink to this headline") #### 1.4.3.1. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") Turing features a unified L1 / Shared Memory cache similar to the one introduced in Volta, but with a smaller size. The total size of the unified L1 / Shared Memory cache in Turing is 96 KB. The portion of the cache dedicated to shared memory or L1 (known as the _carveout_) can be changed at runtime, either automatically by the driver, or manually using the `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. Turing supports two carveout configurations, either with 64 KB of shared memory and 32 KB of L1, or with 32 KB of shared memory and 64 KB of L1. Turing allows a single thread block to address the full 64 KB of shared memory. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the _CUDA C++ Programming Guide_ for details. Like Pascal and Volta, Turing combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. The state-of-the-art L1 cache in Volta and Turing offers lower latency, higher bandwidth, and higher capacity compared to the earlier architectures. Like Volta, Turing’s L1 can cache write operations (write-through). The result is that for many applications Volta and Turing narrow the performance gap between explicitly managed shared memory and direct access to device memory. Also, the cost of register spills is lowered compared to Pascal, and the balance of occupancy versus spilling should be re-evaluated to ensure best performance. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release **Version 1.1** * Updated references to the _CUDA C++ Programming Guide_ and _CUDA C++ Best Practices Guide_. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id2) Throughout this guide, _Kepler_ refers to devices of compute capability 3.x, _Maxwell_ refers to devices of compute capability 5.x, _Pascal_ refers to devices of compute capability 6.x, _Volta_ refers to devices of compute capability 7.0, and _Turing_ refers to devices of compute capability 7.5. [2](#id6) The term warp-synchronous refers to code that implicitly assumes threads in the same warp are synchronized at every instruction. --- # 1. Volta Tuning Guide — Volta Tuning Guide 12.8 documentation * [](../index.html) » * 1\. Volta Tuning Guide * v12.8 | [PDF](../pdf/Volta_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for Volta The programming guide to tuning CUDA Applications for GPUs based on the NVIDIA Volta Architecture. 1\. Volta Tuning Guide[](#volta-tuning-guide "Permalink to this headline") ============================================================================ 1.1. NVIDIA Volta Compute Architecture[](#nvidia-volta-compute-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------- Volta is NVIDIA’s latest architecture for CUDA compute applications. Volta retains and extends the same CUDA programming model provided by previous NVIDIA architectures such as Maxwell and Pascal, and applications that follow the best practices for those architectures should typically see speedups on the Volta architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging Volta architectural features.[1](#fn1) Volta architecture comprises a single variant: GV100. A detailed overview of the major improvements in GV100 over earlier NVIDIA architectures is provided in a white paper entitled [NVIDIA Tesla V100 GPU Architecture: The World’s Most Advanced Datacenter GPU](http://www.nvidia.com/object/volta-architecture-whitepaper.html) . For further details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) and the [CUDA C++ Best Practices Guide](http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code, * Minimize data transfers between the host and the device, * Adjust kernel launch configuration to maximize device utilization, * Ensure global memory accesses are coalesced, * Minimize redundant accesses to global memory whenever possible, * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Volta Compatibility Guide for CUDA Applications](http://docs.nvidia.com/cuda/volta-compatibility-guide/index.html) to ensure that your application is compiled in a way that is compatible with Volta. 1.4. Volta Tuning[](#volta-tuning "Permalink to this headline") ----------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The Volta Streaming Multiprocessor (SM) provides the following improvements over Pascal. #### 1.4.1.1. Instruction Scheduling[](#instruction-scheduling "Permalink to this headline") Each Volta SM includes 4 warp-scheduler units. Each scheduler handles a static set of warps and issues to a dedicated set of arithmetic instruction units. Instructions are performed over two cycles, and the schedulers can issue independent instructions every cycle. Dependent instruction issue latency for core FMA math operations are reduced to four clock cycles, compared to six cycles on Pascal. As a result, execution latencies of core math operations can be hidden by as few as 4 warps per SM, assuming 4-way instruction-level parallelism _ILP_ per warp. Many more warps are, of course, recommended to cover the much greater latency of memory transactions and control-flow operations. Similar to GP100, the GV100 SM provides 64 FP32 cores and 32 FP64 cores. The GV100 SM additionally includes 64 INT32 cores and 8 mixed-precision Tensor Cores. GV100 provides up to 84 SMs. #### 1.4.1.2. Independent Thread Scheduling[](#independent-thread-scheduling "Permalink to this headline") The Volta architecture introduces _Independent Thread Scheduling_ among threads in a warp. This feature enables intra-warp synchronization patterns previously unavailable and simplifies code changes when porting CPU code. However, Independent Thread Scheduling can also lead to a rather different set of threads participating in the executed code than intended if the developer made assumptions about warp-synchronicity[2](#fn2) of previous hardware architectures. When porting existing codes to Volta, the following three code patterns need careful attention. For more details see the _CUDA C++ Programming Guide_. * To avoid data corruption, applications using warp intrinsics (`__shfl*`, `__any`, `__all`, and `__ballot`) should transition to the new, safe, synchronizing counterparts, with the `*_sync` suffix. The new warp intrinsics take in a mask of threads that explicitly define which lanes (threads of a warp) must participate in the warp intrinsic. * Applications that assume reads and writes are implicitly visible to other threads in the same warp need to insert the new `__syncwarp()` warp-wide barrier synchronization instruction between steps where data is exchanged between threads via global or shared memory. Assumptions that code is executed in lockstep or that reads/writes from separate threads are visible across a warp without synchronization are invalid. * Applications using `__syncthreads()` or the PTX `bar.sync` (and their derivatives) in such a way that a barrier will not be reached by some non-exited thread in the thread block must be modified to ensure that all non-exited threads reach the barrier. The `racecheck` and `synccheck` tools provided by `compute-sanitizer` can help with locating violations. #### 1.4.1.3. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM remains the same as in Pascal (i.e., 64), and other [factors influencing warp occupancy](http://developer.download.nvidia.com/compute/cuda/CUDA_Occupancy_calculator.xls) remain similar as well: * The register file size is 64k 32-bit registers per SM. * The maximum registers per thread is 255. * The maximum number of thread blocks per SM is 32. * Shared memory capacity per SM is 96KB, similar to GP104, and a 50% increase compared to GP100. Overall, developers can expect similar occupancy as on Pascal without changes to their application. #### 1.4.1.4. Integer Arithmetic[](#integer-arithmetic "Permalink to this headline") Unlike Pascal GPUs, the GV100 SM includes dedicated FP32 and INT32 cores. This enables simultaneous execution of FP32 and INT32 operations. Applications can now interleave pointer arithmetic with floating-point computations. For example, each iteration of a pipelined loop could update addresses and load data for the next iteration while simultaneously processing the current iteration at full FP32 throughput. ### 1.4.2. Tensor Core Operations[](#tensor-core-operations "Permalink to this headline") Each Tensor Core performs the following operation: D = AxB + C, where A, B, C, and D are 4x4 matrices. The matrix multiply inputs A and B are FP16 matrices, while the accumulation matrices C and D may be FP16 or FP32 matrices. When accumulating in FP32, the FP16 multiply results in a full precision product that is then accumulated using FP32 addition with the other intermediate products for a 4x4x4 matrix multiply. In practice, Tensor Cores are used to perform much larger 2D or higher dimensional matrix operations, built up from these smaller elements. The Volta tensor cores are exposed as Warp-Level Matrix Operations in the CUDA 9 C++ API. The API exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use Tensor Cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16x16 size matrices spanning all 32 threads of the warp. See the _CUDA C++ Programming Guide_ for more information. ### 1.4.3. Memory Throughput[](#memory-throughput "Permalink to this headline") #### 1.4.3.1. High Bandwidth Memory[](#high-bandwidth-memory "Permalink to this headline") GV100 uses up to eight memory dies per HBM2 stack and four stacks, with a maximum of 32 GB of GPU memory. A faster and more efficient HBM2 implementation delivers up to 900 GB/s of peak memory bandwidth, compared to 732 GB/s for GP100. This combination of a new generation HBM2 memory, and a new generation memory controller, in Volta provides 1.5x delivered memory bandwidth, compared to Pascal GP100—and a greater than 95% memory bandwidth efficiency running many workloads. In order to hide the DRAM latencies at full HBM2 bandwidth more memory accesses must be kept in flight, compared to GPUs equipped with traditional GDDR5. This is accomplished by the large complement of SMs in GV100, which typically boost the number of concurrent threads, and thus the reads-in-flight, compared to previous architectures. Resource-constrained kernels that are limited to low occupancy may benefit from increasing the number of concurrent memory accesses per thread. #### 1.4.3.2. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") In Volta the L1 cache, texture cache, and shared memory are backed by a combined 128 KB data cache. As in previous architectures, the portion of the cache dedicated to shared memory (known as the _carveout_) can be selected at runtime using `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. Volta supports shared memory capacities of 0, 8, 16, 32, 64, or 96 KB per SM. A new feature, Volta enables a single thread block to address the full 96 KB of shared memory. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the _CUDA C++ Programming Guide_ for details. Like Pascal, Volta combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. Volta increases the maximum capacity of the L1 cache to 128 KB, more than 7x larger than the GP100 L1. Another benefit of its union with shared memory, the Volta L1 improves in terms of both latency and bandwidth compared to Pascal. The result is that for many applications Volta narrows the performance gap between explicitly managed shared memory and direct access to device memory. Also, the cost of register spills is lowered compared to Pascal, and the balance of occupancy versus spilling should be re-evaluated to ensure best performance. ### 1.4.4. Cooperative Groups[](#cooperative-groups "Permalink to this headline") The Volta architecture introduced Independent Thread Scheduling, which enables intra-warp synchronization patterns that were previously not possible. To efficiently express these new patterns, CUDA 9 introduces Cooperative Groups. This is an extension to the CUDA programming model for organizing groups of communicating threads. Cooperative Groups allows developers to express the granularity at which threads are communicating, helping them to express richer, more efficient parallel decompositions. See the _CUDA C++ Programming Guide_ for more information. ### 1.4.5. Multi-Process Service[](#multi-process-service "Permalink to this headline") The Volta Multi-Process Service is significantly improved compared to previous architecutres, both in terms of performance and robustness. Intermediary software schedulers, used for MPS with previous architectures, have been replaced by hardware accelerated units within the GPU. MPS clients now submit tasks directly to the GPU work queues, significantly decreasing submission latency and increasing aggregate throughput. The limit on the number of MPS clients has also been increased by 3x to 48. Volta MPS also provides each client with an isolated address space,[3](#fn3) and extends Unified Memory support for MPS applications. Volta MPS also provides control for clients to restrict each client to a fraction of the GPU execution resources. Developers can use this feature to reduce or eliminate head-of-line blocking where work from one MPS client overwhelms GPU execution resources and prevents other clients from making progress, and thus improve average latency and jitter accross the system. ### 1.4.6. NVLink Interconnect[](#nvlink-interconnect "Permalink to this headline") NVLink is NVIDIA’s high-speed data interconnect. NVLink can be used to significantly increase performance for both GPU-to-GPU communication and for GPU access to system memory. GV100 supports up to six NVLink connections with each connection carrying up to 50 GB/s of bi-directional bandwidth. NVLink operates transparently within the existing CUDA model. Transfers between NVLink-connected endpoints are automatically routed through NVLink, rather than PCIe. The `cudaDeviceEnablePeerAccess()` API call remains necessary to enable direct transfers (over either PCIe or NVLink) between GPUs. The `cudaDeviceCanAccessPeer()` can be used to determine if peer access is possible between any pair of GPUs. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release **Version 1.1** * Added Cooperative Groups section. * Updated references to the _CUDA C++ Programming Guide_ and _CUDA C++ Best Practices Guide_. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id2) Throughout this guide, _Maxwell_ refers to devices of compute capability 5.x, _Pascal_ refers to device of compute capability 6.x, and _Volta_ refers to devices of compute capability 7.x. [2](#id6) The term warp-synchronous refers to code that implicitly assumes threads in the same warp are synchronized at every instruction. [3](#id10) As with previous architectures, MPS does not provide fatal fault isolation between clients. --- # 1. NVIDIA Ampere GPU Architecture Tuning Guide — NVIDIA Ampere Tuning Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Ampere GPU Architecture Tuning Guide * v12.8 | [PDF](../pdf/NVIDIA_Ampere_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for NVIDIA Ampere GPU Architecture The programming guide for tuning CUDA Applications for GPUs based on the NVIDIA Ampere GPU Architecture. 1\. NVIDIA Ampere GPU Architecture Tuning Guide[](#nvidia-ampere-gpu-architecture-tuning-guide "Permalink to this headline") ============================================================================================================================== 1.1. NVIDIA Ampere GPU Architecture[](#nvidia-ampere-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------- The NVIDIA Ampere GPU architecture is NVIDIA’s latest architecture for CUDA compute applications. The NVIDIA Ampere GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as Turing and Volta, and applications that follow the best practices for those architectures should typically see speedups on the NVIDIA A100 GPU without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Ampere GPU architecture’s features.[1](#fn1) For further details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and the [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code. * Minimize data transfers between the host and the device. * Adjust kernel launch configuration to maximize device utilization. * Ensure global memory accesses are coalesced. * Minimize redundant accesses to global memory whenever possible. * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [NVIDIA Ampere GPU Architecture Compatibility Guide for CUDA Applications](https://docs.nvidia.com/cuda/ampere-compatibility-guide/) to ensure that your application is compiled in a way that is compatible with the NVIDIA Ampere GPU Architecture. 1.4. NVIDIA Ampere GPU Architecture Tuning[](#nvidia-ampere-gpu-architecture-tuning "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The NVIDIA Ampere GPU architecture’s Streaming Multiprocessor (SM) provides the following improvements over Volta and Turing. #### 1.4.1.1. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM remains the same as in Volta (i.e., 64) for compute capability 8.0, while for compute capability 8.6 it is 48. Other [factors influencing warp occupancy](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html#occupancy-calculator) are: * The register file size is 64K 32-bit registers per SM. * The maximum number of registers per thread is 255. * The maximum number of thread blocks per SM is 32 for devices of compute capability 8.0 (i.e., A100 GPUs) and 16 for GPUs with compute capability 8.6. * For devices of compute capability 8.0 (i.e., A100 GPUs) shared memory capacity per SM is 164 KB, a 71% increase compared to V100’s capacity of 96 KB. For GPUs with compute capability 8.6, shared memory capacity per SM is 100 KB. * For devices of compute capability 8.0 (i.e., A100 GPUs) the maximum shared memory per thread block is 163 KB. For GPUs with compute capability 8.6 maximum shared memory per thread block is 99 KB. Overall, developers can expect similar occupancy as on Volta without changes to their application. #### 1.4.1.2. Asynchronous Data Copy from Global Memory to Shared Memory[](#asynchronous-data-copy-from-global-memory-to-shared-memory "Permalink to this headline") The NVIDIA Ampere GPU architecture adds hardware acceleration for copying data from global memory to shared memory. These copy instructions are asynchronous, with respect to computation and allow users to explicitly control overlap of compute with data movement from global memory into the SM. These instructions also avoid using extra registers for memory copies and can also bypass the L1 cache. This new feature is exposed via the `pipeline` API in CUDA. For more information please refer to the section on Async Copy in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#async-copy) . #### 1.4.1.3. Hardware Acceleration for Split Arrive/Wait Barrier[](#hardware-acceleration-for-split-arrive-wait-barrier "Permalink to this headline") The NVIDIA Ampere GPU architecture adds hardware acceleration for a split arrive/wait barrier in shared memory. These barriers can be used to implement fine grained thread controls, producer-consumer computation pipeline and divergence code patterns in CUDA. These barriers can also be used alongside the asynchronous copy. For more information on the Arrive/Wait Barriers refer to the Arrive/Wait Barrier section in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#aw-barrier) . #### 1.4.1.4. Warp level support for Reduction Operations[](#warp-level-support-for-reduction-operations "Permalink to this headline") The NVIDIA Ampere GPU architecture adds native support for warp wide reduction operations for 32-bit signed and unsigned integer operands. The warp wide reduction operations support arithmetic `add`, `min`, and `max` operations on 32-bit signed and unsigned integers and bitwise `and`, `or` and `xor` operations on 32-bit unsigned integers. For more details on the new warp wide reduction operations refer to Warp Reduce Functions in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#warp-reduce-functions) . #### 1.4.1.5. Improved Tensor Core Operations[](#improved-tensor-core-operations "Permalink to this headline") The NVIDIA Ampere GPU architecture includes new Third Generation Tensor Cores that are more powerful than the Tensor Cores used in Volta and Turing SMs. The new Tensor Cores use a larger base matrix size and add powerful new math modes including: * Support for FP64 Tensor Core, using new DMMA instructions. * Support for Bfloat16 Tensor Core, through HMMA instructions. BFloat16 format is especially effective for DL training scenarios. Bfloat16 provides 8-bit exponent i.e., same range as FP32, 7-bit mantissa and 1 sign-bit. * Support for TF32 Tensor Core, through HMMA instructions. TF32 is a new 19-bit Tensor Core format that can be easily integrated into programs for more accurate DL training than 16-bit HMMA formats. TF32 provides 8-bit exponent, 10-bit mantissa and 1 sign-bit. * Support for bitwise `AND` along with bitwise `XOR` which was introduced in Turing, through BMMA instructions. The following table presents the evolution of matrix instruction sizes and supported data types for Tensor Cores across different GPU architecture generations. | Instruction | GPU Architecture | Input Matrix format | Output Accumulator format | Matrix Instruction Size (MxNxK) | | --- | --- | --- | --- | --- | | HMMA (16-bit precision) | NVIDIA Volta Architecture | FP16 | FP16 / FP32 | 8x8x4 | | NVIDIA Turing Architecture | FP16 | FP16 / FP32 | 8x8x4 / 16x8x8 / 16x8x16 | | NVIDIA Ampere Architecture | FP16 / BFloat16 | FP16 / FP32 (BFloat16 only supports FP32 as accumulator) | 16x8x8 / 16x8x16 | | HMMA (19-bit precision) | NVIDIA Volta Architecture | N/A | N/A | N/A | | NVIDIA Turing Architecture | N/A | N/A | N/A | | NVIDIA Ampere Architecture | TF32 (19-bits) | FP32 | 16x8x4 | | IMMA (Integer MMA) | NVIDIA Volta Architecture | N/A | N/A | N/A | | NVIDIA Turing Architecture | unsigned char/signed char (8-bit precision) | int32 | 8x8x16 | | NVIDIA Ampere Architecture | unsigned char/signed char (8-bit precision) | int32 | 8x8x16 / 16x8x16 / 16x8x32 | | IMMA (Integer sub-byte MMA) | NVIDIA Volta Architecture | N/A | N/A | N/A | | NVIDIA Turing Architecture | unsigned u4/signed u4 (4-bit precision) | int32 | 8x8x32 | | NVIDIA Ampere Architecture | unsigned u4/signed u4 (4-bit precision) | int32 | 8x8x32 / 16x8x32 / 16x8x64 | | BMMA (Binary MMA) | NVIDIA Volta Architecture | N/A | N/A | N/A | | NVIDIA Turing Architecture | single bit | int32 | 8x8x128 | | NVIDIA Ampere Architecture | single bit | int32 | 8x8x128 / 16x8x128 / 16x8x256 | | DMMA (64-bit precision) | NVIDIA Volta Architecture | N/A | N/A | N/A | | NVIDIA Turing Architecture | N/A | N/A | N/A | | NVIDIA Ampere Architecture | FP64 | FP64 | 8x8x4 | For more details on the new Tensor Core operations refer to the Warp Matrix Multiply section in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#wmma) . #### 1.4.1.6. Improved FP32 throughput[](#improved-fp32-throughput "Permalink to this headline") Devices of compute capability 8.6 have 2x more FP32 operations per cycle per SM than devices of compute capability 8.0. While a binary compiled for 8.0 will run as is on 8.6, it is recommended to compile explicitly for 8.6 to benefit from the increased FP32 throughput. ### 1.4.2. Memory System[](#memory-system "Permalink to this headline") #### 1.4.2.1. Increased Memory Capacity and High Bandwidth Memory[](#increased-memory-capacity-and-high-bandwidth-memory "Permalink to this headline") The NVIDIA A100 GPU increases the HBM2 memory capacity from 32 GB in V100 GPU to 40 GB in A100 GPU. Along with the increased memory capacity, the bandwidth is increased by 72%, from 900 GB/s on Volta V100 to 1550 GB/s on A100. #### 1.4.2.2. Increased L2 capacity and L2 Residency Controls[](#increased-l2-capacity-and-l2-residency-controls "Permalink to this headline") The NVIDIA Ampere GPU architecture increases the capacity of the L2 cache to 40 MB in Tesla A100, which is 7x larger than Tesla V100. Along with the increased capacity, the bandwidth of the L2 cache to the SMs is also increased. The NVIDIA Ampere GPU architecture allows CUDA users to control the persistence of data in L2 cache. For more information on the persistence of data in L2 cache, refer to the section on managing L2 cache in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#L2_access_intro) . #### 1.4.2.3. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") The NVIDIA A100 GPU based on compute capability 8.0 increases the maximum capacity of the combined L1 cache, texture cache and shared memory to 192 KB, 50% larger than the L1 cache in NVIDIA V100 GPU. The combined L1 cache capacity for GPUs with compute capability 8.6 is 128 KB. In the NVIDIA Ampere GPU architecture, the portion of the L1 cache dedicated to shared memory (known as the _carveout_) can be selected at runtime as in previous architectures such as Volta, using `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. The NVIDIA A100 GPU supports shared memory capacity of 0, 8, 16, 32, 64, 100, 132 or 164 KB per SM. GPUs with compute capability 8.6 support shared memory capacity of 0, 8, 16, 32, 64 or 100 KB per SM. CUDA reserves 1 KB of shared memory per thread block. Hence, the A100 GPU enables a single thread block to address up to 163 KB of shared memory and GPUs with compute capability 8.6 can address up to 99 KB of shared memory in a single thread block. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) for details. Like Volta, the NVIDIA Ampere GPU architecture combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. Another benefit of its union with shared memory, similar to Volta L1 is improvement in terms of both latency and bandwidth. ### 1.4.3. Third Generation NVLink[](#third-generation-nvlink "Permalink to this headline") The third generation of NVIDIA’s high-speed NVLink interconnect is implemented in A100 GPUs, which significantly enhances multi-GPU scalability, performance, and reliability with more links per GPU, much faster communication bandwidth, and improved error-detection and recovery features. The third generation NVLink has the same bi-directional data rate of 50 GB/s per link, but uses half the number of signal pairs to achieve this bandwidth. Therefore, the total number of links available is increased to twelve in A100, versus six in V100, yielding 600 GB/s bidirectional bandwidth versus 300 GB/s for V100. NVLink operates transparently within the existing CUDA model. Transfers between NVLink-connected endpoints are automatically routed through NVLink, rather than PCIe. The `cudaDeviceEnablePeerAccess()` API call remains necessary to enable direct transfers (over either PCIe or NVLink) between GPUs. The `cudaDeviceCanAccessPeer()` can be used to determine if peer access is possible between any pair of GPUs. In the NVIDIA Ampere GPU architecture remote NVLINK accesses go through a Link TLB on the remote GPU. This Link TLB has a reach of 64 GB to the remote GPU’s memory. Applications with remote random accesses may want to constrain the remotely accessed region to 64 GB for each peer GPU. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.1** * Initial Public Release * Added support for compute capability 8.6 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id1) Throughout this guide, _Kepler_ refers to devices of compute capability 3.x, _Maxwell_ refers to devices of compute capability 5.x, _Pascal_ refers to device of compute capability 6.x, _Volta_ refers to devices of compute capability 7.0, _Turing_ refers to devices of compute capability 7.5, and _NVIDIA Ampere GPU Architecture_ refers to devices of compute capability 8.x --- # 1. NVIDIA Hopper Tuning Guide — Hopper Tuning Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Hopper Tuning Guide * v12.8 | [PDF](../pdf/Hopper_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for Hopper GPU Architecture The programming guide for tuning CUDA Applications for GPUs based on the Hopper GPU Architecture. 1\. NVIDIA Hopper Tuning Guide[](#nvidia-hopper-tuning-guide "Permalink to this headline") ============================================================================================ 1.1. NVIDIA Hopper GPU Architecture[](#nvidia-hopper-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------- The NVIDIA® Hopper GPU architecture is NVIDIA’s latest architecture for CUDA® compute applications. The NVIDIA Hopper GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as NVIDIA Ampere GPU architecture and NVIDIA Turing, and applications that follow the best practices for those architectures should typically see speedups on the NVIDIA H100 GPU without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Hopper GPU architecture’s features.[1](#fn1) For further details on the programming features discussed in this guide, refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and the [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code. * Minimize data transfers between the host and the device. * Adjust kernel launch configuration to maximize device utilization. * Ensure that global memory accesses are coalesced. * Minimize redundant accesses to global memory whenever possible. * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Hopper Compatibility Guide for CUDA Applications](https://docs.nvidia.com/cuda/hopper-compatibility-guide/) to ensure that your application is compiled in a way that is compatible with NVIDIA Hopper. 1.4. NVIDIA Hopper Tuning[](#nvidia-hopper-tuning "Permalink to this headline") --------------------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The NVIDIA Hopper Streaming Multiprocessor (SM) provides the following improvements over Turing and NVIDIA Ampere GPU architectures. #### 1.4.1.1. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM remains the same as in NVIDIA Ampere GPU architecture (that is, 64), and other [factors influencing warp occupancy](https://docs.nvidia.com/cuda/cuda-occupancy-calculator/CUDA_Occupancy_Calculator.xls) are: * The register file size is 64K 32-bit registers per SM. * The maximum number of registers per thread is 255. * The maximum number of thread blocks per SM is 32 for devices of compute capability 9.0 (that is, H100 GPUs). * For devices of compute capability 9.0 (H100 GPUs), shared memory capacity per SM is 228 KB, a 39% increase compared to A100’s capacity of 164 KB. * For devices of compute capability 9.0 (H100 GPUs), the maximum shared memory per thread block is 227 KB. * For applications using Thread Block Clusters, it is always recommended to compute the occupancy using `cudaOccupancyMaxActiveClusters` and launch cluster-based kernels accordingly. Overall, developers can expect similar occupancy as on NVIDIA Ampere GPU architecture GPUs without changes to their application. #### 1.4.1.2. Tensor Memory Accelerator[](#tensor-memory-accelerator "Permalink to this headline") The Hopper architecture builds on top of the asynchronous copies introduced by NVIDIA Ampere GPU architecture and provides a more sophisticated asynchronous copy engine: the Tensor Memory Accelerator (TMA). TMA allows applications to transfer 1D and up to 5D tensors between global memory and shared memory, in both directions, as well as between the shared memory regions of different SMs in the same cluster (refer to [Thread Block Clusters](#thread-block-clusters) ). Additionally, for writes from shared memory to global memory, it allows specifying element wise reduction operations such as add/min/max as well as bitwise and/or for most common data types. This has several advantages: * Avoids using registers for moving data between the different memory spaces. * Avoids using SM instructions for moving data: a single thread can issue large data movement instructions to the TMA unit. The whole block can then continue working on other instructions while the data is in flight and only wait for the data to be consumed when actually necessary. * Enables users to write warp specialized codes, where specific warps specialize on data movement between the different memory spaces while other warps only work on local data within the SM. This feature will be exposed through `cuda::memcpy_async` along with the `cuda::barrier` and `cuda::pipeline` for synchronizing data movement. #### 1.4.1.3. Thread Block Clusters[](#thread-block-clusters "Permalink to this headline") NVIDIA Hopper Architecture adds a new optional level of hierarchy, Thread Block Clusters, that allows for further possibilities when parallelizing applications. A thread block can read from, write to, and perform atomics in shared memory of other thread blocks within its cluster. This is known as Distributed Shared Memory. As demonstrated in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#distributed-shared-memory) , there are applications that cannot fit required data within shared memory and must use global memory instead. Distributed shared memory can act as an intermediate step between these two options. Distributed Shared Memory can be used by an SM simultaneously with L2 cache accesses. This can benefit applications that need to communicate data between SMs by utilizing the combined bandwidth of both distributed shared memory and L2. In order to achieve best performance for accesses to Distributed Shared Memory, access patterns to those described in the [CUDA C++ Best Practices Guide for Global Memory](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/#coalesced-access-to-global-memory) should be used. Specifically, accesses to Distributed Shared Memory should be coalesced and aligned to 32-byte segments, if possible. Access patterns with non-unit stride should be avoided if possible, which can be achieved by using local shared memory, similar to what is shown in the [CUDA C++ Best Practices Guide for Shared Memory](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/#shared-memory) . The maximum portable cluster size supported is 8; however, NVIDIA Hopper H100 GPU allows for a nonportable cluster size of 16 by opting in. Launching a kernel with a nonportable cluster size requires setting the **cudaFuncAttributeNonPortableClusterSizeAllowed** function attribute. Using larger cluster sizes may reduce the maximum number of active blocks across the GPU (refer to [Occupancy](#sm-occupancy) ). #### 1.4.1.4. Improved FP32 Throughput[](#improved-fp32-throughput "Permalink to this headline") Devices of compute capability 9.0 have 2x more FP32 operations per cycle per SM than devices of compute capability 8.0. #### 1.4.1.5. Dynamic Programming Instructions[](#dynamic-programming-instructions "Permalink to this headline") The NVIDIA Hopper architecture adds support for new instructions to accelerate dynamic programming algorithms, such as the Smith-Waterman algorithm for sequence alignment in bioinformatics, and algorithms in graph theory, game theory, ML, and finance problems. The new instructions permit computation of max and min values among three operands, max and min operations yielding predicates, combined add operation with max or min, operating on signed and unsigned 32-bit int and 16-bit short2 types, and half2. All DPX instructions with 16-bit short types DPX instructions enable 128 operations per cycle per SM. ### 1.4.2. Memory System[](#memory-system "Permalink to this headline") #### 1.4.2.1. High-Bandwidth Memory HBM3 Subsystem[](#high-bandwidth-memory-hbm3-subsystem "Permalink to this headline") The NVIDIA H100 GPU has support for HBM3 and HBM2e memory, with capacity up to 80 GB. GPUs HBM3 memory system supports up to 3 TB/s memory bandwidth, a 93% increase over the 1.55 TB/s on A100-40GB. #### 1.4.2.2. Increased L2 Capacity[](#increased-l2-capacity "Permalink to this headline") The NVIDIA Hopper architecture increases the L2 cache capacity from 40 MB in the A100 GPU to 50 MB in the H100 GPU. Along with the increased capacity, the bandwidth of the L2 cache to the SMs is also increased. The NVIDIA Hopper architecture allows CUDA users to control the persistence of data in L2 cache similar to the NVIDIA Ampere GPU Architecture. For more information on the persistence of data in L2 cache, refer to the section on managing L2 cache in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#L2_access_intro) . #### 1.4.2.3. Inline Compression[](#inline-compression "Permalink to this headline") The NVIDIA Hopper architecture allows CUDA compute kernels to benefit from the new inline compression (ILC). This feature can be applied to individual memory allocation, and the compressor automatically chooses between several possible compression algorithms, or none if there is no suitable pattern. In case compression can be used, this feature allows accessing global memory at significantly higher bandwidth than global memory bandwidth, since only compressed data needs to be transferred between global memory and SMs. However, the feature does not allow for reducing memory footprint: since compression is automatic, even if compression is active, the memory region will use the same footprint as if there was no compression. This is because underlying data may be changed by the user application and may not be compressible during the entire duration of the application. The feature is available through the CUDA driver API. See the [CUDA C++ Programming Guide section on compressible memory](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#physical-memory-type-compression) : CUmemGenericAllocationHandle allocationHandle; CUmemAllocationProp prop \= {}; memset(prop, 0, sizeof(CUmemAllocationProp)); prop\->type \= CU\_MEM\_ALLOCATION\_TYPE\_PINNED; prop\->location.type \= CU\_MEM\_LOCATION\_TYPE\_DEVICE; prop\->location.id \= currentDevice; prop\->allocFlags.compressionType \= CU\_MEM\_ALLOCATION\_COMP\_GENERIC; cuMemCreate(&allocationHandle, size, &prop, 0); One can check whether compressible memory is available on the given device with: cuDeviceGetAttribute(&compressionAvailable, CU\_DEVICE\_ATTRIBUTE\_GENERIC\_COMPRESSION\_SUPPORTED, currentDevice) Note that this example code does not handle errors and compiling this code requires linking against the CUDA library (`libcuda.so`). #### 1.4.2.4. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") The NVIDIA H100 GPU based on compute capability 9.0 increases the maximum capacity of the combined L1 cache, texture cache, and shared memory to 256 KB, from 192 KB in NVIDIA Ampere Architecture, an increase of 33%. In the NVIDIA Hopper GPU architecture, the portion of the L1 cache dedicated to shared memory (known as the carveout) can be selected at runtime as in previous architectures such as NVIDIA Ampere Architecture and NVIDIA Volta, using `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. The NVIDIA H100 GPU supports shared memory capacities of 0, 8, 16, 32, 64, 100, 132, 164, 196 and 228 KB per SM. CUDA reserves 1 KB of shared memory per thread block. Hence, the H100 GPU enables a single thread block to address up to 227 KB of shared memory. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) for details. Like the NVIDIA Ampere Architecture and NVIDIA Volta GPU architectures, the NVIDIA Hopper GPU architecture combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp before delivery of that data to the warp. Another benefit of its union with shared memory, similar to previous architectures, is improvement in terms of both latency and bandwidth. ### 1.4.3. Fourth-Generation NVLink[](#fourth-generation-nvlink "Permalink to this headline") The fourth generation of NVIDIA’s high-speed NVLink interconnect is implemented in H100 GPUs, which significantly enhances multi-GPU scalability, performance, and reliability with more links per GPU, much faster communication bandwidth, and improved error-detection and recovery features. The fourth-generation NVLink has the same bidirectional data rate of 50 GB/s per link. The total number of links available is increased to 18 in H100, compared to 12 in A100, yielding 900 GB/s bidirectional bandwidth compared to 600 GB/s for A100. NVLink operates transparently within the existing CUDA model. Transfers between NVLink-connected endpoints are automatically routed through NVLink, rather than PCIe. The `cudaDeviceEnablePeerAccess()` API call remains necessary to enable direct transfers (over either PCIe or NVLink) between GPUs. The `cudaDeviceCanAccessPeer()` can be used to determine if peer access is possible between any pair of GPUs. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release * Added support for compute capability 9.0 [1](#id1) Throughout this guide, NVIDIA Volta refers to devices of compute capability 7.0, NVIDIA Turing refers to devices of compute capability 7.5, NVIDIA Ampere GPU Architecture refers to devices of compute capability 8.x, and NVIDIA Hopper refers to devices of compute capability 9.0. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. NVIDIA Ada GPU Architecture Tuning Guide — Ada Tuning Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Ada GPU Architecture Tuning Guide * v12.8 | [PDF](../pdf/Ada_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for NVIDIA Ada GPU Architecture The programming guide for tuning CUDA Applications for GPUs based on the NVIDIA Ada GPU Architecture. 1\. NVIDIA Ada GPU Architecture Tuning Guide[](#nvidia-ada-gpu-architecture-tuning-guide "Permalink to this headline") ======================================================================================================================== 1.1. NVIDIA Ada GPU Architecture[](#nvidia-ada-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------- The NVIDIA® Ada GPU architecture is NVIDIA’s latest architecture for CUDA® compute applications. The NVIDIA Ada GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as NVIDIA Ampere and Turing, and applications that follow the best practices for those architectures should typically see speedups on the NVIDIA Ada architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Ada GPU architecture’s features.[1](#fn1) For further details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and the [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code. * Minimize data transfers between the host and the device. * Adjust kernel launch configuration to maximize device utilization. * Ensure global memory accesses are coalesced. * Minimize redundant accesses to global memory whenever possible. * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [NVIDIA Ada GPU Architecture Compatibility Guide for CUDA Applications](https://docs.nvidia.com/cuda/ada-compatibility-guide/) to ensure that your application is compiled in a way that is compatible with the NVIDIA Ada GPU Architecture. 1.4. NVIDIA Ada GPU Architecture Tuning[](#nvidia-ada-gpu-architecture-tuning "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The NVIDIA Ada GPU architecture’s Streaming Multiprocessor (SM) provides the following improvements over Turing and NVIDIA Ampere GPU architectures. #### 1.4.1.1. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM is 48, remaining the same compared to compute capability 8.6 GPUs, and other [factors influencing warp occupancy](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html#occupancy-calculator) are: * The register file size is 64K 32-bit registers per SM. * The maximum number of registers per thread is 255. * The maximum number of thread blocks per SM is 24. * The shared memory capacity per SM is 100 KB. * The maximum shared memory per thread block is 99 KB. Overall, developers can expect similar occupancy as on compute capability 8.6 GPUs without changes to their application. #### 1.4.1.2. Improved Tensor Core Operations[](#improved-tensor-core-operations "Permalink to this headline") The NVIDIA Ada GPU architecture includes new Ada Fourth Generation Tensor Cores featuring the Hopper FP8 Transformer Engine. #### 1.4.1.3. Improved FP32 throughput[](#improved-fp32-throughput "Permalink to this headline") Devices of compute capability 8.9 have 2x more FP32 operations per cycle per SM than devices of compute capability 8.0. While a binary compiled for 8.0 will run as-is on 8.9, it is recommended to compile explicitly for 8.9 to benefit from the increased FP32 throughput. ### 1.4.2. Memory System[](#memory-system "Permalink to this headline") #### 1.4.2.1. Increased L2 capacity[](#increased-l2-capacity "Permalink to this headline") The NVIDIA Ada GPU architecture increases the capacity of the L2 cache to 98304 KB in AD102, 16x larger than GA102. The NVIDIA Ada GPU architecture allows CUDA users to control the persistence of data in the L2 cache. For more information on the persistence of data in the L2 cache, refer to the section on managing the L2 cache in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#L2_access_intro) . #### 1.4.2.2. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") NVIDIA Ada architecture features a unified L1 cache, texture cache, and shared memory similar to that of the NVIDIA Ampere architecture. The combined L1 cache capacity is 128 KB. In the NVIDIA Ada GPU architecture, the portion of the L1 cache dedicated to shared memory (known as the _carveout_) can be selected at runtime as in previous architectures, such as NVIDIA Ampere, using `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. The NVIDIA Ada GPU architecture supports shared memory capacity of 0, 8, 16, 32, 64 or 100 KB per SM. CUDA reserves 1 KB of shared memory per thread block. Hence, GPUs with compute capability 8.9 can address up to 99 KB of shared memory in a single thread block. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) for details. Like the NVIDIA Ampere and NVIDIA Volta GPU architectures, the NVIDIA Ada GPU architecture combines the functionality of the L1 and texture caches into a unified L1/Texture cache that acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. Another benefit of its union with shared memory, similar to previous architectures, is improvement in terms of both latency and bandwidth. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release * Added support for compute capability 8.9 [1](#id1) Throughout this guide, _Volta_ refers to devices of compute capability 7.0, _Turing_ refers to devices of compute capability 7.5, _NVIDIA Ampere GPU Architecture_ refers to devices of compute capability 8.0 and 8.6, _NVIDIA Ada_ refers to devices of compute capability 8.9. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. NVIDIA Blackwell Tuning Guide — Blackwell Tuning Guide 12.8 documentation * [](../index.html) » * 1\. NVIDIA Blackwell Tuning Guide * v12.8 | [PDF](../pdf/Blackwell_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Tuning CUDA Applications for Blackwell GPU Architecture The programming guide for tuning CUDA Applications for GPUs based on the Blackwell GPU Architecture. 1\. NVIDIA Blackwell Tuning Guide[](#nvidia-blackwell-tuning-guide "Permalink to this headline") ================================================================================================== 1.1. NVIDIA Blackwell GPU Architecture[](#nvidia-blackwell-gpu-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------- The NVIDIA® Blackwell GPU architecture is NVIDIA’s latest architecture for CUDA® compute applications. The NVIDIA Blackwell GPU architecture retains and extends the same CUDA programming model provided by previous NVIDIA GPU architectures such as NVIDIA Ampere GPU architecture and NVIDIA Hopper. Applications that follow the best practices for those architectures should typically see speedups on the Blackwell GPUs without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging the NVIDIA Blackwell GPU architecture’s features.[1](#fn1) For further details on the programming features discussed in this guide, refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and the [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code. * Minimize data transfers between the host and the device. * Adjust kernel launch configuration to maximize device utilization. * Ensure that global memory accesses are coalesced. * Minimize redundant accesses to global memory whenever possible. * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Blackwell Compatibility Guide for CUDA Applications](https://docs.nvidia.com/cuda/blackwell-compatibility-guide/) to ensure that your application is compiled in a way that is compatible with NVIDIA Blackwell. 1.4. NVIDIA Blackwell Tuning[](#nvidia-blackwell-tuning "Permalink to this headline") --------------------------------------------------------------------------------------- ### 1.4.1. Streaming Multiprocessor[](#streaming-multiprocessor "Permalink to this headline") The NVIDIA Blackwell Streaming Multiprocessor (SM) provides the following improvements over the NVIDIA Hopper GPU architecture. #### 1.4.1.1. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SM is 64 for compute capability 10.0 and 48 for compute capability 12.0. Other [factors influencing warp occupancy](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html#occupancy-calculator) are: * The register file size is 64K 32-bit registers per SM. * The maximum number of registers per thread is 255. * The maximum number of thread blocks per SM is 32 for devices of compute capability 10.0 and 12.0. * For devices of compute capability 10.0 shared memory capacity per SM is 228 KB. For devices of compute capability 12.0, shared memory capacity per SM is 128KB. * For devices of compute capability 10.0 the maximum shared memory per thread block is 227 KB. For devices of compute capability 12.0 the maximum shared memory per thread block is 99 KB. * For applications using Thread Block Clusters, it is always recommended to compute the occupancy using `cudaOccupancyMaxActiveClusters` and launch cluster-based kernels accordingly. Overall, developers can expect similar occupancy as on NVIDIA Hopper GPU architecture GPUs without changes to their application. #### 1.4.1.2. Thread Block Clusters[](#thread-block-clusters "Permalink to this headline") NVIDIA Hopper Architecture added a new optional level of hierarchy, Thread Block Clusters, that allows for further possibilities when parallelizing applications. Thread block clusters are supported by Blackwell GPUs as well. A thread block can read from, write to, and perform atomics in shared memory of other thread blocks within its cluster. This is known as Distributed Shared Memory. As demonstrated in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#distributed-shared-memory) , there are applications that cannot fit required data within shared memory and must use global memory instead. Distributed shared memory can act as an intermediate step between these two options. Distributed Shared Memory can be used by an SM simultaneously with L2 cache accesses. This can benefit applications that need to communicate data between SMs by utilizing the combined bandwidth of both distributed shared memory and L2. In order to achieve best performance for accesses to Distributed Shared Memory, access patterns to those described in the [CUDA C++ Best Practices Guide for Global Memory](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/#coalesced-access-to-global-memory) should be used. Specifically, accesses to Distributed Shared Memory should be coalesced and aligned to 32-byte segments, if possible. Access patterns with non-unit stride should be avoided if possible, which can be achieved by using local shared memory, similar to what is shown in the [CUDA C++ Best Practices Guide for Shared Memory](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/#shared-memory) . The maximum portable cluster size supported is 8; however, NVIDIA Blackwell B200 GPU allows for a nonportable cluster size of 16 by opting in. Launching a kernel with a nonportable cluster size requires setting the `cudaFuncAttributeNonPortableClusterSizeAllowed` function attribute. Using larger cluster sizes may reduce the maximum number of active blocks across the GPU (refer to [Occupancy](#sm-occupancy) ). ### 1.4.2. Memory System[](#memory-system "Permalink to this headline") #### 1.4.2.1. High-Bandwidth Memory HBM3 Subsystem[](#high-bandwidth-memory-hbm3-subsystem "Permalink to this headline") The NVIDIA B200 GPU has support for HBM3 and HBM3e memory, with capacity up to 180 GB. #### 1.4.2.2. Increased L2 Capacity[](#increased-l2-capacity "Permalink to this headline") The NVIDIA GB200 GPU increases the L2 cache capacity to 126 MB. The NVIDIA Blackwell architecture allows CUDA users to control the persistence of data in L2 cache similar to the NVIDIA Ampere GPU Architecture. For more information on the persistence of data in L2 cache, refer to the section on managing L2 cache in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#L2_access_intro) . #### 1.4.2.3. Unified Shared Memory/L1/Texture Cache[](#unified-shared-memory-l1-texture-cache "Permalink to this headline") The NVIDIA B200 GPU with compute capability 10.0 has the same the maximum capacity of the combined L1 cache, texture cache, and shared memory of 256 KB as the previous NVIDIA Hopper architecture. In the NVIDIA Blackwell GPU architecture, the portion of the L1 cache dedicated to shared memory (known as the carveout) can be selected at runtime as in previous architectures such as NVIDIA Ampere Architecture and NVIDIA Volta, using `cudaFuncSetAttribute()` with the attribute `cudaFuncAttributePreferredSharedMemoryCarveout`. Both the NVIDIA H100 GPU and the NVIDIA B200 GPU support shared memory capacities of 0, 8, 16, 32, 64, 100, 132, 164, 196 and 228 KB per SM. CUDA reserves 1 KB of shared memory per thread block. Hence, the B200 GPU enables a single thread block to address up to 227 KB of shared memory. To maintain architectural compatibility, static shared memory allocations remain limited to 48 KB, and an explicit opt-in is also required to enable dynamic allocations above this limit. See the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) for details. Like GPU architectures going back to NVIDIA Ampere Architecture (compute capability 8.x), the NVIDIA Blackwell GPU architecture combines the functionality of the L1 and texture caches into a unified L1/Texture cache which acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp before delivery of that data to the warp. Another benefit of its union with shared memory, similar to previous architectures, is improvement in terms of both latency and bandwidth. ### 1.4.3. Fifth-Generation NVLink[](#fifth-generation-nvlink "Permalink to this headline") The fifth generation of NVIDIA’s high-speed NVLink interconnect is implemented in B200 GPUs, which significantly enhances multi-GPU scalability, performance, and reliability with more links per GPU, much faster communication bandwidth, and improved error-detection and recovery features. NVLink operates transparently within the existing CUDA model. Transfers between NVLink-connected endpoints are automatically routed through NVLink, rather than PCIe. The `cudaDeviceEnablePeerAccess()` API call remains necessary to enable direct transfers (over either PCIe or NVLink) between GPUs. The `cudaDeviceCanAccessPeer()` can be used to determine if peer access is possible between any pair of GPUs. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release * Added support for compute capability 10.0 and compute capability 12.0 [1](#id1) Throughout this guide, NVIDIA Volta refers to devices of compute capability 7.0, NVIDIA Turing refers to devices of compute capability 7.5, NVIDIA Ampere GPU Architecture refers to devices of compute capability 8.x, NVIDIA Hopper refers to devices of compute capability 9.0., and NVIDIA Blackwell refers to compute capability 10.0. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # NVIDIA Video Decoder (NVCUVID) * [](../index.html) » * 1\. NVIDIA Video Decoder (NVCUVID) * v12.5 | [PDF](../pdf/NVIDIA_Video_Decoder.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * 1\. NVIDIA Video Decoder (NVCUVID)[](#nvidia-video-decoder-nvcuvid "Permalink to this headline") ================================================================================================== For NVIDIA Video Decoder (NVCUVID) use the [NVIDIA Video Codec SDK](https://developer.nvidia.com/nvidia-video-codec-sdk) . 2\. Notices[](#notices "Permalink to this headline") ====================================================== 2.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 2.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 2.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Preface — CUDA C++ Best Practices Guide 12.8 documentation * [](../index.html) » * 1\. Preface * v12.8 | [PDF](../pdf/CUDA_C_Best_Practices_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA C++ Best Practices Guide The programming guide to using the CUDA Toolkit to obtain the best performance from NVIDIA GPUs. 1\. Preface[](#preface "Permalink to this headline") ====================================================== This Best Practices Guide is a manual to help developers obtain the best performance from NVIDIA® CUDA® GPUs. It presents established parallelization and optimization techniques and explains coding metaphors and idioms that can greatly simplify programming for CUDA-capable GPU architectures. While the contents can be used as a reference manual, you should be aware that some topics are revisited in different contexts as various programming and configuration topics are explored. As a result, it is recommended that first-time readers proceed through the guide sequentially. This approach will greatly improve your understanding of effective programming practices and enable you to better use the guide for reference later. 1.1. Who Should Read This Guide?[](#who-should-read-this-guide "Permalink to this headline") ---------------------------------------------------------------------------------------------- The discussions in this guide all use the C++ programming language, so you should be comfortable reading C++ code. This guide refers to and relies on several other documents that you should have at your disposal for reference, all of which are available at no cost from the CUDA website [https://docs.nvidia.com/cuda/](https://docs.nvidia.com/cuda/) . The following documents are especially important resources: * CUDA Installation Guide * CUDA C++ Programming Guide * CUDA Toolkit Reference Manual In particular, the optimization section of this guide assumes that you have already successfully downloaded and installed the CUDA Toolkit (if not, please refer to the relevant CUDA Installation Guide for your platform) and that you have a basic familiarity with the CUDA C++ programming language and environment (if not, please refer to the CUDA C++ Programming Guide). 1.2. Assess, Parallelize, Optimize, Deploy[](#assess-parallelize-optimize-deploy "Permalink to this headline") ---------------------------------------------------------------------------------------------------------------- This guide introduces the _Assess, Parallelize, Optimize, Deploy(APOD)_ design cycle for applications with the goal of helping application developers to rapidly identify the portions of their code that would most readily benefit from GPU acceleration, rapidly realize that benefit, and begin leveraging the resulting speedups in production as early as possible. APOD is a cyclical process: initial speedups can be achieved, tested, and deployed with only minimal initial investment of time, at which point the cycle can begin again by identifying further optimization opportunities, seeing additional speedups, and then deploying the even faster versions of the application into production. ![_images/apod-cycle.png](_images/apod-cycle.png) ### 1.2.1. Assess[](#assess "Permalink to this headline") For an existing project, the first step is to assess the application to locate the parts of the code that are responsible for the bulk of the execution time. Armed with this knowledge, the developer can evaluate these bottlenecks for parallelization and start to investigate GPU acceleration. By understanding the end-user’s requirements and constraints and by applying Amdahl’s and Gustafson’s laws, the developer can determine the upper bound of performance improvement from acceleration of the identified portions of the application. ### 1.2.2. Parallelize[](#parallelize "Permalink to this headline") Having identified the hotspots and having done the basic exercises to set goals and expectations, the developer needs to parallelize the code. Depending on the original code, this can be as simple as calling into an existing GPU-optimized library such as `cuBLAS`, `cuFFT`, or `Thrust`, or it could be as simple as adding a few preprocessor directives as hints to a parallelizing compiler. On the other hand, some applications’ designs will require some amount of refactoring to expose their inherent parallelism. As even CPU architectures will require exposing parallelism in order to improve or simply maintain the performance of sequential applications, the CUDA family of parallel programming languages (CUDA C++, CUDA Fortran, etc.) aims to make the expression of this parallelism as simple as possible, while simultaneously enabling operation on CUDA-capable GPUs designed for maximum parallel throughput. ### 1.2.3. Optimize[](#optimize "Permalink to this headline") After each round of application parallelization is complete, the developer can move to optimizing the implementation to improve performance. Since there are many possible optimizations that can be considered, having a good understanding of the needs of the application can help to make the process as smooth as possible. However, as with APOD as a whole, program optimization is an iterative process (identify an opportunity for optimization, apply and test the optimization, verify the speedup achieved, and repeat), meaning that it is not necessary for a programmer to spend large amounts of time memorizing the bulk of all possible optimization strategies prior to seeing good speedups. Instead, strategies can be applied incrementally as they are learned. Optimizations can be applied at various levels, from overlapping data transfers with computation all the way down to fine-tuning floating-point operation sequences. The available profiling tools are invaluable for guiding this process, as they can help suggest a next-best course of action for the developer’s optimization efforts and provide references into the relevant portions of the optimization section of this guide. ### 1.2.4. Deploy[](#deploy "Permalink to this headline") Having completed the GPU acceleration of one or more components of the application it is possible to compare the outcome with the original expectation. Recall that the initial _assess_ step allowed the developer to determine an upper bound for the potential speedup attainable by accelerating given hotspots. Before tackling other hotspots to improve the total speedup, the developer should consider taking the partially parallelized implementation and carry it through to production. This is important for a number of reasons; for example, it allows the user to profit from their investment as early as possible (the speedup may be partial but is still valuable), and it minimizes risk for the developer and the user by providing an evolutionary rather than revolutionary set of changes to the application. 1.3. Recommendations and Best Practices[](#recommendations-and-best-practices "Permalink to this headline") ------------------------------------------------------------------------------------------------------------- Throughout this guide, specific recommendations are made regarding the design and implementation of CUDA C++ code. These recommendations are categorized by priority, which is a blend of the effect of the recommendation and its scope. Actions that present substantial improvements for most CUDA applications have the highest priority, while small optimizations that affect only very specific situations are given a lower priority. Before implementing lower priority recommendations, it is good practice to make sure all higher priority recommendations that are relevant have already been applied. This approach will tend to provide the best results for the time invested and will avoid the trap of premature optimization. The criteria of benefit and scope for establishing priority will vary depending on the nature of the program. In this guide, they represent a typical case. Your code might reflect different priority factors. Regardless of this possibility, it is good practice to verify that no higher-priority recommendations have been overlooked before undertaking lower-priority items. Note Code samples throughout the guide omit error checking for conciseness. Production code should, however, systematically check the error code returned by each API call and check for failures in kernel launches by calling `cudaGetLastError()`. 1.4. Assessing Your Application[](#assessing-your-application "Permalink to this headline") --------------------------------------------------------------------------------------------- From supercomputers to mobile phones, modern processors increasingly rely on parallelism to provide performance. The core computational unit, which includes control, arithmetic, registers and typically some cache, is replicated some number of times and connected to memory via a network. As a result, all modern processors require parallel code in order to achieve good utilization of their computational power. While processors are evolving to expose more fine-grained parallelism to the programmer, many existing applications have evolved either as serial codes or as coarse-grained parallel codes (for example, where the data is decomposed into regions processed in parallel, with sub-regions shared using MPI). In order to profit from any modern processor architecture, GPUs included, the first steps are to assess the application to identify the hotspots, determine whether they can be parallelized, and understand the relevant workloads both now and in the future. 2\. Heterogeneous Computing[](#heterogeneous-computing "Permalink to this headline") ====================================================================================== CUDA programming involves running code on two different platforms concurrently: a _host_ system with one or more CPUs and one or more CUDA-enabled NVIDIA GPU _devices_. While NVIDIA GPUs are frequently associated with graphics, they are also powerful arithmetic engines capable of running thousands of lightweight threads in parallel. This capability makes them well suited to computations that can leverage parallel execution. However, the device is based on a distinctly different design from the host system, and it’s important to understand those differences and how they determine the performance of CUDA applications in order to use CUDA effectively. 2.1. Differences between Host and Device[](#differences-between-host-and-device "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- The primary differences are in threading model and in separate physical memories: Threading resources Execution pipelines on host systems can support a limited number of concurrent threads. For example, servers that have two 32 core processors can run only 64 threads concurrently (or small multiple of that if the CPUs support simultaneous multithreading). By comparison, the _smallest_ executable unit of parallelism on a CUDA device comprises 32 threads (termed a _warp_ of threads). Modern NVIDIA GPUs can support up to 2048 active threads concurrently per multiprocessor (see Features and Specifications of the CUDA C++ Programming Guide) On GPUs with 80 multiprocessors, this leads to more than 160,000 concurrently active threads. Threads Threads on a CPU are generally heavyweight entities. The operating system must swap threads on and off CPU execution channels to provide multithreading capability. Context switches (when two threads are swapped) are therefore slow and expensive. By comparison, threads on GPUs are extremely lightweight. In a typical system, thousands of threads are queued up for work (in warps of 32 threads each). If the GPU must wait on one warp of threads, it simply begins executing work on another. Because separate registers are allocated to all active threads, no swapping of registers or other state need occur when switching among GPU threads. Resources stay allocated to each thread until it completes its execution. In short, CPU cores are designed to _minimize latency_ for a small number of threads at a time each, whereas GPUs are designed to handle a large number of concurrent, lightweight threads in order to _maximize throughput_. RAM The host system and the device each have their own distinct attached physical memories [1](#fn1) . As the host and device memories are separated, items in the host memory must occasionally be communicated between device memory and host memory as described in [What Runs on a CUDA-Enabled Device?](#what-runs-on-cuda-enabled-device) . These are the primary hardware differences between CPU hosts and GPU devices with respect to parallel programming. Other differences are discussed as they arise elsewhere in this document. Applications composed with these differences in mind can treat the host and device together as a cohesive heterogeneous system wherein each processing unit is leveraged to do the kind of work it does best: sequential work on the host and parallel work on the device. 2.2. What Runs on a CUDA-Enabled Device?[](#what-runs-on-a-cuda-enabled-device "Permalink to this headline") -------------------------------------------------------------------------------------------------------------- The following issues should be considered when determining what parts of an application to run on the device: * The device is ideally suited for computations that can be run on numerous data elements simultaneously in parallel. This typically involves arithmetic on large data sets (such as matrices) where the same operation can be performed across thousands, if not millions, of elements at the same time. This is a requirement for good performance on CUDA: the software must use a large number (generally thousands or tens of thousands) of concurrent threads. The support for running numerous threads in parallel derives from CUDA’s use of a lightweight threading model described above. * To use CUDA, data values must be transferred from the host to the device. These transfers are costly in terms of performance and should be minimized. (See [Data Transfer Between Host and Device](#data-transfer-between-host-and-device) .) This cost has several ramifications: * The complexity of operations should justify the cost of moving data to and from the device. Code that transfers data for brief use by a small number of threads will see little or no performance benefit. The ideal scenario is one in which many threads perform a substantial amount of work. For example, transferring two matrices to the device to perform a matrix addition and then transferring the results back to the host will not realize much performance benefit. The issue here is the number of operations performed per data element transferred. For the preceding procedure, assuming matrices of size NxN, there are N2 operations (additions) and 3N2 elements transferred, so the ratio of operations to elements transferred is 1:3 or O(1). Performance benefits can be more readily achieved when this ratio is higher. For example, a matrix multiplication of the same matrices requires N3 operations (multiply-add), so the ratio of operations to elements transferred is O(N), in which case the larger the matrix the greater the performance benefit. The types of operations are an additional factor, as additions have different complexity profiles than, for example, trigonometric functions. It is important to include the overhead of transferring data to and from the device in determining whether operations should be performed on the host or on the device. * Data should be kept on the device as long as possible. Because transfers should be minimized, programs that run multiple kernels on the same data should favor leaving the data on the device between kernel calls, rather than transferring intermediate results to the host and then sending them back to the device for subsequent calculations. So, in the previous example, had the two matrices to be added already been on the device as a result of some previous calculation, or if the results of the addition would be used in some subsequent calculation, the matrix addition should be performed locally on the device. This approach should be used even if one of the steps in a sequence of calculations could be performed faster on the host. Even a relatively slow kernel may be advantageous if it avoids one or more transfers between host and device memory. [Data Transfer Between Host and Device](#data-transfer-between-host-and-device) provides further details, including the measurements of bandwidth between the host and the device versus within the device proper. * For best performance, there should be some coherence in memory access by adjacent threads running on the device. Certain memory access patterns enable the hardware to coalesce groups of reads or writes of multiple data items into one operation. Data that cannot be laid out so as to enable _coalescing_, or that doesn’t have enough locality to use the L1 or texture caches effectively, will tend to see lesser speedups when used in computations on GPUs. A noteworthy exception to this are completely random memory access patterns. In general, they should be avoided, because compared to peak capabilities any architecture processes these memory access patterns at a low efficiency. However, compared to cache based architectures, like CPUs, latency hiding architectures, like GPUs, tend to cope better with completely random memory access patterns. [1](#id3) On Systems on a Chip with integrated GPUs, such as NVIDIA® Tegra®, host and device memory are physically the same, but there is still a logical distinction between host and device memory. See the [Application Note on CUDA for Tegra](https://docs.nvidia.com/cuda/cuda-for-tegra-appnote) for details. 3\. Application Profiling[](#application-profiling "Permalink to this headline") ================================================================================== 3.1. Profile[](#profile "Permalink to this headline") ------------------------------------------------------- Many codes accomplish a significant portion of the work with a relatively small amount of code. Using a profiler, the developer can identify such hotspots and start to compile a list of candidates for parallelization. ### 3.1.1. Creating the Profile[](#creating-the-profile "Permalink to this headline") There are many possible approaches to profiling the code, but in all cases the objective is the same: to identify the function or functions in which the application is spending most of its execution time. Note **High Priority:** To maximize developer productivity, profile the application to determine hotspots and bottlenecks. The most important consideration with any profiling activity is to ensure that the workload is realistic - i.e., that information gained from the test and decisions based upon that information are relevant to real data. Using unrealistic workloads can lead to sub-optimal results and wasted effort both by causing developers to optimize for unrealistic problem sizes and by causing developers to concentrate on the wrong functions. There are a number of tools that can be used to generate the profile. The following example is based on `gprof`, which is an open-source profiler for Linux platforms from the GNU Binutils collection. $ gcc -O2 -g -pg myprog.c $ gprof ./a.out > profile.txt Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ms/call ms/call name 33.34 0.02 0.02 7208 0.00 0.00 genTimeStep 16.67 0.03 0.01 240 0.04 0.12 calcStats 16.67 0.04 0.01 8 1.25 1.25 calcSummaryData 16.67 0.05 0.01 7 1.43 1.43 write 16.67 0.06 0.01 mcount 0.00 0.06 0.00 236 0.00 0.00 tzset 0.00 0.06 0.00 192 0.00 0.00 tolower 0.00 0.06 0.00 47 0.00 0.00 strlen 0.00 0.06 0.00 45 0.00 0.00 strchr 0.00 0.06 0.00 1 0.00 50.00 main 0.00 0.06 0.00 1 0.00 0.00 memcpy 0.00 0.06 0.00 1 0.00 10.11 print 0.00 0.06 0.00 1 0.00 0.00 profil 0.00 0.06 0.00 1 0.00 50.00 report ### 3.1.2. Identifying Hotspots[](#identifying-hotspots "Permalink to this headline") In the example above, we can clearly see that the function `genTimeStep()` takes one-third of the total running time of the application. This should be our first candidate function for parallelization. [Understanding Scaling](#understanding-scaling) discusses the potential benefit we might expect from such parallelization. It is worth noting that several of the other functions in the above example also take up a significant portion of the overall running time, such as `calcStats()` and `calcSummaryData()`. Parallelizing these functions as well should increase our speedup potential. However, since APOD is a cyclical process, we might opt to parallelize these functions in a subsequent APOD pass, thereby limiting the scope of our work in any given pass to a smaller set of incremental changes. ### 3.1.3. Understanding Scaling[](#understanding-scaling "Permalink to this headline") The amount of performance benefit an application will realize by running on CUDA depends entirely on the extent to which it can be parallelized. Code that cannot be sufficiently parallelized should run on the host, unless doing so would result in excessive transfers between the host and the device. Note **High Priority:** To get the maximum benefit from CUDA, focus first on finding ways to parallelize sequential code. By understanding how applications can scale it is possible to set expectations and plan an incremental parallelization strategy. [Strong Scaling and Amdahl’s Law](#strong-scaling-and-amdahls-law) describes strong scaling, which allows us to set an upper bound for the speedup with a fixed problem size. [Weak Scaling and Gustafson’s Law](#weak-scaling-and-gustafsons-law) describes weak scaling, where the speedup is attained by growing the problem size. In many applications, a combination of strong and weak scaling is desirable. #### 3.1.3.1. Strong Scaling and Amdahl’s Law[](#strong-scaling-and-amdahl-s-law "Permalink to this headline") Strong scaling is a measure of how, for a fixed overall problem size, the time to solution decreases as more processors are added to a system. An application that exhibits linear strong scaling has a speedup equal to the number of processors used. Strong scaling is usually equated with Amdahl’s Law, which specifies the maximum speedup that can be expected by parallelizing portions of a serial program. Essentially, it states that the maximum speedup _S_ of a program is: \\(S = \\frac{1}{(1 - P) + \\frac{P}{N}}\\) Here _P_ is the fraction of the total serial execution time taken by the portion of code that can be parallelized and _N_ is the number of processors over which the parallel portion of the code runs. The larger _N_ is(that is, the greater the number of processors), the smaller the _P/N_ fraction. It can be simpler to view _N_ as a very large number, which essentially transforms the equation into \\(S = 1/(1 - P)\\). Now, if 3/4 of the running time of a sequential program is parallelized, the maximum speedup over serial code is 1 / (1 - 3/4) = 4. In reality, most applications do not exhibit perfectly linear strong scaling, even if they do exhibit some degree of strong scaling. For most purposes, the key point is that the larger the parallelizable portion _P_ is, the greater the potential speedup. Conversely, if _P_ is a small number (meaning that the application is not substantially parallelizable), increasing the number of processors _N_ does little to improve performance. Therefore, to get the largest speedup for a fixed problem size, it is worthwhile to spend effort on increasing _P_, maximizing the amount of code that can be parallelized. #### 3.1.3.2. Weak Scaling and Gustafson’s Law[](#weak-scaling-and-gustafson-s-law "Permalink to this headline") Weak scaling is a measure of how the time to solution changes as more processors are added to a system with a fixed problem size _per processor_; i.e., where the overall problem size increases as the number of processors is increased. Weak scaling is often equated with Gustafson’s Law, which states that in practice, the problem size scales with the number of processors. Because of this, the maximum speedup _S_ of a program is: \\(S = N + (1 - P)(1 - N)\\) Here _P_ is the fraction of the total serial execution time taken by the portion of code that can be parallelized and _N_ is the number of processors over which the parallel portion of the code runs. Another way of looking at Gustafson’s Law is that it is not the problem size that remains constant as we scale up the system but rather the execution time. Note that Gustafson’s Law assumes that the ratio of serial to parallel execution remains constant, reflecting additional cost in setting up and handling the larger problem. #### 3.1.3.3. Applying Strong and Weak Scaling[](#applying-strong-and-weak-scaling "Permalink to this headline") Understanding which type of scaling is most applicable to an application is an important part of estimating speedup. For some applications the problem size will remain constant and hence only strong scaling is applicable. An example would be modeling how two molecules interact with each other, where the molecule sizes are fixed. For other applications, the problem size will grow to fill the available processors. Examples include modeling fluids or structures as meshes or grids and some Monte Carlo simulations, where increasing the problem size provides increased accuracy. Having understood the application profile, the developer should understand how the problem size would change if the computational performance changes and then apply either Amdahl’s or Gustafson’s Law to determine an upper bound for the speedup. 4\. Parallelizing Your Application[](#parallelizing-your-application "Permalink to this headline") ==================================================================================================== Having identified the hotspots and having done the basic exercises to set goals and expectations, the developer needs to parallelize the code. Depending on the original code, this can be as simple as calling into an existing GPU-optimized library such as `cuBLAS`, `cuFFT`, or `Thrust`, or it could be as simple as adding a few preprocessor directives as hints to a parallelizing compiler. On the other hand, some applications’ designs will require some amount of refactoring to expose their inherent parallelism. As even CPU architectures require exposing this parallelism in order to improve or simply maintain the performance of sequential applications, the CUDA family of parallel programming languages (CUDA C++, CUDA Fortran, etc.) aims to make the expression of this parallelism as simple as possible, while simultaneously enabling operation on CUDA-capable GPUs designed for maximum parallel throughput. 5\. Getting Started[](#getting-started "Permalink to this headline") ====================================================================== There are several key strategies for parallelizing sequential code. While the details of how to apply these strategies to a particular application is a complex and problem-specific topic, the general themes listed here apply regardless of whether we are parallelizing code to run on for multicore CPUs or for use on CUDA GPUs. 5.1. Parallel Libraries[](#parallel-libraries "Permalink to this headline") ----------------------------------------------------------------------------- The most straightforward approach to parallelizing an application is to leverage existing libraries that take advantage of parallel architectures on our behalf. The CUDA Toolkit includes a number of such libraries that have been fine-tuned for NVIDIA CUDA GPUs, such as `cuBLAS`, `cuFFT`, and so on. The key here is that libraries are most useful when they match well with the needs of the application. Applications already using other BLAS libraries can often quite easily switch to `cuBLAS`, for example, whereas applications that do little to no linear algebra will have little use for `cuBLAS`. The same goes for other CUDA Toolkit libraries: `cuFFT` has an interface similar to that of `FFTW`, etc. Also of note is the Thrust library, which is a parallel C++ template library similar to the C++ Standard Template Library. Thrust provides a rich collection of data parallel primitives such as scan, sort, and reduce, which can be composed together to implement complex algorithms with concise, readable source code. By describing your computation in terms of these high-level abstractions you provide Thrust with the freedom to select the most efficient implementation automatically. As a result, Thrust can be utilized in rapid prototyping of CUDA applications, where programmer productivity matters most, as well as in production, where robustness and absolute performance are crucial. 5.2. Parallelizing Compilers[](#parallelizing-compilers "Permalink to this headline") --------------------------------------------------------------------------------------- Another common approach to parallelization of sequential codes is to make use of parallelizing compilers. Often this means the use of directives-based approaches, where the programmer uses a pragma or other similar notation to provide hints to the compiler about where parallelism can be found without needing to modify or adapt the underlying code itself. By exposing parallelism to the compiler, directives allow the compiler to do the detailed work of mapping the computation onto the parallel architecture. The OpenACC standard provides a set of compiler directives to specify loops and regions of code in standard C, C++ and Fortran that should be offloaded from a host CPU to an attached accelerator such as a CUDA GPU. The details of managing the accelerator device are handled implicitly by an OpenACC-enabled compiler and runtime. See [http://www.openacc.org/](http://www.openacc.org/) for details. 5.3. Coding to Expose Parallelism[](#coding-to-expose-parallelism "Permalink to this headline") ------------------------------------------------------------------------------------------------- For applications that need additional functionality or performance beyond what existing parallel libraries or parallelizing compilers can provide, parallel programming languages such as CUDA C++ that integrate seamlessly with existing sequential code are essential. Once we have located a hotspot in our application’s profile assessment and determined that custom code is the best approach, we can use CUDA C++ to expose the parallelism in that portion of our code as a CUDA kernel. We can then launch this kernel onto the GPU and retrieve the results without requiring major rewrites to the rest of our application. This approach is most straightforward when the majority of the total running time of our application is spent in a few relatively isolated portions of the code. More difficult to parallelize are applications with a very flat profile - i.e., applications where the time spent is spread out relatively evenly across a wide portion of the code base. For the latter variety of application, some degree of code refactoring to expose the inherent parallelism in the application might be necessary, but keep in mind that this refactoring work will tend to benefit all future architectures, CPU and GPU alike, so it is well worth the effort should it become necessary. 6\. Getting the Right Answer[](#getting-the-right-answer "Permalink to this headline") ======================================================================================== Obtaining the right answer is clearly the principal goal of all computation. On parallel systems, it is possible to run into difficulties not typically found in traditional serial-oriented programming. These include threading issues, unexpected values due to the way floating-point values are computed, and challenges arising from differences in the way CPU and GPU processors operate. This chapter examines issues that can affect the correctness of returned data and points to appropriate solutions. 6.1. Verification[](#verification "Permalink to this headline") ----------------------------------------------------------------- ### 6.1.1. Reference Comparison[](#reference-comparison "Permalink to this headline") A key aspect of correctness verification for modifications to any existing program is to establish some mechanism whereby previous known-good reference outputs from representative inputs can be compared to new results. After each change is made, ensure that the results match using whatever criteria apply to the particular algorithm. Some will expect bitwise identical results, which is not always possible, especially where floating-point arithmetic is concerned; see [Numerical Accuracy and Precision](#numerical-accuracy-and-precision) regarding numerical accuracy. For other algorithms, implementations may be considered correct if they match the reference within some small epsilon. Note that the process used for validating numerical results can easily be extended to validate performance results as well. We want to ensure that each change we make is correct _and_ that it improves performance (and by how much). Checking these things frequently as an integral part of our cyclical APOD process will help ensure that we achieve the desired results as rapidly as possible. ### 6.1.2. Unit Testing[](#unit-testing "Permalink to this headline") A useful counterpart to the reference comparisons described above is to structure the code itself in such a way that is readily verifiable at the unit level. For example, we can write our CUDA kernels as a collection of many short `__device__` functions rather than one large monolithic `__global__` function; each device function can be tested independently before hooking them all together. For example, many kernels have complex addressing logic for accessing memory in addition to their actual computation. If we validate our addressing logic separately prior to introducing the bulk of the computation, then this will simplify any later debugging efforts. (Note that the CUDA compiler considers any device code that does not contribute to a write to global memory as dead code subject to elimination, so we must at least write _something_ out to global memory as a result of our addressing logic in order to successfully apply this strategy.) Going a step further, if most functions are defined as `__host__ __device__` rather than just `__device__` functions, then these functions can be tested on both the CPU and the GPU, thereby increasing our confidence that the function is correct and that there will not be any unexpected differences in the results. If there _are_ differences, then those differences will be seen early and can be understood in the context of a simple function. As a useful side effect, this strategy will allow us a means to reduce code duplication should we wish to include both CPU and GPU execution paths in our application: if the bulk of the work of our CUDA kernels is done in `__host__ __device__` functions, we can easily call those functions from both the host code _and_ the device code without duplication. 6.2. Debugging[](#debugging "Permalink to this headline") ----------------------------------------------------------- CUDA-GDB is a port of the GNU Debugger that runs on Linux and Mac; see: [https://developer.nvidia.com/cuda-gdb](https://developer.nvidia.com/cuda-gdb) . The NVIDIA Nsight Visual Studio Edition is available as a free plugin for Microsoft Visual Studio; see: [https://developer.nvidia.com/nsight-visual-studio-edition](https://developer.nvidia.com/nsight-visual-studio-edition) . Several third-party debuggers support CUDA debugging as well; see: [https://developer.nvidia.com/debugging-solutions](https://developer.nvidia.com/debugging-solutions) for more details. 6.3. Numerical Accuracy and Precision[](#numerical-accuracy-and-precision "Permalink to this headline") --------------------------------------------------------------------------------------------------------- Incorrect or unexpected results arise principally from issues of floating-point accuracy due to the way floating-point values are computed and stored. The following sections explain the principal items of interest. Other peculiarities of floating-point arithmetic are presented in Features and Technical Specifications of the CUDA C++ Programming Guide as well as in a whitepaper and accompanying webinar on floating-point precision and performance available from [https://developer.nvidia.com/content/precision-performance-floating-point-and-ieee-754-compliance-nvidia-gpus](https://developer.nvidia.com/content/precision-performance-floating-point-and-ieee-754-compliance-nvidia-gpus) . ### 6.3.1. Single vs. Double Precision[](#single-vs-double-precision "Permalink to this headline") Devices of [CUDA Compute Capability](#cuda-compute-capability) 1.3 and higher provide native support for double-precision floating-point values (that is, values 64 bits wide). Results obtained using double-precision arithmetic will frequently differ from the same operation performed via single-precision arithmetic due to the greater precision of the former and due to rounding issues. Therefore, it is important to be sure to compare values of like precision and to express the results within a certain tolerance rather than expecting them to be exact. ### 6.3.2. Floating Point Math Is Not Associative[](#floating-point-math-is-not-associative "Permalink to this headline") Each floating-point arithmetic operation involves a certain amount of rounding. Consequently, the order in which arithmetic operations are performed is important. If A, B, and C are floating-point values, (A+B)+C is not guaranteed to equal A+(B+C) as it is in symbolic math. When you parallelize computations, you potentially change the order of operations and therefore the parallel results might not match sequential results. This limitation is not specific to CUDA, but an inherent part of parallel computation on floating-point values. ### 6.3.3. IEEE 754 Compliance[](#ieee-754-compliance "Permalink to this headline") All CUDA compute devices follow the IEEE 754 standard for binary floating-point representation, with some small exceptions. These exceptions, which are detailed in Features and Technical Specifications of the CUDA C++ Programming Guide, can lead to results that differ from IEEE 754 values computed on the host system. One of the key differences is the fused multiply-add (FMA) instruction, which combines multiply-add operations into a single instruction execution. Its result will often differ slightly from results obtained by doing the two operations separately. ### 6.3.4. x86 80-bit Computations[](#x86-80-bit-computations "Permalink to this headline") x86 processors can use an 80-bit _double extended precision_ math when performing floating-point calculations. The results of these calculations can frequently differ from pure 64-bit operations performed on the CUDA device. To get a closer match between values, set the x86 host processor to use regular double or single precision (64 bits and 32 bits, respectively). This is done with the `FLDCW` x86 assembly instruction or the equivalent operating system API. 7\. Optimizing CUDA Applications[](#optimizing-cuda-applications "Permalink to this headline") ================================================================================================ After each round of application parallelization is complete, the developer can move to optimizing the implementation to improve performance. Since there are many possible optimizations that can be considered, having a good understanding of the needs of the application can help to make the process as smooth as possible. However, as with APOD as a whole, program optimization is an iterative process (identify an opportunity for optimization, apply and test the optimization, verify the speedup achieved, and repeat), meaning that it is not necessary for a programmer to spend large amounts of time memorizing the bulk of all possible optimization strategies prior to seeing good speedups. Instead, strategies can be applied incrementally as they are learned. Optimizations can be applied at various levels, from overlapping data transfers with computation all the way down to fine-tuning floating-point operation sequences. The available profiling tools are invaluable for guiding this process, as they can help suggest a next-best course of action for the developer’s optimization efforts and provide references into the relevant portions of the optimization section of this guide. 8\. Performance Metrics[](#performance-metrics "Permalink to this headline") ============================================================================== When attempting to optimize CUDA code, it pays to know how to measure performance accurately and to understand the role that bandwidth plays in performance measurement. This chapter discusses how to correctly measure performance using CPU timers and CUDA events. It then explores how bandwidth affects performance metrics and how to mitigate some of the challenges it poses. 8.1. Timing[](#timing "Permalink to this headline") ----------------------------------------------------- CUDA calls and kernel executions can be timed using either CPU or GPU timers. This section examines the functionality, advantages, and pitfalls of both approaches. ### 8.1.1. Using CPU Timers[](#using-cpu-timers "Permalink to this headline") Any CPU timer can be used to measure the elapsed time of a CUDA call or kernel execution. The details of various CPU timing approaches are outside the scope of this document, but developers should always be aware of the resolution their timing calls provide. When using CPU timers, it is critical to remember that many CUDA API functions are asynchronous; that is, they return control back to the calling CPU thread prior to completing their work. All kernel launches are asynchronous, as are memory-copy functions with the `Async` suffix on their names. Therefore, to accurately measure the elapsed time for a particular call or sequence of CUDA calls, it is necessary to synchronize the CPU thread with the GPU by calling `cudaDeviceSynchronize()` immediately before starting and stopping the CPU timer. `cudaDeviceSynchronize()`blocks the calling CPU thread until all CUDA calls previously issued by the thread are completed. Although it is also possible to synchronize the CPU thread with a particular stream or event on the GPU, these synchronization functions are not suitable for timing code in streams other than the default stream. `cudaStreamSynchronize()` blocks the CPU thread until all CUDA calls previously issued into the given stream have completed. `cudaEventSynchronize()` blocks until a given event in a particular stream has been recorded by the GPU. Because the driver may interleave execution of CUDA calls from other non-default streams, calls in other streams may be included in the timing. Because the default stream, stream 0, exhibits serializing behavior for work on the device (an operation in the default stream can begin only after all preceding calls in any stream have completed; and no subsequent operation in any stream can begin until it finishes), these functions can be used reliably for timing in the default stream. Be aware that CPU-to-GPU synchronization points such as those mentioned in this section imply a stall in the GPU’s processing pipeline and should thus be used sparingly to minimize their performance impact. ### 8.1.2. Using CUDA GPU Timers[](#using-cuda-gpu-timers "Permalink to this headline") The CUDA event API provides calls that create and destroy events, record events (including a timestamp), and convert timestamp differences into a floating-point value in milliseconds. [How to time code using CUDA events](#how-to-time-code-using-cuda-events-figure) illustrates their use. How to time code using CUDA events cudaEvent\_t start, stop; float time; cudaEventCreate(&start); cudaEventCreate(&stop); cudaEventRecord( start, 0 ); kernel<<>> ( d\_odata, d\_idata, size\_x, size\_y, NUM\_REPS); cudaEventRecord( stop, 0 ); cudaEventSynchronize( stop ); cudaEventElapsedTime( &time, start, stop ); cudaEventDestroy( start ); cudaEventDestroy( stop ); Here `cudaEventRecord()` is used to place the `start` and `stop` events into the default stream, stream 0. The device will record a timestamp for the event when it reaches that event in the stream. The `cudaEventElapsedTime()` function returns the time elapsed between the recording of the `start` and `stop` events. This value is expressed in milliseconds and has a resolution of approximately half a microsecond. Like the other calls in this listing, their specific operation, parameters, and return values are described in the _CUDA Toolkit Reference Manual_. Note that the timings are measured on the GPU clock, so the timing resolution is operating-system-independent. 8.2. Bandwidth[](#bandwidth "Permalink to this headline") ----------------------------------------------------------- Bandwidth - the rate at which data can be transferred - is one of the most important gating factors for performance. Almost all changes to code should be made in the context of how they affect bandwidth. As described in [Memory Optimizations](#memory-optimizations) of this guide, bandwidth can be dramatically affected by the choice of memory in which data is stored, how the data is laid out and the order in which it is accessed, as well as other factors. To measure performance accurately, it is useful to calculate theoretical and effective bandwidth. When the latter is much lower than the former, design or implementation details are likely to reduce bandwidth, and it should be the primary goal of subsequent optimization efforts to increase it. Note **High Priority:** Use the effective bandwidth of your computation as a metric when measuring performance and optimization benefits. ### 8.2.1. Theoretical Bandwidth Calculation[](#theoretical-bandwidth-calculation "Permalink to this headline") Theoretical bandwidth can be calculated using hardware specifications available in the product literature. For example, the NVIDIA Tesla V100 uses HBM2 (double data rate) RAM with a memory clock rate of 877 MHz and a 4096-bit-wide memory interface. Using these data items, the peak theoretical memory bandwidth of the NVIDIA Tesla V100 is 898 GB/s: \\(\\left. \\left( 0.877 \\times 10^{9} \\right. \\times (4096/8) \\times 2 \\right) \\div 10^{9} = 898\\text{GB/s}\\) In this calculation, the memory clock rate is converted in to Hz, multiplied by the interface width (divided by 8, to convert bits to bytes) and multiplied by 2 due to the double data rate. Finally, this product is divided by 109 to convert the result to GB/s. Note Some calculations use 10243 instead of 109 for the final calculation. In such a case, the bandwidth would be 836.4 GiB/s. It is important to use the same divisor when calculating theoretical and effective bandwidth so that the comparison is valid. Note On GPUs with GDDR memory with ECC enabled the available DRAM is reduced by 6.25% to allow for the storage of ECC bits. Fetching ECC bits for each memory transaction also reduced the effective bandwidth by approximately 20% compared to the same GPU with ECC disabled, though the exact impact of ECC on bandwidth can be higher and depends on the memory access pattern. HBM2 memories, on the other hand, provide dedicated ECC resources, allowing overhead-free ECC protection.[2](#fn2) ### 8.2.2. Effective Bandwidth Calculation[](#effective-bandwidth-calculation "Permalink to this headline") Effective bandwidth is calculated by timing specific program activities and by knowing how data is accessed by the program. To do so, use this equation: \\(\\text{Effective\\ bandwidth} = \\left( {\\left( B\_{r} + B\_{w} \\right) \\div 10^{9}} \\right) \\div \\text{time}\\) Here, the effective bandwidth is in units of GB/s, Br is the number of bytes read per kernel, Bw is the number of bytes written per kernel, and time is given in seconds. For example, to compute the effective bandwidth of a 2048 x 2048 matrix copy, the following formula could be used: \\(\\text{Effective\\ bandwidth} = \\left( {\\left( 2048^{2} \\times 4 \\times 2 \\right) \\div 10^{9}} \\right) \\div \\text{time}\\) The number of elements is multiplied by the size of each element (4 bytes for a float), multiplied by 2 (because of the read _and_ write), divided by 109 (or 1,0243) to obtain GB of memory transferred. This number is divided by the time in seconds to obtain GB/s. ### 8.2.3. Throughput Reported by Visual Profiler[](#throughput-reported-by-visual-profiler "Permalink to this headline") For devices with [compute capability](#cuda-compute-capability) of 2.0 or greater, the Visual Profiler can be used to collect several different memory throughput measures. The following throughput metrics can be displayed in the Details or Detail Graphs view: * Requested Global Load Throughput * Requested Global Store Throughput * Global Load Throughput * Global Store Throughput * DRAM Read Throughput * DRAM Write Throughput The Requested Global Load Throughput and Requested Global Store Throughput values indicate the global memory throughput requested by the kernel and therefore correspond to the effective bandwidth obtained by the calculation shown under [Effective Bandwidth Calculation](#effective-bandwidth-calculation) . Because the minimum memory transaction size is larger than most word sizes, the actual memory throughput required for a kernel can include the transfer of data not used by the kernel. For global memory accesses, this actual throughput is reported by the Global Load Throughput and Global Store Throughput values. It’s important to note that both numbers are useful. The actual memory throughput shows how close the code is to the hardware limit, and a comparison of the effective or requested bandwidth to the actual bandwidth presents a good estimate of how much bandwidth is wasted by suboptimal coalescing of memory accesses (see [Coalesced Access to Global Memory](#coalesced-access-to-global-memory) ). For global memory accesses, this comparison of requested memory bandwidth to actual memory bandwidth is reported by the Global Memory Load Efficiency and Global Memory Store Efficiency metrics. [2](#id24) As an exception, scattered writes to HBM2 see some overhead from ECC but much less than the overhead with similar access patterns on ECC-protected GDDR5 memory. 9\. Memory Optimizations[](#memory-optimizations "Permalink to this headline") ================================================================================ Memory optimizations are the most important area for performance. The goal is to maximize the use of the hardware by maximizing bandwidth. Bandwidth is best served by using as much fast memory and as little slow-access memory as possible. This chapter discusses the various kinds of memory on the host and device and how best to set up data items to use the memory effectively. 9.1. Data Transfer Between Host and Device[](#data-transfer-between-host-and-device "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------- The peak theoretical bandwidth between the device memory and the GPU is much higher (898 GB/s on the NVIDIA Tesla V100, for example) than the peak theoretical bandwidth between host memory and device memory (16 GB/s on the PCIe x16 Gen3). Hence, for best overall application performance, it is important to minimize data transfer between the host and the device, even if that means running kernels on the GPU that do not demonstrate any speedup compared with running them on the host CPU. Note **High Priority:** Minimize data transfer between the host and the device, even if it means running some kernels on the device that do not show performance gains when compared with running them on the host CPU. Intermediate data structures should be created in device memory, operated on by the device, and destroyed without ever being mapped by the host or copied to host memory. Also, because of the overhead associated with each transfer, batching many small transfers into one larger transfer performs significantly better than making each transfer separately, even if doing so requires packing non-contiguous regions of memory into a contiguous buffer and then unpacking after the transfer. Finally, higher bandwidth between the host and the device is achieved when using _page-locked_ (or _pinned_) memory, as discussed in the CUDA C++ Programming Guide and the [Pinned Memory](#pinned-memory) section of this document. ### 9.1.1. Pinned Memory[](#pinned-memory "Permalink to this headline") Page-locked or pinned memory transfers attain the highest bandwidth between the host and the device. On PCIe x16 Gen3 cards, for example, pinned memory can attain roughly 12 GB/s transfer rates. Pinned memory is allocated using the `cudaHostAlloc()` functions in the Runtime API. The `bandwidthTest` CUDA Sample shows how to use these functions as well as how to measure memory transfer performance. For regions of system memory that have already been pre-allocated, `cudaHostRegister()` can be used to pin the memory on-the-fly without the need to allocate a separate buffer and copy the data into it. Pinned memory should not be overused. Excessive use can reduce overall system performance because pinned memory is a scarce resource, but how much is too much is difficult to know in advance. Furthermore, the pinning of system memory is a heavyweight operation compared to most normal system memory allocations, so as with all optimizations, test the application and the systems it runs on for optimal performance parameters. ### 9.1.2. Asynchronous and Overlapping Transfers with Computation[](#asynchronous-and-overlapping-transfers-with-computation "Permalink to this headline") Data transfers between the host and the device using `cudaMemcpy()` are blocking transfers; that is, control is returned to the host thread only after the data transfer is complete. The `cudaMemcpyAsync()` function is a non-blocking variant of `cudaMemcpy()` in which control is returned immediately to the host thread. In contrast with `cudaMemcpy()`, the asynchronous transfer version _requires_ pinned host memory (see [Pinned Memory](#pinned-memory) ), and it contains an additional argument, a stream ID. A _stream_ is simply a sequence of operations that are performed in order on the device. Operations in different streams can be interleaved and in some cases overlapped - a property that can be used to hide data transfers between the host and the device. Asynchronous transfers enable overlap of data transfers with computation in two different ways. On all CUDA-enabled devices, it is possible to overlap host computation with asynchronous data transfers and with device computations. For example, [Asynchronous and Overlapping Transfers with Computation](#asynchronous-transfers-and-overlapping-transfers-with-computation) demonstrates how host computation in the routine `cpuFunction()` is performed while data is transferred to the device and a kernel using the device is executed. Overlapping computation and data transfers cudaMemcpyAsync(a\_d, a\_h, size, cudaMemcpyHostToDevice, 0); kernel<<>>(a\_d); cpuFunction(); The last argument to the `cudaMemcpyAsync()` function is the stream ID, which in this case uses the default stream, stream 0. The kernel also uses the default stream, and it will not begin execution until the memory copy completes; therefore, no explicit synchronization is needed. Because the memory copy and the kernel both return control to the host immediately, the host function `cpuFunction()` overlaps their execution. In [Asynchronous and Overlapping Transfers with Computation](#asynchronous-transfers-and-overlapping-transfers-with-computation) , the memory copy and kernel execution occur sequentially. On devices that are capable of concurrent copy and compute, it is possible to overlap kernel execution on the device with data transfers between the host and the device. Whether a device has this capability is indicated by the `asyncEngineCount` field of the `cudaDeviceProp` structure (or listed in the output of the `deviceQuery` CUDA Sample). On devices that have this capability, the overlap once again requires pinned host memory, and, in addition, the data transfer and kernel must use different, non-default streams (streams with non-zero stream IDs). Non-default streams are required for this overlap because memory copy, memory set functions, and kernel calls that use the default stream begin only after all preceding calls on the device (in any stream) have completed, and no operation on the device (in any stream) commences until they are finished. [Asynchronous and Overlapping Transfers with Computation](#asynchronous-transfers-and-overlapping-transfers-with-computation) illustrates the basic technique. Concurrent copy and execute cudaStreamCreate(&stream1); cudaStreamCreate(&stream2); cudaMemcpyAsync(a\_d, a\_h, size, cudaMemcpyHostToDevice, stream1); kernel<<>>(otherData\_d); In this code, two streams are created and used in the data transfer and kernel executions as specified in the last arguments of the `cudaMemcpyAsync` call and the kernel’s execution configuration. [Asynchronous and Overlapping Transfers with Computation](#asynchronous-transfers-and-overlapping-transfers-with-computation) demonstrates how to overlap kernel execution with asynchronous data transfer. This technique could be used when the data dependency is such that the data can be broken into chunks and transferred in multiple stages, launching multiple kernels to operate on each chunk as it arrives. [Sequential copy and execute](#sequential-copy-and-execute) and [Staged concurrent copy and execute](#staged-concurrent-copy-and-execute) demonstrate this. They produce equivalent results. The first segment shows the reference sequential implementation, which transfers and operates on an array of _N_ floats (where _N_ is assumed to be evenly divisible by nThreads). Sequential copy and execute cudaMemcpy(a\_d, a\_h, N\*sizeof(float), dir); kernel<<>>(a\_d); [Staged concurrent copy and execute](#staged-concurrent-copy-and-execute) shows how the transfer and kernel execution can be broken up into nStreams stages. This approach permits some overlapping of the data transfer and execution. Staged concurrent copy and execute size\=N\*sizeof(float)/nStreams; for (i\=0; i>>(a\_d+offset); } (In [Staged concurrent copy and execute](#staged-concurrent-copy-and-execute) , it is assumed that _N_ is evenly divisible by `nThreads*nStreams`.) Because execution within a stream occurs sequentially, none of the kernels will launch until the data transfers in their respective streams complete. Current GPUs can simultaneously process asynchronous data transfers and execute kernels. GPUs with a single copy engine can perform one asynchronous data transfer and execute kernels whereas GPUs with two copy engines can simultaneously perform one asynchronous data transfer from the host to the device, one asynchronous data transfer from the device to the host, and execute kernels. The number of copy engines on a GPU is given by the `asyncEngineCount` field of the `cudaDeviceProp` structure, which is also listed in the output of the `deviceQuery` CUDA Sample. (It should be mentioned that it is not possible to overlap a blocking transfer with an asynchronous transfer, because the blocking transfer occurs in the default stream, so it will not begin until all previous CUDA calls complete. It will not allow any other CUDA call to begin until it has completed.) A diagram depicting the timeline of execution for the two code segments is shown in [Figure 1](#timeline-comparison-for-copy-and-kernel-execution-figure) , and `nStreams` is equal to 4 for [Staged concurrent copy and execute](#staged-concurrent-copy-and-execute) in the bottom half of the figure. ![Timeline comparison for copy and kernel execution](_images/timeline-comparison-for-copy-and-kernel-execution.png) Figure 1 Timeline comparison for copy and kernel execution[](#timeline-comparison-for-copy-and-kernel-execution-figure "Permalink to this image") Top Sequential Bottom Concurrent For this example, it is assumed that the data transfer and kernel execution times are comparable. In such cases, and when the execution time (_tE_) exceeds the transfer time (_tT_), a rough estimate for the overall time is _tE + tT/nStreams_ for the staged version versus _tE + tT_ for the sequential version. If the transfer time exceeds the execution time, a rough estimate for the overall time is _tT + tE/nStreams_. ### 9.1.3. Zero Copy[](#zero-copy "Permalink to this headline") _Zero copy_ is a feature that was added in version 2.2 of the CUDA Toolkit. It enables GPU threads to directly access host memory. For this purpose, it requires mapped pinned (non-pageable) memory. On integrated GPUs (i.e., GPUs with the integrated field of the CUDA device properties structure set to 1), mapped pinned memory is always a performance gain because it avoids superfluous copies as integrated GPU and CPU memory are physically the same. On discrete GPUs, mapped pinned memory is advantageous only in certain cases. Because the data is not cached on the GPU, mapped pinned memory should be read or written only once, and the global loads and stores that read and write the memory should be coalesced. Zero copy can be used in place of streams because kernel-originated data transfers automatically overlap kernel execution without the overhead of setting up and determining the optimal number of streams. Note **Low Priority:** Use zero-copy operations on integrated GPUs for CUDA Toolkit version 2.2 and later. The host code in [Zero-copy host code](#zero-copy-host-code) shows how zero copy is typically set up. Zero-copy host code float \*a\_h, \*a\_map; ... cudaGetDeviceProperties(&prop, 0); if (!prop.canMapHostMemory) exit(0); cudaSetDeviceFlags(cudaDeviceMapHost); cudaHostAlloc(&a\_h, nBytes, cudaHostAllocMapped); cudaHostGetDevicePointer(&a\_map, a\_h, 0); kernel<<>>(a\_map); In this code, the `canMapHostMemory` field of the structure returned by `cudaGetDeviceProperties()` is used to check that the device supports mapping host memory to the device’s address space. Page-locked memory mapping is enabled by calling `cudaSetDeviceFlags()` with `cudaDeviceMapHost`. Note that `cudaSetDeviceFlags()` must be called prior to setting a device or making a CUDA call that requires state (that is, essentially, before a context is created). Page-locked mapped host memory is allocated using `cudaHostAlloc()`, and the pointer to the mapped device address space is obtained via the function `cudaHostGetDevicePointer()`. In the code in [Zero-copy host code](#zero-copy-host-code) , `kernel()` can reference the mapped pinned host memory using the pointer `a_map` in exactly the same was as it would if a\_map referred to a location in device memory. Note Mapped pinned host memory allows you to overlap CPU-GPU memory transfers with computation while avoiding the use of CUDA streams. But since any repeated access to such memory areas causes repeated CPU-GPU transfers, consider creating a second area in device memory to manually cache the previously read host memory data. ### 9.1.4. Unified Virtual Addressing[](#unified-virtual-addressing "Permalink to this headline") Devices of [compute capability](#cuda-compute-capability) 2.0 and later support a special addressing mode called _Unified Virtual Addressing_ (UVA) on 64-bit Linux and Windows. With UVA, the host memory and the device memories of all installed supported devices share a single virtual address space. Prior to UVA, an application had to keep track of which pointers referred to device memory (and for which device) and which referred to host memory as a separate bit of metadata (or as hard-coded information in the program) for each pointer. Using UVA, on the other hand, the physical memory space to which a pointer points can be determined simply by inspecting the value of the pointer using `cudaPointerGetAttributes()`. Under UVA, pinned host memory allocated with `cudaHostAlloc()` will have identical host and device pointers, so it is not necessary to call `cudaHostGetDevicePointer()` for such allocations. Host memory allocations pinned after-the-fact via `cudaHostRegister()`, however, will continue to have different device pointers than their host pointers, so `cudaHostGetDevicePointer()` remains necessary in that case. UVA is also a necessary precondition for enabling peer-to-peer (P2P) transfer of data directly across the PCIe bus or NVLink for supported GPUs in supported configurations, bypassing host memory. See the CUDA C++ Programming Guide for further explanations and software requirements for UVA and P2P. 9.2. Device Memory Spaces[](#device-memory-spaces "Permalink to this headline") --------------------------------------------------------------------------------- CUDA devices use several memory spaces, which have different characteristics that reflect their distinct usages in CUDA applications. These memory spaces include global, local, shared, texture, and registers, as shown in [Figure 2](#memory-spaces-cuda-device-figure) . ![Memory spaces on a CUDA device](_images/memory-spaces-on-cuda-device.png) Figure 2 Memory spaces on a CUDA device[](#memory-spaces-cuda-device-figure "Permalink to this image") Of these different memory spaces, global memory is the most plentiful; see Features and Technical Specifications of the CUDA C++ Programming Guide for the amounts of memory available in each memory space at each [compute capability](#cuda-compute-capability) level. Global, local, and texture memory have the greatest access latency, followed by constant memory, shared memory, and the register file. The various principal traits of the memory types are shown in [Table 1](#salient-features-device-memory-table) . | | | | | | | | --- | --- | --- | --- | --- | --- |Table 1 Salient Features of Device Memory[](#salient-features-device-memory-table "Permalink to this table") | Memory | Location on/off chip | Cached | Access | Scope | Lifetime | | --- | --- | --- | --- | --- | --- | | Register | On | n/a | R/W | 1 thread | Thread | | Local | Off | Yes†† | R/W | 1 thread | Thread | | Shared | On | n/a | R/W | All threads in block | Block | | Global | Off | † | R/W | All threads + host | Host allocation | | Constant | Off | Yes | R | All threads + host | Host allocation | | Texture | Off | Yes | R | All threads + host | Host allocation | | † Cached in L1 and L2 by default on devices of compute capability 6.0 and 7.x; cached only in L2 by default on devices of lower compute capabilities, though some allow opt-in to caching in L1 as well via compilation flags. | | | | | | | †† Cached in L1 and L2 by default except on devices of compute capability 5.x; devices of compute capability 5.x cache locals only in L2. | | | | | | In the case of texture access, if a texture reference is bound to a linear array in global memory, then the device code can write to the underlying array. Texture references that are bound to CUDA arrays can be written to via surface-write operations by binding a surface to the same underlying CUDA array storage). Reading from a texture while writing to its underlying global memory array in the same kernel launch should be avoided because the texture caches are read-only and are not invalidated when the associated global memory is modified. ### 9.2.1. Coalesced Access to Global Memory[](#coalesced-access-to-global-memory "Permalink to this headline") A very important performance consideration in programming for CUDA-capable GPU architectures is the coalescing of global memory accesses. Global memory loads and stores by threads of a warp are coalesced by the device into as few as possible transactions. Note **High Priority:** Ensure global memory accesses are coalesced whenever possible. The access requirements for coalescing depend on the compute capability of the device and are documented in the CUDA C++ Programming Guide. For devices of compute capability 6.0 or higher, the requirements can be summarized quite easily: the concurrent accesses of the threads of a warp will coalesce into a number of transactions equal to the number of 32-byte transactions necessary to service all of the threads of the warp. For certain devices of compute capability 5.2, L1-caching of accesses to global memory can be optionally enabled. If L1-caching is enabled on these devices, the number of required transactions is equal to the number of required 128-byte aligned segments. Note On devices of compute capability 6.0 or higher, L1-caching is the default, however the data access unit is 32-byte regardless of whether global loads are cached in L1 or not. On devices with GDDR memory, accessing memory in a coalesced way is even more important when ECC is turned on. Scattered accesses increase ECC memory transfer overhead, especially when writing data to global memory. Coalescing concepts are illustrated in the following simple examples. These examples assume compute capability 6.0 or higher and that accesses are for 4-byte words, unless otherwise noted. #### 9.2.1.1. A Simple Access Pattern[](#a-simple-access-pattern "Permalink to this headline") The first and simplest case of coalescing can be achieved by any CUDA-enabled device of compute capability 6.0 or higher: the _k_\-th thread accesses the _k_\-th word in a 32-byte aligned array. Not all threads need to participate. For example, if the threads of a warp access adjacent 4-byte words (e.g., adjacent `float` values), four coalesced 32-byte transactions will service that memory access. Such a pattern is shown in Figure 3 . ![Coalesced access](_images/coalesced-access.png) Figure 3 Coalesced access[](#coalesced-access-figure "Permalink to this image") This access pattern results in four 32-byte transactions, indicated by the red rectangles. If from any of the four 32-byte segments only a subset of the words are requested (e.g. if several threads had accessed the same word or if some threads did not participate in the access), the full segment is fetched anyway. Furthermore, if accesses by the threads of the warp had been permuted within or accross the four segments, still only four 32-byte transactions would have been performed by a device with [compute capability](#cuda-compute-capability) 6.0 or higher. #### 9.2.1.2. A Sequential but Misaligned Access Pattern[](#a-sequential-but-misaligned-access-pattern "Permalink to this headline") If sequential threads in a warp access memory that is sequential but not aligned with a 32-byte segment, five 32-byte segments will be requested, as shown in [Figure 4](#misaligned-sequential-addresses-fall-5-32-byte-l2-cache-seqments) . ![Misaligned sequential addresses that fall within five 32-byte segments](_images/misaligned-sequential-addresses.png) Figure 4 Misaligned sequential addresses that fall within five 32-byte segments[](#misaligned-sequential-addresses-fall-5-32-byte-l2-cache-seqments "Permalink to this image") Memory allocated through the CUDA Runtime API, such as via `cudaMalloc()`, is guaranteed to be aligned to at least 256 bytes. Therefore, choosing sensible thread block sizes, such as multiples of the warp size (i.e., 32 on current GPUs), facilitates memory accesses by warps that are properly aligned. (Consider what would happen to the memory addresses accessed by the second, third, and subsequent thread blocks if the thread block size was not a multiple of warp size, for example.) #### 9.2.1.3. Effects of Misaligned Accesses[](#effects-of-misaligned-accesses "Permalink to this headline") It is easy and informative to explore the ramifications of misaligned accesses using a simple copy kernel, such as the one in [A copy kernel that illustrates misaligned accesses](#a-copy-kernel-that-illustrates-misaligned-accesses) . A copy kernel that illustrates misaligned accesses \_\_global\_\_ void offsetCopy(float \*odata, float\* idata, int offset) { int xid \= blockIdx.x \* blockDim.x + threadIdx.x + offset; odata\[xid\] \= idata\[xid\]; } In [A copy kernel that illustrates misaligned accesses](#a-copy-kernel-that-illustrates-misaligned-accesses) , data is copied from the input array `idata` to the output array, both of which exist in global memory. The kernel is executed within a loop in host code that varies the parameter `offset` from 0 to 32 (for example, [Figure 4](#misaligned-sequential-addresses-fall-5-32-byte-l2-cache-seqments) corresponds to this misalignments). The effective bandwidth for the copy with various offsets on an NVIDIA Tesla V100 ([compute capability](#cuda-compute-capability) 7.0) is shown in [Figure 5](#performance-offsetcopy-kernel-figure) . ![Performance of offsetCopy kernel](_images/performance-of-offsetcopy-kernel.png) Figure 5 Performance of offsetCopy kernel[](#performance-offsetcopy-kernel-figure "Permalink to this image") For the NVIDIA Tesla V100, global memory accesses with no offset or with offsets that are multiples of 8 words result in four 32-byte transactions. The achieved bandwidth is approximately 790 GB/s. Otherwise, five 32-byte segments are loaded per warp, and we would expect approximately 4/5th of the memory throughput achieved with no offsets. In this particular example, the offset memory throughput achieved is, however, approximately 9/10th, because adjacent warps reuse the cache lines their neighbors fetched. So while the impact is still evident it is not as large as we might have expected. It would have been more so if adjacent warps had not exhibited such a high degree of reuse of the over-fetched cache lines. #### 9.2.1.4. Strided Accesses[](#strided-accesses "Permalink to this headline") As seen above, in the case of misaligned sequential accesses, caches help to alleviate the performance impact. It may be different with non-unit-strided accesses, however, and this is a pattern that occurs frequently when dealing with multidimensional data or matrices. For this reason, ensuring that as much as possible of the data in each cache line fetched is actually used is an important part of performance optimization of memory accesses on these devices. To illustrate the effect of strided access on effective bandwidth, see the kernel `strideCopy()` in [A kernel to illustrate non-unit stride data copy](#a-kernel-to-illustrate-non-unit-stride-data-copy) , which copies data with a stride of stride elements between threads from `idata` to `odata`. A kernel to illustrate non-unit stride data copy \_\_global\_\_ void strideCopy(float \*odata, float\* idata, int stride) { int xid \= (blockIdx.x\*blockDim.x + threadIdx.x)\*stride; odata\[xid\] \= idata\[xid\]; } [Figure 6](#adjacent-threads-accessing-memory-with-stride-of-2-figure) illustrates such a situation; in this case, threads within a warp access words in memory with a stride of 2. This action leads to a load of eight L2 cache segments per warp on the Tesla V100 (compute capability 7.0). ![Adjacent threads accessing memory with a stride of 2](_images/adjacent-threads-accessing-memory-with-stride-of-2.png) Figure 6 Adjacent threads accessing memory with a stride of 2[](#adjacent-threads-accessing-memory-with-stride-of-2-figure "Permalink to this image") A stride of 2 results in a 50% of load/store efficiency since half the elements in the transaction are not used and represent wasted bandwidth. As the stride increases, the effective bandwidth decreases until the point where 32 32-byte segments are loaded for the 32 threads in a warp, as indicated in [Figure 7](#performance-of-stridecopy-kernel-figure) . ![Performance of strideCopy kernel](_images/performance-of-stridecopy-kernel.png) Figure 7 Performance of strideCopy kernel[](#performance-of-stridecopy-kernel-figure "Permalink to this image") As illustrated in [Figure 7](#performance-of-stridecopy-kernel-figure) , non-unit-stride global memory accesses should be avoided whenever possible. One method for doing so utilizes shared memory, which is discussed in the next section. ### 9.2.2. L2 Cache[](#l2-cache "Permalink to this headline") Starting with CUDA 11.0, devices of compute capability 8.0 and above have the capability to influence persistence of data in the L2 cache. Because L2 cache is on-chip, it potentially provides higher bandwidth and lower latency accesses to global memory. For more details refer to the L2 Access Management section in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#L2_access_intro) . #### 9.2.2.1. L2 Cache Access Window[](#l2-cache-access-window "Permalink to this headline") When a CUDA kernel accesses a data region in the global memory repeatedly, such data accesses can be considered to be _persisting_. On the other hand, if the data is only accessed once, such data accesses can be considered to be _streaming_. A portion of the L2 cache can be set aside for persistent accesses to a data region in global memory. If this set-aside portion is not used by persistent accesses, then streaming or normal data accesses can use it. The L2 cache set-aside size for persisting accesses may be adjusted, within limits: cudaGetDeviceProperties(&prop, device\_id); cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, prop.persistingL2CacheMaxSize); /\* Set aside max possible size of L2 cache for persisting accesses \*/ Mapping of user data to L2 set-aside portion can be controlled using an access policy window on a CUDA stream or CUDA graph kernel node. The example below shows how to use the access policy window on a CUDA stream. cudaStreamAttrValue stream\_attribute; // Stream level attributes data structure stream\_attribute.accessPolicyWindow.base\_ptr \= reinterpret\_cast(ptr); // Global Memory data pointer stream\_attribute.accessPolicyWindow.num\_bytes \= num\_bytes; // Number of bytes for persisting accesses. // (Must be less than cudaDeviceProp::accessPolicyMaxWindowSize) stream\_attribute.accessPolicyWindow.hitRatio \= 1.0; // Hint for L2 cache hit ratio for persisting accesses in the num\_bytes region stream\_attribute.accessPolicyWindow.hitProp \= cudaAccessPropertyPersisting; // Type of access property on cache hit stream\_attribute.accessPolicyWindow.missProp \= cudaAccessPropertyStreaming; // Type of access property on cache miss. //Set the attributes to a CUDA stream of type cudaStream\_t cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &stream\_attribute); The access policy window requires a value for `hitRatio` and `num_bytes`. Depending on the value of the `num_bytes` parameter and the size of L2 cache, one may need to tune the value of `hitRatio` to avoid thrashing of L2 cache lines. #### 9.2.2.2. Tuning the Access Window Hit-Ratio[](#tuning-the-access-window-hit-ratio "Permalink to this headline") The `hitRatio` parameter can be used to specify the fraction of accesses that receive the `hitProp` property. For example, if the `hitRatio` value is 0.6, 60% of the memory accesses in the global memory region \[ptr..ptr+num\_bytes) have the persisting property and 40% of the memory accesses have the streaming property. To understand the effect of `hitRatio` and `num_bytes`, we use a sliding window micro benchmark.\ \ This microbenchmark uses a 1024 MB region in GPU global memory. First, we set aside 30 MB of the L2 cache for persisting accesses using `cudaDeviceSetLimit()`, as discussed above. Then, as shown in the figure below, we specify that the accesses to the first `freqSize * sizeof(int)` bytes of the memory region are persistent. This data will thus use the L2 set-aside portion. In our experiment, we vary the size of this persistent data region from 10 MB to 60 MB to model various scenarios where data fits in or exceeds the available L2 set-aside portion of 30 MB. Note that the NVIDIA Tesla A100 GPU has 40 MB of total L2 cache capacity. Accesses to the remaining data of the memory region (i.e., streaming data) are considered normal or streaming accesses and will thus use the remaining 10 MB of the non set-aside L2 portion (unless part of the L2 set-aside portion is unused).\ \ [![Mapping Persistent data accesses to set-aside L2 in sliding window experiment](_images/sliding-window-l2.png)](_images/sliding-window-l2.png)\ \ Figure 8 Mapping Persistent data accesses to set-aside L2 in sliding window experiment[](#l2-cache-hit-ratio-sliding-window-l2 "Permalink to this image")\ \ Consider the following kernel code and access window parameters, as the implementation of the sliding window experiment.\ \ \_\_global\_\_ void kernel(int \*data\_persistent, int \*data\_streaming, int dataSize, int freqSize) {\ int tid \= blockIdx.x \* blockDim.x + threadIdx.x;\ \ /\*Each CUDA thread accesses one element in the persistent data section\ and one element in the streaming data section.\ Because the size of the persistent memory region (freqSize \* sizeof(int) bytes) is much\ smaller than the size of the streaming memory region (dataSize \* sizeof(int) bytes), data\ in the persistent region is accessed more frequently\*/\ \ data\_persistent\[tid % freqSize\] \= 2 \* data\_persistent\[tid % freqSize\];\ data\_streaming\[tid % dataSize\] \= 2 \* data\_streaming\[tid % dataSize\];\ }\ \ stream\_attribute.accessPolicyWindow.base\_ptr \= reinterpret\_cast(data\_persistent);\ stream\_attribute.accessPolicyWindow.num\_bytes \= freqSize \* sizeof(int); //Number of bytes for persisting accesses in range 10-60 MB\ stream\_attribute.accessPolicyWindow.hitRatio \= 1.0; //Hint for cache hit ratio. Fixed value 1.0\ \ The performance of the above kernel is shown in the chart below. When the persistent data region fits well into the 30 MB set-aside portion of the L2 cache, a performance increase of as much as 50% is observed. However, once the size of this persistent data region exceeds the size of the L2 set-aside cache portion, approximately 10% performance drop is observed due to thrashing of L2 cache lines.\ \ [![The performance of the sliding-window benchmark with fixed hit-ratio of 1.0](_images/l2-hitratio-before.png)](_images/l2-hitratio-before.png)\ \ Figure 9 The performance of the sliding-window benchmark with fixed hit-ratio of 1.0[](#l2-cache-hit-ratio-l2-hitratio-before "Permalink to this image")\ \ In order to optimize the performance, when the size of the persistent data is more than the size of the set-aside L2 cache portion, we tune the `num_bytes` and `hitRatio` parameters in the access window as below.\ \ stream\_attribute.accessPolicyWindow.base\_ptr \= reinterpret\_cast(data\_persistent);\ stream\_attribute.accessPolicyWindow.num\_bytes \= 20\*1024\*1024; //20 MB\ stream\_attribute.accessPolicyWindow.hitRatio \= (20\*1024\*1024)/((float)freqSize\*sizeof(int)); //Such that up to 20MB of data is resident.\ \ We fix the `num_bytes` in the access window to 20 MB and tune the `hitRatio` such that a random 20 MB of the total persistent data is resident in the L2 set-aside cache portion. The remaining portion of this persistent data will be accessed using the streaming property. This helps in reducing cache thrashing. The results are shown in the chart below, where we see good performance regardless of whether the persistent data fits in the L2 set-aside or not.\ \ [![The performance of the sliding-window benchmark with tuned hit-ratio](_images/l2-hitratio-after.png)](_images/l2-hitratio-after.png)\ \ Figure 10 The performance of the sliding-window benchmark with tuned hit-ratio[](#l2-cache-hit-ratio-l2-hitratio-after "Permalink to this image")\ \ ### 9.2.3. Shared Memory[](#shared-memory "Permalink to this headline")\ \ Because it is on-chip, shared memory has much higher bandwidth and lower latency than local and global memory - provided there are no bank conflicts between the threads, as detailed in the following section.\ \ #### 9.2.3.1. Shared Memory and Memory Banks[](#shared-memory-and-memory-banks "Permalink to this headline")\ \ To achieve high memory bandwidth for concurrent accesses, shared memory is divided into equally sized memory modules (_banks_) that can be accessed simultaneously. Therefore, any memory load or store of _n_ addresses that spans _n_ distinct memory banks can be serviced simultaneously, yielding an effective bandwidth that is _n_ times as high as the bandwidth of a single bank.\ \ However, if multiple addresses of a memory request map to the same memory bank, the accesses are serialized. The hardware splits a memory request that has bank conflicts into as many separate conflict-free requests as necessary, decreasing the effective bandwidth by a factor equal to the number of separate memory requests. The one exception here is when multiple threads in a warp address the same shared memory location, resulting in a broadcast. In this case, multiple broadcasts from different banks are coalesced into a single multicast from the requested shared memory locations to the threads.\ \ To minimize bank conflicts, it is important to understand how memory addresses map to memory banks and how to optimally schedule memory requests.\ \ On devices of compute capability 5.x or newer, each bank has a bandwidth of 32 bits every clock cycle, and successive 32-bit words are assigned to successive banks. The warp size is 32 threads and the number of banks is also 32, so bank conflicts can occur between any threads in the warp. See [Compute Capability 5.x](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability-5-x)\ for further details.\ \ #### 9.2.3.2. Shared Memory in Matrix Multiplication (C=AB)[](#shared-memory-in-matrix-multiplication-c-ab "Permalink to this headline")\ \ Shared memory enables cooperation between threads in a block. When multiple threads in a block use the same data from global memory, shared memory can be used to access the data from global memory only once. Shared memory can also be used to avoid uncoalesced memory accesses by loading and storing data in a coalesced pattern from global memory and then reordering it in shared memory. Aside from memory bank conflicts, there is no penalty for non-sequential or unaligned accesses by a warp in shared memory.\ \ The use of shared memory is illustrated via the simple example of a matrix multiplication C = AB for the case with A of dimension Mxw, B of dimension wxN, and C of dimension MxN. To keep the kernels simple, M and N are multiples of 32, since the warp size (w) is 32 for current devices.\ \ A natural decomposition of the problem is to use a block and tile size of wxw threads. Therefore, in terms of wxw tiles, A is a column matrix, B is a row matrix, and C is their outer product; see [Figure 11](#shared-memory-in-matrix-multiplication-c-ab-block-column-matrix-a-multiplied-block-row-matrix-b-product-matrix-c)\ . A grid of N/w by M/w blocks is launched, where each thread block calculates the elements of a different tile in C from a single tile of A and a single tile of B.\ \ ![Block-column matrix multiplied by block-row matrix. Block-column matrix (A) multiplied by block-row matrix (B) with resulting product matrix (C).](_images/matrix-multiplication-block-column-by-block-row.png)\ \ Figure 11 Block-column matrix multiplied by block-row matrix. Block-column matrix (A) multiplied by block-row matrix (B) with resulting product matrix (C).[](#shared-memory-in-matrix-multiplication-c-ab-block-column-matrix-a-multiplied-block-row-matrix-b-product-matrix-c "Permalink to this image")\ \ To do this, the `simpleMultiply` kernel ([Unoptimized matrix multiplication](#unoptimized-matrix-multiplication-example)\ ) calculates the output elements of a tile of matrix C.\ \ Unoptimized matrix multiplication\ \ \_\_global\_\_ void simpleMultiply(float \*a, float\* b, float \*c,\ int N)\ {\ int row \= blockIdx.y \* blockDim.y + threadIdx.y;\ int col \= blockIdx.x \* blockDim.x + threadIdx.x;\ float sum \= 0.0f;\ for (int i \= 0; i < TILE\_DIM; i++) {\ sum += a\[row\*TILE\_DIM+i\] \* b\[i\*N+col\];\ }\ c\[row\*N+col\] \= sum;\ }\ \ In [Unoptimized matrix multiplication](#unoptimized-matrix-multiplication-example)\ , `a`, `b`, and `c` are pointers to global memory for the matrices A, B, and C, respectively; `blockDim.x`, `blockDim.y`, and `TILE_DIM` are all equal to w. Each thread in the wxw-thread block calculates one element in a tile of C. `row` and `col` are the row and column of the element in C being calculated by a particular thread. The `for` loop over `i` multiplies a row of A by a column of B, which is then written to C.\ \ The effective bandwidth of this kernel is 119.9 GB/s on an NVIDIA Tesla V100. To analyze performance, it is necessary to consider how warps access global memory in the `for` loop. Each warp of threads calculates one row of a tile of C, which depends on a single row of A and an entire tile of B as illustrated in [Figure 12](#shared-memory-in-matrix-multiplication-c-ab-computing-row-c-tile-c-row-a-tile-b)\ .\ \ ![Computing a row of a tile. Computing a row of a tile in C using one row of A and an entire tile of B.](_images/computing-row-of-tile.png)\ \ Figure 12 Computing a row of a tile. Computing a row of a tile in C using one row of A and an entire tile of B.[](#shared-memory-in-matrix-multiplication-c-ab-computing-row-c-tile-c-row-a-tile-b "Permalink to this image")\ \ For each iteration _i_ of the `for` loop, the threads in a warp read a row of the B tile, which is a sequential and coalesced access for all compute capabilities.\ \ However, for each iteration _i_, all threads in a warp read the same value from global memory for matrix A, as the index `row*TILE_DIM+i` is constant within a warp. Even though such an access requires only 1 transaction on devices of compute capability 2.0 or higher, there is wasted bandwidth in the transaction, because only one 4-byte word out of 8 words in a 32-byte cache segment is used. We can reuse this cache line in subsequent iterations of the loop, and we would eventually utilize all 8 words; however, when many warps execute on the same multiprocessor simultaneously, as is generally the case, the cache line may easily be evicted from the cache between iterations _i_ and _i+1_.\ \ The performance on a device of any compute capability can be improved by reading a tile of A into shared memory as shown in [Using shared memory to improve the global memory load efficiency in matrix multiplication](#using-shared-memory-to-improve-the-global-memory-load-efficiency-in-matrix-multiplication)\ .\ \ Using shared memory to improve the global memory load efficiency in matrix multiplication\ \ \_\_global\_\_ void coalescedMultiply(float \*a, float\* b, float \*c,\ int N)\ {\ \_\_shared\_\_ float aTile\[TILE\_DIM\]\[TILE\_DIM\];\ \ int row \= blockIdx.y \* blockDim.y + threadIdx.y;\ int col \= blockIdx.x \* blockDim.x + threadIdx.x;\ float sum \= 0.0f;\ aTile\[threadIdx.y\]\[threadIdx.x\] \= a\[row\*TILE\_DIM+threadIdx.x\];\ \_\_syncwarp();\ for (int i \= 0; i < TILE\_DIM; i++) {\ sum += aTile\[threadIdx.y\]\[i\]\* b\[i\*N+col\];\ }\ c\[row\*N+col\] \= sum;\ }\ \ In [Using shared memory to improve the global memory load efficiency in matrix multiplication](#using-shared-memory-to-improve-the-global-memory-load-efficiency-in-matrix-multiplication)\ , each element in a tile of A is read from global memory only once, in a fully coalesced fashion (with no wasted bandwidth), to shared memory. Within each iteration of the `for` loop, a value in shared memory is broadcast to all threads in a warp. Instead of a `__syncthreads()`synchronization barrier call, a `__syncwarp()` is sufficient after reading the tile of A into shared memory because only threads within the warp that write the data into shared memory read this data. This kernel has an effective bandwidth of 144.4 GB/s on an NVIDIA Tesla V100. This illustrates the use of the shared memory as a _user-managed cache_ when the hardware L1 cache eviction policy does not match up well with the needs of the application or when L1 cache is not used for reads from global memory.\ \ A further improvement can be made to how [Using shared memory to improve the global memory load efficiency in matrix multiplication](#using-shared-memory-to-improve-the-global-memory-load-efficiency-in-matrix-multiplication)\ deals with matrix B. In calculating each of the rows of a tile of matrix C, the entire tile of B is read. The repeated reading of the B tile can be eliminated by reading it into shared memory once ([Improvement by reading additional data into shared memory](#improvement-by-reading-additional-data-into-shared-memory)\ ).\ \ Improvement by reading additional data into shared memory\ \ \_\_global\_\_ void sharedABMultiply(float \*a, float\* b, float \*c,\ int N)\ {\ \_\_shared\_\_ float aTile\[TILE\_DIM\]\[TILE\_DIM\],\ bTile\[TILE\_DIM\]\[TILE\_DIM\];\ int row \= blockIdx.y \* blockDim.y + threadIdx.y;\ int col \= blockIdx.x \* blockDim.x + threadIdx.x;\ float sum \= 0.0f;\ aTile\[threadIdx.y\]\[threadIdx.x\] \= a\[row\*TILE\_DIM+threadIdx.x\];\ bTile\[threadIdx.y\]\[threadIdx.x\] \= b\[threadIdx.y\*N+col\];\ \_\_syncthreads();\ for (int i \= 0; i < TILE\_DIM; i++) {\ sum += aTile\[threadIdx.y\]\[i\]\* bTile\[i\]\[threadIdx.x\];\ }\ c\[row\*N+col\] \= sum;\ }\ \ Note that in [Improvement by reading additional data into shared memory](#improvement-by-reading-additional-data-into-shared-memory)\ , a `__syncthreads()` call is required after reading the B tile because a warp reads data from shared memory that were written to shared memory by different warps. The effective bandwidth of this routine is 195.5 GB/s on an NVIDIA Tesla V100. Note that the performance improvement is not due to improved coalescing in either case, but to avoiding redundant transfers from global memory.\ \ The results of the various optimizations are summarized in [Table 2](#performance-improvements-optimizing-c-ab-matrix-table)\ .\ \ | | |\ | --- | --- |Table 2 Performance Improvements Optimizing C = AB Matrix Multiply[](#performance-improvements-optimizing-c-ab-matrix-table "Permalink to this table")\ \ | Optimization | NVIDIA Tesla V100 |\ | --- | --- |\ | No optimization | 119.9 GB/s |\ | Coalesced using shared memory to store a tile of A | 144.4 GB/s |\ | Using shared memory to eliminate redundant reads of a tile of B | 195.5 GB/s |\ \ Note\ \ **Medium Priority:** Use shared memory to avoid redundant transfers from global memory.\ \ #### 9.2.3.3. Shared Memory in Matrix Multiplication (C=AAT)[](#shared-memory-in-matrix-multiplication-c-aat "Permalink to this headline")\ \ A variant of the previous matrix multiplication can be used to illustrate how strided accesses to global memory, as well as shared memory bank conflicts, are handled. This variant simply uses the transpose of A in place of B, so C = AAT.\ \ A simple implementation for C = AAT is shown in [Unoptimized handling of strided accesses to global memory](#unoptimized-handling-of-strided-accesses-to-global-memory)\ .\ \ Unoptimized handling of strided accesses to global memory\ \ \_\_global\_\_ void simpleMultiply(float \*a, float \*c, int M)\ {\ int row \= blockIdx.y \* blockDim.y + threadIdx.y;\ int col \= blockIdx.x \* blockDim.x + threadIdx.x;\ float sum \= 0.0f;\ for (int i \= 0; i < TILE\_DIM; i++) {\ sum += a\[row\*TILE\_DIM+i\] \* a\[col\*TILE\_DIM+i\];\ }\ c\[row\*M+col\] \= sum;\ }\ \ In the example above, the _row_\-th, _col_\-th element of C is obtained by taking the dot product of the _row_\-th and _col_\-th rows of A. The effective bandwidth for this kernel is 12.8 GB/s on an NVIDIA Tesla V100. These results are substantially lower than the corresponding measurements for the C = AB kernel. The difference is in how threads in a half warp access elements of A in the second term, `a[col*TILE_DIM+i]`, for each iteration `i`. For a warp of threads, `col` represents sequential columns of the transpose of A, and therefore `col*TILE_DIM` represents a strided access of global memory with a stride of w, resulting in plenty of wasted bandwidth.\ \ The way to avoid strided access is to use shared memory as before, except in this case a warp reads a row of A into a column of a shared memory tile, as shown in [An optimized handling of strided accesses using coalesced reads from global memory](#an-optimized-handling-of-strided-accesses-using-coalesced-reads-from-global-memory)\ .\ \ An optimized handling of strided accesses using coalesced reads from global memory\ \ \_\_global\_\_ void coalescedMultiply(float \*a, float \*c, int M)\ {\ \_\_shared\_\_ float aTile\[TILE\_DIM\]\[TILE\_DIM\],\ transposedTile\[TILE\_DIM\]\[TILE\_DIM\];\ int row \= blockIdx.y \* blockDim.y + threadIdx.y;\ int col \= blockIdx.x \* blockDim.x + threadIdx.x;\ float sum \= 0.0f;\ aTile\[threadIdx.y\]\[threadIdx.x\] \= a\[row\*TILE\_DIM+threadIdx.x\];\ transposedTile\[threadIdx.x\]\[threadIdx.y\] \=\ a\[(blockIdx.x\*blockDim.x + threadIdx.y)\*TILE\_DIM +\ threadIdx.x\];\ \_\_syncthreads();\ for (int i \= 0; i < TILE\_DIM; i++) {\ sum += aTile\[threadIdx.y\]\[i\]\* transposedTile\[i\]\[threadIdx.x\];\ }\ c\[row\*M+col\] \= sum;\ }\ \ [An optimized handling of strided accesses using coalesced reads from global memory](#an-optimized-handling-of-strided-accesses-using-coalesced-reads-from-global-memory)\ uses the shared `transposedTile` to avoid uncoalesced accesses in the second term in the dot product and the shared `aTile` technique from the previous example to avoid uncoalesced accesses in the first term. The effective bandwidth of this kernel is 140.2 GB/s on an NVIDIA Tesla V100.These results are lower than those obtained by the final kernel for C = AB. The cause of the difference is shared memory bank conflicts.\ \ The reads of elements in `transposedTile` within the for loop are free of conflicts, because threads of each half warp read across rows of the tile, resulting in unit stride across the banks. However, bank conflicts occur when copying the tile from global memory into shared memory. To enable the loads from global memory to be coalesced, data are read from global memory sequentially. However, this requires writing to shared memory in columns, and because of the use of wxw tiles in shared memory, this results in a stride between threads of w banks - every thread of the warp hits the same bank (Recall that w is selected as 32). These many-way bank conflicts are very expensive. The simple remedy is to pad the shared memory array so that it has an extra column, as in the following line of code.\ \ \_\_shared\_\_ float transposedTile\[TILE\_DIM\]\[TILE\_DIM+1\];\ \ This padding eliminates the conflicts entirely, because now the stride between threads is w+1 banks (i.e., 33 for current devices), which, due to modulo arithmetic used to compute bank indices, is equivalent to a unit stride. After this change, the effective bandwidth is 199.4 GB/s on an NVIDIA Tesla V100, which is comparable to the results from the last C = AB kernel.\ \ The results of these optimizations are summarized in [Table 3](#performance-inmprovements-optimizing-c-aa-matrix-multiplication-table)\ .\ \ | | |\ | --- | --- |Table 3 Performance Improvements Optimizing C = AAT Matrix Multiplication[](#performance-inmprovements-optimizing-c-aa-matrix-multiplication-table "Permalink to this table")\ \ | Optimization | NVIDIA Tesla V100 |\ | --- | --- |\ | No optimization | 12.8 GB/s |\ | Using shared memory to coalesce global reads | 140.2 GB/s |\ | Removing bank conflicts | 199.4 GB/s |\ \ These results should be compared with those in [Table 2](#performance-improvements-optimizing-c-ab-matrix-table)\ . As can be seen from these tables, judicious use of shared memory can dramatically improve performance.\ \ The examples in this section have illustrated three reasons to use shared memory:\ \ * To enable coalesced accesses to global memory, especially to avoid large strides (for general matrices, strides are much larger than 32)\ \ * To eliminate (or reduce) redundant loads from global memory\ \ * To avoid wasted bandwidth\ \ \ #### 9.2.3.4. Asynchronous Copy from Global Memory to Shared Memory[](#asynchronous-copy-from-global-memory-to-shared-memory "Permalink to this headline")\ \ CUDA 11.0 introduces an _async-copy_ feature that can be used within device code to explicitly manage the asynchronous copying of data from global memory to shared memory. This feature enables CUDA kernels to overlap copying data from global to shared memory with computation. It also avoids an intermediary register file access traditionally present between the global memory read and the shared memory write.\ \ For more details refer to the `memcpy_async` section in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#async_data_operations)\ .\ \ To understand the performance difference between synchronous copy and asynchronous copy of data from global memory to shared memory, consider the following micro benchmark CUDA kernels for demonstrating the synchronous and asynchronous approaches. Asynchronous copies are hardware accelerated for NVIDIA A100 GPU.\ \ template \ \_\_global\_\_ void pipeline\_kernel\_sync(T \*global, uint64\_t \*clock, size\_t copy\_count) {\ extern \_\_shared\_\_ char s\[\];\ T \*shared \= reinterpret\_cast(s);\ \ uint64\_t clock\_start \= clock64();\ \ for (size\_t i \= 0; i < copy\_count; ++i) {\ shared\[blockDim.x \* i + threadIdx.x\] \= global\[blockDim.x \* i + threadIdx.x\];\ }\ \ uint64\_t clock\_end \= clock64();\ \ atomicAdd(reinterpret\_cast(clock),\ clock\_end \- clock\_start);\ }\ \ template \ \_\_global\_\_ void pipeline\_kernel\_async(T \*global, uint64\_t \*clock, size\_t copy\_count) {\ extern \_\_shared\_\_ char s\[\];\ T \*shared \= reinterpret\_cast(s);\ \ uint64\_t clock\_start \= clock64();\ \ //pipeline pipe;\ for (size\_t i \= 0; i < copy\_count; ++i) {\ \_\_pipeline\_memcpy\_async(&shared\[blockDim.x \* i + threadIdx.x\],\ &global\[blockDim.x \* i + threadIdx.x\], sizeof(T));\ }\ \_\_pipeline\_commit();\ \_\_pipeline\_wait\_prior(0);\ \ uint64\_t clock\_end \= clock64();\ \ atomicAdd(reinterpret\_cast(clock),\ clock\_end \- clock\_start);\ }\ \ The synchronous version for the kernel loads an element from global memory to an intermediate register and then stores the intermediate register value to shared memory. In the asynchronous version of the kernel, instructions to load from global memory and store directly into shared memory are issued as soon as `__pipeline_memcpy_async()` function is called. The `__pipeline_wait_prior(0)` will wait until all the instructions in the pipe object have been executed. Using asynchronous copies does not use any intermediate register. Not using intermediate registers can help reduce register pressure and can increase kernel occupancy. Data copied from global memory to shared memory using asynchronous copy instructions can be cached in the L1 cache or the L1 cache can be optionally bypassed. If individual CUDA threads are copying elements of 16 bytes, the L1 cache can be bypassed. This difference is illustrated in [Figure 13](#async-copy-sync-vs-async-figure)\ .\ \ [![Comparing Synchronous vs Asynchronous Copy from Global Memory to Shared Memory](_images/sync-vs-async.png)](_images/sync-vs-async.png)\ \ Figure 13 Comparing Synchronous vs Asynchronous Copy from Global Memory to Shared Memory[](#async-copy-sync-vs-async-figure "Permalink to this image")\ \ We evaluate the performance of both kernels using elements of size 4B, 8B and 16B per thread i.e., using `int`, `int2` and `int4` for the template parameter. We adjust the `copy_count` in the kernels such that each thread block copies from 512 bytes up to 48 MB. The performance of the kernels is shown in [Figure 14](#async-copy-async-perf-figure)\ .\ \ [![Comparing Performance of Synchronous vs Asynchronous Copy from Global Memory to Shared Memory](_images/async-perf.png)](_images/async-perf.png)\ \ Figure 14 Comparing Performance of Synchronous vs Asynchronous Copy from Global Memory to Shared Memory[](#async-copy-async-perf-figure "Permalink to this image")\ \ From the performance chart, the following observations can be made for this experiment.\ \ * Best performance with synchronous copy is achieved when the `copy_count` parameter is a multiple of 4 for all three element sizes. The compiler can optimize groups of 4 load and store instructions. This is evident from the saw tooth curves.\ \ * Asynchronous copy achieves better performance in nearly all cases.\ \ * The async-copy does not require the `copy_count` parameter to be a multiple of 4, to maximize performance through compiler optimizations.\ \ * Overall, best performance is achieved when using asynchronous copies with an element of size 8 or 16 bytes.\ \ \ ### 9.2.4. Local Memory[](#local-memory "Permalink to this headline")\ \ Local memory is so named because its scope is local to the thread, not because of its physical location. In fact, local memory is off-chip. Hence, access to local memory is as expensive as access to global memory. In other words, the term _local_ in the name does not imply faster access.\ \ Local memory is used only to hold automatic variables. This is done by the `nvcc` compiler when it determines that there is insufficient register space to hold the variable. Automatic variables that are likely to be placed in local memory are large structures or arrays that would consume too much register space and arrays that the compiler determines may be indexed dynamically.\ \ Inspection of the PTX assembly code (obtained by compiling with `-ptx` or `-keep` command-line options to `nvcc`) reveals whether a variable has been placed in local memory during the first compilation phases. If it has, it will be declared using the `.local` mnemonic and accessed using the `ld.local` and `st.local` mnemonics. If it has not, subsequent compilation phases might still decide otherwise, if they find the variable consumes too much register space for the targeted architecture. There is no way to check this for a specific variable, but the compiler reports total local memory usage per kernel (lmem) when run with the`--ptxas-options=-v` option.\ \ ### 9.2.5. Texture Memory[](#texture-memory "Permalink to this headline")\ \ The read-only texture memory space is cached. Therefore, a texture fetch costs one device memory read only on a cache miss; otherwise, it just costs one read from the texture cache. The texture cache is optimized for 2D spatial locality, so threads of the same warp that read texture addresses that are close together will achieve best performance. Texture memory is also designed for streaming fetches with a constant latency; that is, a cache hit reduces DRAM bandwidth demand, but not fetch latency.\ \ In certain addressing situations, reading device memory through texture fetching can be an advantageous alternative to reading device memory from global or constant memory.\ \ #### 9.2.5.1. Additional Texture Capabilities[](#additional-texture-capabilities "Permalink to this headline")\ \ If textures are fetched using `tex1D()`,`tex2D()`, or `tex3D()` rather than `tex1Dfetch()`, the hardware provides other capabilities that might be useful for some applications such as image processing, as shown in [Table 4](#useful-features-for-tex1d-tex2d-tex3d-fetches-table)\ .\ \ | | | |\ | --- | --- | --- |Table 4 Useful Features for tex1D(), tex2D(), and tex3D() Fetches[](#useful-features-for-tex1d-tex2d-tex3d-fetches-table "Permalink to this table")\ \ | Feature | Use | Caveat |\ | --- | --- | --- |\ | Filtering | Fast, low-precision interpolation between texels | Valid only if the texture reference returns floating-point data |\ | Normalized texture coordinates | Resolution-independent coding | None |\ | Addressing modes | Automatic handling of boundary cases1 | Can be used only with normalized texture coordinates |\ | 1 The automatic handling of boundary cases in the bottom row of Table 4 refers to how a texture coordinate is resolved when it falls outside the valid addressing range. There are two options: _clamp_ and _wrap_. If _x_ is the coordinate and _N_ is the number of texels for a one-dimensional texture, then with clamp, _x_ is replaced by _0_ if _x_ < 0 and by 1-1/_N_ if 1 _<__x_. With wrap, _x_ is replaced by _frac(x)_ where _frac(x) = x - floor(x)_. Floor returns the largest integer less than or equal to _x_. So, in clamp mode where _N_ = 1, an _x_ of 1.3 is clamped to 1.0; whereas in wrap mode, it is converted to 0.3 | | |\ \ Within a kernel call, the texture cache is not kept coherent with respect to global memory writes, so texture fetches from addresses that have been written via global stores in the same kernel call return undefined data. That is, a thread can safely read a memory location via texture if the location has been updated by a previous kernel call or memory copy, but not if it has been previously updated by the same thread or another thread within the same kernel call.\ \ ### 9.2.6. Constant Memory[](#constant-memory "Permalink to this headline")\ \ There is a total of 64 KB constant memory on a device. The constant memory space is cached. As a result, a read from constant memory costs one memory read from device memory only on a cache miss; otherwise, it just costs one read from the constant cache. Accesses to different addresses by threads within a warp are serialized, thus the cost scales linearly with the number of unique addresses read by all threads within a warp. As such, the constant cache is best when threads in the same warp accesses only a few distinct locations. If all threads of a warp access the same location, then constant memory can be as fast as a register access.\ \ ### 9.2.7. Registers[](#registers "Permalink to this headline")\ \ Generally, accessing a register consumes zero extra clock cycles per instruction, but delays may occur due to register read-after-write dependencies and register memory bank conflicts.\ \ The compiler and hardware thread scheduler will schedule instructions as optimally as possible to avoid register memory bank conflicts. An application has no direct control over these bank conflicts. In particular, there is no register-related reason to pack data into vector data types such as `float4` or `int4` types.\ \ #### 9.2.7.1. Register Pressure[](#register-pressure "Permalink to this headline")\ \ Register pressure occurs when there are not enough registers available for a given task. Even though each multiprocessor contains thousands of 32-bit registers (see Features and Technical Specifications of the CUDA C++ Programming Guide), these are partitioned among concurrent threads. To prevent the compiler from allocating too many registers, use the `-maxrregcount=N` compiler command-line option or the launch bounds kernel definition qualifier (see Execution Configuration of the CUDA C++ Programming Guide) to control the maximum number of registers to allocated per thread.\ \ 9.3. Allocation[](#allocation "Permalink to this headline")\ \ -------------------------------------------------------------\ \ Device memory allocation and de-allocation via `cudaMalloc()` and `cudaFree()` are expensive operations. It is recommended to use `cudaMallocAsync()` and `cudaFreeAsync()` which are stream ordered pool allocators to manage device memory.\ \ 9.4. NUMA Best Practices[](#numa-best-practices "Permalink to this headline")\ \ -------------------------------------------------------------------------------\ \ Some recent Linux distributions enable automatic NUMA balancing (or “[AutoNUMA](https://lwn.net/Articles/488709/)\ ”) by default. In some instances, operations performed by automatic NUMA balancing may degrade the performance of applications running on NVIDIA GPUs. For optimal performance, users should manually tune the NUMA characteristics of their application.\ \ The optimal NUMA tuning will depend on the characteristics and desired hardware affinities of each application and node, but in general applications computing on NVIDIA GPUs are advised to choose a policy that disables automatic NUMA balancing. For example, on IBM Newell POWER9 nodes (where the CPUs correspond to NUMA nodes 0 and 8), use:\ \ numactl --membind=0,8\ \ to bind memory allocations to the CPUs.\ \ 10\. Execution Configuration Optimizations[](#execution-configuration-optimizations "Permalink to this headline")\ \ ===================================================================================================================\ \ One of the keys to good performance is to keep the multiprocessors on the device as busy as possible. A device in which work is poorly balanced across the multiprocessors will deliver suboptimal performance. Hence, it’s important to design your application to use threads and blocks in a way that maximizes hardware utilization and to limit practices that impede the free distribution of work. A key concept in this effort is occupancy, which is explained in the following sections.\ \ Hardware utilization can also be improved in some cases by designing your application so that multiple, independent kernels can execute at the same time. Multiple kernels executing at the same time is known as concurrent kernel execution. Concurrent kernel execution is described below.\ \ Another important concept is the management of system resources allocated for a particular task. How to manage this resource utilization is discussed in the final sections of this chapter.\ \ 10.1. Occupancy[](#occupancy "Permalink to this headline")\ \ ------------------------------------------------------------\ \ Thread instructions are executed sequentially in CUDA, and, as a result, executing other warps when one warp is paused or stalled is the only way to hide latencies and keep the hardware busy. Some metric related to the number of active warps on a multiprocessor is therefore important in determining how effectively the hardware is kept busy. This metric is _occupancy_.\ \ Occupancy is the ratio of the number of active warps per multiprocessor to the maximum number of possible active warps. (To determine the latter number, see the `deviceQuery` CUDA Sample or refer to [Compute Capabilities](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities)\ .) Another way to view occupancy is the percentage of the hardware’s ability to process warps that is actively in use.\ \ Higher occupancy does not always equate to higher performance-there is a point above which additional occupancy does not improve performance. However, low occupancy always interferes with the ability to hide memory latency, resulting in performance degradation.\ \ Per thread resources required by a CUDA kernel might limit the maximum block size in an unwanted way. In order to maintain forward compatibility to future hardware and toolkits and to ensure that at least one thread block can run on an SM, developers should include the single argument `__launch_bounds__(maxThreadsPerBlock)` which specifies the largest block size that the kernel will be launched with. Failure to do so could lead to “too many resources requested for launch” errors. Providing the two argument version of `__launch_bounds__(maxThreadsPerBlock,minBlocksPerMultiprocessor)` can improve performance in some cases. The right value for `minBlocksPerMultiprocessor` should be determined using a detailed per kernel analysis.\ \ ### 10.1.1. Calculating Occupancy[](#calculating-occupancy "Permalink to this headline")\ \ One of several factors that determine occupancy is register availability. Register storage enables threads to keep local variables nearby for low-latency access. However, the set of registers (known as the _register file_) is a limited commodity that all threads resident on a multiprocessor must share. Registers are allocated to an entire block all at once. So, if each thread block uses many registers, the number of thread blocks that can be resident on a multiprocessor is reduced, thereby lowering the occupancy of the multiprocessor. The maximum number of registers per thread can be set manually at compilation time per-file using the `-maxrregcount` option or per-kernel using the `__launch_bounds__` qualifier (see [Register Pressure](#register-pressure)\ ).\ \ For purposes of calculating occupancy, the number of registers used by each thread is one of the key factors. For example, on devices of [CUDA Compute Capability](#cuda-compute-capability)\ 7.0 each multiprocessor has 65,536 32-bit registers and can have a maximum of 2048 simultaneous threads resident (64 warps x 32 threads per warp). This means that in one of these devices, for a multiprocessor to have 100% occupancy, each thread can use at most 32 registers. However, this approach of determining how register count affects occupancy does not take into account the register allocation granularity. For example, on a device of compute capability 7.0, a kernel with 128-thread blocks using 37 registers per thread results in an occupancy of 75% with 12 active 128-thread blocks per multi-processor, whereas a kernel with 320-thread blocks using the same 37 registers per thread results in an occupancy of 63% because only four 320-thread blocks can reside on a multiprocessor. Furthermore, register allocations are rounded up to the nearest 256 registers per warp.\ \ The number of registers available, the maximum number of simultaneous threads resident on each multiprocessor, and the register allocation granularity vary over different compute capabilities. Because of these nuances in register allocation and the fact that a multiprocessor’s shared memory is also partitioned between resident thread blocks, the exact relationship between register usage and occupancy can be difficult to determine. The `--ptxas options=v` option of `nvcc` details the number of registers used per thread for each kernel. See Hardware Multithreading of the CUDA C++ Programming Guide for the register allocation formulas for devices of various compute capabilities and Features and Technical Specifications of the CUDA C++ Programming Guide for the total number of registers available on those devices. Alternatively, NVIDIA provides an occupancy calculator as part of Nsight Compute; refer to [https://docs.nvidia.com/nsight-compute/NsightCompute/index.html#occupancy-calculator](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html#occupancy-calculator)\ .\ \ ![Using the CUDA Occupancy Calculator to project GPU multiprocessor occupancy](_images/using-cuda-occupancy-calculator-usage.png)\ \ Figure 15 Using the CUDA Occupancy Calculator to project GPU multiprocessor occupancy[](#cuda-occupancy-calculator-usage-project-gpu-multi-occupancy-figure "Permalink to this image")\ \ An application can also use the Occupancy API from the CUDA Runtime, e.g. `cudaOccupancyMaxActiveBlocksPerMultiprocessor`, to dynamically select launch configurations based on runtime parameters.\ \ 10.2. Hiding Register Dependencies[](#hiding-register-dependencies "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------\ \ Note\ \ **Medium Priority:** To hide latency arising from register dependencies, maintain sufficient numbers of active threads per multiprocessor (i.e., sufficient occupancy).\ \ Register dependencies arise when an instruction uses a result stored in a register written by an instruction before it. The latency of most arithmetic instructions is typically 4 cycles on devices of compute capability 7.0. So threads must wait approximatly 4 cycles before using an arithmetic result. However, this latency can be completely hidden by the execution of threads in other warps. See [Registers](index.html#registers)\ for details.\ \ 10.3. Thread and Block Heuristics[](#thread-and-block-heuristics "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------------\ \ Note\ \ **Medium Priority:** The number of threads per block should be a multiple of 32 threads, because this provides optimal computing efficiency and facilitates coalescing.\ \ The dimension and size of blocks per grid and the dimension and size of threads per block are both important factors. The multidimensional aspect of these parameters allows easier mapping of multidimensional problems to CUDA and does not play a role in performance. As a result, this section discusses size but not dimension.\ \ Latency hiding and occupancy depend on the number of active warps per multiprocessor, which is implicitly determined by the execution parameters along with resource (register and shared memory) constraints. Choosing execution parameters is a matter of striking a balance between latency hiding (occupancy) and resource utilization.\ \ Choosing the execution configuration parameters should be done in tandem; however, there are certain heuristics that apply to each parameter individually. When choosing the first execution configuration parameter-the number of blocks per grid, or _grid size_ - the primary concern is keeping the entire GPU busy. The number of blocks in a grid should be larger than the number of multiprocessors so that all multiprocessors have at least one block to execute. Furthermore, there should be multiple active blocks per multiprocessor so that blocks that aren’t waiting for a `__syncthreads()` can keep the hardware busy. This recommendation is subject to resource availability; therefore, it should be determined in the context of the second execution parameter - the number of threads per block, or _block size_ - as well as shared memory usage. To scale to future devices, the number of blocks per kernel launch should be in the thousands.\ \ When choosing the block size, it is important to remember that multiple concurrent blocks can reside on a multiprocessor, so occupancy is not determined by block size alone. In particular, a larger block size does not imply a higher occupancy.\ \ As mentioned in [Occupancy](index.html#occupancy)\ , higher occupancy does not always equate to better performance. For example, improving occupancy from 66 percent to 100 percent generally does not translate to a similar increase in performance. A lower occupancy kernel will have more registers available per thread than a higher occupancy kernel, which may result in less register spilling to local memory; in particular, with a high degree of exposed instruction-level parallelism (ILP) it is, in some cases, possible to fully cover latency with a low occupancy.\ \ There are many such factors involved in selecting block size, and inevitably some experimentation is required. However, a few rules of thumb should be followed:\ \ * Threads per block should be a multiple of warp size to avoid wasting computation on under-populated warps and to facilitate coalescing.\ \ * A minimum of 64 threads per block should be used, and only if there are multiple concurrent blocks per multiprocessor.\ \ * Between 128 and 256 threads per block is a good initial range for experimentation with different block sizes.\ \ * Use several smaller thread blocks rather than one large thread block per multiprocessor if latency affects performance. This is particularly beneficial to kernels that frequently call `__syncthreads()`.\ \ \ Note that when a thread block allocates more registers than are available on a multiprocessor, the kernel launch fails, as it will when too much shared memory or too many threads are requested.\ \ 10.4. Effects of Shared Memory[](#effects-of-shared-memory "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------\ \ Shared memory can be helpful in several situations, such as helping to coalesce or eliminate redundant access to global memory. However, it also can act as a constraint on occupancy. In many cases, the amount of shared memory required by a kernel is related to the block size that was chosen, but the mapping of threads to shared memory elements does not need to be one-to-one. For example, it may be desirable to use a 64x64 element shared memory array in a kernel, but because the maximum number of threads per block is 1024, it is not possible to launch a kernel with 64x64 threads per block. In such cases, kernels with 32x32 or 64x16 threads can be launched with each thread processing four elements of the shared memory array. The approach of using a single thread to process multiple elements of a shared memory array can be beneficial even if limits such as threads per block are not an issue. This is because some operations common to each element can be performed by the thread once, amortizing the cost over the number of shared memory elements processed by the thread.\ \ A useful technique to determine the sensitivity of performance to occupancy is through experimentation with the amount of dynamically allocated shared memory, as specified in the third parameter of the execution configuration. By simply increasing this parameter (without modifying the kernel), it is possible to effectively reduce the occupancy of the kernel and measure its effect on performance.\ \ 10.5. Concurrent Kernel Execution[](#concurrent-kernel-execution "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------------\ \ As described in [Asynchronous and Overlapping Transfers with Computation](index.html#asynchronous-transfers-and-overlapping-transfers-with-computation)\ , CUDA streams can be used to overlap kernel execution with data transfers. On devices that are capable of concurrent kernel execution, streams can also be used to execute multiple kernels simultaneously to more fully take advantage of the device’s multiprocessors. Whether a device has this capability is indicated by the `concurrentKernels` field of the `cudaDeviceProp` structure (or listed in the output of the `deviceQuery` CUDA Sample). Non-default streams (streams other than stream 0) are required for concurrent execution because kernel calls that use the default stream begin only after all preceding calls on the device (in any stream) have completed, and no operation on the device (in any stream) commences until they are finished.\ \ The following example illustrates the basic technique. Because `kernel1` and `kernel2` are executed in different, non-default streams, a capable device can execute the kernels at the same time.\ \ cudaStreamCreate(&stream1);\ cudaStreamCreate(&stream2);\ kernel1<<>>(data\_1);\ kernel2<<>>(data\_2);\ \ 10.6. Multiple contexts[](#multiple-contexts "Permalink to this headline")\ \ ----------------------------------------------------------------------------\ \ CUDA work occurs within a process space for a particular GPU known as a _context_. The context encapsulates kernel launches and memory allocations for that GPU as well as supporting constructs such as the page tables. The context is explicit in the CUDA Driver API but is entirely implicit in the CUDA Runtime API, which creates and manages contexts automatically.\ \ With the CUDA Driver API, a CUDA application process can potentially create more than one context for a given GPU. If multiple CUDA application processes access the same GPU concurrently, this almost always implies multiple contexts, since a context is tied to a particular host process unless [Multi-Process Service](https://docs.nvidia.com/deploy/mps/index.html)\ is in use.\ \ While multiple contexts (and their associated resources such as global memory allocations) can be allocated concurrently on a given GPU, only one of these contexts can execute work at any given moment on that GPU; contexts sharing the same GPU are time-sliced. Creating additional contexts incurs memory overhead for per-context data and time overhead for context switching. Furthermore, the need for context switching can reduce utilization when work from several contexts could otherwise execute concurrently (see also [Concurrent Kernel Execution](index.html#concurrent-kernel-execution)\ ).\ \ Therefore, it is best to avoid multiple contexts per GPU within the same CUDA application. To assist with this, the CUDA Driver API provides methods to access and manage a special context on each GPU called the _primary context_. These are the same contexts used implicitly by the CUDA Runtime when there is not already a current context for a thread.\ \ // When initializing the program/library\ CUcontext ctx;\ cuDevicePrimaryCtxRetain(&ctx, dev);\ \ // When the program/library launches work\ cuCtxPushCurrent(ctx);\ kernel<<<...\>>>(...);\ cuCtxPopCurrent(&ctx);\ \ // When the program/library is finished with the context\ cuDevicePrimaryCtxRelease(dev);\ \ Note\ \ NVIDIA-SMI can be used to configure a GPU for [exclusive process mode](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-modes)\ , which limits the number of contexts per GPU to one. This context can be current to as many threads as desired within the creating process, and `cuDevicePrimaryCtxRetain` will fail if a non-primary context that was created with the CUDA driver API already exists on the device.\ \ 11\. Instruction Optimization[](#instruction-optimization "Permalink to this headline")\ \ =========================================================================================\ \ Awareness of how instructions are executed often permits low-level optimizations that can be useful, especially in code that is run frequently (the so-called hot spot in a program). Best practices suggest that this optimization be performed after all higher-level optimizations have been completed.\ \ 11.1. Arithmetic Instructions[](#arithmetic-instructions "Permalink to this headline")\ \ ----------------------------------------------------------------------------------------\ \ Single-precision floats provide the best performance, and their use is highly encouraged. The throughput of individual arithmetic operations is detailed in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html)\ .\ \ ### 11.1.1. Division Modulo Operations[](#division-modulo-operations "Permalink to this headline")\ \ Note\ \ **Low Priority:** Use shift operations to avoid expensive division and modulo calculations.\ \ Integer division and modulo operations are particularly costly and should be avoided or replaced with bitwise operations whenever possible: If \\(n\\) is a power of 2, ( \\(i/n\\) ) is equivalent to ( \\(i \\gg {log2}(n)\\) ) and ( \\(i\\% n\\) ) is equivalent to ( \\(i\\&\\left( {n - 1} \\right)\\) ).\ \ The compiler will perform these conversions if n is literal. (For further information, refer to Performance Guidelines in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html)\ ).\ \ ### 11.1.2. Loop Counters Signed vs. Unsigned[](#loop-counters-signed-vs-unsigned "Permalink to this headline")\ \ Note\ \ **Low Medium Priority:** Use signed integers rather than unsigned integers as loop counters.\ \ In the C language standard, unsigned integer overflow semantics are well defined, whereas signed integer overflow causes undefined results. Therefore, the compiler can optimize more aggressively with signed arithmetic than it can with unsigned arithmetic. This is of particular note with loop counters: since it is common for loop counters to have values that are always positive, it may be tempting to declare the counters as unsigned. For slightly better performance, however, they should instead be declared as signed.\ \ For example, consider the following code:\ \ for (i \= 0; i < n; i++) {\ out\[i\] \= in\[offset + stride\*i\];\ }\ \ Here, the sub-expression `stride*i` could overflow a 32-bit integer, so if `i` is declared as unsigned, the overflow semantics prevent the compiler from using some optimizations that might otherwise have applied, such as strength reduction. If instead `i` is declared as signed, where the overflow semantics are undefined, the compiler has more leeway to use these optimizations.\ \ ### 11.1.3. Reciprocal Square Root[](#reciprocal-square-root "Permalink to this headline")\ \ The reciprocal square root should always be invoked explicitly as `rsqrtf()` for single precision and `rsqrt()` for double precision. The compiler optimizes `1.0f/sqrtf(x)` into `rsqrtf()` only when this does not violate IEEE-754 semantics.\ \ ### 11.1.4. Other Arithmetic Instructions[](#other-arithmetic-instructions "Permalink to this headline")\ \ Note\ \ **Low Priority:** Avoid automatic conversion of doubles to floats.\ \ The compiler must on occasion insert conversion instructions, introducing additional execution cycles. This is the case for:\ \ * Functions operating on `char` or `short` whose operands generally need to be converted to an `int`\ \ * Double-precision floating-point constants (defined without any type suffix) used as input to single-precision floating-point computations\ \ \ The latter case can be avoided by using single-precision floating-point constants, defined with an `f` suffix such as `3.141592653589793f`, `1.0f`, `0.5f`.\ \ For single-precision code, use of the float type and the single-precision math functions are highly recommended.\ \ It should also be noted that the CUDA math library’s complementary error function, `erfcf()`, is particularly fast with full single-precision accuracy.\ \ ### 11.1.5. Exponentiation With Small Fractional Arguments[](#exponentiation-with-small-fractional-arguments "Permalink to this headline")\ \ For some fractional exponents, exponentiation can be accelerated significantly compared to the use of `pow()` by using square roots, cube roots, and their inverses. For those exponentiations where the exponent is not exactly representable as a floating-point number, such as 1/3, this can also provide much more accurate results, as use of `pow()` magnifies the initial representational error.\ \ The formulas in the table below are valid for `x >= 0, x != -0`, that is, `signbit(x) == 0`.\ \ | | |\ | --- | --- |Table 5 Formulae for exponentiation by small fractions[](#id85 "Permalink to this table")\ \ | Computation | Formula |\ | --- | --- |\ | x1/9 | `r = rcbrt(rcbrt(x))` |\ | x\-1/9 | `r = cbrt(rcbrt(x))` |\ | x1/6 | `r = rcbrt(rsqrt(x))` |\ | x\-1/6 | `r = rcbrt(sqrt(x))` |\ | x1/4 | `r = rsqrt(rsqrt(x))` |\ | x\-1/4 | `r = sqrt(rsqrt(x))` |\ | x1/3 | `r = cbrt(x)` |\ | x\-1/3 | `r = rcbrt(x)` |\ | x1/2 | `r = sqrt(x)` |\ | x\-1/2 | `r = rsqrt(x)` |\ | x2/3 | `r = cbrt(x); r = r*r` |\ | x\-2/3 | `r = rcbrt(x); r = r*r` |\ | x3/4 | `r = sqrt(x); r = r*sqrt(r)` |\ | x\-3/4 | `r = rsqrt(x); r = r*sqrt(r)` |\ | x7/6 | `r = x*rcbrt(rsqrt(x))` |\ | x\-7/6 | `r = (1/x) * rcbrt(sqrt(x))` |\ | x5/4 | `r = x*rsqrt(rsqrt(x))` |\ | x\-5/4 | `r = (1/x)*sqrt(rsqrt(x))` |\ | x4/3 | `r = x*cbrt(x)` |\ | x\-4/3 | `r = (1/x)*rcbrt(x)` |\ | x3/2 | `r = x*sqrt(x)` |\ | x\-3/2 | `r = (1/x)*rsqrt(x)` |\ \ ### 11.1.6. Math Libraries[](#math-libraries "Permalink to this headline")\ \ Note\ \ **Medium Priority:** Use the fast math library whenever speed trumps precision.\ \ Two types of runtime math operations are supported. They can be distinguished by their names: some have names with prepended underscores, whereas others do not (e.g., `__functionName()` versus `functionName()`). Functions following the `__functionName()` naming convention map directly to the hardware level. They are faster but provide somewhat lower accuracy (e.g., `__sinf(x)` and `__expf(x)`). Functions following `functionName()` naming convention are slower but have higher accuracy (e.g., `sinf(x)` and `expf(x)`). The throughput of `__sinf(x)`, `__cosf(x)`, and`__expf(x)` is much greater than that of `sinf(x)`, `cosf(x)`, and `expf(x)`. The latter become even more expensive (about an order of magnitude slower) if the magnitude of the argument `x` needs to be reduced. Moreover, in such cases, the argument-reduction code uses local memory, which can affect performance even more because of the high latency of local memory. More details are available in the _CUDA C++ Programming Guide_.\ \ Note also that whenever sine and cosine of the same argument are computed, the `sincos` family of instructions should be used to optimize performance:\ \ * `__sincosf()` for single-precision fast math (see next paragraph)\ \ * `sincosf()` for regular single-precision\ \ * `sincos()` for double precision\ \ \ The `-use_fast_math` compiler option of `nvcc` coerces every `functionName()` call to the equivalent `__functionName()` call. It also disables single-precision denormal support and lowers the precision of single-precision division in general. This is an aggressive optimization that can both reduce numerical accuracy and alter special case handling. A more robust approach is to selectively introduce calls to fast intrinsic functions only if merited by performance gains and where altered behavior can be tolerated. Note this switch is effective only on single-precision floating point.\ \ Note\ \ **Medium Priority:** Prefer faster, more specialized math functions over slower, more general ones when possible.\ \ For small integer powers (e.g., _x2_ or _x3_), explicit multiplication is almost certainly faster than the use of general exponentiation routines such as `pow()`. While compiler optimization improvements continually seek to narrow this gap, explicit multiplication (or the use of an equivalent purpose-built inline function or macro) can have a significant advantage. This advantage is increased when several powers of the same base are needed (e.g., where both _x2_ and _x5_ are calculated in close proximity), as this aids the compiler in its common sub-expression elimination (CSE) optimization.\ \ For exponentiation using base 2 or 10, use the functions `exp2()` or `expf2()` and `exp10()` or `expf10()` rather than the functions `pow()` or `powf()`. Both `pow()` and `powf()` are heavy-weight functions in terms of register pressure and instruction count due to the numerous special cases arising in general exponentiation and the difficulty of achieving good accuracy across the entire ranges of the base and the exponent. The functions `exp2()`, `exp2f()`, `exp10()`, and `exp10f()`, on the other hand, are similar to `exp()` and `expf()` in terms of performance, and can be as much as ten times faster than their `pow()`/`powf()` equivalents.\ \ For exponentiation with an exponent of 1/3, use the `cbrt()` or `cbrtf()` function rather than the generic exponentiation functions `pow()` or `powf()`, as the former are significantly faster than the latter. Likewise, for exponentation with an exponent of -1/3, use `rcbrt()` or `rcbrtf()`.\ \ Replace `sin(π*)` with `sinpi()`, `cos(π*)` with `cospi()`, and `sincos(π*)` with `sincospi()`. This is advantageous with regard to both accuracy and performance. As a particular example, to evaluate the sine function in degrees instead of radians, use `sinpi(x/180.0)`. Similarly, the single-precision functions `sinpif()`, `cospif()`, and `sincospif()` should replace calls to `sinf()`, `cosf()`, and `sincosf()` when the function argument is of the form `π*`. (The performance advantage `sinpi()` has over `sin()` is due to simplified argument reduction; the accuracy advantage is because `sinpi()` multiplies by `π` only implicitly, effectively using an infinitely precise mathematical `π` rather than a single- or double-precision approximation thereof.)\ \ ### 11.1.7. Precision-related Compiler Flags[](#precision-related-compiler-flags "Permalink to this headline")\ \ By default, the `nvcc` compiler generates IEEE-compliant code, but it also provides options to generate code that somewhat less accurate but faster:\ \ * `-ftz=true` (denormalized numbers are flushed to zero)\ \ * `-prec-div=false` (less precise division)\ \ * `-prec-sqrt=false` (less precise square root)\ \ \ Another, more aggressive, option is `-use_fast_math`, which coerces every `functionName()` call to the equivalent `__functionName()` call. This makes the code run faster at the cost of diminished precision and accuracy. See [Math Libraries](#math-libraries)\ .\ \ 11.2. Memory Instructions[](#memory-instructions "Permalink to this headline")\ \ --------------------------------------------------------------------------------\ \ Note\ \ **High Priority:** Minimize the use of global memory. Prefer shared memory access where possible.\ \ Memory instructions include any instruction that reads from or writes to shared, local, or global memory. When accessing uncached local or global memory, there are hundreds of clock cycles of memory latency.\ \ As an example, the assignment operator in the following sample code has a high throughput, but, crucially, there is a latency of hundreds of clock cycles to read data from global memory:\ \ \_\_shared\_\_ float shared\[32\];\ \_\_device\_\_ float device\[32\];\ shared\[threadIdx.x\] \= device\[threadIdx.x\];\ \ Much of this global memory latency can be hidden by the thread scheduler if there are sufficient independent arithmetic instructions that can be issued while waiting for the global memory access to complete. However, it is best to avoid accessing global memory whenever possible.\ \ 12\. Control Flow[](#control-flow "Permalink to this headline")\ \ =================================================================\ \ 12.1. Branching and Divergence[](#branching-and-divergence "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------\ \ Note\ \ **High Priority:** Avoid different execution paths within the same warp.\ \ Flow control instructions (`if`, `switch`, `do`, `for`, `while`) can significantly affect the instruction throughput by causing threads of the same warp to diverge; that is, to follow different execution paths. If this happens, the different execution paths must be executed separately; this increases the total number of instructions executed for this warp.\ \ To obtain best performance in cases where the control flow depends on the thread ID, the controlling condition should be written so as to minimize the number of divergent warps.\ \ This is possible because the distribution of the warps across the block is deterministic as mentioned in SIMT Architecture of the CUDA C++ Programming Guide. A trivial example is when the controlling condition depends only on (`threadIdx` / `WSIZE`) where `WSIZE` is the warp size.\ \ In this case, no warp diverges because the controlling condition is perfectly aligned with the warps.\ \ For branches including just a few instructions, warp divergence generally results in marginal performance losses. For example, the compiler may use predication to avoid an actual branch. Instead, all instructions are scheduled, but a per-thread condition code or predicate controls which threads execute the instructions. Threads with a false predicate do not write results, and also do not evaluate addresses or read operands.\ \ Starting with the Volta architecture, Independent Thread Scheduling allows a warp to remain diverged outside of the data-dependent conditional block. An explicit `__syncwarp()` can be used to guarantee that the warp has reconverged for subsequent instructions.\ \ 12.2. Branch Predication[](#branch-predication "Permalink to this headline")\ \ ------------------------------------------------------------------------------\ \ Note\ \ **Low Priority:** Make it easy for the compiler to use branch predication in lieu of loops or control statements.\ \ Sometimes, the compiler may unroll loops or optimize out `if` or `switch` statements by using branch predication instead. In these cases, no warp can ever diverge. The programmer can also control loop unrolling using\ \ #pragma unroll\ \ For more information on this pragma, refer to the CUDA C++ Programming Guide.\ \ When using branch predication, none of the instructions whose execution depends on the controlling condition is skipped. Instead, each such instruction is associated with a per-thread condition code or predicate that is set to true or false according to the controlling condition. Although each of these instructions is scheduled for execution, only the instructions with a true predicate are actually executed. Instructions with a false predicate do not write results, and they also do not evaluate addresses or read operands.\ \ The compiler replaces a branch instruction with predicated instructions only if the number of instructions controlled by the branch condition is less than or equal to a certain threshold.\ \ 13\. Deploying CUDA Applications[](#deploying-cuda-applications "Permalink to this headline")\ \ ===============================================================================================\ \ Having completed the GPU acceleration of one or more components of the application it is possible to compare the outcome with the original expectation. Recall that the initial _assess_ step allowed the developer to determine an upper bound for the potential speedup attainable by accelerating given hotspots.\ \ Before tackling other hotspots to improve the total speedup, the developer should consider taking the partially parallelized implementation and carry it through to production. This is important for a number of reasons; for example, it allows the user to profit from their investment as early as possible (the speedup may be partial but is still valuable), and it minimizes risk for the developer and the user by providing an evolutionary rather than revolutionary set of changes to the application.\ \ 14\. Understanding the Programming Environment[](#understanding-the-programming-environment "Permalink to this headline")\ \ ===========================================================================================================================\ \ With each generation of NVIDIA processors, new features are added to the GPU that CUDA can leverage. Consequently, it’s important to understand the characteristics of the architecture.\ \ Programmers should be aware of two version numbers. The first is the [compute capability](#cuda-compute-capability)\ , and the second is the version number of the CUDA Runtime and CUDA Driver APIs.\ \ 14.1. CUDA Compute Capability[](#cuda-compute-capability "Permalink to this headline")\ \ ----------------------------------------------------------------------------------------\ \ The _compute capability_ describes the features of the hardware and reflects the set of instructions supported by the device as well as other specifications, such as the maximum number of threads per block and the number of registers per multiprocessor. Higher compute capability versions are supersets of lower (that is, earlier) versions, so they are backward compatible.\ \ The compute capability of the GPU in the device can be queried programmatically as illustrated in the `deviceQuery` CUDA Sample. The output for that program is shown in [Figure 16](#sample-cuda-configuration-data-reported-devicequery-figure)\ . This information is obtained by calling `cudaGetDeviceProperties()` and accessing the information in the structure it returns.\ \ [![Sample CUDA configuration data reported by deviceQuery](_images/sample-cuda-configuration-data.png)](_images/sample-cuda-configuration-data.png)\ \ Figure 16 Sample CUDA configuration data reported by deviceQuery[](#sample-cuda-configuration-data-reported-devicequery-figure "Permalink to this image")\ \ The major and minor revision numbers of the compute capability are shown on the seventh line of [Figure 16](#sample-cuda-configuration-data-reported-devicequery-figure)\ . Device 0 of this system has compute capability 7.0.\ \ More details about the compute capabilities of various GPUs are in CUDA-Enabled GPUs and Compute Capabilities of the CUDA C++ Programming Guide. In particular, developers should note the number of multiprocessors on the device, the number of registers and the amount of memory available, and any special capabilities of the device.\ \ 14.2. Additional Hardware Data[](#additional-hardware-data "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------\ \ Certain hardware features are not described by the compute capability. For example, the ability to overlap kernel execution with asynchronous data transfers between the host and the device is available on most but not all GPUs irrespective of the compute capability. In such cases, call `cudaGetDeviceProperties()` to determine whether the device is capable of a certain feature. For example, the `asyncEngineCount` field of the device property structure indicates whether overlapping kernel execution and data transfers is possible (and, if so, how many concurrent transfers are possible); likewise, the `canMapHostMemory` field indicates whether zero-copy data transfers can be performed.\ \ 14.3. Which Compute Capability Target[](#which-compute-capability-target "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------------\ \ To target specific versions of NVIDIA hardware and CUDA software, use the `-arch`, `-code`, and `-gencode` options of `nvcc`. Code that uses the warp shuffle operation, for example, must be compiled with `-arch=sm_30` (or higher compute capability).\ \ See [Building for Maximum Compatibility](#building-for-maximum-compatibility)\ for further discussion of the flags used for building code for multiple generations of CUDA-capable device simultaneously.\ \ 14.4. CUDA Runtime[](#cuda-runtime "Permalink to this headline")\ \ ------------------------------------------------------------------\ \ The host runtime component of the CUDA software environment can be used only by host functions. It provides functions to handle the following:\ \ * Device management\ \ * Context management\ \ * Memory management\ \ * Code module management\ \ * Execution control\ \ * Texture reference management\ \ * Interoperability with OpenGL and Direct3D\ \ \ As compared to the lower-level CUDA Driver API, the CUDA Runtime greatly eases device management by providing implicit initialization, context management, and device code module management. The C++ host code generated by `nvcc` utilizes the CUDA Runtime, so applications that link to this code will depend on the CUDA Runtime; similarly, any code that uses the `cuBLAS`, `cuFFT`, and other CUDA Toolkit libraries will also depend on the CUDA Runtime, which is used internally by these libraries.\ \ The functions that make up the CUDA Runtime API are explained in the CUDA Toolkit Reference Manual.\ \ The CUDA Runtime handles kernel loading and setting up kernel parameters and launch configuration before the kernel is launched. The implicit driver version checking, code initialization, CUDA context management, CUDA module management (cubin to function mapping), kernel configuration, and parameter passing are all performed by the CUDA Runtime.\ \ It comprises two principal parts:\ \ * A C-style function interface (`cuda_runtime_api.h`).\ \ * C++-style convenience wrappers (`cuda_runtime.h`) built on top of the C-style functions.\ \ \ For more information on the Runtime API, refer to CUDA Runtime of the CUDA C++ Programming Guide.\ \ 15\. CUDA Compatibility Developer’s Guide[](#cuda-compatibility-developer-s-guide "Permalink to this headline")\ \ =================================================================================================================\ \ CUDA Toolkit is released on a monthly release cadence to deliver new features, performance improvements, and critical bug fixes. CUDA compatibility allows users to update the latest CUDA Toolkit software (including the compiler, libraries, and tools) without requiring update to the entire driver stack.\ \ The CUDA software environment consists of three parts:\ \ * CUDA Toolkit (libraries, CUDA runtime and developer tools) - SDK for developers to build CUDA applications.\ \ * CUDA driver - User-mode driver component used to run CUDA applications (e.g. libcuda.so on Linux systems).\ \ * NVIDIA GPU device driver - Kernel-mode driver component for NVIDIA GPUs.\ \ \ On Linux systems, the CUDA driver and kernel mode components are delivered together in the NVIDIA display driver package. This is shown in Figure 1.\ \ ![Components of CUDA](_images/CUDA-components.png)\ \ Figure 17 Components of CUDA[](#id86 "Permalink to this image")\ \ The CUDA compiler (nvcc), provides a way to handle CUDA and non-CUDA code (by splitting and steering compilation), along with the CUDA runtime, is part of the CUDA compiler toolchain. The CUDA Runtime API provides developers with high-level C++ interface for simplified management of devices, kernel executions etc., While the CUDA driver API provides ([CUDA Driver API](https://docs.nvidia.com/cuda/cuda-driver-api/index.html)\ ) a low-level programming interface for applications to target NVIDIA hardware.\ \ Built on top of these technologies are CUDA libraries, some of which are included in the CUDA Toolkit, while others such as cuDNN may be released independently of the CUDA Toolkit.\ \ 15.1. CUDA Toolkit Versioning[](#cuda-toolkit-versioning "Permalink to this headline")\ \ ----------------------------------------------------------------------------------------\ \ Starting with CUDA 11, the toolkit versions are based on an industry-standard semantic versioning scheme: .X.Y.Z, where:\ \ * .X stands for the major version - APIs have changed and binary compatibility is broken.\ \ * .Y stands for the minor version - Introduction of new APIs, deprecation of old APIs, and source compatibility might be broken but binary compatibility is maintained.\ \ * .Z stands for the release/patch version - new updates and patches will increment this.\ \ \ Each component in the toolkit is recommended to be semantically versioned. From CUDA 11.3 NVRTC is also semantically versioned. We will note some of them later on in the document. The versions of the components in the toolkit are available in this [table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions)\ .\ \ Compatibility of the CUDA platform is thus intended to address a few scenarios:\ \ 1. NVIDIA driver upgrades to systems with GPUs running in production for enterprises or datacenters can be complex and may need advance planning. Delays in rolling out new NVIDIA drivers could mean that users of such systems may not have access to new features available in CUDA releases. Not requiring driver updates for new CUDA releases can mean that new versions of the software can be made available faster to users.\ \ 2. Many software libraries and applications built on top of CUDA (e.g. math libraries or deep learning frameworks) do not have a direct dependency on the CUDA runtime, compiler or driver. In such cases, users or developers can still benefit from not having to upgrade the entire CUDA Toolkit or driver to use these libraries or frameworks.\ \ 3. Upgrading dependencies is error-prone and time consuming, and in some corner cases, can even change the semantics of a program. Constantly recompiling with the latest CUDA Toolkit means forcing upgrades on the end-customers of an application product. Package managers facilitate this process but unexpected issues can still arise and if a bug is found, it necessitates a repeat of the above upgrade process.\ \ \ CUDA supports several compatibility choices:\ \ 1. First introduced in CUDA 10, the **CUDA Forward Compatible Upgrade** is designed to allow users to get access to new CUDA features and run applications built with new CUDA releases on systems with older installations of the NVIDIA datacenter driver.\ \ 2. First introduced in CUDA 11.1, **CUDA Enhanced Compatibility** provides two benefits:\ \ * By leveraging semantic versioning across components in the CUDA Toolkit, an application can be built for one CUDA minor release (for example 11.1) and work across all future minor releases within the major family (i.e. 11.x).\ \ * The CUDA runtime has relaxed the minimum driver version check and thus no longer requires a driver upgrade when moving to a new minor release.\ \ 3. The CUDA driver ensures backward Binary Compatibility is maintained for compiled CUDA applications. Applications compiled with CUDA toolkit versions as old as 3.2 will run on newer drivers.\ \ \ 15.2. Source Compatibility[](#source-compatibility "Permalink to this headline")\ \ ----------------------------------------------------------------------------------\ \ We define source compatibility as a set of guarantees provided by the library, where a well-formed application built against a specific version of the library (using the SDK) will continue to build and run without errors when a newer version of the SDK is installed.\ \ Both the CUDA driver and the CUDA runtime are not source compatible across the different SDK releases. APIs can be deprecated and removed. Therefore, an application that compiled successfully on an older version of the toolkit may require changes in order to compile against a newer version of the toolkit.\ \ Developers are notified through deprecation and documentation mechanisms of any current or upcoming changes. This does not mean that application binaries compiled using an older toolkit will not be supported anymore. Application binaries rely on CUDA Driver API interface and even though the CUDA Driver API itself may also have changed across toolkit versions, CUDA guarantees Binary Compatibility of the CUDA Driver API interface.\ \ 15.3. Binary Compatibility[](#binary-compatibility "Permalink to this headline")\ \ ----------------------------------------------------------------------------------\ \ We define binary compatibility as a set of guarantees provided by the library, where an application targeting the said library will continue to work when dynamically linked against a different version of the library.\ \ The CUDA Driver API has a versioned C-style ABI, which guarantees that applications that were running against an older driver (for example CUDA 3.2) will still run and function correctly against a modern driver (for example one shipped with CUDA 11.0). This means that even though an application source might need to be changed if it has to be recompiled against a newer CUDA Toolkit in order to use the newer features, replacing the driver components installed in a system with a newer version will always support existing applications and its functions.\ \ The CUDA Driver API thus is binary-compatible (the OS loader can pick up a newer version and the application continues to work) but not source-compatible (rebuilding your application against a newer SDK might require source changes).\ \ ![CUDA Toolkit and Minimum Driver Versions](_images/CTK-and-min-driver-versions.png)\ \ Figure 18 CUDA Toolkit and Minimum Driver Versions[](#id87 "Permalink to this image")\ \ Before we proceed further on this topic, it’s important for developers to understand the concept of Minimum Driver Version and how that may affect them.\ \ Each version of the CUDA Toolkit (and runtime) requires a minimum version of the NVIDIA driver. Applications compiled against a CUDA Toolkit version will only run on systems with the specified minimum driver version for that toolkit version. Prior to CUDA 11.0, the minimum driver version for a toolkit was the same as the driver shipped with that version of the CUDA Toolkit.\ \ So, when an application is built with CUDA 11.0, it can only run on a system with an R450 or later driver. If such an application is run on a system with the R418 driver installed, CUDA initialization will return an error as can be seen in the example below.\ \ In this example, the deviceQuery sample is compiled with CUDA 11.1 and is run on a system with R418. In this scenario, CUDA initialization returns an error due to the minimum driver requirement.\ \ ubuntu@:~/samples/1\_Utilities/deviceQuery\ $ make\ /usr/local/cuda-11.1/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute\_35,code=sm\_35 -gencode arch=compute\_37,code=sm\_37 -gencode arch=compute\_50,code=sm\_50 -gencode arch=compute\_52,code=sm\_52 -gencode arch=compute\_60,code=sm\_60 -gencode arch=compute\_61,code=sm\_61 -gencode arch=compute\_70,code=sm\_70 -gencode arch=compute\_75,code=sm\_75 -gencode arch=compute\_80,code=sm\_80 -gencode arch=compute\_86,code=sm\_86 -gencode arch=compute\_86,code=compute\_86 -o deviceQuery.o -c deviceQuery.cpp\ \ /usr/local/cuda-11.1/bin/nvcc -ccbin g++ -m64 -gencode arch=compute\_35,code=sm\_35 -gencode arch=compute\_37,code=sm\_37 -gencode arch=compute\_50,code=sm\_50 -gencode arch=compute\_52,code=sm\_52 -gencode arch=compute\_60,code=sm\_60 -gencode arch=compute\_61,code=sm\_61 -gencode arch=compute\_70,code=sm\_70 -gencode arch=compute\_75,code=sm\_75 -gencode arch=compute\_80,code=sm\_80 -gencode arch=compute\_86,code=sm\_86 -gencode arch=compute\_86,code=compute\_86 -o deviceQuery deviceQuery.o\ \ $ nvidia-smi\ \ +-----------------------------------------------------------------------------+\ | NVIDIA-SMI 418.165.02 Driver Version: 418.165.02 CUDA Version: 10.1 |\ |-------------------------------+----------------------+----------------------+\ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\ | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\ |===============================+======================+======================|\ | 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |\ | N/A 42C P0 28W / 70W | 0MiB / 15079MiB | 0% Default |\ +-------------------------------+----------------------+----------------------+\ \ +-----------------------------------------------------------------------------+\ | Processes: GPU Memory |\ | GPU PID Type Process name Usage |\ |=============================================================================|\ | No running processes found |\ +-----------------------------------------------------------------------------+\ \ \ $ samples/bin/x86\_64/linux/release/deviceQuery\ samples/bin/x86\_64/linux/release/deviceQuery Starting...\ \ CUDA Device Query (Runtime API) version (CUDART static linking)\ \ cudaGetDeviceCount returned 3\ -> initialization error\ Result = FAIL\ \ Refer to the [CUDA Toolkit Release Notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html)\ for details for the minimum driver version and the version of the driver shipped with the toolkit.\ \ ### 15.3.1. CUDA Binary (cubin) Compatibility[](#cuda-binary-cubin-compatibility "Permalink to this headline")\ \ A slightly related but important topic is one of application binary compatibility across GPU architectures in CUDA.\ \ CUDA C++ provides a simple path for users familiar with the C++ programming language to easily write programs for execution by the device. Kernels can be written using the CUDA instruction set architecture, called PTX, which is described in the PTX reference manual. It is however usually more effective to use a high-level programming language such as C++. In both cases, kernels must be compiled into binary code by nvcc (called cubins) to execute on the device.\ \ The cubins are architecture-specific. Binary compatibility for cubins is guaranteed from one compute capability minor revision to the next one, but not from one compute capability minor revision to the previous one or across major compute capability revisions. In other words, a cubin object generated for compute capability _X.y_ will only execute on devices of compute capability _X.z_ where _z≥y_.\ \ To execute code on devices of specific compute capability, an application must load binary or PTX code that is compatible with this compute capability. For portability, that is, to be able to execute code on future GPU architectures with higher compute capability (for which no binary code can be generated yet), an application must load PTX code that will be just-in-time compiled by the NVIDIA driver for these future devices.\ \ More information on cubins, PTX and application compatibility can be found in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#binary-compatibility)\ .\ \ 15.4. CUDA Compatibility Across Minor Releases[](#cuda-compatibility-across-minor-releases "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------------------------------\ \ By leveraging the semantic versioning, starting with CUDA 11, components in the CUDA Toolkit will remain binary compatible across the minor versions of the toolkit. In order to maintain binary compatibility across minor versions, the CUDA runtime no longer bumps up the minimum driver version required for every minor release - this only happens when a major release is shipped.\ \ One of the main reasons a new toolchain requires a new minimum driver is to handle the JIT compilation of PTX code and the JIT linking of binary code.\ \ In this section, we will review the usage patterns that may require new user workflows when taking advantage of the compatibility features of the CUDA platform.\ \ ### 15.4.1. Existing CUDA Applications within Minor Versions of CUDA[](#existing-cuda-applications-within-minor-versions-of-cuda "Permalink to this headline")\ \ $ nvidia\-smi\ \ +-----------------------------------------------------------------------------+\ | NVIDIA\-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |\ |-------------------------------+----------------------+----------------------+\ | GPU Name Persistence\-M| Bus\-Id Disp.A | Volatile Uncorr. ECC |\ | Fan Temp Perf Pwr:Usage/Cap| Memory\-Usage | GPU\-Util Compute M. |\ | | | MIG M. |\ |===============================+======================+======================|\ | 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |\ | N/A 39C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |\ | | | N/A |\ +-------------------------------+----------------------+----------------------+\ \ +-----------------------------------------------------------------------------+\ | Processes: |\ | GPU GI CI PID Type Process name GPU Memory |\ | ID ID Usage |\ |=============================================================================|\ | No running processes found |\ +-----------------------------------------------------------------------------+\ \ When our CUDA 11.1 application (i.e. cudart 11.1 is statically linked) is run on the system, we see that it runs successfully even when the driver reports a 11.0 version - that is, without requiring the driver or other toolkit components to be updated on the system.\ \ $ samples/bin/x86\_64/linux/release/deviceQuery\ samples/bin/x86\_64/linux/release/deviceQuery Starting...\ \ CUDA Device Query (Runtime API) version (CUDART static linking)\ \ Detected 1 CUDA Capable device(s)\ \ Device 0: "Tesla T4"\ CUDA Driver Version / Runtime Version 11.0 / 11.1\ CUDA Capability Major/Minor version number: 7.5\ \ ......\ \ deviceQuery, CUDA Driver \= CUDART, CUDA Driver Version \= 11.0, CUDA Runtime Version \= 11.1, NumDevs \= 1\ Result \= PASS\ \ By using new CUDA versions, users can benefit from new CUDA programming model APIs, compiler optimizations and math library features.\ \ The following sections discuss some caveats and considerations.\ \ #### 15.4.1.1. Handling New CUDA Features and Driver APIs[](#handling-new-cuda-features-and-driver-apis "Permalink to this headline")\ \ A subset of CUDA APIs don’t need a new driver and they can all be used without any driver dependencies. For example, `cuMemMap` APIs or any of APIs introduced prior to CUDA 11.0, such as `cudaDeviceSynchronize`, do not require a driver upgrade. To use other CUDA APIs introduced in a minor release (that require a new driver), one would have to implement fallbacks or fail gracefully. This situation is not different from what is available today where developers use macros to compile out features based on CUDA versions. Users should refer to the CUDA headers and documentation for new CUDA APIs introduced in a release.\ \ When working with a feature exposed in a minor version of the toolkit, the feature might not be available at runtime if the application is running against an older CUDA driver. Users wishing to take advantage of such a feature should query its availability with a dynamic check in the code:\ \ static bool hostRegisterFeatureSupported \= false;\ static bool hostRegisterIsDeviceAddress \= false;\ \ static error\_t cuFooFunction(int \*ptr)\ {\ int \*dptr \= null;\ if (hostRegisterFeatureSupported) {\ cudaHostRegister(ptr, size, flags);\ if (hostRegisterIsDeviceAddress) {\ qptr \= ptr;\ }\ else {\ cudaHostGetDevicePointer(&qptr, ptr, 0);\ }\ }\ else {\ // cudaMalloc();\ // cudaMemcpy();\ }\ gemm<<<1,1\>>>(dptr);\ cudaDeviceSynchronize();\ }\ \ int main()\ {\ // rest of code here\ cudaDeviceGetAttribute(\ &hostRegisterFeatureSupported,\ cudaDevAttrHostRegisterSupported,\ 0);\ cudaDeviceGetAttribute(\ &hostRegisterIsDeviceAddress,\ cudaDevAttrCanUseHostPointerForRegisteredMem,\ 0);\ cuFooFunction(/\* malloced pointer \*/);\ }\ \ Alternatively the application’s interface might not work at all without a new CUDA driver and then its best to return an error right away:\ \ #define MIN\_VERSION 11010\ cudaError\_t foo()\ {\ int version \= 0;\ cudaGetDriverVersion(&version);\ if (version < MIN\_VERSION) {\ return CUDA\_ERROR\_INSUFFICIENT\_DRIVER;\ }\ // proceed as normal\ }\ \ A new error code is added to indicate that the functionality is missing from the driver you are running against: `cudaErrorCallRequiresNewerDriver`.\ \ #### 15.4.1.2. Using PTX[](#using-ptx "Permalink to this headline")\ \ PTX defines a virtual machine and ISA for general purpose parallel thread execution. PTX programs are translated at load time to the target hardware instruction set via the JIT Compiler which is part of the CUDA driver. As PTX is compiled by the CUDA driver, new toolchains will generate PTX that is not compatible with the older CUDA driver. This is not a problem when PTX is used for future device compatibility (the most common case), but can lead to issues when used for runtime compilation.\ \ For codes continuing to make use of PTX, in order to support compiling on an older driver, your code must be first transformed into device code via the static ptxjitcompiler library or NVRTC with the option of generating code for a specific architecture (e.g. sm\_80) rather than a virtual architecture (e.g. compute\_80). For this workflow, a new nvptxcompiler\_static library is shipped with the CUDA Toolkit.\ \ We can see this usage in the following example:\ \ char\* compilePTXToNVElf()\ {\ nvPTXCompilerHandle compiler \= NULL;\ nvPTXCompileResult status;\ \ size\_t elfSize, infoSize, errorSize;\ char \*elf, \*infoLog, \*errorLog;\ int minorVer, majorVer;\ \ const char\* compile\_options\[\] \= { "--gpu-name=sm\_80",\ "--device-debug"\ };\ \ nvPTXCompilerGetVersion(&majorVer, &minorVer);\ nvPTXCompilerCreate(&compiler, (size\_t)strlen(ptxCode), ptxCode);\ status \= nvPTXCompilerCompile(compiler, 2, compile\_options);\ if (status != NVPTXCOMPILE\_SUCCESS) {\ nvPTXCompilerGetErrorLogSize(compiler, (void\*)&errorSize);\ \ if (errorSize != 0) {\ errorLog \= (char\*)malloc(errorSize+1);\ nvPTXCompilerGetErrorLog(compiler, (void\*)errorLog);\ printf("Error log: %s\\n", errorLog);\ free(errorLog);\ }\ exit(1);\ }\ \ nvPTXCompilerGetCompiledProgramSize(compiler, &elfSize));\ elf \= (char\*)malloc(elfSize);\ nvPTXCompilerGetCompiledProgram(compiler, (void\*)elf);\ nvPTXCompilerGetInfoLogSize(compiler, (void\*)&infoSize);\ \ if (infoSize != 0) {\ infoLog \= (char\*)malloc(infoSize+1);\ nvPTXCompilerGetInfoLog(compiler, (void\*)infoLog);\ printf("Info log: %s\\n", infoLog);\ free(infoLog);\ }\ \ nvPTXCompilerDestroy(&compiler);\ return elf;\ }\ \ #### 15.4.1.3. Dynamic Code Generation[](#dynamic-code-generation "Permalink to this headline")\ \ NVRTC is a runtime compilation library for CUDA C++. It accepts CUDA C++ source code in character string form and creates handles that can be used to obtain the PTX. The PTX string generated by NVRTC can be loaded by cuModuleLoadData and cuModuleLoadDataEx.\ \ Dealing with relocatable objects is not yet supported, therefore the `cuLink`\* set of APIs in the CUDA driver will not work with enhanced compatibility. An upgraded driver matching the CUDA runtime version is currently required for those APIs.\ \ As mentioned in the PTX section, the compilation of PTX to device code lives along with the CUDA driver, hence the generated PTX might be newer than what is supported by the driver on the deployment system. When using NVRTC, it is recommended that the resulting PTX code is first transformed to the final device code via the steps outlined by the PTX user workflow. This ensures your code is compatible. Alternatively, NVRTC can generate cubins directly starting with CUDA 11.1. Applications using the new API can load the final device code directly using driver APIs `cuModuleLoadData` and `cuModuleLoadDataEx`.\ \ NVRTC used to support only virtual architectures through the option -arch, since it was only emitting PTX. It will now support actual architectures as well to emit SASS. The interface is augmented to retrieve either the PTX or cubin if an actual architecture is specified.\ \ The example below shows how an existing example can be adapted to use the new features, guarded by the `USE_CUBIN` macro in this case:\ \ #include \ #include \ #include \ \ void NVRTC\_SAFE\_CALL(nvrtcResult result) {\ if (result != NVRTC\_SUCCESS) {\ std::cerr << "\\nnvrtc error: " << nvrtcGetErrorString(result) << '\\n';\ std::exit(1);\ }\ }\ \ void CUDA\_SAFE\_CALL(CUresult result) {\ if (result != CUDA\_SUCCESS) {\ const char \*msg;\ cuGetErrorName(result, &msg);\ std::cerr << "\\ncuda error: " << msg << '\\n';\ std::exit(1);\ }\ }\ \ const char \*hello \= " \\n\\\ extern \\"C\\" \_\_global\_\_ void hello() { \\n\\\ printf(\\"hello world\\\\n\\"); \\n\\\ } \\n";\ \ int main()\ {\ nvrtcProgram prog;\ NVRTC\_SAFE\_CALL(nvrtcCreateProgram(&prog, hello, "hello.cu", 0, NULL, NULL));\ #ifdef USE\_CUBIN\ const char \*opts\[\] \= {"-arch=sm\_70"};\ #else\ const char \*opts\[\] \= {"-arch=compute\_70"};\ #endif\ nvrtcResult compileResult \= nvrtcCompileProgram(prog, 1, opts);\ size\_t logSize;\ NVRTC\_SAFE\_CALL(nvrtcGetProgramLogSize(prog, &logSize));\ char \*log \= new char\[logSize\];\ NVRTC\_SAFE\_CALL(nvrtcGetProgramLog(prog, log));\ std::cout << log << '\\n';\ delete\[\] log;\ if (compileResult != NVRTC\_SUCCESS)\ exit(1);\ size\_t codeSize;\ #ifdef USE\_CUBIN\ NVRTC\_SAFE\_CALL(nvrtcGetCUBINSize(prog, &codeSize));\ char \*code \= new char\[codeSize\];\ NVRTC\_SAFE\_CALL(nvrtcGetCUBIN(prog, code));\ #else\ NVRTC\_SAFE\_CALL(nvrtcGetPTXSize(prog, &codeSize));\ char \*code \= new char\[codeSize\];\ NVRTC\_SAFE\_CALL(nvrtcGetPTX(prog, code));\ #endif\ NVRTC\_SAFE\_CALL(nvrtcDestroyProgram(&prog));\ CUdevice cuDevice;\ CUcontext context;\ CUmodule module;\ CUfunction kernel;\ CUDA\_SAFE\_CALL(cuInit(0));\ CUDA\_SAFE\_CALL(cuDeviceGet(&cuDevice, 0));\ CUDA\_SAFE\_CALL(cuCtxCreate(&context, 0, cuDevice));\ CUDA\_SAFE\_CALL(cuModuleLoadDataEx(&module, code, 0, 0, 0));\ CUDA\_SAFE\_CALL(cuModuleGetFunction(&kernel, module, "hello"));\ CUDA\_SAFE\_CALL(cuLaunchKernel(kernel, 1, 1, 1, 1, 1, 1, 0, NULL, NULL, 0));\ CUDA\_SAFE\_CALL(cuCtxSynchronize());\ CUDA\_SAFE\_CALL(cuModuleUnload(module));\ CUDA\_SAFE\_CALL(cuCtxDestroy(context));\ delete\[\] code;\ }\ \ #### 15.4.1.4. Recommendations for building a minor-version compatible library[](#recommendations-for-building-a-minor-version-compatible-library "Permalink to this headline")\ \ We recommend that the CUDA runtime be statically linked to minimize dependencies. Verify that your library doesn’t leak dependencies, breakages, namespaces, etc. outside your established ABI contract.\ \ Follow semantic versioning for your library’s soname. Having a semantically versioned ABI means the interfaces need to be maintained and versioned. The library should follow semantic rules and increment the version number when a change is made that affects this ABI contract. Missing dependencies is also a binary compatibility break, hence you should provide fallbacks or guards for functionality that depends on those interfaces. Increment major versions when there are ABI breaking changes such as API deprecation and modifications. New APIs can be added in minor versions.\ \ Conditionally use features to remain compatible against older drivers. If no new features are used (or if they are used conditionally with fallbacks provided) you’ll be able to remain compatible.\ \ Don’t expose ABI structures that can change. A pointer to a structure with a size embedded is a better solution.\ \ When linking with dynamic libraries from the toolkit, the library must be equal to or newer than what is needed by any one of the components involved in the linking of your application. For example, if you link against the CUDA 11.1 dynamic runtime, and use functionality from 11.1, as well as a separate shared library that was linked against the CUDA 11.2 dynamic runtime that requires 11.2 functionality, the final link step must include a CUDA 11.2 or newer dynamic runtime.\ \ #### 15.4.1.5. Recommendations for taking advantage of minor version compatibility in your application[](#recommendations-for-taking-advantage-of-minor-version-compatibility-in-your-application "Permalink to this headline")\ \ Certain functionality might not be available so you should query where applicable. This is common for building applications that are GPU architecture, platform and compiler agnostic. However we now add “the underlying driver” to that mix.\ \ As with the previous section on library building recommendations, if using the CUDA runtime, we recommend linking to the CUDA runtime statically when building your application. When using the driver APIs directly, we recommend using the new driver entry point access API (`cuGetProcAddress`) documented here: [CUDA Driver API :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__DRIVER__ENTRY__POINT.html#group__CUDA__DRIVER__ENTRY__POINT)\ .\ \ When using a shared or static library, follow the release notes of said library to determine if the library supports minor version compatibility.\ \ 16\. Preparing for Deployment[](#preparing-for-deployment "Permalink to this headline")\ \ =========================================================================================\ \ 16.1. Testing for CUDA Availability[](#testing-for-cuda-availability "Permalink to this headline")\ \ ----------------------------------------------------------------------------------------------------\ \ When deploying a CUDA application, it is often desirable to ensure that the application will continue to function properly even if the target machine does not have a CUDA-capable GPU and/or a sufficient version of the NVIDIA Driver installed. (Developers targeting a single machine with known configuration may choose to skip this section.)\ \ **Detecting a CUDA-Capable GPU**\ \ When an application will be deployed to target machines of arbitrary/unknown configuration, the application should explicitly test for the existence of a CUDA-capable GPU in order to take appropriate action when no such device is available. The `cudaGetDeviceCount()` function can be used to query for the number of available devices. Like all CUDA Runtime API functions, this function will fail gracefully and return `cudaErrorNoDevice` to the application if there is no CUDA-capable GPU or `cudaErrorInsufficientDriver` if there is not an appropriate version of the NVIDIA Driver installed. If `cudaGetDeviceCount()` reports an error, the application should fall back to an alternative code path.\ \ A system with multiple GPUs may contain GPUs of different hardware versions and capabilities. When using multiple GPUs from the same application, it is recommended to use GPUs of the same type, rather than mixing hardware generations. The `cudaChooseDevice()` function can be used to select the device that most closely matches a desired set of features.\ \ **Detecting Hardware and Software Configuration**\ \ When an application depends on the availability of certain hardware or software capabilities to enable certain functionality, the CUDA API can be queried for details about the configuration of the available device and for the installed software versions.\ \ The `cudaGetDeviceProperties()` function reports various features of the available devices, including the [CUDA Compute Capability](#cuda-compute-capability)\ of the device (see also the Compute Capabilities section of the CUDA C++ Programming Guide). See [Version Management](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html#group__CUDART____VERSION)\ for details on how to query the available CUDA software API versions.\ \ 16.2. Error Handling[](#error-handling "Permalink to this headline")\ \ ----------------------------------------------------------------------\ \ All CUDA Runtime API calls return an error code of type `cudaError_t`; the return value will be equal to `cudaSuccess` if no errors have occurred. (The exceptions to this are kernel launches, which return void, and `cudaGetErrorString()`, which returns a character string describing the `cudaError_t` code that was passed into it.) The CUDA Toolkit libraries (`cuBLAS`, `cuFFT`, etc.) likewise return their own sets of error codes.\ \ Since some CUDA API calls and all kernel launches are asynchronous with respect to the host code, errors may be reported to the host asynchronously as well; often this occurs the next time the host and device synchronize with each other, such as during a call to `cudaMemcpy()` or to `cudaDeviceSynchronize()`.\ \ Always check the error return values on all CUDA API functions, even for functions that are not expected to fail, as this will allow the application to detect and recover from errors as soon as possible should they occur. To check for errors occurring during kernel launches using the `<<<...>>>` syntax, which does not return any error code, the return code of `cudaGetLastError()` should be checked immediately after the kernel launch. Applications that do not check for CUDA API errors could at times run to completion without having noticed that the data calculated by the GPU is incomplete, invalid, or uninitialized.\ \ Note\ \ The CUDA Toolkit Samples provide several helper functions for error checking with the various CUDA APIs; these helper functions are located in the `samples/common/inc/helper_cuda.h` file in the CUDA Toolkit.\ \ 16.3. Building for Maximum Compatibility[](#building-for-maximum-compatibility "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------------------\ \ Each generation of CUDA-capable device has an associated _compute capability_ version that indicates the feature set supported by the device (see [CUDA Compute Capability](#cuda-compute-capability)\ ). One or more compute capability versions can be specified to the nvcc compiler while building a file; compiling for the native compute capability for the target GPU(s) of the application is important to ensure that application kernels achieve the best possible performance and are able to use the features that are available on a given generation of GPU.\ \ When an application is built for multiple compute capabilities simultaneously (using several instances of the `-gencode` flag to nvcc), the binaries for the specified compute capabilities are combined into the executable, and the CUDA Driver selects the most appropriate binary at runtime according to the compute capability of the present device. If an appropriate native binary (_cubin_) is not available, but the intermediate _PTX_ code (which targets an abstract virtual instruction set and is used for forward-compatibility) is available, then the kernel will be compiled _Just In Time_ (JIT) (see [Compiler JIT Cache Management Tools](#compiler-jit-cache-management)\ ) from the PTX to the native cubin for the device. If the PTX is also not available, then the kernel launch will fail.\ \ **Windows**\ \ nvcc.exe -ccbin "C:\\vs2008\\VC\\bin"\ -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT"\ -gencode=arch=compute\_30,code=sm\_30\ -gencode=arch=compute\_35,code=sm\_35\ -gencode=arch=compute\_50,code=sm\_50\ -gencode=arch=compute\_60,code=sm\_60\ -gencode=arch=compute\_70,code=sm\_70\ -gencode=arch=compute\_75,code=sm\_75\ -gencode=arch=compute\_75,code=compute\_75\ --compile -o "Release\\mykernel.cu.obj" "mykernel.cu"\ \ **Mac/Linux**\ \ /usr/local/cuda/bin/nvcc\ -gencode=arch=compute\_30,code=sm\_30\ -gencode=arch=compute\_35,code=sm\_35\ -gencode=arch=compute\_50,code=sm\_50\ -gencode=arch=compute\_60,code=sm\_60\ -gencode=arch=compute\_70,code=sm\_70\ -gencode=arch=compute\_75,code=sm\_75\ -gencode=arch=compute\_75,code=compute\_75\ -O2 -o mykernel.o -c mykernel.cu\ \ Alternatively, the `nvcc` command-line option `-arch=sm_XX` can be used as a shorthand equivalent to the following more explicit `-gencode=` command-line options described above:\ \ \-gencode=arch=compute\_XX,code=sm\_XX\ -gencode=arch=compute\_XX,code=compute\_XX\ \ However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target by default (due to the `code=compute_XX` target it implies), it can only specify a single target `cubin` architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly.\ \ 16.4. Distributing the CUDA Runtime and Libraries[](#distributing-the-cuda-runtime-and-libraries "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------------------------------------\ \ CUDA applications are built against the CUDA Runtime library, which handles device, memory, and kernel management. Unlike the CUDA Driver, the CUDA Runtime guarantees neither forward nor backward binary compatibility across versions. It is therefore best to [redistribute](#redistribution)\ the CUDA Runtime library with the application when using dynamic linking or else to statically link against the CUDA Runtime. This will ensure that the executable will be able to run even if the user does not have the same CUDA Toolkit installed that the application was built against.\ \ Note\ \ When statically linking to the CUDA Runtime, multiple versions of the runtime can peacably coexist in the same application process simultaneously; for example, if an application uses one version of the CUDA Runtime, and a plugin to that application is statically linked to a different version, that is perfectly acceptable, as long as the installed NVIDIA Driver is sufficient for both.\ \ Statically-linked CUDA Runtime\ \ The easiest option is to statically link against the CUDA Runtime. This is the default if using `nvcc` to link in CUDA 5.5 and later. Static linking makes the executable slightly larger, but it ensures that the correct version of runtime library functions are included in the application binary without requiring separate redistribution of the CUDA Runtime library.\ \ Dynamically-linked CUDA Runtime\ \ If static linking against the CUDA Runtime is impractical for some reason, then a dynamically-linked version of the CUDA Runtime library is also available. (This was the default and only option provided in CUDA versions 5.0 and earlier.)\ \ To use dynamic linking with the CUDA Runtime when using the `nvcc` from CUDA 5.5 or later to link the application, add the `--cudart=shared` flag to the link command line; otherwise the [statically-linked CUDA Runtime library](#statically-linked-cuda-runtime)\ is used by default.\ \ After the application is dynamically linked against the CUDA Runtime, this version of the runtime library should be [bundled with](#redistribution)\ the application. It can be copied into the same directory as the application executable or into a subdirectory of that installation path.\ \ Other CUDA Libraries\ \ Although the CUDA Runtime provides the option of static linking, some libraries included in the CUDA Toolkit are available only in dynamically-linked form. As with the [dynamically-linked version of the CUDA Runtime library](#dynamically-linked-cuda-runtime)\ , these libraries should be [bundled with](#redistribution)\ the application executable when distributing that application.\ \ ### 16.4.1. CUDA Toolkit Library Redistribution[](#cuda-toolkit-library-redistribution "Permalink to this headline")\ \ The CUDA Toolkit’s End-User License Agreement (EULA) allows for redistribution of many of the CUDA libraries under certain terms and conditions. This allows applications that depend on these libraries [to redistribute the exact versions](#redistribution-which-files)\ of the libraries against which they were built and tested, thereby avoiding any trouble for end users who might have a different version of the CUDA Toolkit (or perhaps none at all) installed on their machines. Please refer to the EULA for details.\ \ Note\ \ This does _not_ apply to the NVIDIA Driver; the end user must still download and install an NVIDIA Driver appropriate to their GPU(s) and operating system.\ \ #### 16.4.1.1. Which Files to Redistribute[](#which-files-to-redistribute "Permalink to this headline")\ \ When redistributing the dynamically-linked versions of one or more CUDA libraries, it is important to identify the exact files that need to be redistributed. The following examples use the cuBLAS library from CUDA Toolkit 5.5 as an illustration:\ \ **Linux**\ \ In a shared library on Linux, there is a string field called the `SONAME` that indicates the binary compatibility level of the library. The `SONAME` of the library against which the application was built must match the filename of the library that is redistributed with the application.\ \ For example, in the standard CUDA Toolkit installation, the files `libcublas.so` and `libcublas.so.5.5` are both symlinks pointing to a specific build of cuBLAS, which is named like `libcublas.so.5.5.x`, where _x_ is the build number (e.g., `libcublas.so.5.5.17`). However, the `SONAME` of this library is given as “`libcublas.so.5.5`”:\ \ $ objdump -p /usr/local/cuda/lib64/libcublas.so | grep SONAME\ SONAME libcublas.so.5.5\ \ Because of this, even if `-lcublas` (with no version number specified) is used when linking the application, the `SONAME` found at link time implies that “`libcublas.so.5.5`” is the name of the file that the dynamic loader will look for when loading the application and therefore must be the name of the file (or a symlink to the same) that is redistributed with the application.\ \ The `ldd` tool is useful for identifying the exact filenames of the libraries that the application expects to find at runtime as well as the path, if any, of the copy of that library that the dynamic loader would select when loading the application given the current library search path:\ \ $ ldd a.out | grep libcublas\ libcublas.so.5.5 => /usr/local/cuda/lib64/libcublas.so.5.5\ \ **Mac**\ \ In a shared library on Mac OS X, there is a field called the `install name` that indicates the expected installation path and filename the library; the CUDA libraries also use this filename to indicate binary compatibility. The value of this field is propagated into an application built against the library and is used to locate the library of the correct version at runtime.\ \ For example, if the install name of the cuBLAS library is given as `@rpath/libcublas.5.5.dylib`, then the library is version 5.5 and the copy of this library redistributed with the application must be named `libcublas.5.5.dylib`, even though only `-lcublas` (with no version number specified) is used at link time. Furthermore, this file should be installed into the `@rpath` of the application; see [Where to Install Redistributed CUDA Libraries](#redistribution-where-to-install)\ .\ \ To view a library’s install name, use the `otool -L` command:\ \ $ otool -L a.out\ a.out:\ @rpath/libcublas.5.5.dylib (...)\ \ **Windows**\ \ The binary compatibility version of the CUDA libraries on Windows is indicated as part of the filename.\ \ For example, a 64-bit application linked to cuBLAS 5.5 will look for `cublas64_55.dll` at runtime, so this is the file that should be redistributed with that application, even though `cublas.lib` is the file that the application is linked against. For 32-bit applications, the file would be `cublas32_55.dll`.\ \ To verify the exact DLL filename that the application expects to find at runtime, use the `dumpbin` tool from the Visual Studio command prompt:\ \ $ dumpbin /IMPORTS a.exe\ Microsoft (R) COFF/PE Dumper Version 10.00.40219.01\ Copyright (C) Microsoft Corporation. All rights reserved.\ \ \ Dump of file a.exe\ \ File Type: EXECUTABLE IMAGE\ \ Section contains the following imports:\ \ ...\ cublas64\_55.dll\ ...\ \ #### 16.4.1.2. Where to Install Redistributed CUDA Libraries[](#where-to-install-redistributed-cuda-libraries "Permalink to this headline")\ \ Once the correct library files are identified for redistribution, they must be configured for installation into a location where the application will be able to find them.\ \ On Windows, if the CUDA Runtime or other dynamically-linked CUDA Toolkit library is placed in the same directory as the executable, Windows will locate it automatically. On Linux and Mac, the `-rpath` linker option should be used to instruct the executable to search its local path for these libraries before searching the system paths:\ \ **Linux/Mac**\ \ nvcc -I $(CUDA\_HOME)/include\ -Xlinker "-rpath '$ORIGIN'" --cudart=shared\ -o myprogram myprogram.cu\ \ **Windows**\ \ nvcc.exe -ccbin "C:\\vs2008\\VC\\bin"\ -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" --cudart=shared\ -o "Release\\myprogram.exe" "myprogram.cu"\ \ Note\ \ It may be necessary to adjust the value of `-ccbin` to reflect the location of your Visual Studio installation.\ \ To specify an alternate path where the libraries will be distributed, use linker options similar to those below:\ \ **Linux/Mac**\ \ nvcc -I $(CUDA\_HOME)/include\ -Xlinker "-rpath '$ORIGIN/lib'" --cudart=shared\ -o myprogram myprogram.cu\ \ **Windows**\ \ nvcc.exe -ccbin "C:\\vs2008\\VC\\bin"\ -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT /DELAY" --cudart=shared\ -o "Release\\myprogram.exe" "myprogram.cu"\ \ For Linux and Mac, the `-rpath` option is used as before. For Windows, the `/DELAY` option is used; this requires that the application call `SetDllDirectory()` before the first call to any CUDA API function in order to specify the directory containing the CUDA DLLs.\ \ Note\ \ For Windows 8, `SetDefaultDLLDirectories()` and `AddDllDirectory()` should be used instead of `SetDllDirectory()`. Please see the MSDN documentation for these routines for more information.\ \ 17\. Deployment Infrastructure Tools[](#deployment-infrastructure-tools "Permalink to this headline")\ \ =======================================================================================================\ \ 17.1. Nvidia-SMI[](#nvidia-smi "Permalink to this headline")\ \ --------------------------------------------------------------\ \ The NVIDIA System Management Interface (`nvidia-smi`) is a command line utility that aids in the management and monitoring of NVIDIA GPU devices. This utility allows administrators to query GPU device state and, with the appropriate privileges, permits administrators to modify GPU device state. `nvidia-smi` is targeted at Tesla and certain Quadro GPUs, though limited support is also available on other NVIDIA GPUs. `nvidia-smi` ships with NVIDIA GPU display drivers on Linux, and with 64-bit Windows Server 2008 R2 and Windows 7. `nvidia-smi` can output queried information as XML or as human-readable plain text either to standard output or to a file. See the nvidia-smi documenation for details. Please note that new versions of nvidia-smi are not guaranteed to be backward-compatible with previous versions.\ \ ### 17.1.1. Queryable state[](#queryable-state "Permalink to this headline")\ \ ECC error counts\ \ Both correctable single-bit and detectable double-bit errors are reported. Error counts are provided for both the current boot cycle and the lifetime of the GPU.\ \ GPU utilization\ \ Current utilization rates are reported for both the compute resources of the GPU and the memory interface.\ \ Active compute process\ \ The list of active processes running on the GPU is reported, along with the corresponding process name/ID and allocated GPU memory.\ \ Clocks and performance state\ \ Max and current clock rates are reported for several important clock domains, as well as the current GPU performance state (_pstate_).\ \ Temperature and fan speed\ \ The current GPU core temperature is reported, along with fan speeds for products with active cooling.\ \ Power management\ \ The current board power draw and power limits are reported for products that report these measurements.\ \ Identification\ \ Various dynamic and static information is reported, including board serial numbers, PCI device IDs, VBIOS/Inforom version numbers and product names.\ \ ### 17.1.2. Modifiable state[](#modifiable-state "Permalink to this headline")\ \ ECC mode\ \ Enable and disable ECC reporting.\ \ ECC reset\ \ Clear single-bit and double-bit ECC error counts.\ \ Compute mode\ \ Indicate whether compute processes can run on the GPU and whether they run exclusively or concurrently with other compute processes.\ \ Persistence mode\ \ Indicate whether the NVIDIA driver stays loaded when no applications are connected to the GPU. It is best to enable this option in most circumstances.\ \ GPU reset\ \ Reinitialize the GPU hardware and software state via a secondary bus reset.\ \ 17.2. NVML[](#nvml "Permalink to this headline")\ \ --------------------------------------------------\ \ The NVIDIA Management Library (NVML) is a C-based interface that provides direct access to the queries and commands exposed via `nvidia-smi` intended as a platform for building 3rd-party system management applications. The NVML API is shipped with the CUDA Toolkit (since version 8.0) and is also available standalone on the NVIDIA developer website as part of the GPU Deployment Kit through a single header file accompanied by PDF documentation, stub libraries, and sample applications; see [https://developer.nvidia.com/gpu-deployment-kit](https://developer.nvidia.com/gpu-deployment-kit)\ . Each new version of NVML is backward-compatible.\ \ An additional set of Perl and Python bindings are provided for the NVML API. These bindings expose the same features as the C-based interface and also provide backwards compatibility. The Perl bindings are provided via CPAN and the Python bindings via PyPI.\ \ All of these products (`nvidia-smi`, NVML, and the NVML language bindings) are updated with each new CUDA release and provide roughly the same functionality.\ \ See [https://developer.nvidia.com/nvidia-management-library-nvml](https://developer.nvidia.com/nvidia-management-library-nvml)\ for additional information.\ \ 17.3. Cluster Management Tools[](#cluster-management-tools "Permalink to this headline")\ \ ------------------------------------------------------------------------------------------\ \ Managing your GPU cluster will help achieve maximum GPU utilization and help you and your users extract the best possible performance. Many of the industry’s most popular cluster management tools support CUDA GPUs via NVML. For a listing of some of these tools, see [https://developer.nvidia.com/cluster-management](https://developer.nvidia.com/cluster-management)\ .\ \ 17.4. Compiler JIT Cache Management Tools[](#compiler-jit-cache-management-tools "Permalink to this headline")\ \ ----------------------------------------------------------------------------------------------------------------\ \ Any PTX device code loaded by an application at runtime is compiled further to binary code by the device driver. This is called _just-in-time compilation_ (_JIT_). Just-in-time compilation increases application load time but allows applications to benefit from latest compiler improvements. It is also the only way for applications to run on devices that did not exist at the time the application was compiled.\ \ When JIT compilation of PTX device code is used, the NVIDIA driver caches the resulting binary code on disk. Some aspects of this behavior such as cache location and maximum cache size can be controlled via the use of environment variables; see Just in Time Compilation of the CUDA C++ Programming Guide.\ \ 17.5. CUDA\_VISIBLE\_DEVICES[](#cuda-visible-devices "Permalink to this headline")\ \ ------------------------------------------------------------------------------------\ \ It is possible to rearrange the collection of installed CUDA devices that will be visible to and enumerated by a CUDA application prior to the start of that application by way of the `CUDA_VISIBLE_DEVICES` environment variable.\ \ Devices to be made visible to the application should be included as a comma-separated list in terms of the system-wide list of enumerable devices. For example, to use only devices 0 and 2 from the system-wide list of devices, set `CUDA_VISIBLE_DEVICES=0,2` before launching the application. The application will then enumerate these devices as device 0 and device 1, respectively.\ \ 18\. Recommendations and Best Practices[](#id84 "Permalink to this headline")\ \ ===============================================================================\ \ This chapter contains a summary of the recommendations for optimization that are explained in this document.\ \ 18.1. Overall Performance Optimization Strategies[](#overall-performance-optimization-strategies "Permalink to this headline")\ \ --------------------------------------------------------------------------------------------------------------------------------\ \ Performance optimization revolves around three basic strategies:\ \ * Maximizing parallel execution\ \ * Optimizing memory usage to achieve maximum memory bandwidth\ \ * Optimizing instruction usage to achieve maximum instruction throughput\ \ \ Maximizing parallel execution starts with structuring the algorithm in a way that exposes as much parallelism as possible. Once the parallelism of the algorithm has been exposed, it needs to be mapped to the hardware as efficiently as possible. This is done by carefully choosing the execution configuration of each kernel launch. The application should also maximize parallel execution at a higher level by explicitly exposing concurrent execution on the device through streams, as well as maximizing concurrent execution between the host and the device.\ \ Optimizing memory usage starts with minimizing data transfers between the host and the device because those transfers have much lower bandwidth than internal device data transfers. Kernel access to global memory also should be minimized by maximizing the use of shared memory on the device. Sometimes, the best optimization might even be to avoid any data transfer in the first place by simply recomputing the data whenever it is needed.\ \ The effective bandwidth can vary by an order of magnitude depending on the access pattern for each type of memory. The next step in optimizing memory usage is therefore to organize memory accesses according to the optimal memory access patterns. This optimization is especially important for global memory accesses, because latency of access costs hundreds of clock cycles. Shared memory accesses, in counterpoint, are usually worth optimizing only when there exists a high degree of bank conflicts.\ \ As for optimizing instruction usage, the use of arithmetic instructions that have low throughput should be avoided. This suggests trading precision for speed when it does not affect the end result, such as using intrinsics instead of regular functions or single precision instead of double precision. Finally, particular attention must be paid to control flow instructions due to the SIMT (single instruction multiple thread) nature of the device.\ \ 19\. nvcc Compiler Switches[](#nvcc-compiler-switches "Permalink to this headline")\ \ =====================================================================================\ \ 19.1. nvcc[](#nvcc "Permalink to this headline")\ \ --------------------------------------------------\ \ The NVIDIA `nvcc` compiler driver converts `.cu` files into C++ for the host system and CUDA assembly or binary instructions for the device. It supports a number of command-line parameters, of which the following are especially useful for optimization and related best practices:\ \ * `-maxrregcount=N` specifies the maximum number of registers kernels can use at a per-file level. See [Register Pressure](#register-pressure)\ . (See also the`__launch_bounds__` qualifier discussed in Execution Configuration of the CUDA C++ Programming Guide to control the number of registers used on a per-kernel basis.)\ \ * `--ptxas-options=-v` or `-Xptxas=-v` lists per-kernel register, shared, and constant memory usage.\ \ * `-ftz=true` (denormalized numbers are flushed to zero)\ \ * `-prec-div=false` (less precise division)\ \ * `-prec-sqrt=false` (less precise square root)\ \ * `-use_fast_math` compiler option of `nvcc` coerces every `functionName()` call to the equivalent `__functionName()` call. This makes the code run faster at the cost of diminished precision and accuracy. See [Math Libraries](#math-libraries)\ .\ \ \ 20\. Notices[](#notices "Permalink to this headline")\ \ =======================================================\ \ 20.1. Notice[](#notice "Permalink to this headline")\ \ ------------------------------------------------------\ \ This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.\ \ NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.\ \ Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.\ \ NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.\ \ NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.\ \ NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.\ \ No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.\ \ Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.\ \ THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.\ \ 20.2. OpenCL[](#opencl "Permalink to this headline")\ \ ------------------------------------------------------\ \ OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc.\ \ 20.3. Trademarks[](#trademarks "Permalink to this headline")\ \ --------------------------------------------------------------\ \ NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Pascal Compatibility — Pascal Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Pascal Compatibility * v12.8 | [PDF](../pdf/Pascal_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Pascal Compatibility Guide for CUDA Applications The guide to building CUDA applications for GPUs based on the NVIDIA Pascal Architecture. 1\. Pascal Compatibility[](#pascal-compatibility "Permalink to this headline") ================================================================================ 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Pascal Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on GPUs based on the NVIDIA® Pascal Architecture. This document provides guidance to developers who are already familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Pascal. 1.2. Application Compatibility on Pascal[](#application-compatibility-on-pascal "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- The NVIDIA CUDA C++ compiler, `nvcc`, can be used to generate both architecture-specific _cubin_ files and forward-compatible _PTX_ versions of each kernel. Each cubin file targets a specific compute-capability version and is forward-compatible _only with GPU architectures of the same major version number_. For example, cubin files that target compute capability 3.0 are supported on all compute-capability 3.x (Kepler) devices but are _not_ supported on compute-capability 5.x (Maxwell) or 6.x (Pascal) devices. For this reason, to ensure forward compatibility with GPU architectures introduced after the application has been released, it is recommended that all applications include PTX versions of their kernels. Note CUDA Runtime applications containing both cubin and PTX code for a given architecture will automatically use the cubin by default, keeping the PTX path strictly for forward-compatibility purposes. Applications that already include PTX versions of their kernels should work as-is on Pascal-based GPUs. Applications that only support specific GPU architectures via cubin files, however, will need to be updated to provide Pascal-compatible PTX or cubins. 1.3. Verifying Pascal Compatibility for Existing Applications[](#verifying-pascal-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------- The first step is to check that Pascal-compatible device code (at least PTX) is compiled in to the application. The following sections show how to accomplish this for applications built with different CUDA Toolkit versions. ### 1.3.1. Applications Using CUDA Toolkit 7.5 or Earlier[](#applications-using-cuda-toolkit-7-5-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 7.5 are compatible with Pascal as long as they are built to include PTX versions of their kernels. To test that PTX JIT is working for your application, you can do the following: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch your application. When starting a CUDA application for the first time with the above environment flag, the CUDA driver will JIT-compile the PTX for each CUDA kernel that is used into native cubin code. If you set the environment variable above and then launch your program and it works properly, then you have successfully verified Pascal compatibility. Note Be sure to unset the CUDA\_FORCE\_PTX\_JIT environment variable when you are done testing. ### 1.3.2. Applications Using CUDA Toolkit 8.0[](#applications-using-cuda-toolkit-8-0 "Permalink to this headline") CUDA applications built using CUDA Toolkit 8.0 are compatible with Pascal as long as they are built to include kernels in either Pascal-native cubin format (see [Building Applications with Pascal Support](#building-applications-with-pascal-support) ) or PTX format (see [Applications Using CUDA Toolkit 7.5 or Earlier](#verifying-pascal-compatibility-using-cuda-7-5) ) or both. 1.4. Building Applications with Pascal Support[](#building-applications-with-pascal-support "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------- When a CUDA application launches a kernel, the CUDA Runtime determines the compute capability of each GPU in the system and uses this information to automatically find the best matching cubin or PTX version of the kernel that is available. If a cubin file supporting the architecture of the target GPU is available, it is used; otherwise, the CUDA Runtime will load the PTX and JIT-compile that PTX to the GPU’s native cubin format before launching it. If neither is available, then the kernel launch will fail. The method used to build your application with either native cubin or at least PTX support for Pascal depend on the version of the CUDA Toolkit used. The main advantages of providing native cubins are as follows: * It saves the end user the time it takes to JIT-compile kernels that are available only as PTX. All kernels compiled into the application must have native binaries at load time or else they will be built just-in-time from PTX, including kernels from all libraries linked to the application, even if those kernels are never launched by the application. Especially when using large libraries, this JIT compilation can take a significant amount of time. The CUDA driver will cache the cubins generated as a result of the PTX JIT, so this is mostly a one-time cost for a given user, but it is time best avoided whenever possible. * PTX JIT-compiled kernels often cannot take advantage of architectural features of newer GPUs, meaning that native-compiled code may be faster or of greater accuracy. ### 1.4.1. Applications Using CUDA Toolkit 7.5 or Earlier[](#building-pascal-compatible-apps-using-cuda-7-5 "Permalink to this headline") The compilers included in CUDA Toolkit 7.5 or earlier generate cubin files native to earlier NVIDIA architectures such as Kepler and Maxwell, but they _cannot_ generate cubin files native to the Pascal architecture. To allow support for Pascal and future architectures when using version 7.5 or earlier of the CUDA Toolkit, the compiler must generate a PTX version of each kernel. Below are compiler settings that could be used to build `mykernel.cu` to run on Kepler or Maxwell devices natively and on Pascal devices via PTX JIT. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one must be PTX to provide Pascal compatibility. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_52,code=compute\_52 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_52,code=compute\_52 -O2 -o mykernel.o -c mykernel.cu Alternatively, you may be familiar with the simplified `nvcc` command-line option `-arch=sm_XX`, which is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. ### 1.4.2. Applications Using CUDA Toolkit 8.0[](#building-pascal-compatible-apps-using-cuda-8-0 "Permalink to this headline") With version 8.0 of the CUDA Toolkit, `nvcc` can generate cubin files native to the Pascal architectures (compute capability 6.0 and 6.1). When using CUDA Toolkit 8.0, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Mac/Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_30,code=sm\_30 -gencode=arch=compute\_35,code=sm\_35 -gencode=arch=compute\_50,code=sm\_50 -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_61,code=compute\_61 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. Only the back-end target version(s) specified by the `code=` clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. **Version 1.1** * Use CUDA C++ instead of CUDA C/C++ 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Blackwell Architecture Compatibility — Blackwell Compatibility Guide 12.8 documentation * [](../index.html) » * 1\. Blackwell Architecture Compatibility * v12.8 | [PDF](../pdf/Blackwell_Compatibility_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * Blackwell Compatibility Guide for CUDA Applications The guide to building CUDA applications for Blackwell GPUs 1\. Blackwell Architecture Compatibility[](#blackwell-architecture-compatibility "Permalink to this headline") ================================================================================================================ 1.1. About this Document[](#about-this-document "Permalink to this headline") ------------------------------------------------------------------------------- This application note, Blackwell Architecture Compatibility Guide for CUDA Applications, is intended to help developers ensure that their NVIDIA® CUDA® applications will run on the NVIDIA® Blackwell architecture based GPUs. This document provides guidance to developers who are familiar with programming in CUDA C++ and want to make sure that their software applications are compatible with Blackwell architecture. 1.2. Application Compatibility on Blackwell Architecture[](#application-compatibility-on-blackwell-architecture "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------- A CUDA application binary (with one or more GPU kernels) can contain the compiled GPU code in two forms, binary cubin objects and forward-compatible PTX assembly for each kernel. Both cubin and PTX are generated for a certain target compute capability. A cubin generated for a certain compute capability is supported to run on any GPU with the same major revision and same or higher minor revision of compute capability. For example, a cubin generated for compute capability 8.0 is supported to run on a GPU with compute capability 8.6, however a cubin generated for compute capability 8.6 is _not_ supported to run on a GPU with compute capability 8.0, and a cubin generated with compute capability 8.x is _not_ supported to run on a GPU with compute capability 9.0. Kernel can also be compiled to a PTX form. PTX is compiled at runtime to cubin and the cubin is used for kernel execution. Unlike cubin, PTX is forward-compatible. Meaning PTX is supported to run on any GPU with compute capability higher than the compute capability assumed for generation of that PTX. For example, PTX code generated for compute capability 9.x is supported to run on compute capability 9.x or any higher revision (major or minor), including compute capability 10.0. Therefore although it is optional, **it is recommended that all applications should include PTX of the kernels to ensure forward-compatibility.** To read more about cubin and PTX compatibilities see [Compilation with NVCC](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-with-nvcc) from the _CUDA C++ Programming Guide_. When a CUDA application launches a kernel on a GPU, the CUDA Runtime determines the compute capability of the GPU in the system and uses this information to find the best matching cubin or PTX version of the kernel. If a cubin compatible with that GPU is present in the binary, the cubin is used as-is for execution. Otherwise, the CUDA Runtime first generates compatible cubin by JIT-compiling [1](#fn1) the PTX and then the cubin is used for the execution. If neither compatible cubin nor PTX is available, kernel launch results in a failure. Application binaries that include PTX version of kernels, should work as-is on the Blackwell GPUs. In such cases, rebuilding the application is not required. However application binaries which do not include PTX (only include cubins), need to be rebuilt to run on the Blackwell GPUs. To know more about building compatible applications read [Building Applications with Blackwell Architecture Support](#building-applications-with-blackwell-support) . Application binaries that include PTX version of kernels with architecture conditional features using `sm_100a` or `compute_100a` in order to take full advantage of Blackwell GPU architecture, are not forward or backward compatible. For example, PTX compiled for `compute_90a` (Hopper) are not supported on the Blackwell architecture. 1.3. Verifying Blackwell Compatibility for Existing Applications[](#verifying-blackwell-compatibility-for-existing-applications "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------------- The first step towards making a CUDA application compatible with Blackwell architecture is to check if the application binary already contains compatible GPU code (at least the PTX). The following sections explain how to accomplish this for an already built CUDA application. ### 1.3.1. Applications Built Using CUDA Toolkit 12.8 or Earlier[](#applications-built-using-cuda-toolkit-12-8-or-earlier "Permalink to this headline") CUDA applications built using CUDA Toolkit versions 2.1 through 12.8 are compatible with Blackwell GPUs as long as they are built to include PTX versions of their kernels. This can be tested by forcing the PTX to JIT-compile at application load time with following the steps: * Download and install the latest driver from [https://www.nvidia.com/drivers](https://www.nvidia.com/drivers) . * Set the environment variable `CUDA_FORCE_PTX_JIT=1`. * Launch the application. With `CUDA_FORCE_PTX_JIT=1`, GPU binary code embedded in an application binary is ignored. Instead PTX code for each kernel is JIT-compiled to produce GPU binary code. An application fails to execute if it does not include PTX. This means the application is not Blackwell architecture compatible and needs to be rebuilt for compatibility. On the other hand, if the application works properly with this environment variable set, then the application is Blackwell compatible. Note Be sure to unset the `CUDA_FORCE_PTX_JIT` environment variable after testing is done. ### 1.3.2. Applications Built Using CUDA Toolkit 12.8[](#applications-built-using-cuda-toolkit-12-8 "Permalink to this headline") CUDA applications built using CUDA Toolkit 12.8 are compatible with Blackwell architecture as long as they are built to include kernels in native cubin (compute capability 10.0) or PTX form or both. 1.4. Building Applications with Blackwell Architecture Support[](#building-applications-with-blackwell-architecture-support "Permalink to this headline") ----------------------------------------------------------------------------------------------------------------------------------------------------------- Depending on the version of the CUDA Toolkit used for building the application, it can be built to include PTX and/or native cubin for the Blackwell architecture. Although it is enough to just include PTX, including native cubin is can avoid the need to JIT compile the PTX at runtime. [2](#fn2) ### 1.4.1. Building Applications Using CUDA Toolkit 12.7 or Earlier[](#building-applications-using-cuda-toolkit-12-7-or-earlier "Permalink to this headline") The `nvcc` compiler included with version 12.7 or earlier (11.8-12.7) of the CUDA Toolkit can generate cubins native to the NVIDIA Hopper GPU architectures (compute capability 9.x). When using CUDA Toolkit 12.7 or earlier, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_90,code=compute\_90 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_90,code=compute\_90 -O2 -o mykernel.o -c mykernel.cu Alternatively, the simplified `nvcc` command-line option `-arch=sm_XX` can be used. It is a shorthand equivalent to the following more explicit `-gencode=` command-line options used above. `-arch=sm_XX` expands to the following: \-gencode=arch=compute\_XX,code=sm\_XX -gencode=arch=compute\_XX,code=compute\_XX However, while the `-arch=sm_XX` command-line option does result in inclusion of a PTX back-end target binary by default, it can only specify a single target cubin architecture at a time, and it is not possible to use multiple `-arch=` options on the same `nvcc` command line, which is why the examples above use `-gencode=` explicitly. For CUDA toolkits prior to 11.0, one or more of the `-gencode` options need to be removed according to the architectures supported by the specific toolkit version (for example, CUDA toolkit 10.x supports architectures up to sm\_72 and sm\_75). The final `-gencode` to generate PTX also needs to be updated. For further information and examples see the documentation for the specific CUDA toolkit version. Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. **Only the back-end target version(s) specified by the code= clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures.** ### 1.4.2. Building Applications Using CUDA Toolkit 12.8[](#building-applications-using-cuda-toolkit-12-8 "Permalink to this headline") With versions 12.8 of the CUDA Toolkit, `nvcc` can generate cubin native to the Blackwell architecture (compute capability 10.0). When using CUDA Toolkit 12.8, to ensure that `nvcc` will generate cubin files for all recent GPU architectures as well as a PTX version for forward compatibility with future GPU architectures, specify the appropriate `-gencode=` parameters on the `nvcc` command line as shown in the examples below. **Windows** nvcc.exe -ccbin "C:\\vs2010\\VC\\bin" -Xcompiler "/EHsc /W3 /nologo /O2 /Zi /MT" -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_100,code=sm\_100 -gencode=arch=compute\_100,code=compute\_100 --compile -o "Release\\mykernel.cu.obj" "mykernel.cu" **Linux** /usr/local/cuda/bin/nvcc -gencode=arch=compute\_52,code=sm\_52 -gencode=arch=compute\_60,code=sm\_60 -gencode=arch=compute\_61,code=sm\_61 -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_75,code=sm\_75 -gencode=arch=compute\_80,code=sm\_80 -gencode=arch=compute\_90,code=sm\_90 -gencode=arch=compute\_100,code=sm\_100 -gencode=arch=compute\_100,code=compute\_100 -O2 -o mykernel.o -c mykernel.cu Note `compute_XX` refers to a PTX version and `sm_XX` refers to a cubin version. The `arch=` clause of the `-gencode=` command-line option to `nvcc` specifies the front-end compilation target and must always be a PTX version. The `code=` clause specifies the back-end compilation target and can either be cubin or PTX or both. **Only the back-end target version(s) specified by the code= clause will be retained in the resulting binary; at least one should be PTX to provide compatibility with future architectures.** ### 1.4.3. Independent Thread Scheduling Compatibility[](#independent-thread-scheduling-compatibility "Permalink to this headline") NVIDIA GPUs since Volta architecture have Independent Thread Scheduling among threads in a warp. If the developer made assumptions about warp-synchronicity[3](#fn3) , this feature can alter the set of threads participating in the executed code compared to previous architectures. Please see [Compute Capability 7.x](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability-7-x) in the _CUDA C++ Programming Guide_ for details and corrective actions. To aid migration to the Blackwell architecture, developers can opt-in to the Pascal scheduling model with the following combination of compiler options. nvcc -gencode=arch=compute\_60,code=sm\_100 ... 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial public release. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id1) Just-in-time compilation. [2](#id2) The CUDA driver caches the cubins generated as a result of the PTX JIT, so this is often a one-time cost. [3](#id3) Warp-synchronous refers to an assumption that threads in the same warp are synchronized at every instruction and can, for example, communicate values without explicit synchronization. --- # 1. Maxwell Tuning Guide — Maxwell Tuning Guide 12.8 documentation * [](../index.html) » * 1\. Maxwell Tuning Guide * v12.8 | [PDF](../pdf/Maxwell_Tuning_Guide.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * [Tuning CUDA Applications for Maxwell](#abstract) The programming guide to tuning CUDA Applications for GPUs based on the NVIDIA Maxwell Architecture. 1\. Maxwell Tuning Guide[](#tuning-cuda-applications-for-maxwell "Permalink to this headline") ================================================================================================ 1.1. NVIDIA Maxwell Compute Architecture[](#nvidia-maxwell-compute-architecture "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- Maxwell is NVIDIA’s next-generation architecture for CUDA compute applications. Maxwell retains and extends the same CUDA programming model as in previous NVIDIA architectures such as Fermi and Kepler, and applications that follow the best practices for those architectures should typically see speedups on the Maxwell architecture without any code changes. This guide summarizes the ways that an application can be fine-tuned to gain additional speedups by leveraging Maxwell architectural features.[1](#fn1) Maxwell introduces an all-new design for the Streaming Multiprocessor (_SM_) that dramatically improves energy efficiency. Although the Kepler SMX design was extremely efficient for its generation, through its development, NVIDIA’s GPU architects saw an opportunity for another big leap forward in architectural efficiency; the Maxwell SM is the realization of that vision. Improvements to control logic partitioning, workload balancing, clock-gating granularity, compiler-based scheduling, number of instructions issued per clock cycle, and many other enhancements allow the Maxwell SM (also called _SMM_) to far exceed Kepler SMX efficiency. The first Maxwell-based GPU is codenamed _GM107_ and is designed for use in power-limited environments like notebooks and small form factor (SFF) PCs. GM107 is described in a whitepaper entitled [NVIDIA GeForce GTX 750 Ti: Featuring First-Generation Maxwell GPU Technology, Designed for Extreme Performance per Watt](http://international.download.nvidia.com/geforce-com/international/pdfs/GeForce-GTX-750-Ti-Whitepaper.pdf) .[2](#fn2) The first GPU using the second-generation Maxwell architecture is codenamed _GM204_. Second-generation Maxwell GPUs retain the power efficiency of the earlier generation while delivering significantly higher performance. GM204 is described in a whitepaper entitled [NVIDIA GeForce GTX 980: Featuring Maxwell, The Most Advanced GPU Ever Made](http://international.download.nvidia.com/geforce-com/international/pdfs/GeForce_GTX_980_Whitepaper_FINAL.PDF) . Compute programming features of GM204 are similar to those of GM107, except where explicitly noted in this guide. For details on the programming features discussed in this guide, please refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) . 1.2. CUDA Best Practices[](#cuda-best-practices "Permalink to this headline") ------------------------------------------------------------------------------- The performance guidelines and best practices described in the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) and the [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) apply to all CUDA-capable GPU architectures. Programmers must primarily focus on following those recommendations to achieve the best performance. The high-priority recommendations from those guides are as follows: * Find ways to parallelize sequential code, * Minimize data transfers between the host and the device, * Adjust kernel launch configuration to maximize device utilization, * Ensure global memory accesses are coalesced, * Minimize redundant accesses to global memory whenever possible, * Avoid long sequences of diverged execution by threads within the same warp. 1.3. Application Compatibility[](#application-compatibility "Permalink to this headline") ------------------------------------------------------------------------------------------- Before addressing specific performance tuning issues covered in this guide, refer to the [Maxwell Compatibility Guide for CUDA Applications](https://docs.nvidia.com/cuda/maxwell-compatibility-guide/) to ensure that your application is compiled in a way that is compatible with Maxwell. 1.4. Maxwell Tuning[](#maxwell-tuning "Permalink to this headline") --------------------------------------------------------------------- ### 1.4.1. SMM[](#smm "Permalink to this headline") The Maxwell Streaming Multiprocessor, SMM, is similar in many respects to the Kepler architecture’s SMX. The key enhancements of SMM over SMX are geared toward improving efficiency without requiring significant increases in available parallelism per SM from the application. #### 1.4.1.1. Occupancy[](#occupancy "Permalink to this headline") The maximum number of concurrent warps per SMM remains the same as in SMX (i.e., 64), and [factors influencing warp occupancy](https://developer.download.nvidia.com/compute/cuda/CUDA_Occupancy_calculator.xls) remain similar or improved over SMX: * The register file size (64k 32-bit registers) is the same as that of SMX. * The maximum registers per thread, 255, matches that of Kepler GK110. As with Kepler, experimentation should be used to determine the optimum balance of register spilling vs. occupancy, however. * The maximum number of thread blocks per SM has been increased from 16 to 32. This should result in an automatic occupancy improvement for kernels with small thread blocks of 64 or fewer threads (shared memory and register file resource requirements permitting). Such kernels would have tended to under-utilize SMX, but less so SMM. * Shared memory capacity is increased (see [Shared Memory Capacity](#shared-memory-capacity) ). As such, developers can expect similar or improved occupancy on SMM without changes to their application. At the same time, warp occupancy requirements (i.e., available parallelism) for maximum device utilization are similar to or less than those of SMX (see [Instruction Latencies](#smm-latencies) ). #### 1.4.1.2. Instruction Scheduling[](#instruction-scheduling "Permalink to this headline") The number of CUDA Cores per SM has been reduced to a power of two, however with Maxwell’s improved execution efficiency, performance per SM is usually within 10% of Kepler performance, and the improved area efficiency of SMM means CUDA Cores per GPU will be substantially higher vs. comparable Fermi or Kepler chips. SMM retains the same number of instruction issue slots per clock and reduces arithmetic latencies compared to the Kepler design. As with SMX, each SMM has four warp schedulers. Unlike SMX, however, all SMM core functional units are assigned to a particular scheduler, with no shared units. Along with the selection of a power-of-two number of CUDA Cores per SM, which simplifies scheduling and reduces stall cycles, this partitioning of SM computational resources in SMM is a major component of the streamlined efficiency of SMM. The power-of-two number of CUDA Cores per partition simplifies scheduling, as each of SMM’s warp schedulers issue to a dedicated set of CUDA Cores equal to the warp width. Each warp scheduler still has the flexibility to dual-issue (such as issuing a math operation to a CUDA Core in the same cycle as a memory operation to a load/store unit), but single-issue is now sufficient to fully utilize all CUDA Cores. #### 1.4.1.3. Instruction Latencies[](#instruction-latencies "Permalink to this headline") Another major improvement of SMM is that dependent math latencies have been significantly reduced; a consequence of this is a further reduction of stall cycles, as the available warp-level parallelism (i.e., occupancy) on SMM should be equal to or greater than that of SMX (see [Occupancy](#smm-occupancy) ), while at the same time each math operation takes _less_ time to complete, improving utilization and throughput. #### 1.4.1.4. Instruction Throughput[](#instruction-throughput "Permalink to this headline") The most significant changes to peak instruction throughputs in SMM are as follows: * The change in [number of CUDA Cores per SM](#smm-scheduling) brings with it a corresponding change in peak single-precision floating point operations per clock per SM. However, since the number of SMs is typically increased, the result is an increase in aggregate peak throughput; furthermore, the scheduling and latency improvements also discussed above make this peak easier to approach. * The throughput of many integer operations including multiply, logical operations and shift is improved. In addition, there are now specialized integer instructions that can accelerate pointer arithmetic. These instructions are most efficient when data structures are a power of two in size. Note As was already the recommended best practice, signed arithmetic should be preferred over unsigned arithmetic wherever possible for best throughput on SMM. The C language standard places more restrictions on overflow behavior for unsigned math, limiting compiler optimization opportunities. ### 1.4.2. Memory Throughput[](#memory-throughput "Permalink to this headline") #### 1.4.2.1. Unified L1/Texture Cache[](#unified-l1-texture-cache "Permalink to this headline") Maxwell combines the functionality of the L1 and texture caches into a single unit. As with Kepler, global loads in Maxwell are cached in L2 only, unless using the _LDG_ read-only data cache mechanism introduced in Kepler. In a manner similar to Kepler GK110B, GM204 retains this behavior by default but also allows applications to opt-in to caching of global loads in its unified L1/Texture cache. The opt-in mechanism is the same as with GK110B: pass the `-Xptxas -dlcm=ca` flag to `nvcc` at compile time. Local loads also are cached in L2 only, which could increase the cost of register spilling if L1 local load hit rates were high with Kepler. The balance of occupancy versus spilling should therefore be reevaluated to ensure best performance. Especially given the improvements to arithmetic latencies, code built for Maxwell may benefit from somewhat lower occupancy (due to increased registers per thread) in exchange for lower spilling. The unified L1/texture cache acts as a coalescing buffer for memory accesses, gathering up the data requested by the threads of a warp prior to delivery of that data to the warp. This function previously was served by the separate L1 cache in Fermi and Kepler. Two new device attributes were added in CUDA Toolkit 6.0: `globalL1CacheSupported` and `localL1CacheSupported`. Developers who wish to have separately-tuned paths for various architecture generations can use these fields to simplify the path selection process. Note Enabling caching of globals in GM204 can affect occupancy. If per-thread-block SM resource usage would result in zero occupancy with caching enabled, the CUDA driver will override the caching selection to allow the kernel launch to succeed. This situation is reported by the profiler. ### 1.4.3. Shared Memory[](#shared-memory "Permalink to this headline") #### 1.4.3.1. Shared Memory Capacity[](#shared-memory-capacity "Permalink to this headline") With Fermi and Kepler, shared memory and the L1 cache shared the same on-chip storage. Maxwell, by contrast, provides dedicated space to the shared memory of each SMM, since the functionality of the L1 and texture caches have been merged in SMM. This increases the shared memory space available per SMM as compared to SMX: GM107 provides 64 KB shared memory per SMM, and GM204 further increases this to 96 KB shared memory per SMM. This presents several benefits to application developers: * Algorithms with significant shared memory capacity requirements (e.g., radix sort) see an automatic 33% to 100% boost in capacity per SM on top of the aggregate boost from higher SM count. * Applications no longer need to select a preference of the L1/shared split for optimal performance. For purposes of backward compatibility with Fermi and Kepler, applications may optionally continue to specify such a preference, but the preference will be ignored on Maxwell, with the full 64 KB per SMM always going to shared memory. Note While the per-SM shared memory capacity is increased in SMM, the per-thread-block limit remains 48 KB. For maximum flexibility on possible future GPUs, NVIDIA recommends that applications use at most 32 KB of shared memory in any one thread block, which would for example allow at least two such thread blocks to fit per SMM. #### 1.4.3.2. Shared Memory Bandwidth[](#shared-memory-bandwidth "Permalink to this headline") Kepler SMX introduced an optional 8-byte shared memory banking mode, which had the potential to increase shared memory bandwidth per SM over Fermi for shared memory accesses of 8 or 16 bytes. However, applications could only benefit from this when storing these larger elements in shared memory (i.e., integers and fp32 values saw no benefit), and only when the developer explicitly opted into the 8-byte bank mode via the API. To simplify this, Maxwell returns to the Fermi style of shared memory banking, where banks are always four bytes wide. Aggregate shared memory bandwidth across the chip remains comparable to that of corresponding Kepler chips, given increased SM count. In this way, all applications using shared memory can now benefit from the higher bandwidth, even when storing only four-byte items into shared memory and without specifying any particular preference via the API. #### 1.4.3.3. Fast Shared Memory Atomics[](#fast-shared-memory-atomics "Permalink to this headline") Kepler introduced a dramatically higher throughput for atomic operations to _global_ memory as compared to Fermi. However, atomic operations to _shared_ memory remained essentially unchanged: both architectures implemented shared memory atomics using a lock/update/unlock pattern that could be expensive in the case of high contention for updates to particular locations in shared memory. Maxwell improves upon this by implementing native shared memory atomic operations for 32-bit integers and native shared memory 32-bit and 64-bit compare-and-swap (CAS), which can be used to implement other atomic functions with reduced overhead compared to the Fermi and Kepler methods. Note Refer to the [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) for an example implementation of an fp64 `atomicAdd()` using `atomicCAS()`. ### 1.4.4. Dynamic Parallelism[](#dynamic-parallelism "Permalink to this headline") GK110 introduced a new architectural feature called Dynamic Parallelism, which allows the GPU to create additional work for itself. A programming model enhancement leveraging this feature was introduced in CUDA 5.0 to enable kernels running on GK110 to launch additional kernels onto the same GPU. SMM brings Dynamic Parallelism into the mainstream by supporting it across the product line, even in lower-power chips such as GM107. This will benefit developers, as it means that applications will no longer need special-case algorithm implementations for high-end GPUs that differ from those usable in more power-constrained environments. 2\. Revision History[](#revision-history "Permalink to this headline") ======================================================================== **Version 1.0** * Initial Public Release **Version 1.1** * Updated for second-generation Maxwell (compute capability 5.2). **Version 1.2** * Updated references to the CUDA C++ Programming Guide and CUDA C++ Best Practices Guide. 3\. Notices[](#notices "Permalink to this headline") ====================================================== 3.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 3.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 3.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. [1](#id3) Throughout this guide, _Fermi_ refers to devices of compute capability 2.x, _Kepler_ refers to devices of compute capability 3.x, and _Maxwell_ refers to devices of compute capability 5.x. [2](#id4) The features of GM108 are similar to those of GM107. --- # CUDA Driver API :: CUDA Toolkit Documentation CUDA Driver API ([PDF](../pdf/CUDA_Driver_API.pdf) ) - v12.8.1 ([older](https://developer.nvidia.com/cuda-toolkit-archive) ) - Last updated March 04, 2025 - [Send Feedback](mailto:CUDAIssues@nvidia.com?subject=CUDA%20Toolkit%20Documentation%20Feedback:%20CUDA%20Driver%20API) Table of Contents ================= * [1. Difference between the driver and runtime APIs](driver-vs-runtime-api.html#driver-vs-runtime-api) * [2. API synchronization behavior](api-sync-behavior.html#api-sync-behavior) * [3. Stream synchronization behavior](stream-sync-behavior.html#stream-sync-behavior) * [4. Graph object thread safety](graphs-thread-safety.html#graphs-thread-safety) * [5. Rules for version mixing](version-mixing-rules.html#version-mixing-rules) * [6. Modules](modules.html#modules) * [6.1. Data types used by CUDA driver](group__CUDA__TYPES.html#group__CUDA__TYPES) * [6.2. Error Handling](group__CUDA__ERROR.html#group__CUDA__ERROR) * [6.3. Initialization](group__CUDA__INITIALIZE.html#group__CUDA__INITIALIZE) * [6.4. Version Management](group__CUDA__VERSION.html#group__CUDA__VERSION) * [6.5. Device Management](group__CUDA__DEVICE.html#group__CUDA__DEVICE) * [6.6. Device Management \[DEPRECATED\]](group__CUDA__DEVICE__DEPRECATED.html#group__CUDA__DEVICE__DEPRECATED) * [6.7. Primary Context Management](group__CUDA__PRIMARY__CTX.html#group__CUDA__PRIMARY__CTX) * [6.8. Context Management](group__CUDA__CTX.html#group__CUDA__CTX) * [6.9. Context Management \[DEPRECATED\]](group__CUDA__CTX__DEPRECATED.html#group__CUDA__CTX__DEPRECATED) * [6.10. Module Management](group__CUDA__MODULE.html#group__CUDA__MODULE) * [6.11. Module Management \[DEPRECATED\]](group__CUDA__MODULE__DEPRECATED.html#group__CUDA__MODULE__DEPRECATED) * [6.12. Library Management](group__CUDA__LIBRARY.html#group__CUDA__LIBRARY) * [6.13. Memory Management](group__CUDA__MEM.html#group__CUDA__MEM) * [6.14. Virtual Memory Management](group__CUDA__VA.html#group__CUDA__VA) * [6.15. Stream Ordered Memory Allocator](group__CUDA__MALLOC__ASYNC.html#group__CUDA__MALLOC__ASYNC) * [6.16. Multicast Object Management](group__CUDA__MULTICAST.html#group__CUDA__MULTICAST) * [6.17. Unified Addressing](group__CUDA__UNIFIED.html#group__CUDA__UNIFIED) * [6.18. Stream Management](group__CUDA__STREAM.html#group__CUDA__STREAM) * [6.19. Event Management](group__CUDA__EVENT.html#group__CUDA__EVENT) * [6.20. External Resource Interoperability](group__CUDA__EXTRES__INTEROP.html#group__CUDA__EXTRES__INTEROP) * [6.21. Stream Memory Operations](group__CUDA__MEMOP.html#group__CUDA__MEMOP) * [6.22. Execution Control](group__CUDA__EXEC.html#group__CUDA__EXEC) * [6.23. Execution Control \[DEPRECATED\]](group__CUDA__EXEC__DEPRECATED.html#group__CUDA__EXEC__DEPRECATED) * [6.24. Graph Management](group__CUDA__GRAPH.html#group__CUDA__GRAPH) * [6.25. Occupancy](group__CUDA__OCCUPANCY.html#group__CUDA__OCCUPANCY) * [6.26. Texture Reference Management \[DEPRECATED\]](group__CUDA__TEXREF__DEPRECATED.html#group__CUDA__TEXREF__DEPRECATED) * [6.27. Surface Reference Management \[DEPRECATED\]](group__CUDA__SURFREF__DEPRECATED.html#group__CUDA__SURFREF__DEPRECATED) * [6.28. Texture Object Management](group__CUDA__TEXOBJECT.html#group__CUDA__TEXOBJECT) * [6.29. Surface Object Management](group__CUDA__SURFOBJECT.html#group__CUDA__SURFOBJECT) * [6.30. Tensor Map Object Managment](group__CUDA__TENSOR__MEMORY.html#group__CUDA__TENSOR__MEMORY) * [6.31. Peer Context Memory Access](group__CUDA__PEER__ACCESS.html#group__CUDA__PEER__ACCESS) * [6.32. Graphics Interoperability](group__CUDA__GRAPHICS.html#group__CUDA__GRAPHICS) * [6.33. Driver Entry Point Access](group__CUDA__DRIVER__ENTRY__POINT.html#group__CUDA__DRIVER__ENTRY__POINT) * [6.34. Coredump Attributes Control API](group__CUDA__COREDUMP.html#group__CUDA__COREDUMP) * [6.35. Green Contexts](group__CUDA__GREEN__CONTEXTS.html#group__CUDA__GREEN__CONTEXTS) * [6.36. CUDA Checkpointing](group__CUDA__CHECKPOINT.html#group__CUDA__CHECKPOINT) * [6.37. Profiler Control \[DEPRECATED\]](group__CUDA__PROFILER__DEPRECATED.html#group__CUDA__PROFILER__DEPRECATED) * [6.38. Profiler Control](group__CUDA__PROFILER.html#group__CUDA__PROFILER) * [6.39. OpenGL Interoperability](group__CUDA__GL.html#group__CUDA__GL) * [6.39.1. OpenGL Interoperability \[DEPRECATED\]](group__CUDA__GL__DEPRECATED.html#group__CUDA__GL__DEPRECATED) * [6.40. Direct3D 9 Interoperability](group__CUDA__D3D9.html#group__CUDA__D3D9) * [6.40.1. Direct3D 9 Interoperability \[DEPRECATED\]](group__CUDA__D3D9__DEPRECATED.html#group__CUDA__D3D9__DEPRECATED) * [6.41. Direct3D 10 Interoperability](group__CUDA__D3D10.html#group__CUDA__D3D10) * [6.41.1. Direct3D 10 Interoperability \[DEPRECATED\]](group__CUDA__D3D10__DEPRECATED.html#group__CUDA__D3D10__DEPRECATED) * [6.42. Direct3D 11 Interoperability](group__CUDA__D3D11.html#group__CUDA__D3D11) * [6.42.1. Direct3D 11 Interoperability \[DEPRECATED\]](group__CUDA__D3D11__DEPRECATED.html#group__CUDA__D3D11__DEPRECATED) * [6.43. VDPAU Interoperability](group__CUDA__VDPAU.html#group__CUDA__VDPAU) * [6.44. EGL Interoperability](group__CUDA__EGL.html#group__CUDA__EGL) * [7. Data Structures](annotated.html#annotated) * [7.1. CUaccessPolicyWindow\_v1](structCUaccessPolicyWindow__v1.html#structCUaccessPolicyWindow__v1) * [7.2. CUarrayMapInfo\_v1](structCUarrayMapInfo__v1.html#structCUarrayMapInfo__v1) * [7.3. CUasyncNotificationInfo](structCUasyncNotificationInfo.html#structCUasyncNotificationInfo) * [7.4. CUcheckpointCheckpointArgs](structCUcheckpointCheckpointArgs.html#structCUcheckpointCheckpointArgs) * [7.5. CUcheckpointLockArgs](structCUcheckpointLockArgs.html#structCUcheckpointLockArgs) * [7.6. CUcheckpointRestoreArgs](structCUcheckpointRestoreArgs.html#structCUcheckpointRestoreArgs) * [7.7. CUcheckpointUnlockArgs](structCUcheckpointUnlockArgs.html#structCUcheckpointUnlockArgs) * [7.8. CUctxCigParam](structCUctxCigParam.html#structCUctxCigParam) * [7.9. CUctxCreateParams](structCUctxCreateParams.html#structCUctxCreateParams) * [7.10. CUDA\_ARRAY3D\_DESCRIPTOR\_v2](structCUDA__ARRAY3D__DESCRIPTOR__v2.html#structCUDA__ARRAY3D__DESCRIPTOR__v2) * [7.11. CUDA\_ARRAY\_DESCRIPTOR\_v2](structCUDA__ARRAY__DESCRIPTOR__v2.html#structCUDA__ARRAY__DESCRIPTOR__v2) * [7.12. CUDA\_ARRAY\_MEMORY\_REQUIREMENTS\_v1](structCUDA__ARRAY__MEMORY__REQUIREMENTS__v1.html#structCUDA__ARRAY__MEMORY__REQUIREMENTS__v1) * [7.13. CUDA\_ARRAY\_SPARSE\_PROPERTIES\_v1](structCUDA__ARRAY__SPARSE__PROPERTIES__v1.html#structCUDA__ARRAY__SPARSE__PROPERTIES__v1) * [7.14.](structCUDA__BATCH__MEM__OP__NODE__PARAMS__v2.html)   * [7.15. CUDA\_CHILD\_GRAPH\_NODE\_PARAMS](structCUDA__CHILD__GRAPH__NODE__PARAMS.html#structCUDA__CHILD__GRAPH__NODE__PARAMS) * [7.16. CUDA\_CONDITIONAL\_NODE\_PARAMS](structCUDA__CONDITIONAL__NODE__PARAMS.html#structCUDA__CONDITIONAL__NODE__PARAMS) * [7.17. CUDA\_EVENT\_RECORD\_NODE\_PARAMS](structCUDA__EVENT__RECORD__NODE__PARAMS.html#structCUDA__EVENT__RECORD__NODE__PARAMS) * [7.18. CUDA\_EVENT\_WAIT\_NODE\_PARAMS](structCUDA__EVENT__WAIT__NODE__PARAMS.html#structCUDA__EVENT__WAIT__NODE__PARAMS) * [7.19. CUDA\_EXT\_SEM\_SIGNAL\_NODE\_PARAMS\_v1](structCUDA__EXT__SEM__SIGNAL__NODE__PARAMS__v1.html#structCUDA__EXT__SEM__SIGNAL__NODE__PARAMS__v1) * [7.20. CUDA\_EXT\_SEM\_SIGNAL\_NODE\_PARAMS\_v2](structCUDA__EXT__SEM__SIGNAL__NODE__PARAMS__v2.html#structCUDA__EXT__SEM__SIGNAL__NODE__PARAMS__v2) * [7.21. CUDA\_EXT\_SEM\_WAIT\_NODE\_PARAMS\_v1](structCUDA__EXT__SEM__WAIT__NODE__PARAMS__v1.html#structCUDA__EXT__SEM__WAIT__NODE__PARAMS__v1) * [7.22. CUDA\_EXT\_SEM\_WAIT\_NODE\_PARAMS\_v2](structCUDA__EXT__SEM__WAIT__NODE__PARAMS__v2.html#structCUDA__EXT__SEM__WAIT__NODE__PARAMS__v2) * [7.23. CUDA\_EXTERNAL\_MEMORY\_BUFFER\_DESC\_v1](structCUDA__EXTERNAL__MEMORY__BUFFER__DESC__v1.html#structCUDA__EXTERNAL__MEMORY__BUFFER__DESC__v1) * [7.24. CUDA\_EXTERNAL\_MEMORY\_HANDLE\_DESC\_v1](structCUDA__EXTERNAL__MEMORY__HANDLE__DESC__v1.html#structCUDA__EXTERNAL__MEMORY__HANDLE__DESC__v1) * [7.25. CUDA\_EXTERNAL\_MEMORY\_MIPMAPPED\_ARRAY\_DESC\_v1](structCUDA__EXTERNAL__MEMORY__MIPMAPPED__ARRAY__DESC__v1.html#structCUDA__EXTERNAL__MEMORY__MIPMAPPED__ARRAY__DESC__v1) * [7.26. CUDA\_EXTERNAL\_SEMAPHORE\_HANDLE\_DESC\_v1](structCUDA__EXTERNAL__SEMAPHORE__HANDLE__DESC__v1.html#structCUDA__EXTERNAL__SEMAPHORE__HANDLE__DESC__v1) * [7.27. CUDA\_EXTERNAL\_SEMAPHORE\_SIGNAL\_PARAMS\_v1](structCUDA__EXTERNAL__SEMAPHORE__SIGNAL__PARAMS__v1.html#structCUDA__EXTERNAL__SEMAPHORE__SIGNAL__PARAMS__v1) * [7.28. CUDA\_EXTERNAL\_SEMAPHORE\_WAIT\_PARAMS\_v1](structCUDA__EXTERNAL__SEMAPHORE__WAIT__PARAMS__v1.html#structCUDA__EXTERNAL__SEMAPHORE__WAIT__PARAMS__v1) * [7.29. CUDA\_GRAPH\_INSTANTIATE\_PARAMS](structCUDA__GRAPH__INSTANTIATE__PARAMS.html#structCUDA__GRAPH__INSTANTIATE__PARAMS) * [7.30. CUDA\_HOST\_NODE\_PARAMS\_v1](structCUDA__HOST__NODE__PARAMS__v1.html#structCUDA__HOST__NODE__PARAMS__v1) * [7.31. CUDA\_HOST\_NODE\_PARAMS\_v2](structCUDA__HOST__NODE__PARAMS__v2.html#structCUDA__HOST__NODE__PARAMS__v2) * [7.32. CUDA\_KERNEL\_NODE\_PARAMS\_v1](structCUDA__KERNEL__NODE__PARAMS__v1.html#structCUDA__KERNEL__NODE__PARAMS__v1) * [7.33. CUDA\_KERNEL\_NODE\_PARAMS\_v2](structCUDA__KERNEL__NODE__PARAMS__v2.html#structCUDA__KERNEL__NODE__PARAMS__v2) * [7.34. CUDA\_KERNEL\_NODE\_PARAMS\_v3](structCUDA__KERNEL__NODE__PARAMS__v3.html#structCUDA__KERNEL__NODE__PARAMS__v3) * [7.35. CUDA\_LAUNCH\_PARAMS\_v1](structCUDA__LAUNCH__PARAMS__v1.html#structCUDA__LAUNCH__PARAMS__v1) * [7.36. CUDA\_MEM\_ALLOC\_NODE\_PARAMS\_v1](structCUDA__MEM__ALLOC__NODE__PARAMS__v1.html#structCUDA__MEM__ALLOC__NODE__PARAMS__v1) * [7.37. CUDA\_MEM\_ALLOC\_NODE\_PARAMS\_v2](structCUDA__MEM__ALLOC__NODE__PARAMS__v2.html#structCUDA__MEM__ALLOC__NODE__PARAMS__v2) * [7.38. CUDA\_MEM\_FREE\_NODE\_PARAMS](structCUDA__MEM__FREE__NODE__PARAMS.html#structCUDA__MEM__FREE__NODE__PARAMS) * [7.39. CUDA\_MEMCPY2D\_v2](structCUDA__MEMCPY2D__v2.html#structCUDA__MEMCPY2D__v2) * [7.40. CUDA\_MEMCPY3D\_PEER\_v1](structCUDA__MEMCPY3D__PEER__v1.html#structCUDA__MEMCPY3D__PEER__v1) * [7.41. CUDA\_MEMCPY3D\_v2](structCUDA__MEMCPY3D__v2.html#structCUDA__MEMCPY3D__v2) * [7.42. CUDA\_MEMCPY\_NODE\_PARAMS](structCUDA__MEMCPY__NODE__PARAMS.html#structCUDA__MEMCPY__NODE__PARAMS) * [7.43. CUDA\_MEMSET\_NODE\_PARAMS\_v1](structCUDA__MEMSET__NODE__PARAMS__v1.html#structCUDA__MEMSET__NODE__PARAMS__v1) * [7.44. CUDA\_MEMSET\_NODE\_PARAMS\_v2](structCUDA__MEMSET__NODE__PARAMS__v2.html#structCUDA__MEMSET__NODE__PARAMS__v2) * [7.45. CUDA\_POINTER\_ATTRIBUTE\_P2P\_TOKENS\_v1](structCUDA__POINTER__ATTRIBUTE__P2P__TOKENS__v1.html#structCUDA__POINTER__ATTRIBUTE__P2P__TOKENS__v1) * [7.46. CUDA\_RESOURCE\_DESC\_v1](structCUDA__RESOURCE__DESC__v1.html#structCUDA__RESOURCE__DESC__v1) * [7.47. CUDA\_RESOURCE\_VIEW\_DESC\_v1](structCUDA__RESOURCE__VIEW__DESC__v1.html#structCUDA__RESOURCE__VIEW__DESC__v1) * [7.48. CUDA\_TEXTURE\_DESC\_v1](structCUDA__TEXTURE__DESC__v1.html#structCUDA__TEXTURE__DESC__v1) * [7.49. CUdevprop\_v1](structCUdevprop__v1.html#structCUdevprop__v1) * [7.50. CUdevResource](structCUdevResource.html#structCUdevResource) * [7.51. CUdevSmResource](structCUdevSmResource.html#structCUdevSmResource) * [7.52. CUeglFrame\_v1](structCUeglFrame__v1.html#structCUeglFrame__v1) * [7.53. CUexecAffinityParam\_v1](structCUexecAffinityParam__v1.html#structCUexecAffinityParam__v1) * [7.54. CUexecAffinitySmCount\_v1](structCUexecAffinitySmCount__v1.html#structCUexecAffinitySmCount__v1) * [7.55. CUextent3D\_v1](structCUextent3D__v1.html#structCUextent3D__v1) * [7.56. CUgraphEdgeData](structCUgraphEdgeData.html#structCUgraphEdgeData) * [7.57. CUgraphExecUpdateResultInfo\_v1](structCUgraphExecUpdateResultInfo__v1.html#structCUgraphExecUpdateResultInfo__v1) * [7.58. CUgraphNodeParams](structCUgraphNodeParams.html#structCUgraphNodeParams) * [7.59. CUipcEventHandle\_v1](structCUipcEventHandle__v1.html#structCUipcEventHandle__v1) * [7.60. CUipcMemHandle\_v1](structCUipcMemHandle__v1.html#structCUipcMemHandle__v1) * [7.61. CUlaunchAttribute](structCUlaunchAttribute.html#structCUlaunchAttribute) * [7.62. CUlaunchAttributeValue](unionCUlaunchAttributeValue.html#unionCUlaunchAttributeValue) * [7.63. CUlaunchConfig](structCUlaunchConfig.html#structCUlaunchConfig) * [7.64. CUlaunchMemSyncDomainMap](structCUlaunchMemSyncDomainMap.html#structCUlaunchMemSyncDomainMap) * [7.65. CUmemAccessDesc\_v1](structCUmemAccessDesc__v1.html#structCUmemAccessDesc__v1) * [7.66. CUmemAllocationProp\_v1](structCUmemAllocationProp__v1.html#structCUmemAllocationProp__v1) * [7.67. CUmemcpy3DOperand\_v1](structCUmemcpy3DOperand__v1.html#structCUmemcpy3DOperand__v1) * [7.68. CUmemcpyAttributes\_v1](structCUmemcpyAttributes__v1.html#structCUmemcpyAttributes__v1) * [7.69. CUmemDecompressParams](structCUmemDecompressParams.html#structCUmemDecompressParams) * [7.70. CUmemFabricHandle\_v1](structCUmemFabricHandle__v1.html#structCUmemFabricHandle__v1) * [7.71. CUmemLocation\_v1](structCUmemLocation__v1.html#structCUmemLocation__v1) * [7.72. CUmemPoolProps\_v1](structCUmemPoolProps__v1.html#structCUmemPoolProps__v1) * [7.73. CUmemPoolPtrExportData\_v1](structCUmemPoolPtrExportData__v1.html#structCUmemPoolPtrExportData__v1) * [7.74. CUmulticastObjectProp\_v1](structCUmulticastObjectProp__v1.html#structCUmulticastObjectProp__v1) * [7.75. CUoffset3D\_v1](structCUoffset3D__v1.html#structCUoffset3D__v1) * [7.76. CUstreamBatchMemOpParams\_v1](unionCUstreamBatchMemOpParams__v1.html#unionCUstreamBatchMemOpParams__v1) * [7.77. CUtensorMap](structCUtensorMap.html#structCUtensorMap) * [8. Data Fields](functions.html#functions) * [9. Deprecated List](deprecated.html#deprecated) * * * --- # CUDA Runtime API :: CUDA Toolkit Documentation CUDA Runtime API ([PDF](../pdf/CUDA_Runtime_API.pdf) ) - v12.8.1 ([older](https://developer.nvidia.com/cuda-toolkit-archive) ) - Last updated March 04, 2025 - [Send Feedback](mailto:CUDAIssues@nvidia.com?subject=CUDA%20Toolkit%20Documentation%20Feedback:%20CUDA%20Runtime%20API) Table of Contents ================= * [1. Difference between the driver and runtime APIs](driver-vs-runtime-api.html#driver-vs-runtime-api) * [2. API synchronization behavior](api-sync-behavior.html#api-sync-behavior) * [3. Stream synchronization behavior](stream-sync-behavior.html#stream-sync-behavior) * [4. Graph object thread safety](graphs-thread-safety.html#graphs-thread-safety) * [5. Rules for version mixing](version-mixing-rules.html#version-mixing-rules) * [6. Modules](modules.html#modules) * [6.1.](group__CUDART__DEVICE.html)   * [6.2. Device Management \[DEPRECATED\]](group__CUDART__DEVICE__DEPRECATED.html#group__CUDART__DEVICE__DEPRECATED) * [6.3. Thread Management \[DEPRECATED\]](group__CUDART__THREAD__DEPRECATED.html#group__CUDART__THREAD__DEPRECATED) * [6.4. Error Handling](group__CUDART__ERROR.html#group__CUDART__ERROR) * [6.5. Stream Management](group__CUDART__STREAM.html#group__CUDART__STREAM) * [6.6. Event Management](group__CUDART__EVENT.html#group__CUDART__EVENT) * [6.7. External Resource Interoperability](group__CUDART__EXTRES__INTEROP.html#group__CUDART__EXTRES__INTEROP) * [6.8. Execution Control](group__CUDART__EXECUTION.html#group__CUDART__EXECUTION) * [6.9. Execution Control \[DEPRECATED\]](group__CUDART__EXECUTION__DEPRECATED.html#group__CUDART__EXECUTION__DEPRECATED) * [6.10. Occupancy](group__CUDART__OCCUPANCY.html#group__CUDART__OCCUPANCY) * [6.11. Memory Management](group__CUDART__MEMORY.html#group__CUDART__MEMORY) * [6.12. Memory Management \[DEPRECATED\]](group__CUDART__MEMORY__DEPRECATED.html#group__CUDART__MEMORY__DEPRECATED) * [6.13. Stream Ordered Memory Allocator](group__CUDART__MEMORY__POOLS.html#group__CUDART__MEMORY__POOLS) * [6.14. Unified Addressing](group__CUDART__UNIFIED.html#group__CUDART__UNIFIED) * [6.15. Peer Device Memory Access](group__CUDART__PEER.html#group__CUDART__PEER) * [6.16. OpenGL Interoperability](group__CUDART__OPENGL.html#group__CUDART__OPENGL) * [6.17. OpenGL Interoperability \[DEPRECATED\]](group__CUDART__OPENGL__DEPRECATED.html#group__CUDART__OPENGL__DEPRECATED) * [6.18. Direct3D 9 Interoperability](group__CUDART__D3D9.html#group__CUDART__D3D9) * [6.19. Direct3D 9 Interoperability \[DEPRECATED\]](group__CUDART__D3D9__DEPRECATED.html#group__CUDART__D3D9__DEPRECATED) * [6.20. Direct3D 10 Interoperability](group__CUDART__D3D10.html#group__CUDART__D3D10) * [6.21. Direct3D 10 Interoperability \[DEPRECATED\]](group__CUDART__D3D10__DEPRECATED.html#group__CUDART__D3D10__DEPRECATED) * [6.22. Direct3D 11 Interoperability](group__CUDART__D3D11.html#group__CUDART__D3D11) * [6.23. Direct3D 11 Interoperability \[DEPRECATED\]](group__CUDART__D3D11__DEPRECATED.html#group__CUDART__D3D11__DEPRECATED) * [6.24. VDPAU Interoperability](group__CUDART__VDPAU.html#group__CUDART__VDPAU) * [6.25. EGL Interoperability](group__CUDART__EGL.html#group__CUDART__EGL) * [6.26. Graphics Interoperability](group__CUDART__INTEROP.html#group__CUDART__INTEROP) * [6.27. Texture Object Management](group__CUDART__TEXTURE__OBJECT.html#group__CUDART__TEXTURE__OBJECT) * [6.28. Surface Object Management](group__CUDART__SURFACE__OBJECT.html#group__CUDART__SURFACE__OBJECT) * [6.29. Version Management](group__CUDART____VERSION.html#group__CUDART____VERSION) * [6.30. Graph Management](group__CUDART__GRAPH.html#group__CUDART__GRAPH) * [6.31. Driver Entry Point Access](group__CUDART__DRIVER__ENTRY__POINT.html#group__CUDART__DRIVER__ENTRY__POINT) * [6.32. Library Management](group__CUDART__LIBRARY.html#group__CUDART__LIBRARY) * [6.33. C++ API Routines](group__CUDART__HIGHLEVEL.html#group__CUDART__HIGHLEVEL) * [6.34. Interactions with the CUDA Driver API](group__CUDART__DRIVER.html#group__CUDART__DRIVER) * [6.35. Profiler Control](group__CUDART__PROFILER.html#group__CUDART__PROFILER) * [6.36. Data types used by CUDA Runtime](group__CUDART__TYPES.html#group__CUDART__TYPES) * [7. Data Structures](annotated.html#annotated) * [7.1. \_\_cudaOccupancyB2DHelper](class____cudaOccupancyB2DHelper.html#class____cudaOccupancyB2DHelper) * [7.2. cudaAccessPolicyWindow](structcudaAccessPolicyWindow.html#structcudaAccessPolicyWindow) * [7.3. cudaArrayMemoryRequirements](structcudaArrayMemoryRequirements.html#structcudaArrayMemoryRequirements) * [7.4. cudaArraySparseProperties](structcudaArraySparseProperties.html#structcudaArraySparseProperties) * [7.5. cudaAsyncNotificationInfo\_t](structcudaAsyncNotificationInfo__t.html#structcudaAsyncNotificationInfo__t) * [7.6. cudaChannelFormatDesc](structcudaChannelFormatDesc.html#structcudaChannelFormatDesc) * [7.7. cudaChildGraphNodeParams](structcudaChildGraphNodeParams.html#structcudaChildGraphNodeParams) * [7.8. cudaConditionalNodeParams](structcudaConditionalNodeParams.html#structcudaConditionalNodeParams) * [7.9. cudaDeviceProp](structcudaDeviceProp.html#structcudaDeviceProp) * [7.10. cudaEglFrame](structcudaEglFrame.html#structcudaEglFrame) * [7.11. cudaEglPlaneDesc](structcudaEglPlaneDesc.html#structcudaEglPlaneDesc) * [7.12. cudaEventRecordNodeParams](structcudaEventRecordNodeParams.html#structcudaEventRecordNodeParams) * [7.13. cudaEventWaitNodeParams](structcudaEventWaitNodeParams.html#structcudaEventWaitNodeParams) * [7.14. cudaExtent](structcudaExtent.html#structcudaExtent) * [7.15. cudaExternalMemoryBufferDesc](structcudaExternalMemoryBufferDesc.html#structcudaExternalMemoryBufferDesc) * [7.16. cudaExternalMemoryHandleDesc](structcudaExternalMemoryHandleDesc.html#structcudaExternalMemoryHandleDesc) * [7.17. cudaExternalMemoryMipmappedArrayDesc](structcudaExternalMemoryMipmappedArrayDesc.html#structcudaExternalMemoryMipmappedArrayDesc) * [7.18. cudaExternalSemaphoreHandleDesc](structcudaExternalSemaphoreHandleDesc.html#structcudaExternalSemaphoreHandleDesc) * [7.19. cudaExternalSemaphoreSignalNodeParams](structcudaExternalSemaphoreSignalNodeParams.html#structcudaExternalSemaphoreSignalNodeParams) * [7.20. cudaExternalSemaphoreSignalNodeParamsV2](structcudaExternalSemaphoreSignalNodeParamsV2.html#structcudaExternalSemaphoreSignalNodeParamsV2) * [7.21. cudaExternalSemaphoreSignalParams](structcudaExternalSemaphoreSignalParams.html#structcudaExternalSemaphoreSignalParams) * [7.22. cudaExternalSemaphoreSignalParams\_v1](structcudaExternalSemaphoreSignalParams__v1.html#structcudaExternalSemaphoreSignalParams__v1) * [7.23. cudaExternalSemaphoreWaitNodeParams](structcudaExternalSemaphoreWaitNodeParams.html#structcudaExternalSemaphoreWaitNodeParams) * [7.24. cudaExternalSemaphoreWaitNodeParamsV2](structcudaExternalSemaphoreWaitNodeParamsV2.html#structcudaExternalSemaphoreWaitNodeParamsV2) * [7.25. cudaExternalSemaphoreWaitParams](structcudaExternalSemaphoreWaitParams.html#structcudaExternalSemaphoreWaitParams) * [7.26. cudaExternalSemaphoreWaitParams\_v1](structcudaExternalSemaphoreWaitParams__v1.html#structcudaExternalSemaphoreWaitParams__v1) * [7.27. cudaFuncAttributes](structcudaFuncAttributes.html#structcudaFuncAttributes) * [7.28. cudaGraphEdgeData](structcudaGraphEdgeData.html#structcudaGraphEdgeData) * [7.29. cudaGraphExecUpdateResultInfo](structcudaGraphExecUpdateResultInfo.html#structcudaGraphExecUpdateResultInfo) * [7.30. cudaGraphInstantiateParams](structcudaGraphInstantiateParams.html#structcudaGraphInstantiateParams) * [7.31. cudaGraphKernelNodeUpdate](structcudaGraphKernelNodeUpdate.html#structcudaGraphKernelNodeUpdate) * [7.32. cudaGraphNodeParams](structcudaGraphNodeParams.html#structcudaGraphNodeParams) * [7.33. cudaHostNodeParams](structcudaHostNodeParams.html#structcudaHostNodeParams) * [7.34. cudaHostNodeParamsV2](structcudaHostNodeParamsV2.html#structcudaHostNodeParamsV2) * [7.35. cudaIpcEventHandle\_t](structcudaIpcEventHandle__t.html#structcudaIpcEventHandle__t) * [7.36. cudaIpcMemHandle\_t](structcudaIpcMemHandle__t.html#structcudaIpcMemHandle__t) * [7.37. cudaKernelNodeParams](structcudaKernelNodeParams.html#structcudaKernelNodeParams) * [7.38. cudaKernelNodeParamsV2](structcudaKernelNodeParamsV2.html#structcudaKernelNodeParamsV2) * [7.39. cudaLaunchAttribute](structcudaLaunchAttribute.html#structcudaLaunchAttribute) * [7.40. cudaLaunchAttributeValue](unioncudaLaunchAttributeValue.html#unioncudaLaunchAttributeValue) * [7.41. cudaLaunchConfig\_t](structcudaLaunchConfig__t.html#structcudaLaunchConfig__t) * [7.42. cudaLaunchMemSyncDomainMap](structcudaLaunchMemSyncDomainMap.html#structcudaLaunchMemSyncDomainMap) * [7.43. cudaLaunchParams](structcudaLaunchParams.html#structcudaLaunchParams) * [7.44. cudaMemAccessDesc](structcudaMemAccessDesc.html#structcudaMemAccessDesc) * [7.45. cudaMemAllocNodeParams](structcudaMemAllocNodeParams.html#structcudaMemAllocNodeParams) * [7.46. cudaMemAllocNodeParamsV2](structcudaMemAllocNodeParamsV2.html#structcudaMemAllocNodeParamsV2) * [7.47. cudaMemcpy3DOperand](structcudaMemcpy3DOperand.html#structcudaMemcpy3DOperand) * [7.48. cudaMemcpy3DParms](structcudaMemcpy3DParms.html#structcudaMemcpy3DParms) * [7.49. cudaMemcpy3DPeerParms](structcudaMemcpy3DPeerParms.html#structcudaMemcpy3DPeerParms) * [7.50. cudaMemcpyAttributes](structcudaMemcpyAttributes.html#structcudaMemcpyAttributes) * [7.51. cudaMemcpyNodeParams](structcudaMemcpyNodeParams.html#structcudaMemcpyNodeParams) * [7.52. cudaMemFreeNodeParams](structcudaMemFreeNodeParams.html#structcudaMemFreeNodeParams) * [7.53. cudaMemLocation](structcudaMemLocation.html#structcudaMemLocation) * [7.54. cudaMemPoolProps](structcudaMemPoolProps.html#structcudaMemPoolProps) * [7.55. cudaMemPoolPtrExportData](structcudaMemPoolPtrExportData.html#structcudaMemPoolPtrExportData) * [7.56. cudaMemsetParams](structcudaMemsetParams.html#structcudaMemsetParams) * [7.57. cudaMemsetParamsV2](structcudaMemsetParamsV2.html#structcudaMemsetParamsV2) * [7.58. cudaOffset3D](structcudaOffset3D.html#structcudaOffset3D) * [7.59. cudaPitchedPtr](structcudaPitchedPtr.html#structcudaPitchedPtr) * [7.60. cudaPointerAttributes](structcudaPointerAttributes.html#structcudaPointerAttributes) * [7.61. cudaPos](structcudaPos.html#structcudaPos) * [7.62. cudaResourceDesc](structcudaResourceDesc.html#structcudaResourceDesc) * [7.63. cudaResourceViewDesc](structcudaResourceViewDesc.html#structcudaResourceViewDesc) * [7.64. cudaTextureDesc](structcudaTextureDesc.html#structcudaTextureDesc) * [7.65. CUuuid\_st](structCUuuid__st.html#structCUuuid__st) * [8. Data Fields](functions.html#functions) * [9. Deprecated List](deprecated.html#deprecated) * * * --- # CUDA Math API Reference Manual — CUDA Math API Reference Manual 12.8 documentation * [](../index.html) » * CUDA Math API Reference Manual * v12.8 | [PDF](../pdf/CUDA_Math_API.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * CUDA Math API Reference Manual[](#cuda-math-api-reference-manual "Permalink to this headline") ================================================================================================ CUDA mathematical functions are always available in device code. Host implementations of the common mathematical functions are mapped in a platform-specific way to standard math library functions, provided by the host compiler and respective host libm where available. Some functions, not available with the host compilers, are implemented in crt/math\_functions.hpp header file. For example, see [erfinv()](cuda_math_api/group__CUDA__MATH__DOUBLE.html#group__cuda__math__double_1gaef012e8d10e9ef980940f65630f77ae3) . Other, less common functions, like [rhypot()](cuda_math_api/group__CUDA__MATH__DOUBLE.html#group__cuda__math__double_1gaf1dfb4d01feaa01b0b1ff15cf57ebbc3) , [cyl\_bessel\_i0()](cuda_math_api/group__CUDA__MATH__DOUBLE.html#group__cuda__math__double_1gaaeae8990c401dc1ad0426de1350560b3) are only available in device code. CUDA Math device functions are no-throw for well-formed CUDA programs. Note that many floating-point and integer functions names are overloaded for different argument types. For example, the [log()](cuda_math_api/group__CUDA__MATH__DOUBLE.html#group__cuda__math__double_1ga28ce8e15ef5149c271eba95663becba2) function has the following prototypes: double log(double x); float log(float x); float logf(float x); Note also that due to implementation constraints, certain math functions from std:: namespace may be callable in device code even via explicitly qualified std:: names. However, such use is discouraged, since this capability is unsupported, unverified, undocumented, not portable, and may change without notice. * [1\. FP4 Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__FP4.html) * [2\. FP6 Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__FP6.html) * [3\. FP8 Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__FP8.html) * [4\. Half Precision Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__HALF.html) * [5\. Bfloat16 Precision Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__BFLOAT16.html) * [6\. Single Precision Mathematical Functions](cuda_math_api/group__CUDA__MATH__SINGLE.html) * [7\. Single Precision Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__SINGLE.html) * [8\. Double Precision Mathematical Functions](cuda_math_api/group__CUDA__MATH__DOUBLE.html) * [9\. Double Precision Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__DOUBLE.html) * [10\. FP128 Quad Precision Mathematical Functions](cuda_math_api/group__CUDA__MATH__QUAD.html) * [11\. Type Casting Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__CAST.html) * [12\. Integer Mathematical Functions](cuda_math_api/group__CUDA__MATH__INT.html) * [13\. Integer Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__INT.html) * [14\. SIMD Intrinsics](cuda_math_api/group__CUDA__MATH__INTRINSIC__SIMD.html) * [15\. Structs](cuda_math_api/structs.html) * [16\. Notices](notices.html) --- # 1. Introduction — NVBLAS 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/NVBLAS_Library.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVBLAS The User guide for NVBLAS, drop-in BLAS replacement, multi-GPUs accelerated 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ The NVBLAS Library is a GPU-accelerated Libary that implements BLAS (Basic Linear Algebra Subprograms). It can accelerate most BLAS Level-3 routines by dynamically routing BLAS calls to one or more NVIDIA GPUs present in the system, when the charateristics of the call make it speed up on a GPU. 2\. NVBLAS Overview[](#nvblas-overview "Permalink to this headline") ====================================================================== The NVBLAS Library is built on top of the cuBLAS Library using only the CUBLASXT API (refer to the CUBLASXT API section of the cuBLAS Documentation for more details). NVBLAS also requires the presence of a CPU BLAS lirbary on the system. Currently NVBLAS intercepts only compute intensive BLAS Level-3 calls (see table below). Depending on the charateristics of those BLAS calls, NVBLAS will redirect the calls to the GPUs present in the system or to CPU. That decision is based on a simple heuristic that estimates if the BLAS call will execute for long enough to amortize the PCI transfers of the input and output data to the GPU. **Because NVBLAS does not support all standard BLAS routines, it might be necessary to associate it with an existing full BLAS Library. Please refer to the Usage section for more details.** 3\. GPU Accelerated Routines[](#gpu-accelerated-routines "Permalink to this headline") ======================================================================================== NVBLAS offloads only the compute-intensive BLAS3 routines which have the best potential for acceleration on GPUs. The following table shows the currently supported routines: | Routine | Types | Operation | | --- | --- | --- | | gemm | S,D,C,Z | Multiplication of 2 matrices | | syrk | S,D,C,Z | Symmetric rank-k update | | herk | C,Z | Hermitian rank-k update | | syr2k | S,D,C,Z | Symmetric rank-2k update | | her2k | C,Z | Hermitian rank-2k update | | trsm | S,D,C,Z | Triangular solve with multiple right-hand sides | | trmm | S,D,C,Z | Triangular matrix-matrix multiplication | | symm | S,D,C,Z | Symmetric matrix-matrix multiplication | | hemm | C,Z | Hermitian matrix-matrix multiplication | 4\. BLAS Symbols Interception[](#blas-symbols-interception "Permalink to this headline") ========================================================================================== Standard BLAS Library implementations usually expose multiple symbols for the same routines. Let’s say `func` is a BLAS routine name, `func_` or/and `func` are usually defined as extern symbols. Some BLAS Libraries might also expose some symbols with a proprietary appended prefix. NVBLAS intercepts only the symbols `func_` and `func`. The user needs to make sure that the application intended to be GPU-accelerated by NVBLAS actually calls those defined symbols. Any other symbols will not be intercepted and the original BLAS routine will be executed for those cases. 5\. Device Memory Support[](#device-memory-support "Permalink to this headline") ================================================================================== Starting with Release 8.0, data can be located on any GPU device, even on GPU devices that are not configured to be part of the computation. When any of the data is located on a GPU, the computation will be exclusively done on GPU whatever the size of the problem. Also, this feature has to be used with caution: the user has to be sure that the BLAS call will indeed be intercepted by NVBLAS, otherwise it will result in a crash when the CPU BLAS tries to execute it. 6\. Security Precaution[](#security-precaution "Permalink to this headline") ============================================================================== Because the NVBLAS Library relies on a symbols interception mechanism, it is essential to make sure it has not been compromised. In that regard, NVBLAS should never be used from a process running at elevated privileges, such as Administrator on Windows or root on Linux. 7\. Configuration[](#configuration "Permalink to this headline") ================================================================== Because NVBLAS is a drop-in replacement of BLAS, it must be configured through an ASCII text file that describes how many and which GPUs can participate in the intercepted BLAS calls. The configuration file is parsed at the time of the loading of the library. The format of the configuration file is based on keywords optionally followed by one or more user-defined parameters. At most one keyword per line is allowed. Blank lines or lines beginning with the character `#` are ignored. 7.1. NVBLAS\_CONFIG\_FILE Environment Variable[](#nvblas-config-file-environment-variable "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- The location and name of the configuration file must be defined by the environment variable `NVBLAS_CONFIG_FILE`. By default, if `NVBLAS_CONFIG_FILE` is not defined, NVBLAS will try to open the file `nvblas.conf` in the current directory. For a safe use of NVBLAS, the configuration file should have have restricted write permissions. 7.2. Configuration Keywords[](#configuration-keywords "Permalink to this headline") ------------------------------------------------------------------------------------- The configuration keywords syntax is described in the following subsections. ### 7.2.1. NVBLAS\_LOGFILE[](#nvblas-logfile "Permalink to this headline") This keyword defines the file where NVBLAS should print status and error messages. By default, if not defined, the standard error output file (eg. stderr) will be used. It is advised to define this keyword early in the configuration to capture errors in parsing that file itself. ### 7.2.2. NVBLAS\_TRACE\_LOG\_ENABLED[](#nvblas-trace-log-enabled "Permalink to this headline") When this keyword is defined, every intercepted BLAS calls will be logged into the NVBLAS\_LOGFILE. This feature, even though intrusive, can be useful for debugging purposes. ### 7.2.3. NVBLAS\_CPU\_BLAS\_LIB[](#nvblas-cpu-blas-lib "Permalink to this headline") This keyword defines the CPU BLAS dynamic library file (for example, `.so` file on Linux or `.dll` on Windows) that NVBLAS should open to find the CPU BLAS symbols definitions. This keyword must be defined for NVBLAS to work. Because CPU Blas libraries are often composed of multiple files, even though this keyword is set to the full path to the main file of the CPU library, it might still be necessary to define the right path to find the rest of the library files in the environment of your system. On Linux, this can be done by setting the environment variable `LD_LIBRARY_PATH` whereas on Windows, this can be done by setting the environment variable `PATH`. For a safe use of NVBLAS, the following precautions are strongly advised: * The CPU BLAS Library should be located where ordinary users do not have write permissions. * The path specified should be absolute, not relative. ### 7.2.4. NVBLAS\_GPU\_LIST[](#nvblas-gpu-list "Permalink to this headline") This keyword defines the list of GPUs that should participate in the computation of the intercepted BLAS calls. If not defined, only GPU device 0 is used, since that is normally the most compute-capable GPU installed in the system. This keyword can be set to a list of device numbers separated by blank characters. Also the following wildcard keywords are also accepted for simplicity : | Keyword | Meaning | | --- | --- | | `ALL` | All compute-capable GPUs detected on the system will be used by NVBLAS | | `ALL0` | GPU device 0, AND all others GPUs detected that have the same compute-capabilities as device 0 will be used by NVBLAS | Note In the current release of CUBLAS, the CUBLASXT API supports two GPUs if they are on the same board such as Tesla K10 or GeForce GTX690 and one GPU otherwise. Because NVBLAS is built on top of the CUBLASXT API, NVBLAS has the same restriction. If access to more GPUs devices is needed, details of the licensing are described at [cublasXt](https://developer.nvidia.com/cublasxt) . ### 7.2.5. NVBLAS\_TILE\_DIM[](#nvblas-tile-dim "Permalink to this headline") This keyword defines the tile dimension that should be used to divide the matrices involved in the computation. This definition maps directly to a call of the cublasXt API routine `cublasXtSetBlockDim`. Refer to [cuBLAS documentation](https://docs.nvidia.com/cuda/cublas/index.html) to understand the tradeoffs associated with setting this to a larger or a smaller value. ### 7.2.6. [NVBLAS\_GPU\_DISABLED\_](#nvblas_gpu_disabled) [](#nvblas-gpu-disabled-blas-func-name "Permalink to this headline") This keyword, appended with the name of a BLAS routine disables NVBLAS from running a specified routine on the GPU. This feature is intended mainly for debugging purposes. By default, all supported BLAS routines are enabled. ### 7.2.7. [NVBLAS\_CPU\_RATIO\_](#nvblas_cpu_ratio) [](#nvblas-cpu-ratio-blas-func-name "Permalink to this headline") This keyword, appended with the name of ta BLAS routine defines the ratio of the workload that should remain on the CPU in the event that the NVBLAS decides to offload work for that routine on the GPU. This functionality is directly mapped to the cublasXt API routine `cublasXtSetCpuRatio`. By default, the ratio is defined to zero for all routines. Please refer to the [cuBLAS documentation](https://docs.nvidia.com/cuda/cublas/index.html) for details and for the list of routines which support this feature. ### 7.2.8. NVBLAS\_AUTOPIN\_MEM\_ENABLED[](#nvblas-autopin-mem-enabled "Permalink to this headline") This keyword enables the Pinning Memory mode. This functionality is directly mapped to the cublasXt API routine `cublasXtSetPinningMemMode`. If this keyowrd is not present in the configuration file, the Pinning Memory mode will be set to `CUBLASXT_PINNING_DISABLED`. Note There are some restrictions to use this feature as specified in the cuBLAS documentation of the underlying routine `cublasXtSetPinningMemMode`. Specifically when NVBLAS is used in a multi-threaded applications, this option should not be used if there is a chance that matrices used by different threads overlaps while calling NVBLAS. Please refer to the cuBLAS Documentation of the routine `` `cublasXtSetPinningMemMode `` <[https://docs.nvidia.com/cuda/cublas/index.html#cublasxt\_setPinningMemMode](https://docs.nvidia.com/cuda/cublas/index.html#cublasxt_setPinningMemMode) \>\`\_\_ for details. ### 7.2.9. Configuration File Example[](#configuration-file-example "Permalink to this headline") The following example shows a typical NVBLAS configuration file : \# This is the configuration file to use NVBLAS Library # Setup the environment variable NVBLAS\_CONFIG\_FILE to specify your own config file. # By default, if NVBLAS\_CONFIG\_FILE is not defined, # NVBLAS Library will try to open the file "nvblas.conf" in its current directory # Example : NVBLAS\_CONFIG\_FILE /home/cuda\_user/my\_nvblas.conf # The config file should have restricted write permissions accesses # Specify which output log file (default is stderr) NVBLAS\_LOGFILE nvblas.log # Enable trace log of every intercepted BLAS calls NVBLAS\_TRACE\_LOG\_ENABLED #Put here the CPU BLAS fallback Library of your choice #It is strongly advised to use full path to describe the location of the CPU Library NVBLAS\_CPU\_BLAS\_LIB /usr/lib/libopenblas.so #NVBLAS\_CPU\_BLAS\_LIB /libmkl\_rt.so # List of GPU devices Id to participate to the computation # Use ALL if you want all your GPUs to contribute # Use ALL0, if you want all your GPUs of the same type as device 0 to contribute # However, NVBLAS consider that all GPU have the same performance and PCI bandwidth # By default if no GPU are listed, only device 0 will be used #NVBLAS\_GPU\_LIST 0 2 4 #NVBLAS\_GPU\_LIST ALL NVBLAS\_GPU\_LIST ALL0 # Tile Dimension NVBLAS\_TILE\_DIM 2048 # Autopin Memory NVBLAS\_AUTOPIN\_MEM\_ENABLED #List of BLAS routines that are prevented from running on GPU (use for debugging purpose # The current list of BLAS routines supported by NVBLAS are # GEMM, SYRK, HERK, TRSM, TRMM, SYMM, HEMM, SYR2K, HER2K #NVBLAS\_GPU\_DISABLED\_SGEMM #NVBLAS\_GPU\_DISABLED\_DGEMM #NVBLAS\_GPU\_DISABLED\_CGEMM #NVBLAS\_GPU\_DISABLED\_ZGEMM # Computation can be optionally hybridized between CPU and GPU # By default, GPU-supported BLAS routines are ran fully on GPU # The option NVBLAS\_CPU\_RATIO\_ give the ratio \[0,1\] # of the amount of computation that should be done on CPU # CAUTION : this option should be used wisely because it can actually # significantly reduced the overall performance if too much work is given to CPU #NVBLAS\_CPU\_RATIO\_CGEMM 0.07 8\. NVBLAS Installation[](#nvblas-installation "Permalink to this headline") ============================================================================== The NVBLAS Library is part of the CUDA Toolkit, and will be installed along all the other CUDA libraries. It is available on 64-bit operating systems. NVBLAS Library is built on top of cuBLAS, so the cuBLAS library needs to be accessible by NVBLAS. 9\. Usage[](#usage "Permalink to this headline") ================================================== To use the NVBLAS Library, the user application must be relinked against NVBLAS in addition to the original CPU Blas (technically only NVBLAS is needed unless some BLAS routines not supported by NVBLAS are used by the application). To be sure that the linker links against the exposed symbols of NVBLAS and not the ones from the CPU BLAS, the NVBLAS Library needs to be put before the CPU BLAS on the linkage command line. On Linux, an alternative way to use NVBLAS Library is to use the `LD_PRELOAD` environment variable; this technique has the advantage of avoiding the relinkage step. However, the user should avoid defining that environment variable globally because it will cause the NVBLAS library to be loaded by every shell command executed on the system, thus leading to a lack of responsiveness of the system. Finally, mathematical tools and libraries often offer the opportunity to specify the BLAS Library to be used through an environment variable or a configuration file. Because NVBLAS does not support all the standard BLAS routines, it might be necessary to pair NVBLAS with a full BLAS library, even though your application only calls supported NVBLAS routines. Fortunately, those tools and libraries usually offer a way to specify multiple BLAS Libraries. Please refer to the documentation of the appropriate tools and libraries for details. 10\. Notices[](#notices "Permalink to this headline") ======================================================= 10.1. Notice[](#notice "Permalink to this headline") ------------------------------------------------------ This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 10.2. OpenCL[](#opencl "Permalink to this headline") ------------------------------------------------------ OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 10.3. Trademarks[](#trademarks "Permalink to this headline") -------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Introduction — nvJPEG 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/nvJPEG.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * nvJPEG A GPU accelerated JPEG codec library. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ 1.1. nvJPEG Decoder[](#nvjpeg-decoder "Permalink to this headline") --------------------------------------------------------------------- The nvJPEG library provides high-performance, GPU accelerated JPEG decoding functionality for image formats commonly used in deep learning and hyperscale multimedia applications. The library offers single and batched JPEG decoding capabilities which efficiently utilize the available GPU resources for optimum performance; and the flexibility for users to manage the memory allocation needed for decoding. The nvJPEG library enables the following functions: use the JPEG image data stream as input; retrieve the width and height of the image from the data stream, and use this retrieved information to manage the GPU memory allocation and the decoding. A dedicated API is provided for retrieving the image information from the raw JPEG image data stream. Note Throughout this document, the terms “CPU” and “Host” are used synonymously. Similarly, the terms “GPU” and “Device” are synonymous. The nvJPEG library supports the following: **JPEG options:** * Baseline and Progressive JPEG decoding/encoding * 8 bits per pixel * Huffman bitstream decoding * Upto 4 channel JPEG bitstreams * 8- and 16-bit quantization tables * The following chroma subsampling for the 3 color channels Y, Cb, Cr (Y, U, V): * 4:4:4 * 4:2:2 * 4:2:0 * 4:4:0 * 4:1:1 * 4:1:0 **Features:** * Hybrid decoding using both the CPU (i.e., host) and the GPU (i.e., device). * Hardware acceleration for baseline JPEG decode on [Hardware Acceleration](#nvjpeg-hardware-accelaration) . * Input to the library is in the host memory, and the output is in the GPU memory. * Single image and batched image decoding. * Single phase and multiple phases decoding. * Color space conversion. * User-provided memory manager for the device and pinned host memory allocations. 1.2. nvJPEG Encoder[](#nvjpeg-encoder "Permalink to this headline") --------------------------------------------------------------------- The encoding functions of the nvJPEG library perform GPU-accelerated compression of user’s image data to the JPEG bitstream. User can provide input data in a number of formats and colorspaces, and control the encoding process with parameters. Encoding functionality will allocate temporary buffers using user-provided memory allocator. Before calling the encoding functions the user should perform a few prerequisite steps using the helper functions described in [nvJPEG Encoder Helper API Reference](#nvjpeg-encoder-helper-api-reference) . 1.3. Thread Safety[](#thread-safety "Permalink to this headline") ------------------------------------------------------------------- Not all nvJPEG types are thread safe. When using decoder APIs across multiple threads, the following decoder types should be instantiated separately for each thread: [nvJPEG Bitstream Handle](#nvjpeg-bitstream-handle) , [nvJPEG Opaque JPEG Decoding State Handle](#nvjpeg-opaque-jpeg-decoding-state-handle) , [nvJPEG Decode Device Buffer Handle](#nvjpeg-decode-device-buffer-handle) , [nvJPEG Decode Pinned Buffer Handle](#nvjpeg-decode-pinned-buffer-handle) When using encoder APIs across multiple threads, [nvjpegEncoderState\_t](#nvjpeg-encoder-state) should be instantiated separately for each thread. For user-provided allocators (inputs to [nvjpegCreateEx()](#nvjpeg-create-ex) ), the user needs to ensure thread safety. 1.4. Multi-GPU support[](#multi-gpu-support "Permalink to this headline") --------------------------------------------------------------------------- The nvJPEG states and handles are bound to the device that was set as current during their creation. Using these states and handles with another device set as current is undefined. The user is responsible of keeping track of the current device. 1.5. Hardware Acceleration[](#hardware-acceleration "Permalink to this headline") ----------------------------------------------------------------------------------- Hardware accelerated JPEG decode is available on the following GPUs architecures - * Ampere (A100, A30) * Hopper * Ada * Blackwell Platforms which support hardware accelerated JPEG decode: * Windows * Linux (x86\_64, PowerPC, ARM64) 2\. JPEG Decoding[](#jpeg-decoding "Permalink to this headline") ================================================================== 2.1. Using JPEG Decoding[](#using-jpeg-decoding "Permalink to this headline") ------------------------------------------------------------------------------- ​The nvJPEG library provides functions for both the decoding of a single image, and batched decoding of multiple images. ### 2.1.1. Single Image Decoding[](#single-image-decoding "Permalink to this headline") For single-image decoding you provide the data size and a pointer to the file data, and the decoded image is placed in the output buffer. To use the nvJPEG library, start by calling the helper functions for initialization. 1. Create nvJPEG library handle with one of the helper functions `nvjpegCreateSimple() or nvjpegCreateEx()`. 2. Create JPEG state with the helper function `nvjpegJpegStateCreate()`. See [nvJPEG Type Declarations](#nvjpeg-type-declarations) and `nvjpegJpegStateCreate()`. The following helper functions are available in the nvJPEG library: * `nvjpegStatus_t nvjpegGetProperty(libraryPropertyType type, int *value);` * `[DEPRECATED] nvjpegStatus_t nvjpegCreate(nvjpegBackend_t backend, nvjpegHandle_t *handle , nvjpeg_dev_allocator allocator);` * `nvjpegStatus_t nvjpegCreateSimple(nvjpegHandle_t *handle);` * `nvjpegStatus_t nvjpegCreateEx(nvjpegBackend_t backend, nvjpegDevAllocator_t *dev_allocator, nvjpegPinnedAllocator_t *pinned_allocator, unsigned int flags, nvjpegHandle_t *handle);` * `nvjpegStatus_t nvjpegDestroy(nvjpegHandle_t handle);` * `nvjpegStatus_t nvjpegJpegStateCreate(nvjpegHandle_t handle, nvjpegJpegState_t *jpeg_handle);` * `nvjpegStatus_t nvjpegJpegStateDestroy(nvjpegJpegState handle);` * Other helper functions such as `nvjpegSet*()` and `nvjpegGet*()` can be used to configure the library functionality on per-handle basis. Refer to the [nvJPEG Helper API Reference](#nvjpeg-helper-api-reference) for more details. 3. Retrieve the width and height information from the JPEG-encoded image by using the `nvjpegGetImageInfo()` function. Below is the signature of `nvjpegGetImageInfo()`function: nvjpegStatus\_t nvjpegGetImageInfo( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, int \*nComponents, nvjpegChromaSubsampling\_t \*subsampling, int \*widths, int \*heights); For each image to be decoded, pass the JPEG data pointer and data length to the above function. The `nvjpegGetImageInfo()` function is thread safe. 4. One of the outputs of the above `nvjpegGetImageInfo()` function is `nvjpegChromaSubsampling_t`. This parameter is an enum type, and its enumerator list is composed of the chroma subsampling property retrieved from the JPEG image. See [nvJPEG Chroma Subsampling](#nvjpeg-chroma-subsampling) . 5. Use the `nvjpegDecode()` function in the nvJPEG library to decode this single JPEG image. See the signature of this function below: nvjpegStatus\_t nvjpegDecode( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*data, size\_t length, nvjpegOutputFormat\_t output\_format, nvjpegImage\_t \*destination, cudaStream\_t stream); In the above `nvjpegDecode()` function, the parameters `nvjpegOutputFormat_t`, `nvjpegImage_t`, and `cudaStream_t` can be used to set the output behavior of the `nvjpegDecode()`function. You provide the `cudaStream_t` parameter to indicate the stream to which your asynchronous tasks are submitted. 6. **The \`\`nvjpegOutputFormat\_t\`\` parameter:** The `nvjpegOutputFormat_t` parameter can be set to one of the `output_format` settings below: | | | | --- | --- | | **output\_format** | **Meaning** | | `NVJPEG_OUTPUT_UNCHANGED` | Return the decoded image planar format. | | `NVJPEG_OUTPUT_RGB` | Convert to planar RGB. | | `NVJPEG_OUTPUT_BGR` | Convert to planar BGR. | | `NVJPEG_OUTPUT_RGBI` | Convert to interleaved RGB. | | `NVJPEG_OUTPUT_BGRI` | Convert to interleaved BGR. | | `NVJPEG_OUTPUT_Y` | Return the Y component only. | | `NVJPEG_OUTPUT_YUV` | Return in the YUV planar format. | | `NVJPEG_OUTPUT_UNCHANGEDI_U16` | Return the decoded image interleaved format. | > For example, if `output_format` is set to `NVJPEG_OUTPUT_Y` or `NVJPEG_OUTPUT_RGBI`, or `NVJPEG_OUTPUT_BGRI` then the output is written only to channel\[0\] of `nvjpegImage_t`, and the other channels are not touched. > > Alternately, in the case of planar output, the data is written to the corresponding channels of the `nvjpegImage_t` destination structure. > > Finally, in the case of grayscale JPEG and RGB output, the luminance is used to create the grayscale RGB. > > The below table explains the combinations of the output formats and the number of channels supported by the library. | | | | | | | --- | --- | --- | --- | --- | | **No of Channels in bitstream** | 1 | 2 | 3 | 4 | | **Output Format** | | NVJPEG\_OUTPUT\_UNCHANGED | Yes | Yes | Yes | Yes | | NVJPEG\_OUTPUT\_YUV | Only the first channel of the output is populated | No | Yes | No | | NVJPEG\_OUTPUT\_Y | Yes | No | Yes | Yes(a) :ref:\`nvjpeg-single-image-decoding\_\_first | | NVJPEG\_OUTPUT\_RGB | Yes(b) :ref:\`nvjpeg-single-image-decoding\_\_second | No | Yes | Yes(a) :ref:\`nvjpeg-single-image-decoding\_\_first | | NVJPEG\_OUTPUT\_BGR | Yes(b) :ref:\`nvjpeg-single-image-decoding\_\_second | No | Yes | Yes(a) :ref:\`nvjpeg-single-image-decoding\_\_first | | NVJPEG\_OUTPUT\_RGBI | Yes(b) :ref:\`nvjpeg-single-image-decoding\_\_second | No | Yes | Yes(a) :ref:\`nvjpeg-single-image-decoding\_\_first | | NVJPEG\_OUTPUT\_BGRI | Yes(b) :ref:\`nvjpeg-single-image-decoding\_\_second | No | Yes | Yes(a) :ref:\`nvjpeg-single-image-decoding\_\_first | | NVJPEG\_OUTPUT\_UNCHANGEDI\_U16 | Yes(c) | Yes | No | No | | **NOTES:**

1. Must be enabled using [nvjpegDecodeParamsSetAllowCMYK()](#nvjpeg-decode-params-set-allow-cmyk)
.

2. Luminance is used to create the grayscale RGB.

3. Supported only by `NVJPEG_BACKEND_LOSSLESS_JPEG` backend. | | | | | 1. As mentioned above, an important benefit of the `nvjpegGetImageInfo()`function is the ability to utilize the image information retrieved from the the input JPEG image to allocate proper GPU memory for your decoding operation. The `nvjpegGetImageInfo()` function returns the `widths`, `heights` and `nComponents` parameters. nvjpegStatus\_t nvjpegGetImageInfo( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, int \*nComponents, nvjpegChromaSubsampling\_t \*subsampling, int \*widths, int \*heights); You can use the retrieved parameters, `widths`, `heights` and `nComponents`, to calculate the required size for the output buffers, either for a single decoded JPEG, or for every decoded JPEG in a batch. To optimally set the `destination` parameter for the `nvjpegDecode()` function, use the following guidelines: | | | | | --- | --- | --- | | **For the output\_format:**

NVJPEG\_OUTPUT\_Y | **destination.pitch\[0\] should be at least:** width\[0\] | **destination.channel\[0\] should be at least of size:** destination.pitch\[0\]\*height\[0\] | | **For the output\_format** | **destination.pitch\[c\] should be at least:** | **destination.channel\[c\] should be at least of size:** | | NVJPEG\_OUTPUT\_YUV | width\[c\] for c = 0, 1, 2 | destination.pitch\[c\]\*height\[c\] for c = 0, 1, 2 | | NVJPEG\_OUTPUT\_RGB and NVJPEG\_OUTPUT\_BGR | width\[0\] for c = 0, 1, 2 | destination.pitch\[0\]\*height\[0\] for c = 0, 1, 2 | | NVJPEG\_OUTPUT\_RGBI and NVJPEG\_OUTPUT\_BGRI | width\[0\]\*3 | destination.pitch\[0\]\*height\[0\] | | NVJPEG\_OUTPUT\_UNCHANGED | width\[c\] for c = \[ 0, nComponents - 1 \] | destination.pitch\[c\]\*height\[c\] for c = \[ 0, nComponents - 1\] | | NVJPEG\_OUTPUT\_UNCHANGEDI\_U16 | width\[c\]\* nComponents\* sizeof(unsigned short) | destination.pitch\[c\]\*height\[c\] for c = \[ 0, nComponents - 1\] | 2. Ensure that the [nvJPEG Image](#nvjpeg-image) structure (or structures, in the case of batched decode) is filled with the pointers and pitches of allocated buffers. The `nvjpegImage_t` structure that holds the output pointers is defined as follows: typedef struct { unsigned char \* channel\[NVJPEG\_MAX\_COMPONENT\]; size\_t pitch\[NVJPEG\_MAX\_COMPONENT\]; } nvjpegImage\_t; NVJPEG\_MAX\_COMPONENT is the maximum number of color components the nvJPEG library supports in the current release. For generic images, this is the maximum number of encoded channels that the library is able to decompress. 3. Finally, when you call the `nvjpegDecode()` function with the parameters as described above, the `nvjpegDecode()` function fills the output buffers with the decoded data. ### 2.1.2. Decode using Decoupled Phases[](#decode-using-decoupled-phases "Permalink to this headline") The nvJPEG library allows further separation of the host and device phases of the decode process. The host phase of the decoding will not need to access to device resources. A few examples of decoupled APIs can be found under [Decode API—Decoupled Decoding](#nvjpeg-decoupled-decode-api) . Below is the sequence of API calls to decode a single image 1. Initialize all the items that are used in the decoding process: 1. Create the library handle using one of the library handle initialization routines. 2. Choose decoder implementation `nvjpegBackend_t`, and create decoder using `nvjpegDecoderCreate()`. 3. Create JPEG decoder state using `nvjpegDecoderStateCreate()`. 4. Create JPEG stream using `nvjpegJpegStreamCreate()`. 5. Create the pinned and device buffers used by the decoder using the below APIs respectively. These buffers are used to store intermediate decoding results. * `nvjpegBufferPinnedCreate()` * `nvjpegBufferDeviceCreate()` 6. Link the buffers to the JPEG state using the following APIs respectively: * `nvjpegStateAttachPinnedBuffer()` * `nvjpegStateAttachDeviceBuffer()` 7. Create decode parameters using the below API. This is used to set the output format, and enable ROI decode: `nvjpegDecodeParamsCreate()` 2. Perform decoding: 1. Parse the jpeg bit-stream using `nvjpegJpegStreamParse()` * Encoded bitstream information, like channel dimensions, can be retrieved using the below API. This information is used to allocate the output pointers in `nvjpegImage_t`. * `nvjpegJpegStreamGetComponentsNum()` * `nvjpegJpegStreamGetComponentDimensions()` 2. Call the decode API in the below sequence to decode the image: * `nvjpegDecodeJpegHost()` * `nvjpegDecodeJpegTransferToDevice()` * `nvjpegDecodeJpegDevice()` ### 2.1.3. Batched Image Decoding[](#batched-image-decoding "Permalink to this headline") For the batched image decoding you provide pointers to multiple file data in the memory, and also provide the buffer sizes for each file data. The nvJPEG library will decode these multiple images, and will place the decoded data in the output buffers that you specified in the parameters. #### 2.1.3.1. Single Phase[](#single-phase "Permalink to this headline") For batched image decoding in single phase, follow these steps: 1. Call `nvjpegDecodeBatchedInitialize()` function to initialize the batched decoder. Specify the batch size in the `batch_size` parameter. See `nvjpegDecodeBatchedInitialize()`. 2. Next, call `nvjpegDecodeBatched()` for each new batch. Make sure to pass the parameters that are correct to the specific batch of images. If the size of the batch changes, or if the batch decoding fails, then call the `nvjpegDecodeBatchedInitialize()` function again. 2.2. nvJPEG Type Declarations[](#nvjpeg-type-declarations "Permalink to this headline") ----------------------------------------------------------------------------------------- ### 2.2.1. nvJPEG Backend[](#nvjpeg-backend "Permalink to this headline") typedef enum { NVJPEG\_BACKEND\_DEFAULT \= 0, NVJPEG\_BACKEND\_HYBRID \= 1, NVJPEG\_BACKEND\_GPU\_HYBRID \= 2, NVJPEG\_BACKEND\_HARDWARE \= 3, NVJPEG\_BACKEND\_GPU\_HYBRID\_DEVICE \= 4, NVJPEG\_BACKEND\_HARDWARE\_DEVICE \= 5, NVJPEG\_BACKEND\_LOSSLESS\_JPEG \= 6 } nvjpegBackend\_t; The `nvjpegBackend_t` enum is used to select either default back-end by default, or use GPU decoding for baseline JPEG images, or use CPU for Huffman decoding. | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_BACKEND\_DEFAULT | Back-end is selected internally. | | NVJPEG\_BACKEND\_HYBRID | Uses CPU for Huffman decoding. | | NVJPEG\_BACKEND\_GPU\_HYBRID | Uses GPU for Huffman decoding. `nvjpegDecodeBatched` will use GPU decoding for baseline JPEG images with interleaved scan when batch size is greater than 50. The [Decode API—Decoupled Decoding](#nvjpeg-decoupled-decode-api)
will use GPU assisted Huffman decoding. | | NVJPEG\_BACKEND\_HARDWARE | Uses [Hardware Acceleration](#nvjpeg-hardware-accelaration)
for decode. Supports baseline JPEG images with single scan with 1 or 3 channels. 410 and 411 chroma subsamplings are not supported. | | NVJPEG\_BACKEND\_GPU\_HYBRID\_DEVICE | Supports input bitstream on device memory. Can be used only with batched decode APIs for baseline JPEG images without restart intervals. | | NVJPEG\_BACKEND\_HARDWARE\_DEVICE | Supports input bitstream on device memory. Can be used only with batched decode APIs. Uses [Hardware Acceleration](#nvjpeg-hardware-accelaration)
for decode. Supports baseline JPEG images with single scan with 1 or 3 channels. 410 and 411 chroma subsamplings are not supported. | | NVJPEG\_BACKEND\_LOSSLESS\_JPEG | Supports lossless jpeg bitstreams as defined in the jpeg 92 standard. Bitstreams with up to 2 channels and prediction mode 1 are supported. | ### 2.2.2. nvJPEG Bitstream Handle[](#nvjpeg-bitstream-handle "Permalink to this headline") struct nvjpegJpegStream; typedef struct nvjpegJpegStream\* nvjpegJpegStream\_t; This handle stores the bit-stream parameters on the host. This helps retrieve bitstream meta-data using APIs defined in [nvJPEG Stream API](#nvjpeg-stream-api) . ### 2.2.3. nvJPEG Decode Device Buffer Handle[](#nvjpeg-decode-device-buffer-handle "Permalink to this headline") struct nvjpegBufferDevice; typedef struct nvjpegBufferDevice\* nvjpegBufferDevice\_t; This `nvjpegBufferDevice_t` is used by decoder states to store the intermediate information in device memory. ### 2.2.4. nvJPEG Decode Parameter Handle[](#nvjpeg-decode-parameter-handle "Permalink to this headline") struct nvjpegDecodeParams; typedef struct nvjpegDecodeParams\* nvjpegDecodeParams\_t; This decoder parameter handle stores the parameters like output format, and the ROI decode parameters that are set using APIs defined in [nvJPEG Chroma Subsampling](#nvjpeg-chroma-subsampling) . ### 2.2.5. nvJPEG Decode Pinned Buffer Handle[](#nvjpeg-decode-pinned-buffer-handle "Permalink to this headline") struct nvjpegBufferPinned; typedef struct nvjpegBufferPinned\* nvjpegBufferPinned\_t; This `nvjpegBufferPinned_t` handle is used by decoder states to store the intermediate information on pinned memory. ### 2.2.6. nvJPEG Decoder Handle[](#nvjpeg-decoder-handle "Permalink to this headline") struct nvjpegJpegDecoder; typedef struct nvjpegJpegDecoder\* nvjpegJpegDecoder\_t; This decoder handle stores the intermediate decoder data, which is shared across the decoding stages. This decoder handle is initialized for a given `nvjpegBackend_t`. It is used as input to the [Decode API—Decoupled Decoding](#nvjpeg-decoupled-decode-api) . ### 2.2.7. nvJPEG Host Pinned Memory Allocator Interface[](#nvjpeg-host-pinned-memory-allocator-interface "Permalink to this headline") typedef int (\*tPinnedMalloc)(void\*\*, size\_t, unsigned int flags); typedef int (\*tPinnedFree)(void\*); typedef struct { tPinnedMalloc pinned\_malloc; tPinnedFree pinned\_free; } nvjpegPinnedAllocator\_t; When the `nvjpegPinnedAllocator_t *allocator` parameter in the `nvjpegCreateEx()` function is set as a pointer to the above `nvjpegPinnedAllocator_t` structure, then this structure will be used for allocating and releasing host pinned memory for copying data to/from device. The function prototypes for the memory allocation and memory freeing functions are similar to the `cudaHostAlloc()` and `cudaFreeHost()` functions. They will return 0 in case of success, and non-zero otherwise. However, if the `nvjpegPinnedAllocator_t *allocator` parameter in the `nvjpegCreateEx()` function is set to NULL, then the default memory allocation functions `cudaHostAlloc()` and `cudaFreeHost()` will be used. When using `nvjpegCreate()` or `nvjpegCreateSimple()` function to create library handle, the default host pinned memory allocator will be used. ### 2.2.8. nvJPEG Extended Host Pinned Memory Allocator Interface[](#nvjpeg-extended-host-pinned-memory-allocator-interface "Permalink to this headline") typedef int (\*tPinnedMallocV2)(void\* ctx, void \*\*ptr, size\_t size, cudaStream\_t stream); typedef int (\*tPinnedFreeV2)(void\* ctx, void \*ptr, size\_t size, cudaStream\_t stream); typedef struct { tPinnedMallocV2 pinned\_malloc; tPinnedFreeV2 pinned\_free; void \*pinned\_ctx; } nvjpegPinnedAllocatorV2\_t; Extended pinned allocators support stream ordered allocations along with user defined context information `pinned_ctx`. When invoking the allocators, nvJPEG will pass `pinned_ctx` as input to the extended pinned allocators. ### 2.2.9. nvJPEG Image[](#nvjpeg-image "Permalink to this headline") typedef struct { unsigned char \* channel\[NVJPEG\_MAX\_COMPONENT\]; size\_t pitch\[NVJPEG\_MAX\_COMPONENT\]; } nvjpegImage\_t; The `nvjpegImage_t` structure (or structures, in the case of batched decode) is used to fill with the pointers and pitches of allocated buffers. The `nvjpegImage_t` structure that holds the output pointers. | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_MAX\_COMPONENT | Maximum number of color components the nvJPEG library supports. For generic images, this is the maximum number of encoded channels that the library is able to decompress. | ### 2.2.10. nvJPEG Device Memory Allocator Interface[](#nvjpeg-device-memory-allocator-interface "Permalink to this headline") typedef int (\*tDevMalloc)(void\*\*, size\_t); typedef int (\*tDevFree)(void\*); typedef struct { tDevMalloc dev\_malloc; tDevFree dev\_free; } nvjpegDevAllocator\_t; Users can tell the library to use their own device memory allocator. The function prototypes for the memory allocation and memory freeing functions are similar to the `cudaMalloc()`and `cudaFree()` functions. They should return 0 in case of success, and non-zero otherwise. A pointer to the `nvjpegDevAllocator_t` structure, with properly filled fields, should be provided to the `nvjpegCreate()` function. NULL is accepted, in which case the default memory allocation functions `cudaMalloc()` and `cudaFree()` is used. When the `nvjpegDevAllocator_t *allocator` parameter in the `nvjpegCreate()` or `nvjpegCreateEx()` function is set as a pointer to the above `nvjpegDevAllocator_t` structure, then this structure is used for allocating and releasing the device memory. The function prototypes for the memory allocation and memory freeing functions are similar to the `cudaMalloc()` and `cudaFree()` functions. They should return 0 in case of success, and non-zero otherwise. However, if the `nvjpegDevAllocator_t *allocator` parameter in the `nvjpegCreate()` or `nvjpegCreateEx()` function is set to NULL, then the default memory allocation functions `cudaMalloc()` and `cudaFree()` will be used. When using `nvjpegCreateSimple()` function to create library handle the default device memory allocator will be used. ### 2.2.11. nvJPEG Extended Device Memory Allocator Interface[](#nvjpeg-extended-device-memory-allocator-interface "Permalink to this headline") typedef int (\*tDevMallocV2)(void\* ctx, void \*\*ptr, size\_t size, cudaStream\_t stream); typedef int (\*tDevFreeV2)(void\* ctx, void \*ptr, size\_t size, cudaStream\_t stream); typedef struct { tDevMallocV2 dev\_malloc; tDevFreeV2 dev\_free; void \*dev\_ctx; } nvjpegDevAllocatorV2\_t; Extended device allocators support stream ordered allocations along with user defined context information `dev_ctx`. When invoking the allocators, nvJPEG will pass `dev_ctx` as input to the extended device allocators. ### 2.2.12. nvJPEG Opaque JPEG Decoding State Handle[](#nvjpeg-opaque-jpeg-decoding-state-handle "Permalink to this headline") struct nvjpegJpegState; typedef struct nvjpegJpegState\* nvjpegJpegState\_t; The `nvjpegJpegState` structure stores the temporary JPEG information. It should be initialized before any usage. This JPEG state handle can be reused after being used in another decoding. The same JPEG handle should be used across the decoding phases for the same image or batch. Multiple threads are allowed to share the JPEG state handle only when processing same batch during first phase (`nvjpegDecodePhaseOne`) . ### 2.2.13. nvJPEG Opaque Library Handle Struct[](#nvjpeg-opaque-library-handle-struct "Permalink to this headline") struct nvjpegHandle; typedef struct nvjpegHandle\* nvjpegHandle\_t; The library handle is used in any consecutive nvJPEG library calls, and should be initialized first. The library handle is thread safe, and can be used by multiple threads simultaneously. ### 2.2.14. nvJPEG Output Pointer Struct[](#nvjpeg-output-pointer-struct "Permalink to this headline") typedef struct { unsigned char \* channel\[NVJPEG\_MAX\_COMPONENT\]; size\_t pitch\[NVJPEG\_MAX\_COMPONENT\]; } nvjpegImage\_t; The `nvjpegImage_t`struct holds the pointers to the output buffers, and holds the corresponding strides of those buffers for the image decoding. Refer to [Single Image Decoding](#nvjpeg-single-image-decoding) on how to set up the `nvjpegImage_t` struct. ### 2.2.15. nvJPEG Jpeg Encoding[](#nvjpeg-jpeg-encoding "Permalink to this headline") typedef enum { NVJPEG\_ENCODING\_UNKNOWN \= 0x0, NVJPEG\_ENCODING\_BASELINE\_DCT \= 0xc0, NVJPEG\_ENCODING\_EXTENDED\_SEQUENTIAL\_DCT\_HUFFMAN \= 0xc1, NVJPEG\_ENCODING\_PROGRESSIVE\_DCT\_HUFFMAN \= 0xc2, NVJPEG\_ENCODING\_LOSSLESS\_HUFFMAN \= 0xc3 } nvjpegJpegEncoding\_t; The `nvjpegJpegEncoding_t` enum lists the JPEG encoding types that are supported by the nvJPEG library The enum values are based on the markers defined in the JPEG specification | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_ENCODING\_UNKNOWN | This value is returned for all the JPEG markers not supported by the nvJPEG library. | | NVJPEG\_ENCODING\_BASELINE\_DCT | Corresponds to the JPEG marker 0xc0, refer to the JPEG spec for more details. | | NVJPEG\_ENCODING\_EXTENDED\_SEQUENTIAL\_DCT\_HUFFMAN | Corresponds to the JPEG marker 0xc1, refer to the JPEG spec for more details. | | NVJPEG\_ENCODING\_PROGRESSIVE\_DCT\_HUFFMAN | Corresponds to the JPEG marker 0xc2, refer to the JPEG spec for more details. | | NVJPEG\_ENCODING\_LOSSLESS\_HUFFMAN | Corresponds to the JPEG marker 0xc3, refer to the JPEG spec for more details. | ### 2.2.16. nvJPEG Scale Factor[](#nvjpeg-scale-factor "Permalink to this headline") typedef enum { NVJPEG\_SCALE\_NONE \= 0, NVJPEG\_SCALE\_1\_BY\_2 \= 1, NVJPEG\_SCALE\_1\_BY\_4 \= 2, NVJPEG\_SCALE\_1\_BY\_8 \= 3 } nvjpegScaleFactor\_t; The `nvjpegScaleFactor_t` enum lists all the scale factors supported by the library. This feature is supported when nvjpeg handles are intstaniated using NVJPEG\_BACKEND\_HARDWARE | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_SCALE\_NONE | Decoded output is not scaled | | NVJPEG\_SCALE\_1\_BY\_2 | Decoded output width and height are scaled by a factor of 1/2 | | NVJPEG\_SCALE\_1\_BY\_4 | Decoded output width and height are scaled by a factor of 1/4 | | NVJPEG\_SCALE\_1\_BY\_8 | Decoded output width and height are scaled by a factor of 1/8 | ### 2.2.17. nvJPEG Flags[](#nvjpeg-flags "Permalink to this headline") #define NVJPEG\_FLAGS\_DEFAULT 0 #define NVJPEG\_FLAGS\_HW\_DECODE\_NO\_PIPELINE 1 #define NVJPEG\_FLAGS\_ENABLE\_MEMORY\_POOLS 2 #define NVJPEG\_FLAGS\_BITSTREAM\_STRICT 4 #define NVJPEG\_FLAGS\_REDUCED\_MEMORY\_DECODE 8 #define NVJPEG\_FLAGS\_REDUCED\_MEMORY\_DECODE\_ZERO\_COPY 16 #define NVJPEG\_FLAGS\_UPSAMPLING\_WITH\_INTERPOLATION 32 nvJPEG flags provide additional controls when initializing the library using [nvjpegCreateEx()](#nvjpeg-create-ex) or [nvjpegCreateExV2()](#nvjpeg-create-ex-v2) . It is possible to combine the flags as they are bit fields. | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_FLAGS\_DEFAULT | Corresponds to default library behavior. | | NVJPEG\_FLAGS\_HW\_DECODE\_NO\_PIPELINE | To be used when the library is initialized with NVJPEG\_BACKEND\_HARDWARE. It will be ignored for other back-ends. nvjpeg in batched decode mode buffers additional images to achieve optimal performance. Use this flag to disable buffering of additional images. | | NVJPEG\_FLAGS\_ENABLE\_MEMORY\_POOLS \[Deprecated\] | Starting with CUDA 11.1 this flag will be ignored. | | NVJPEG\_FLAGS\_BITSTREAM\_STRICT | nvJPEG library will try to decode a bitstream even if it doesn’t strictly follow the JPEG specification. Using this flag will return an error in such cases. | | NVJPEG\_FLAGS\_REDUCED\_MEMORY\_DECODE | When using `NVJPEG_BACKEND_HYBRID` or `NVJPEG_BACKEND_GPU_HYBRID` backends, enabling this flag will reduce the memory usage of the decoding whenever possible. | | NVJPEG\_FLAGS\_REDUCED\_MEMORY\_DECODE\_ZERO\_COPY | Using this flag enables zero-copy memory when feasible on supported platforms. | | NVJPEG\_FLAGS\_UPSAMPLING\_WITH\_INTERPOLATION | Using this flag enables the decoder to use interpolation when performing chroma upsampling during the YCbCr to RGB conversion stage. | ### 2.2.18. nvJPEG Exif Orientation[](#nvjpeg-exif-orientation "Permalink to this headline") typedef enum { NVJPEG\_ORIENTATION\_UNKNOWN \= 0, NVJPEG\_ORIENTATION\_NORMAL \= 1, NVJPEG\_ORIENTATION\_FLIP\_HORIZONTAL \= 2, NVJPEG\_ORIENTATION\_ROTATE\_180 \= 3, NVJPEG\_ORIENTATION\_FLIP\_VERTICAL \= 4, NVJPEG\_ORIENTATION\_TRANSPOSE \= 5, NVJPEG\_ORIENTATION\_ROTATE\_90 \= 6, NVJPEG\_ORIENTATION\_TRANSVERSE \= 7, NVJPEG\_ORIENTATION\_ROTATE\_270 \= 8 } nvjpegExifOrientation\_t; The `nvjpegExifOrientation_t` enum represents the exif orientation in a jfif(jpeg) file. Exif orientation information is typically used to denote the digital camera sensor orientation at the time of image capture. | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_ORIENTATION\_UNKNOWN | Exif orientation information is not available in the bitstream. | | NVJPEG\_ORIENTATION\_NORMAL | Decode output remains unchanged. | | NVJPEG\_ORIENTATION\_FLIP\_HORIZONTAL | Decoded output should be mirrored/flipped horizontally. | | NVJPEG\_ORIENTATION\_ROTATE\_180 | Decoded output should be rotated 180 degrees. | | NVJPEG\_ORIENTATION\_FLIP\_VERTICAL | Decoded output should be mirrored/flipped vertically. | | NVJPEG\_ORIENTATION\_TRANSPOSE | Decoded output should be flipped/mirrored horizontally followed by a 90 degrees counter-clockwise rotation. | | NVJPEG\_ORIENTATION\_ROTATE\_90 | Decoded output should be rotated 90 degrees counter-clockwise. | | NVJPEG\_ORIENTATION\_TRANSVERSE | Decoded output should be flipped/mirrored horizontally followed by a 270 degrees counter-clockwise rotation. | | NVJPEG\_ORIENTATION\_ROTATE\_270 | Decoded output should be rotated 270 degrees counter-clockwise. | 2.3. nvJPEG API Reference[](#nvjpeg-api-reference "Permalink to this headline") --------------------------------------------------------------------------------- This section describes the nvJPEG decoder API. ### 2.3.1. nvJPEG Helper API Reference[](#nvjpeg-helper-api-reference "Permalink to this headline") #### 2.3.1.1. nvjpegGetProperty()[](#nvjpeggetproperty "Permalink to this headline") Gets the numeric value for the major or minor version, or the patch level, of the nvJPEG library. **Signature:** nvjpegStatus\_t nvjpegGetProperty( libraryPropertyType type, int \*value); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `libraryPropertyType type` | Input | Host | One of the supported `libraryPropertyType` values, that is, MAJOR\_VERSION, MINOR\_VERSION or PATCH\_LEVEL. | | `int *value` | Output | Host | The numeric value corresponding to the specific `libraryPropertyType` requested. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.2. nvjpegGetCudartProperty()[](#nvjpeggetcudartproperty "Permalink to this headline") Gets the numeric value for the major version, minor version, or the patch level of the CUDA toolkit that was used to build nvJPEG library. For the same information on the nvJPEG library itself, see [nvjpegGetProperty()](#nvjpeg-get-property) . **Signature:** nvjpegStatus\_t nvjpegGetCudartProperty( libraryPropertyType type, int \*value); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `libraryPropertyType type` | Input | Host | One of the supported `libraryPropertyType` values, that is, MAJOR\_VERSION, MINOR\_VERSION or PATCH\_LEVEL. | | `int *value` | Output | Host | The numeric value corresponding to the specific `libraryPropertyType` requested. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.3. nvjpegCreate() \[DEPRECATED\][](#nvjpegcreate-deprecated "Permalink to this headline") Allocates and initializes the library handle. Note This function is deprecated. Use either `nvjpegCreateSimple()` or `nvjpegCreateEx()` functions to create the library handle. **Signature:** nvjpegStatus\_t nvjpegCreate( nvjpegBackend\_t backend, nvjpegDevAllocator\_t \*allocator, nvjpegHandle\_t \*handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBackend_t backend` | Input | Host | Backend parameter for [​nvjpegDecodeBatched()](#nvjpeg-decode-batched)
API. If this is set to DEFAULT then it automatically chooses one of the underlying algorithms. | | `nvjpegDevAllocator_t *allocator` | Input | Host | Device memory allocator. See [nvJPEG Device Memory Allocator Interface](#nvjpeg-memory-allocator-interface)
structure description. If NULL is provided, then the default CUDA runtime `cudaMalloc()`and `cudaFree()` functions will be used. | | `nvjpegHandle_t *handle` | Input/Output | Host | The library handle. | The `nvjpegBackend_t` parameter is an `enum` type, with the below enumerated list values: typedef enum { NVJPEG\_BACKEND\_DEFAULT \= 0, NVJPEG\_BACKEND\_HYBRID \= 1, } nvjpegBackend\_t; **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.4. nvjpegCreateSimple()[](#nvjpegcreatesimple "Permalink to this headline") Allocates and initializes the library handle, with default codec implementations selected by library and default memory allocators. **Signature:** nvjpegStatus\_t nvjpegCreateSimple(nvjpegHandle\_t \*handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t *handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.5. nvjpegCreateEx()[](#nvjpegcreateex "Permalink to this headline") Allocates and initializes the library handle using the provided arguments. **Signature:** nvjpegStatus\_t nvjpegCreateEx(nvjpegBackend\_t backend, nvjpegDevAllocator\_t \*dev\_allocator, nvjpegPinnedAllocator\_t \*pinned\_allocator, unsigned int flags, nvjpegHandle\_t \*handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBackend_t backend` | Input | Host | Backend parameter for [​nvjpegDecodeBatched()](#nvjpeg-decode-batched)
API. If this is set to DEFAULT then it automatically chooses one of the underlying algorithms. | | `nvjpegDevAllocator_t *dev_allocator` | Input | Host | Device memory allocator. See `nvjpegDevAllocator_t` structure description. If NULL is provided, then the default CUDA runtime functions `cudaMalloc()` and `cudaFree()` will be used. | | `nvjpegPinnedAllocator_t *pinned_allocator` | Input | Host | Pinned host memory allocator. See `nvjpegPinnedAllocator_t`structure description. If NULL is provided, then the default CUDA runtime functions `cudaHostAlloc()` and `cudaFreeHost()` will be used. | | `unsigned int flags` | Input | Host | Refer to [nvJPEG Flags](#nvjpeg-flags)
for details. | | `nvjpegHandle_t *handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.6. nvjpegCreateExV2()[](#nvjpegcreateexv2 "Permalink to this headline") Allocates and initializes the library handle using the provided arguments. **Signature:** nvjpegStatus\_t nvjpegCreateExV2(nvjpegBackend\_t backend, nvjpegDevAllocatorV2\_t \*dev\_allocator, nvjpegPinnedAllocatorV2\_t \*pinned\_allocator, unsigned int flags, nvjpegHandle\_t \*handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBackend_t backend` | Input | Host | Backend parameter for [​nvjpegDecodeBatched()](#nvjpeg-decode-batched)
API. If this is set to DEFAULT then it automatically chooses one of the underlying algorithms. | | `nvjpegDevAllocatorV2_t *dev_allocator` | Input | Host | Extended device memory allocator. See `nvjpegDevAllocatorV2_t_t` structure description. Cannot be NULL. | | `nvjpegPinnedAllocatorV2_t *pinned_allocator` | Input | Host | Extended pinned memory allocator. See `nvjpegPinnedAllocatorV2_t`structure description. Cannot be NULL. | | `unsigned int flags` | Input | Host | Refer to [nvJPEG Flags](#nvjpeg-flags)
for details. | | `nvjpegHandle_t *handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.7. nvjpegDestroy()[](#nvjpegdestroy "Permalink to this headline") Releases the library handle. **Signature:** nvjpegStatus\_t nvjpegDestroy(nvjpegHandle\_t handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input/Output | Host | The library handle to release. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.8. nvjpegSetDeviceMemoryPadding()[](#nvjpegsetdevicememorypadding "Permalink to this headline") Use the provided padding for all device memory allocations with specified library handle. A large number will help to amortize the need for device memory reallocations when needed. **Signature:** nvjpegStatus\_t nvjpegSetDeviceMemoryPadding( size\_t padding, nvjpegHandle\_t handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `size_t padding` | Input | Host | Device memory padding to use for all further device memory allocations. | | `nvjpegHandle_t handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.9. nvjpegGetDeviceMemoryPadding()[](#nvjpeggetdevicememorypadding "Permalink to this headline") Retrieve the device memory padding that is currently used for the specified library handle. **Signature:** nvjpegStatus\_t nvjpegGetDeviceMemoryPadding( size\_t \*padding, nvjpegHandle\_t handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `size_t *padding` | Output | Host | Device memory padding that is currently used for device memory allocations. | | `nvjpegHandle_t handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.10. nvjpegSetPinnedMemoryPadding()[](#nvjpegsetpinnedmemorypadding "Permalink to this headline") Use the provided padding for all pinned host memory allocations with specified library handle. A large number will help to amortize the need for pinned host memory reallocations when needed. **Signature:** nvjpegStatus\_t nvjpegSetPinnedMemoryPadding( size\_t padding, nvjpegHandle\_t handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `size_t padding` | Input | Host | Pinned host memory padding to use for all further pinned host memory allocations. | | `nvjpegHandle_t handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.11. nvjpegGetPinnedMemoryPadding()[](#nvjpeggetpinnedmemorypadding "Permalink to this headline") Retrieve the pinned host memory padding that is currently used for specified library handle. **Signature:** nvjpegStatus\_t nvjpegGetPinnedMemoryPadding( size\_t \*padding, nvjpegHandle\_t handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `size_t *padding` | Output | Host | Pinned host memory padding that is currently used for pinned host memory allocations. | | `nvjpegHandle_t handle` | Input/Output | Host | The library handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.12. nvjpegGetHardwareDecoderInfo()[](#nvjpeggethardwaredecoderinfo "Permalink to this headline") Retrieve hardware decoder details such as number of engines and number of cores available in each engine. **Signature:** nvjpegStatus\_t nvjpegGetHardwareDecoderInfo(nvjpegHandle\_t handle, unsigned int\* num\_engines, unsigned int\* num\_cores\_per\_engine); **Parameters:** | | | | | | --- | --- | --- | --- | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `unsigned int* num_engines` | Input/Output | Host | Retrieves number of engines available for decode. Return value of 0 indicates that hardware decoder is not available. | | `unsigned int* num_cores_per_engine` | Input/Output | Host | Retrieves number of cores per engine. Return value of 0 indicates that hardware decoder is not available. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.13. nvjpegJpegStateCreate()[](#nvjpegjpegstatecreate "Permalink to this headline") Allocates and initializes the internal structure required for the JPEG processing. **Signature:** nvjpegStatus\_t nvjpegJpegStateCreate( nvjpegHandle\_t handle, nvjpegJpegState\_t \*jpeg\_handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t *jpeg_handle` | Input/Output | Host | The image state handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.14. nvjpegJpegStateDestroy()[](#nvjpegjpegstatedestroy "Permalink to this headline") Releases the image internal structure. **Signature:** nvjpegStatus\_t nvjpegJpegStateDestroy(nvjpegJpegState handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegState handle` | Input/Output | Host | The image state handle. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.15. nvjpegDecoderCreate()[](#nvjpegdecodercreate "Permalink to this headline") Creates a decoder handle. **Signature:** nvjpegStatus\_t nvjpegDecoderCreate( nvjpegHandle\_t nvjpeg\_handle, nvjpegBackend\_t implementation, nvjpegJpegDecoder\_t\* decoder\_handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t nvjpeg_handle` | Input | Host | Library handle. | | `nvjpegBackend_t backend` | Input | Host | Backend parameter for the decoder\_handle.The back end applies to all the functions under the [decoupled API](#nvjpeg-decoupled-decode-api)
, when called with this handle. | | `nvjpegJpegDecoder_t decoder_handle` | Input/Output | Host | Decoder state handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.16. nvjpegDecoderDestroy()[](#nvjpegdecoderdestroy "Permalink to this headline") Destroys the decoder handle. **Signature:** nvjpegStatus\_t nvjpegDecoderDestroy( nvjpegJpegDecoder\_t decoder\_handle); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegDecoder_t decoder_handle` | Input/Output | Host | Decoder handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.17. nvjpegDecoderJpegSupported()[](#nvjpegdecoderjpegsupported "Permalink to this headline") Determines whether the `decoder_handle` is able to handle the bit-stream stored in `jpeg_stream`. **Signature:** nvjpegStatus\_t nvjpegDecoderJpegSupported( nvjpegJpegDecoder\_t decoder\_handle, nvjpegJpegStream\_t jpeg\_stream, nvjpegDecodeParams\_t decode\_params, int\* is\_supported); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegDecoder_t decoder_handle` | Input | Host | Decoder state handle | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bit stream meta-data | | `nvjpegDecodeParams_t decode_params` | Input | Host | Decoder output configuration | | `int* is_supported` | Output | Host | Return value of 0 indicates bitstream can be decoded by the `decoder_handle`, non zero value indicates that the bitstream is not supported | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.18. nvjpegDecoderStateCreate()[](#nvjpegdecoderstatecreate "Permalink to this headline") Creates the `decoder_state` internal structure. The `decoder_state` is associated with the [nvJPEG Backend](#nvjpeg-backend) implementation that was used to create the `decoder_handle`. **Signature:** nvjpegStatus\_t nvjpegDecoderStateCreate( nvjpegHandle\_t nvjpeg\_handle, nvjpegJpegDecoder\_t decoder\_handle, nvjpegJpegState\_t\* decoder\_state); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t nvjpeg_handle` | Input | Host | Library handle. | | `nvjpegJpegDecoder_t decoder_handle` | Input | Host | Decoder handle. | | `nvjpegJpegState_t* decoder_state` | Input/Output | Host | nvJPEG Image State Handle. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.19. nvjpegJpegStreamCreate()[](#nvjpegjpegstreamcreate "Permalink to this headline") Creates `jpeg_stream` that is used to parse the JPEG bitstream and store bitstream parameters. **Signature:** nvjpegStatus\_t nvjpegJpegStreamCreate( nvjpegHandle\_t handle, nvjpegJpegStream\_t \*jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | Library handle | | `nvjpegJpegStream_t *jpeg_stream` | Input | Host | Bitstream handle | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.20. nvjpegJpegStreamDestroy()[](#nvjpegjpegstreamdestroy "Permalink to this headline") Destroys the `jpeg_stream` structure. **Signature:** nvjpegStatus\_t nvjpegJpegStreamDestroy( nvjpegJpegStream\_t \*jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t *jpeg_stream` | Input | Host | Bitstream handle | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.21. nvjpegBufferPinnedCreate()[](#nvjpegbufferpinnedcreate "Permalink to this headline") Creates a pinned buffer handle. **Signature:** nvjpegStatus\_t nvjpegBufferPinnedCreate( nvjpegHandle\_t handle, nvjpegPinnedAllocator\_t\* pinned\_allocator, nvjpegBufferPinned\_t\* buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | Library handle. | | `nvjpegPinnedAllocator_t* pinned_allocator` | Input | Host | Pinned host memory allocator. See `nvjpegPinnedAllocator_t` structure description. | | `nvjpegBufferPinned_t* buffer` | Input/Output | Host | nvJPEG pinned buffer object. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.22. nvjpegBufferPinnedCreateV2()[](#nvjpegbufferpinnedcreatev2 "Permalink to this headline") Creates a pinned buffer handle using extended allocators. **Signature:** nvjpegStatus\_t nvjpegBufferPinnedCreateV2( nvjpegHandle\_t handle, nvjpegPinnedAllocatorV2\_t\* pinned\_allocator, nvjpegBufferPinned\_t\* buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | nvjpegHandle\_t handle | Input | Host | Library handle. | | nvjpegPinnedAllocatorV2\_t\* pinned\_allocator | Input | Host | Extended pinned host memory allocator. See [nvJPEG Extended Host Pinned Memory Allocator Interface](#nvjpeg-host-pinned-memory-allocator-interface-v2)
structure description. | | nvjpegBufferPinned\_t\* buffer | Input/Output | Host | nvJPEG pinned buffer object. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.23. nvjpegBufferPinnedDestroy()[](#nvjpegbufferpinneddestroy "Permalink to this headline") Destroys a pinned buffer handle. **Signature:** nvjpegStatus\_t nvjpegBufferPinnedDestroy( nvjpegBufferPinned\_t buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferPinned_t buffer` | Input | Host | nvJPEG pinned buffer object. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.24. nvjpegStateAttachPinnedBuffer()[](#nvjpegstateattachpinnedbuffer "Permalink to this headline") Link the nvJPEG pinned buffer handle to `decoder_state`. The `pinned_buffer` is used by the decoder to store the intermediate information that is used across the decoding stages. Pinned buffer can be attached to different decoder states, which helps to switch between implementations without allocating extra memory. **Signature:** nvjpegStatus\_t nvjpegStateAttachPinnedBuffer( nvjpegJpegState\_t decoder\_state, nvjpegBufferPinned\_t pinned\_buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegState_t decoder_state` | Input | Host | nvJPEG decoder state. | | `nvjpegBufferPinned_t pinned_buffer` | Input | Host | nvJPEG pinned buffer container. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.25. nvjpegBufferPinnedRetrieve()[](#nvjpegbufferpinnedretrieve "Permalink to this headline") Retrieves the pinned memory pointer and size from the nvJPEG pinned buffer handle. Allows the application to re-use the memory once the decode is complete. **Signature:** nvjpegStatus\_t nvjpegBufferPinnedRetrieve( nvjpegBufferPinned\_t buffer, size\_t\* size, void\*\* ptr); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferPinned_t buffer` | Input | Host | nvJPEG pinned buffer container. | | `size_t* size` | Input/Output | Host | Size in bytes of the pinned buffer. | | `void** ptr` | Input/Output | Host | Pointer to the pinned buffer. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.26. nvjpegBufferPinnedResize()[](#nvjpegbufferpinnedresize "Permalink to this headline") Resize the pinned buffer to the specified size in bytes. This API can be used to pre-allocate the pinned buffer to a large value and avoid allocator calls during decode. **Signature:** nvjpegStatus\_t nvjpegBufferPinnedResize(nvjpegBufferPinned\_t buffer, size\_t size, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferPinned_t buffer` | Input | Host | nvJPEG pinned buffer container. | | `size_t* size` | Input | Host | Size in bytes of the pinned buffer. | | `cudaStream_t stream` | Input | Host | CUDA stream to use when `nvjpegBufferPinned_t buffer` is initialized using stream ordered allocators. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.27. nvjpegBufferDeviceCreate()[](#nvjpegbufferdevicecreate "Permalink to this headline") Creates the device buffer handle. **Signature:** nvjpegStatus\_t nvjpegBufferDeviceCreate( nvjpegHandle\_t handle, nvjpegDevAllocator\_t\* device\_allocator, nvjpegBufferDevice\_t\* buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | Library handle. | | `nvjpegDevAllocator_t* device_allocator` | Input | Host | Device memory allocator. See the [nvJPEG Device Memory Allocator Interface](#nvjpeg-memory-allocator-interface)
structure description. | | `nvjpegBufferDevice_t* buffer` | Input/Output | Host | nvJPEG device buffer container. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.28. nvjpegBufferDeviceCreateV2()[](#nvjpegbufferdevicecreatev2 "Permalink to this headline") Creates the device buffer handle using extended allocators. **Signature:** nvjpegStatus\_t nvjpegBufferDeviceCreateV2( nvjpegHandle\_t handle, nvjpegDevAllocatorV2\_t\* device\_allocator, nvjpegBufferDevice\_t\* buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | nvjpegHandle\_t handle | Input | Host | Library handle. | | nvjpegDevAllocatorV2\_t\* device\_allocator | Input | Host | Extended device memory allocator. See `nvjpegDevAllocatorV2_t_t` structure description. | | nvjpegBufferDevice\_t\* buffer | Input/Output | Host | nvJPEG device buffer container. | **Returns:** `nvjpegStatus_t` - An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.29. nvjpegBufferDeviceDestroy()[](#nvjpegbufferdevicedestroy "Permalink to this headline") Destroys the device buffer handle. **Signature:** nvjpegStatus\_t nvjpegBufferDeviceDestroy( nvjpegBufferDevice\_t buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferDevice_t buffer` | Input | Host/Device | nvJPEG device buffer container. Device pointers are stored within the host structures. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.30. nvjpegStateAttachDeviceBuffer()[](#nvjpegstateattachdevicebuffer "Permalink to this headline") Link the nvJPEG device buffer handle to the `decoder_state`. The `device_buffer` is used by the decoder to store the intermediate information that is used across the decoding stages. Device buffer can be attached to different decoder states, which helps to switch between implementations without allocating extra memory. **Signature:** nvjpegStatus\_t nvjpegStateAttachDeviceBuffer( nvjpegJpegState\_t decoder\_state, nvjpegBufferDevice\_t device\_buffer); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegState_t decoder_state` | Input | Host | nvJPEG decoder state. | | `nvjpegBufferDevice_t device buffer` | Input | Host/Device | nvJPEG device buffer container. Device pointers are stored within the host structures. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.31. nvjpegBufferDeviceRetrieve()[](#nvjpegbufferdeviceretrieve "Permalink to this headline") Retrieve the device memory pointer and size from the nvJPEG device buffer handle. Allows the application to re-use the memory after the decode is complete. **Signature:** nvjpegStatus\_t nvjpegBufferDeviceRetrieve( nvjpegBufferDevice\_t buffer, size\_t\* size, void\*\* ptr); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferDevice_t buffer` | Input | Host | nvJPEG device buffer container. | | `size_t* size` | Input/Output | Host | Device buffer size in bytes. | | `void** ptr` | Input/Output | Host | Pointer to the device buffer. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.32. nvjpegBufferDeviceResize()[](#nvjpegbufferdeviceresize "Permalink to this headline") Resize the device buffer to the specified size in bytes. This API can be used to pre-allocate the device buffer to a large value and avoid allocator calls during decode. **Signature:** nvjpegStatus\_t nvjpegBufferDeviceResize(nvjpegBufferDevice\_t buffer, size\_t size, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegBufferDevice_t buffer` | Input | Host | nvJPEG device buffer container. | | `size_t* size` | Input | Host | Size in bytes of the device buffer. | | `cudaStream_t stream` | Input | Host | CUDA stream to use when `nvjpegBufferDevice_t buffer` is initialized using stream ordered allocators. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.33. nvjpegDecodeParamsCreate()[](#nvjpegdecodeparamscreate "Permalink to this headline") Creates a handle for the parameters. The parameters that can be programmed include: output format, ROI decode, CMYK to RGB conversion. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsCreate( nvjpegHandle\_t handle, nvjpegDecodeParams\_t \*decode\_params); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | Library handle. | | `nvjpegDecodeParams_t *decode_params` | Input/Output | Host | Decode output parameters. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.1.34. nvjpegDecodeParamsDestroy()[](#nvjpegdecodeparamsdestroy "Permalink to this headline") Destroys the `decode_params` handle. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsDestroy( nvjpegDecodeParams\_t \*decode\_params); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t *decode_params` | Input/Output | Host | Decode output parameters. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ### 2.3.2. Retrieve Encoded Image Information API[](#retrieve-encoded-image-information-api "Permalink to this headline") The helper functions for retrieving the encoded image information. #### 2.3.2.1. nvjpegGetImageInfo()[](#nvjpeggetimageinfo "Permalink to this headline") Decodes the JPEG header and retrieves the basic information about the image. **Signature:** nvjpegStatus\_t nvjpegGetImageInfo( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, int \*nComponents, nvjpegChromaSubsampling\_t \*subsampling, int \*widths, int \*heights); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `const unsigned char *data` | Input | Host | Pointer to the encoded data. | | `size_t length` | Input | Host | Size of the encoded data in bytes. | | `int *nComponents` | Output | Host | Number of channels in the jpeg encoded data. | | `nvjpegChromaSubsampling_t *subsampling` | Output | Host | Chroma subsampling for the 1- or 3- channel encoding. | | `int *widths` | Output | Host | Pointer to the first element of array of size NVJPEG\_MAX\_COMPONENT, where the width of each channel (up to NVJPEG\_MAX\_COMPONENT) will be saved. If the channel is not encoded, then the corresponding value would be zero. | | `int *heights` | Output | Host | Pointer to the first element of array of size NVJPEG\_MAX\_COMPONENT, where the height of each channel (up to NVJPEG\_MAX\_COMPONENT) will be saved. If the channel is not encoded, then the corresponding value would be zero. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.2.2. nvJPEG Stream API[](#nvjpeg-stream-api "Permalink to this headline") These functions store the parsed bit-stream data on the host. ##### 2.3.2.2.1. nvjpegJpegStreamParse()[](#nvjpegjpegstreamparse "Permalink to this headline") Parses the bitstream and stores the metadata in the`jpeg_stream` struct. **Signature:** nvjpegStatus\_t nvjpegJpegStreamParse( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, int save\_metadata, int save\_stream, nvjpegJpegStream\_t jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `const unsigned char *data` | Input | Host | Pointer to the bit-stream. | | `size_t length` | Input | Host | Bit-stream size. | | `int save_metadata` | Input | Host | (Not enabled. Marked for future use). If not 0, then the JPEG stream metadata (headers, app markers, etc.) will be saved in the internal `JpegStream` structure for future usage. If 0, then the meta data (headers, app markerms etc.) will be discarded. | | `int save_stream` | Input | Host | If not 0, then the whole jpeg stream will be copied to the internal JpegStream structure, and the pointer to the JPEG file data will not be needed after this call. If 0, then `JpegStream` will just save the pointers (to JPEG file data), and these pointers will be used later during the image decoding. | | `nvjpegJpegStream_t jpeg_stream` | Input/Output | Host/Device | The nvJPEG bitstream handle that stores the parsed bitstream information. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.2. nvjpegJpegStreamParseHeader()[](#nvjpegjpegstreamparseheader "Permalink to this headline") Parses only the header of the bit-stream and stores the header information in the`jpeg_stream` struct. **Signature:** nvjpegStatus\_t nvjpegJpegStreamParseHeader( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, nvjpegJpegStream\_t jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `const unsigned char *data` | Input | Host | Pointer to the bit-stream. | | `size_t length` | Input | Host | Bit-stream size. | | `nvjpegJpegStream_t jpeg_stream` | Input/Output | Host/Device | The nvJPEG bitstream handle that stores the parsed bitstream information. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.3. nvjpegJpegStreamParseTables()[](#nvjpegjpegstreamparsetables "Permalink to this headline") To be used when decoding TIFF files with JPEG compression. Parses the JPEG tables bitstream and stores the jpeg tables in `jpeg_stream` **Signature:** nvjpegStatus\_t nvjpegJpegStreamParseHeader( nvjpegHandle\_t handle, const unsigned char \*data, size\_t length, nvjpegJpegStream\_t jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `const unsigned char *data` | Input | Host | Pointer to the JPEG tables bitstream. Can be set to NULL to reset the JPEG tables. | | `size_t length` | Input | Host | JPEG tables bitstream size. | | `nvjpegJpegStream_t jpeg_stream` | Input/Output | Host | The nvJPEG bitstream handle that stores the parsed bitstream information. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.4. nvjpegJpegStreamGetFrameDimensions()[](#nvjpegjpegstreamgetframedimensions "Permalink to this headline") Extracts the JPEG frame dimensions from the bitstream. **Signature:** nvjpegStatus\_t nvjpegJpegStreamGetFrameDimensions( nvjpegJpegStream\_t jpeg\_stream, unsigned int\* width, unsigned int\* height); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `unsigned int* width` | Output | Host | Frame height. | | `unsigned int* height` | Output | Host | Frame width. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.5. nvjpegJpegStreamGetComponentsNum()[](#nvjpegjpegstreamgetcomponentsnum "Permalink to this headline") Extracts the JPEG frame dimensions from the bitstream. **Signature:** nvjpegStatus\_t nvjpegJpegStreamGetComponentsNum( nvjpegJpegStream\_t jpeg\_stream, unsigned int\* components\_num); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `unsigned int* components_num` | Output | Host | Number of encoded channels in the input. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.6. nvjpegJpegStreamGetComponentDimensions()[](#nvjpegjpegstreamgetcomponentdimensions "Permalink to this headline") Extracts the component dimensions from the bitstream. **Signature:** nvjpegStatus\_t nvjpegJpegStreamGetComponentDimensions( nvjpegJpegStream\_t jpeg\_stream, unsigned int component, unsigned int\* width, unsigned int\* height) **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `unsigned int component` | Input | Host | Component index. | | `unsigned int* width` | Output | Host | Component height. | | `unsigned int* height` | Output | Host | Component width. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.7. nvjpegJpegStreamGetChromaSubsampling()[](#nvjpegjpegstreamgetchromasubsampling "Permalink to this headline") Gets the chroma subsampling from the `jpeg_stream`. For grayscale (single channel) images it returns NVJPEG\_CSS\_GRAY. For 3-channel images it tries to assign one of the known chroma sub-sampling values based on the sampling information present in the bitstream, else it returns NVJPEG\_CSS\_UNKNOWN. If the number of channels is 2 or 4, then it returns NVJPEG\_CSS\_UNKNOWN. **Signature:** nvjpegStatus\_t nvjpegJpegStreamGetChromaSubsampling( nvjpegJpegStream\_t jpeg\_stream, nvjpegChromaSubsampling\_t\* chroma\_subsampling); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `nvjpegChromaSubsampling_t* chroma_subsampling` | Output | Host | Chroma subsampling for the 1- or 3- channel encoding. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.8. nvjpegJpegStreamGetJpegEncoding()[](#nvjpegjpegstreamgetjpegencoding "Permalink to this headline") This function obtains the JPEG encoding type from the `jpeg_stream`. For baseline images it returns NVJPEG\_ENCODING\_BASELINE\_DCT. For progressive images it returns NVJPEG\_ENCODING\_PROGRESSIVE\_DCT\_HUFFMAN. **Signature:** nvjpegStatus\_t nvjpegJpegStreamGetJpegEncoding( nvjpegJpegStream\_t jpeg\_stream, nvjpegJpegEncoding\_t\* jpeg\_encoding); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `jpeg_stream` | In | Host | Input bitstream handle. | | `jpeg_encoding` | Out | Host | Encoding type obtained—baseline or progressive. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.9. nvjpegJpegStreamGetExifOrientation()[](#nvjpegjpegstreamgetexiforientation "Permalink to this headline") Extracts the exif orientation from the bitstream. Returns `NVJPEG_ORIENTATION_UNKNOWN` if the exif marker/orientation information is not present. **Signature:** nvjpegStatus\_t NVJPEGAPI nvjpegJpegStreamGetExifOrientation( nvjpegJpegStream\_t jpeg\_stream, nvjpegExifOrientation\_t \*orientation\_flag); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `nvjpegExifOrientation_t *orientation_flag` | Output | Host | Exif orientation in JPEG stream. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ##### 2.3.2.2.10. nvjpegJpegStreamGetSamplePrecision()[](#nvjpegjpegstreamgetsampleprecision "Permalink to this headline") Extracts the sample precision(bit depth) from the bitstream. **Signature:** nvjpegStatus\_t NVJPEGAPI nvjpegJpegStreamGetSamplePrecision( nvjpegJpegStream\_t jpeg\_stream, unsigned int \*precision); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bitstream handle. | | `unsigned int *precision` | Output | Host | Sample precision value. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ### 2.3.3. Decode API—Single Phase[](#decode-apisingle-phase "Permalink to this headline") Functions for decoding single image or batched images in a single phase. #### 2.3.3.1. ​nvjpegDecode()[](#nvjpegdecode "Permalink to this headline") Decodes a single image, and writes the decoded image in the desired format to the output buffers. This function is asynchronous with respect to the host. All GPU tasks for this function will be submitted to the provided stream. From CUDA 11 onwards, `nvjpegDecode()` picks the best available back-end for a given image, user no longer has control on this. If there is a need to select the back-end, then consider using [nvjpegDecodeJpeg()](#nvjpeg-decode-jpeg) . This is a new API added in CUDA 11 which allows user to control the back-end. **Signature:** nvjpegStatus\_t nvjpegDecode( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*data, size\_t length, nvjpegOutputFormat\_t output\_format, nvjpegImage\_t \*destination, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t jpeg_handle` | Input | Host | The image state handle. | | `const unsigned char *data` | Input | Host | Pointer to the encoded data. | | `size_t length` | Input | Host | Size of the encoded data in bytes. | | `nvjpegOutputFormat_t output_format` | Input | Host | Format in which the decoded output will be saved. | | `nvjpegImage_t *destination` | Input/Output | Host/Device | Pointer to the structure that describes the output destination. This structure should be on the host (CPU), but the pointers in this structure should be pointing to the device (i.e., GPU) memory. See `nvjpegImage_t.` | | `cudaStream_t stream` | Input | Host | The CUDA stream where all of the GPU work will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.2. ​nvjpegDecodeBatchedInitialize()[](#nvjpegdecodebatchedinitialize "Permalink to this headline") This function initializes the batched decoder state. The initialization parameters include the batch size, the maximum number of CPU threads, and the specific output format in which the decoded image will be saved. This function should be called once, prior to decoding the batches of images. Any currently running batched decoding should be finished before calling this function. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedInitialize( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, int batch\_size, int max\_cpu\_threads, nvjpegOutputFormat\_t output\_format); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t jpeg_handle` | Input | Host | The image state handle. | | `int batch_size` | Input | Host | Batch size. | | `int max_cpu_threads` | Input | Host | This parameter is no longer used by the library. | | `nvjpegOutputFormat_t output_format` | Input | Host | Format in which the decoded output will be saved. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.3. ​nvjpegDecodeBatched()[](#nvjpegdecodebatched "Permalink to this headline") Decodes the batch of images, and writes them to the buffers described in the `destination` parameter in a format provided to `nvjpegDecodeBatchedInitialize()` function. This function is asynchronous with respect to the host. All GPU tasks for this function will be submitted to the provided stream. **Signature:** nvjpegStatus\_t nvjpegDecodeBatched( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*const \*data, const size\_t \*lengths, nvjpegImage\_t \*destinations, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t jpeg_handle` | Input | Host | The image state handle. | | `const unsigned char *const *data` | Input | Host | Pointer to the first element of array of the input data. The size of the array is assumed to be batch\_size provided to `nvjpegDecodeBatchedInitialize()` batch initialization function. | | `const size_t *lengths` | Input | Host | Pointer to the first element of array of input sizes. Size of array is assumed to be batch\_size provided to `nvjpegDecodeBatchedInitialize()`, the batch initialization function. | | `nvjpegImage_t *destinations` | Input/Output | Host/Device | Pointer to the first element of array of output descriptors. The size of array is assumed to be batch\_size provided to `nvjpegDecodeBatchedInitialize(),` the batch initialization function. See also `nvjpegImage_t`. | | `cudaStream_t stream` | Input | Host | The CUDA stream where all the GPU work will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.4. nvjpegDecodeBatchedEx()[](#nvjpegdecodebatchedex "Permalink to this headline") This API helps to Decodes the batch of images with ROI, and writes them to the buffers described in the `destination` parameter in a format provided to `nvjpegDecodeBatchedInitialize()` function. This function is asynchronous with respect to the host. All GPU tasks for this function will be submitted to the provided stream. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedEx( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*const \*data, const size\_t \*lengths, nvjpegImage\_t \*destinations, nvjpegDecodeParams\_t \*decode\_params, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | nvjpeg library handle. | | `nvjpegJpegState_t jpeg_handle` | Input | Host | The image state handle. | | `const unsigned char *const *data` | Input | Host | Pointer to the first element of array of the input data. The size of the array is assumed to be `batch_size` provided to `nvjpegDecodeBatchedInitialize()` batch initialization function. | | `const size_t *lengths` | Input | Host | Pointer to the first element of array of input sizes. | | `nvjpegImage_t *destinations` | Input/Output | Host/Device | Pointer to the first element of array of output descriptors. The size of array is assumed to be `batch_size` provided to `nvjpegDecodeBatchedInitialize()`, the batch initialization function. See also `nvjpegImage_t`. | | `nvjpegDecodeParams_t *decode_params` | Input | Host | Setting ROI Decode parameters | | `cudaStream_t stream` | Input | Host | The CUDA stream where all the GPU work will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.5. nvjpegDecodeBatchedSupported()[](#nvjpegdecodebatchedsupported "Permalink to this headline") This API helps determine whether an image can be decoded by [​nvjpegDecodeBatched()](#nvjpeg-decode-batched) . User can parse the bitstream header using [nvjpegJpegStreamParseHeader()](#nvjpeg-jpeg-stream-parse-header) and then call this API to determine whether the image can be decoded. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedSupported( nvjpegHandle\_t handle, nvjpegJpegStream\_t jpeg\_stream, int\* is\_supported); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | nvjpeg library handle. | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bit stream meta-data. | | `int* is_supported` | Output | Host | Return value of 0 indicates bitstream can be decoded by the `decoder_handle`, non zero value indicates that the bitstream is not supported. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.6. nvjpegDecodeBatchedSupportedEx()[](#nvjpegdecodebatchedsupportedex "Permalink to this headline") This API helps determine whether an image can be decoded by [​nvjpegDecodeBatched()](#nvjpeg-decode-batched) . User can parse the bitstream header using [nvjpegJpegStreamParseHeader()](#nvjpeg-jpeg-stream-parse-header) and set the ROI in the decode params then call this API to determine whether the image can be decoded. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedSupportedEx( nvjpegHandle\_t handle, nvjpegJpegStream\_t jpeg\_stream, nvjpegDecodeParams\_t decode\_params, int\* is\_supported); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | nvjpeg library handle. | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Bit stream meta-data. | | `nvjpegDecodeParams_t decode_params` | Input | Host | Setting ROI Decode parameters. | | `int* is_supported` | Output | Host | Return value of 0 indicates bitstream can be decoded by the `decoder_handle`, a non zero value indicates that the bitstream is not supported. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.7. nvjpegDecodeBatchedPreAllocate()[](#nvjpegdecodebatchedpreallocate "Permalink to this headline") This is an experimental API that can be used with [​nvjpegDecodeBatched()](#nvjpeg-decode-batched) . When decoding images with varying sizes and chroma subsampling, performance is limited by the repeated cuda calls made by the library to free/allocate device memory. This API attempts to avoid this problem by allocating device memory prior to the actual decoding. Users have the option to call this API with values that are unlikely to be exceeded when [​nvjpegDecodeBatched()](#nvjpeg-decode-batched) is called. Note Note: This functionality is available only when the `nvjpegHandle_t`is instantiated using NVJPEG\_BACKEND\_HARDWARE. It is currently a No Op for other backends. This API only provides a hint for initial allocation. If the image dimensions at the time of decode exceed what was provided, then the library will resize the device buffers. If the images being decoded have different chroma subsamplings, then the `chroma_subsampling` field should be set to NVJPEG\_CSS\_444 to ensure that the device memory can be reused. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedPreAllocate( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, int batch\_size, int width, int height, nvjpegChromaSubsampling\_t chroma\_subsampling, nvjpegOutputFormat\_t output\_format); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t jpeg_handle` | Input | Host | The image state handle. | | `int batch_size` | Input | Host | Batch size. | | `int width` | Input | Host | Maximum width of image that will be decoded. | | `int height` | Input | Host | Maximum height of image that will be decoded. | | `nvjpegChromaSubsampling_t chroma_subsampling` | Input | Host | Chroma-subsampling of the images. | | `nvjpegOutputFormat_t output_format` | Input | Host | Format in which the decoded output will be saved. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.3.8. nvjpegDecodeBatchedParseJpegTables()[](#nvjpegdecodebatchedparsejpegtables "Permalink to this headline") To be used along with batched decode APIs when decoding JPEG bitstreams from a TIFF file. This function parses the JPEG tables bitstream to extract the JPEG tables. The external Huffman and quantization tables will be applied to all the JPEG bitstreams in the batch. **Signature:** nvjpegStatus\_t nvjpegDecodeBatchedParseJpegTables( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*data, const size\_t length); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegState_t jpeg_handle` | Input/Output | Host/Device | The image state handle. | | `const unsigned char *data` | Input | Host | Pointer to the JPEG tables bitstream. Can be set to NULL to reset the jpeg tables. | | `size_t length` | Input | Host | JPEG tables bitstream size. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ### 2.3.4. Decode API—Decoupled Decoding[](#decode-apidecoupled-decoding "Permalink to this headline") This set of decoding API works with the bitstream handles, decode parameter handles, pinned and device buffers handles as input, thus decoupling JPEG bitstream parse, buffer management and setting up decoder parameters from the decode process itself. Currently only multiphase decoding is available. Multiphase decoupled single image decoding consists of three phases: * Host * Mixed * Device Each of the above decodings is carried on according to its individual semantics. Phases on different images can be carried out with different decoding state handles simultaneously, while sharing of some helper objects is possible. See the details of semantics in the individual phases descriptions. Below are a couple of examples of using decoupled API. The following snippet explains how to use the API to prefetch the host stage of the processing: first do all of the host work on the host, and then submit the rest of decoding work to the device. #define BATCH\_SIZE 2 nvjpegHandle\_t nvjpeg\_handle; nvjpegJpegState\_t nvjpeg\_decoder\_state\[BATCH\_SIZE\]; nvjpegBufferPinned\_t nvjpeg\_pinned\_buffer\[BATCH\_SIZE\]; nvjpegBufferDevice\_t nvjpeg\_device\_buffer; nvjpegJpegStream\_t nvjpeg\_jpeg\_stream\[BATCH\_SIZE\]; nvjpegDecodeParams\_t nvjpeg\_decode\_params; nvjpegJpegDecoder\_t nvjpeg\_decoder; nvjpegBackend\_t impl \= NVJPEG\_BACKEND\_DEFAULT; unsigned char\* bitstream\[BATCH\_SIZE\] // pointers jpeg bitstreams size\_t length\[BATCH\_SIZE\]; // bitstream sizes nvjpegImage\_t output\_images\[BATCH\_SIZE\]; // all the images in the batch will be decoded as RGBI nvjpegDecodeParamsSetOutputFormat(nvjpeg\_decode\_params,NVJPEG\_OUTPUT\_RGBI ); // call host phase for two bitstreams for (int i \= 0; i < BATCH\_SIZE; i++) { nvjpegJpegStreamParse(nvjpeg\_handle, bitstream\[i\], length\[i\], 0, 0, nvjpeg\_jpeg\_stream\[i\]); nvjpegStateAttachPinnedBuffer(nvjpeg\_decoder\_state\[i\], nvjpeg\_pinned\_buffer\[i\]); nvjpegDecodeJpegHost(nvjpeg\_handle, nvjpeg\_decoder, nvjpeg\_decoder\_state\[i\], nvjpeg\_decode\_params, nvjpeg\_jpeg\_stream\[i\]) } for (int i \= 0; i < BATCH\_SIZE; i++) { // same device buffer being used for decoding bitstreams nvjpegStateAttachDeviceBuffer(nvjpeg\_decoder\_state\[i\], nvjpeg\_device\_buffer); // cuda stream set to NULL nvjpegDecodeJpegTransferToDevice(nvjpeg\_handle, nvjpeg\_decoder, nvjpeg\_decoder\_state\[i\], nvjpeg\_jpeg\_stream\[i\], NULL); // cuda stream set to NULL nvjpegDecodeJpegDevice(nvjpeg\_handle, nvjpeg\_decoder, nvjpeg\_decoder\_state\[i\], &output\_images\[i\], NULL); cudaDeviceSynchronize(); } The following snippet explains how pinned and device buffers can be shared across two instances of [nvJPEG Decoder Handle](#nvjpeg-decoder-handle) . #define BATCH\_SIZE 4 nvjpegHandle\_t nvjpeg\_handle; nvjpegJpegDecoder\_t nvjpeg\_decoder\_impl1; nvjpegJpegDecoder\_t nvjpeg\_decoder\_impl2; nvjpegJpegState\_t nvjpeg\_decoder\_state\_impl1; nvjpegJpegState\_t nvjpeg\_decoder\_state\_impl2; nvjpegBufferPinned\_t nvjpeg\_pinned\_buffer; nvjpegBufferDevice\_t nvjpeg\_device\_buffer; nvjpegJpegStream\_t nvjpeg\_jpeg\_stream; nvjpegDecodeParams\_t nvjpeg\_decode\_params; unsigned char\* bitstream\[BATCH\_SIZE\] // pointers jpeg bitstreams size\_t length\[BATCH\_SIZE\]; // bitstream sizes // populate bitstream and length correctly for this code to work nvjpegImage\_t output\_images\[BATCH\_SIZE\]; // allocate device memory for output images, for this snippet to work nvjpegStateAttachPinnedBuffer(nvjpeg\_decoder\_state\_impl1, nvjpeg\_pinned\_buffer); nvjpegStateAttachPinnedBuffer(nvjpeg\_decoder\_state\_impl2, nvjpeg\_pinned\_buffer); nvjpegStateAttachDeviceBuffer(nvjpeg\_decoder\_state\_impl1, nvjpeg\_device\_buffer); nvjpegStateAttachDeviceBuffer(nvjpeg\_decoder\_state\_impl2, nvjpeg\_device\_buffer); // all the images in the batch will be decoded as RGBI nvjpegDecodeParamsSetOutputFormat(nvjpeg\_decode\_params,NVJPEG\_OUTPUT\_RGBI ); for (int i \= 0; i < BATCH\_SIZE; i++) { nvjpegJpegStreamParse(nvjpeg\_handle,bitstream\[i\],length\[i\],0,0,nvjpeg\_jpeg\_stream); // decide which implementation to use, based on image size unsigned int frame\_width; unsigned int frame\_height; nvjpegJpegStreamGetFrameDimensions(nvjpeg\_jpeg\_stream,&frame\_width, &frame\_height)); nvjpegJpegDecoder\_t& decoder \= (frame\_height\*frame\_width \> 1024 \* 768 ) ? nvjpeg\_decoder\_impl2: nvjpeg\_decoder\_impl1; nvjpegJpegState\_t& decoder\_state \= (frame\_height \* frame\_width \> 1024 \* 768) ? nvjpeg\_decoder\_state\_impl2:nvjpeg\_decoder\_state\_impl1; nvjpegDecodeJpegHost(nvjpeg\_handle,decoder,decoder\_state,nvjpeg\_decode\_params,nvjpeg\_jpeg\_stream); // cuda stream set to NULL nvjpegDecodeJpegTransferToDevice(nvjpeg\_handle,decoder,decoder\_state,nvjpeg\_jpeg\_stream,NULL); // cuda stream set to NULL nvjpegDecodeJpegDevice(nvjpeg\_handle,nvjpeg\_decoder,decoder\_state,&output\_images, NULL); cudaDeviceSynchronize(); } #### 2.3.4.1. nvjpegDecodeJpegHost()[](#nvjpegdecodejpeghost "Permalink to this headline") This is the first stage of the decoupled decoding process. It is done entirely on the host, hence it is synchronous with respect to the host. If a pinned buffer is attached to the decoder state, then the pinned buffer object will be used to allocate the pinned memory required for the host decoding phase. There wouldn’t be allocation if the pinned buffer object already handles the required amount of pinned memory. If a pinned buffer object is not attached, then the state will use heap host memory to allocate the memory required for the host processing. In this phase, device is not participating. Hence the device selection, device initialization, and device memory initialization can be done later in the decoding process. This function works on a parsed stream. The parsed stream handle that is available after calling the [nvjpegJpegStreamParse()](#nvjpeg-jpeg-stream-parse) function should be provided to this function. **Signature:** nnvjpegStatus\_t nvjpegDecodeJpegHost( nvjpegHandle\_t handle, nvjpegJpegDecoder\_t decoder, nvjpegJpegState\_t decoder\_state, nvjpegDecodeParams\_t decode\_params, nvjpegJpegStream\_t jpeg\_stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegDecoder_t decoder` | Input | Host | The nvJPEG decoder handle. | | `nvjpegJpegState_t decoder_state` | Input | Host | The nvJPEG decoder state handle. | | `nvjpegDecodeParams_t decode_params` | Input | Host | Handle to decode the output properties. | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Handle to the parsed bitstream data. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.4.2. nvjpegDecodeJpegTransferToDevice()[](#nvjpegdecodejpegtransfertodevice "Permalink to this headline") This phase contains both host and device operations. Hence it is a mix of synchronous and asynchronous operations with respect to the host. All the device operations will be submitted to the provided stream. This phase should be called only after the host phase with the same decoder handle, decoder state handle and parsed jpeg stream handle. Device should be initialized and device buffer should be attached to `decoder_state` handle using [nvjpegStateAttachDeviceBuffer()](#nvjpeg-state-attach-device-buffer) prior to calling this API. This device buffer object will be resized to the required amount of memory if needed. For the host memory buffer, this phase will use whatever was used in the host phase: either the attached pinned buffer or the state’s host memory buffer. **Signature:** nvjpegStatus\_t nvjpegDecodeJpegTransferToDevice( nvjpegHandle\_t handle, nvjpegJpegDecoder\_t decoder, nvjpegJpegState\_t decoder\_state, nvjpegJpegStream\_t jpeg\_stream, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegDecoder_t decoder` | Input | Host | The nvJPEG decoder handle. | | `nvjpegJpegState_t decoder_state` | Input | Host | The nvJPEG decoder state handle. | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Handle to the parsed bitstream data. | | `cudaStream_t stream` | Input | Host | The CUDA stream to which all the GPU tasks will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.4.3. nvjpegDecodeJpegDevice()[](#nvjpegdecodejpegdevice "Permalink to this headline") This phase consists of decode operations that take place mainly on the device (no significant host side computation is done). Hence this phase is asynchronous with respect to the host. This phase should be called after [nvjpegDecodeJpegTransferToDevice()](#nvjpeg-decode-jpeg-transfer-to-device) for a given `decoder_state` handle and decoder handle. In this function call, the host memory buffers are not used, so if the pinned buffer was attached to the state, then it can be reused somewhere else. Note that at this point the Jpeg stream handle is not needed anymore, since parts that are needed for device decoding will be copied to the device memory in the previous phase. **Signature:** nvjpegStatus\_t nvjpegDecodeJpegDevice( nvjpegHandle\_t handle, nvjpegJpegDecoder\_t decoder, nvjpegJpegState\_t decoder\_state, nvjpegImage\_t \*destination, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegDecoder_t decoder` | Input | Host | The nvJPEG decoder handle. | | `nvjpegJpegState_t decoder_state` | Input | Host | The nvJPEG decoder state handle. | | `nvjpegImage_t *destination` | Input/Output | Host/Device | Pointer to a structure that describes the output destination. This structure should be on host, but the pointers in this structure should be pointing to the device memory. See [nvJPEG Image](#nvjpeg-image)
for details. | | `cudaStream_t stream` | Input | Host | The CUDA stream to which all the GPU tasks will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.4.4. nvjpegDecodeJpeg()[](#nvjpegdecodejpeg "Permalink to this headline") This is a single phase API with the flexibility to select nvJPEG back-end when creating an `nvjpegJpegDecoder_t` object. The user has the option to call this API instead of making three separate calls to [nvjpegDecodeJpegHost()](#nvjpeg-decode-jpeg-host) , [nvjpegDecodeJpegTransferToDevice()](#nvjpeg-decode-jpeg-transfer-to-device) , and [nvjpegDecodeJpegDevice()](#nvjpeg-decode-jpeg-device) . It is required to atttach the device buffer to the decoder state before calling this API. The pinned buffer is optional. If the pinned buffer is not attached, then heap memory will be used for host processing. This function works on a parsed stream. The parsed stream handle that is available after calling the [nvjpegJpegStreamParse()](#nvjpeg-jpeg-stream-parse) function should be provided to this function. **Signature:** nvjpegStatus\_t nvjpegDecodeJpeg( nvjpegHandle\_t handle, nvjpegJpegDecoder\_t decoder, nvjpegJpegState\_t decoder\_state, nvjpegJpegStream\_t jpeg\_bitstream, nvjpegImage\_t \*destination, nvjpegDecodeParams\_t decode\_params, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegHandle_t handle` | Input | Host | The library handle. | | `nvjpegJpegDecoder_t decoder` | Input | Host | The nvJPEG decoder handle. | | `nvjpegJpegState_t decoder_state` | Input | Host | The nvJPEG decoder state handle. | | `nvjpegJpegStream_t jpeg_stream` | Input | Host | Handle to the parsed bitstream data. | | `nvjpegImage_t *destination` | Input/Output | Host/Device | Pointer to a structure that describes the output destination. This structure should be on the host, but the pointers in this structure should be pointing to the device memory. See [nvJPEG Image](#nvjpeg-image)
for details. | | `nvjpegDecodeParams_t decode_params` | Input | Host | The handle which stores the decode output properties. | | `cudaStream_t stream` | Input | Host | The CUDA stream to which all the GPU tasks will be submitted. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ### 2.3.5. nvJPEG Decode Parameters[](#nvjpeg-decode-parameters "Permalink to this headline") This category of APIs is used to set the decoding parameters. These APIs should be used with the decode APIs defined in [Decode API—Decoupled Decoding](#nvjpeg-decoupled-decode-api) . #### 2.3.5.1. nvjpegDecodeParamsSetOutputFormat()[](#nvjpegdecodeparamssetoutputformat "Permalink to this headline") This function is used to set the decode output format. See `nvjpegOutputFormat_t` described in step 6 of [Single Image Decoding](#nvjpeg-single-image-decoding) . The output parameter of `nvjpegOutputFormat_t` defaults to `NVJPEG_OUTPUT_UNCHANGED` if not set using this API. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsSetOutputFormat( nvjpegDecodeParams\_t decode\_params, nvjpegOutputFormat\_t output\_format); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t decode_params` | Input | Host | Decode output parameter handle. | | `nvjpegOutputFormat_t output_format` | Input | Host | See step 6 of [Single Image Decoding](#nvjpeg-single-image-decoding)
. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.5.2. nvjpegDecodeParamsSetROI()[](#nvjpegdecodeparamssetroi "Permalink to this headline") This function enables the region of interest-only (ROI-only) decode. To disable the ROI-only, i.e., to decode the whole image, set: * `offset_x` = 0, * `offset_y` = 0, * `roi_width` = -1, and * `roi_height` = -1. Note ROI decode is disabled by default. It is not supported when the nvJPEG decoder handle is created using NVJPEG\_BACKEND\_HARDWARE. The ROI window cannot go out of image bounds. That is: * `offset_x` cannot be lower than zero, or * `offset_x + roi_width` cannot be larger than the JPEG image width. If the output format is NVJPEG\_OUTPUT\_YUV or NVJPEG\_OUTPUT\_UNCHANGED, then the `offset_x`and `offset_y` values have to be multiples of the maximum subsampling factor, as defined in the JPEG standard. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsSetROI( nvjpegDecodeParams\_t decode\_params, int offset\_x, int offset\_y, int roi\_width, int roi\_height); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t decode_params` | Input | Host | The decode output parameter handle. | | `int offset_x` | Input | Host | Image offset along the horizontal direction relative to the top left corner. | | `int offset_y` | Input | Host | Image offset along the vertical direction relative to the top left corner. | | `int roi_width` | Input | Host | Image width relative to `offset_x`. | | `int roi_height` | Input | Host | Image height relative to `offset_y`. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.5.3. nvjpegDecodeParamsSetAllowCMYK()[](#nvjpegdecodeparamssetallowcmyk "Permalink to this headline") If enabled, the nvJPEG library assumes that the JPEG with 4 encoded color components is in CMYK colorspace, and enables the conversion to RGB/YUV colorspace. The CMYK-to-RGB conversion is disabled by default. The conversion is based on the subtractive scheme—this behavior matches OpenCV’s handling of 4-component JPEGs. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsSetAllowCMYK( nvjpegDecodeParams\_t decode\_params, int allow\_cmyk); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t decode_params` | Input | Host | Decode output parameter handle. | | `int allow_cmyk` | Input | Host | Enable CMYK to RGB conversion. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.5.4. nvjpegDecodeParamsSetScaleFactor()[](#nvjpegdecodeparamssetscalefactor "Permalink to this headline") Allows the user to scale decode output. Note This feature is currently supported only when nvJPEG decoder handle is created using NVJPEG\_BACKEND\_HARDWARE. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsSetScaleFactor( nvjpegDecodeParams\_t decode\_params, nvjpegScaleFactor\_t scale\_factor); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t decode_params` | Input | Host | Decode output parameter handle. | | `nvjpegScaleFactor_t scale_factor` | Input | Host | Set the scaling factor for the decode output. | The scale factor is set to NVJPEG\_SCALE\_NONE by default. The supported values are listed [nvJPEG Scale Factor](#nvjpeg-scale-factor) . When setting a scale factor value, the recommended allocation of the destination parameters is as follows: * Use [nvjpegGetImageInfo()](#nvjpeggetimageinfo) , or [nvjpegJpegStreamGetFrameDimensions()](#nvjpeg-jpeg-stream-get-frame-dim) to extract the dimensions of each channel. * Let height\[NVJPEG\_MAX\_COMPONENT\] and width\[NVJPEG\_MAX\_COMPONENT\] be 2 arrays which store the height and width. The index to these arrays correspond to the channel id. * For a channel c, the scaled dimensions are calculated as follows: * scaled\_height\[c\] = (height\[c\] + rounding\_factor - 1)/rounding\_factor * scaled\_width\[c\] = (width\[c\] + rounding\_factor - 1)/rounding\_factor * when scale\_factor = NVJPEG\_SCALE\_NONE, rounding\_factor = 1 * when scale\_factor = NVJPEG\_SCALE\_1\_BY\_2, rounding\_factor = 2 * when scale\_factor = NVJPEG\_SCALE\_1\_BY\_4, rounding\_factor = 4 * when scale\_factor = NVJPEG\_SCALE\_1\_BY\_8, rounding\_factor = 8 | | | | | --- | --- | --- | | **For the output\_format:**

NVJPEG\_OUTPUT\_Y | **destination.pitch\[0\] should be at least:** width\[0\] | **destination.channel\[0\] should be at least of size:** destination.pitch\[0\]\*height\[0\] | | **For the output\_format** | **destination.pitch\[c\] should be at least:** | **destination.channel\[c\] should be at least of size:** | | NVJPEG\_OUTPUT\_YUV | width\[c\] for c = 0, 1, 2 | destination.pitch\[c\]\*height\[c\] for c = 0, 1, 2 | | NVJPEG\_OUTPUT\_RGB and NVJPEG\_OUTPUT\_BGR | width\[0\] for c = 0, 1, 2 | destination.pitch\[0\]\*height\[0\] for c = 0, 1, 2 | | NVJPEG\_OUTPUT\_RGBI and NVJPEG\_OUTPUT\_BGRI | width\[0\]\*3 | destination.pitch\[0\]\*height\[0\] | | NVJPEG\_OUTPUT\_UNCHANGED | width\[c\] for c = \[ 0, nComponents - 1 \] | destination.pitch\[c\]\*height\[c\] for c = \[ 0, nComponents - 1\] | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . #### 2.3.5.5. nvjpegDecodeParamsSetExifOrientation()[](#nvjpegdecodeparamssetexiforientation "Permalink to this headline") This function is used to generate the decoded output based on the exif orientation parameter. When ExifOrientation is enabled, the output buffers should be allocated based on the rotated dimensions. If the orientation is set as `NVJPEG_ORIENTATION_UNKNOWN`, the library will default to `NVJPEG_ORIENTATION_HORIZONTAL`. **ROI Decode and EXIF rotation** Exif rotation and ROI Decode can be enabled together. The ROI coordinates should be in the rotated space. **Signature:** nvjpegStatus\_t nvjpegDecodeParamsSetExifOrientation( nvjpegDecodeParams\_t decode\_params, nvjpegExifOrientation\_t orientation); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `nvjpegDecodeParams_t decode_params` | Input | Host | Decode output parameter handle. | | `nvjpegExifOrientation_t orientation` | Input | Host | Set the exif orientation for the decode output. | **Returns:** `nvjpegStatus_t` — An error code as specified in [nvJPEG API Return Codes](#nvjpeg-api-return-codes) . ### 2.3.6. nvJPEG API Return Codes[](#nvjpeg-api-return-codes "Permalink to this headline") The nvJPEG API adheres to the following return codes and their indicators: typedef enum { NVJPEG\_STATUS\_SUCCESS \= 0, NVJPEG\_STATUS\_NOT\_INITIALIZED \= 1, NVJPEG\_STATUS\_INVALID\_PARAMETER \= 2, NVJPEG\_STATUS\_BAD\_JPEG \= 3, NVJPEG\_STATUS\_JPEG\_NOT\_SUPPORTED \= 4, NVJPEG\_STATUS\_ALLOCATOR\_FAILURE \= 5, NVJPEG\_STATUS\_EXECUTION\_FAILED \= 6, NVJPEG\_STATUS\_ARCH\_MISMATCH \= 7, NVJPEG\_STATUS\_INTERNAL\_ERROR \= 8, NVJPEG\_STATUS\_IMPLEMENTATION\_NOT\_SUPPORTED \= 9 } nvjpegStatus\_t; **Description of the returned error codes:** | | | | --- | --- | | **Returned Error (Returned Code)** | **Description** | | `NVJPEG_STATUS_SUCCESS (0)` | The API call has finished successfully. Note that many of the calls are asynchronous and some of the errors may be seen only after synchronization. | | `NVJPEG_STATUS_NOT_INITIALIZED (1)` | The library handle was not initialized. A call to `nvjpegCreate()` is required to initialize the handle. | | `NVJPEG_STATUS_INVALID_PARAMETER (2)` | Wrong parameter was passed. For example, a null pointer as input data, or an image index not in the allowed range. | | `NVJPEG_STATUS_BAD_JPEG (3)` | Cannot parse the JPEG stream. Check that the encoded JPEG stream and its size parameters are correct. | | `NVJPEG_STATUS_JPEG_NOT_SUPPORTED (4)` | Attempting to decode a JPEG stream that is not supported by the nvJPEG library. | | `NVJPEG_STATUS_ALLOCATOR_FAILURE (5)` | The user-provided allocator functions, for either memory allocation or for releasing the memory, returned a non-zero code. | | `NVJPEG_STATUS_EXECUTION_FAILED (6)` | Error during the execution of the device tasks. | | `NVJPEG_STATUS_ARCH_MISMATCH (7)` | The device capabilities are not enough for the set of input parameters provided (input parameters such as backend, encoded stream parameters, output format). | | `NVJPEG_STATUS_INTERNAL_ERROR (8)` | Error during the execution of the device tasks. | | `NVJPEG_STATUS_IMPLEMENTATION_NOT_SUPPORTED (9)` | Not supported. | | `NVJPEG_STATUS_INCOMPLETE_BITSTREAM (10)` | Bitstream input data incomplete | ### 2.3.7. nvJPEG Chroma Subsampling[](#nvjpeg-chroma-subsampling "Permalink to this headline") One of the outputs of the `nvjpegGetImageInfo()` API is `nvjpegChromaSubsampling_t`. This parameter is an `enum` type, and its enumerator list comprises of the chroma subsampling property retrieved from the encoded JPEG image. The `nvjpegGetImageInfo()` function currently supports the following chroma subsampling types: typedef enum { NVJPEG\_CSS\_444, NVJPEG\_CSS\_422, NVJPEG\_CSS\_420, NVJPEG\_CSS\_440, NVJPEG\_CSS\_411, NVJPEG\_CSS\_410, NVJPEG\_CSS\_GRAY, NVJPEG\_CSS\_410V, NVJPEG\_CSS\_UNKNOWN } nvjpegChromaSubsampling\_t; ### 2.3.8. Reference Documents[](#reference-documents "Permalink to this headline") Refer to the JPEG standard: [https://jpeg.org/jpeg/](https://jpeg.org/jpeg/) 2.4. Examples of nvJPEG[](#examples-of-nvjpeg "Permalink to this headline") ----------------------------------------------------------------------------- nvJPEG Decode sample can be found here: [https://github.com/NVIDIA/CUDALibrarySamples/tree/master/nvJPEG/nvJPEG-Decoder](https://github.com/NVIDIA/CUDALibrarySamples/tree/master/nvJPEG/nvJPEG-Decoder) 3\. JPEG Encoding[](#jpeg-encoding "Permalink to this headline") ================================================================== This section describes the encoding functions of the nvJPEG Library. 3.1. Using the Encoder[](#using-the-encoder "Permalink to this headline") --------------------------------------------------------------------------- The user should perform the below prerequisite steps before calling the nvJPEG encoding functions. See also [nvJPEG Encoder Helper API Reference](#nvjpeg-encoder-helper-api-reference) . ### 3.1.1. Encoding the Parameters[](#encoding-the-parameters "Permalink to this headline") The user should create an encoding parameters structure with `nvjpegEncoderParamsCreate()` function. The function will be initialized with default parameters. User can use an appropriate `nvjpegEncoderParamsSet*()` function to set a specific parameter. The quality parameter can be set, using the `nvjpegEncoderParamsSetQuality()` function, to an integer value between 1 and 100, and this quality parameter will be used as a base for generating the JPEG quantization tables. The parameters structure should be passed to compression functions. Note The encoding parameters structure can be reused to compress multiple images simultaneously, but no changes to the parameters should be made during the ongoing encoding, or the encoding result will be undefined. ### 3.1.2. Encoding the State[](#encoding-the-state "Permalink to this headline") The user should create the encoding state structure using `nvjpegEncoderStateCreate()` function. This function will hold intermediate buffers for the encoding process. This state should be passed to the compression functions. Note The encoding state structure can be reused to encode a series of images, but no encoding should be performed on multiple images with the same encoding state at the same time—otherwise the result of the encodings will be undefined. ### 3.1.3. Encoding the Image[](#encoding-the-image "Permalink to this headline") The nvJPEG library provides a few interfaces for compressing the image in different formats and colorspaces. See below. #### 3.1.3.1. nvjpegEncodeYUV[](#nvjpegencodeyuv "Permalink to this headline") Input for this function is an image in YUV colorspace. See `nvjpegEncodeYUV()`. The `source` argument should be filled with the corresponding YUV planar data. The `chroma_subsampling` argument should have the chroma subsampling of the input data. If the chroma subsampling in the encoding parameters is the same as input chroma subsampling, then the user’s input data will be directly used in the JPEG compression. Otherwise chroma will be resampled to match the chroma subsampling of the encoding parameters. Input data should be provided with respect to the subsampling factors. That is, the chrominance image planes should have sizes aligned to the corresponding subsamplings. For example: * Image dimensions: 123x321 * Input chroma subsampling: NVJPEG\_CSS\_410 * Chroma subsampling factor for this chroma subsampling: 4x2 * Given the above, the encoder library expects the user to provide: * Y plane with size: 123 x 321 * Cb and Cr plane with size: 31 x 161 #### 3.1.3.2. nvjpegEncodeImage[](#nvjpegencodeimage "Permalink to this headline") See `nvjpegEncodeImage()`. Input for this function, i.e., how data should be provided in the `source` argument, is determined by the `input_format` argument. For the interleaved formats (ending with **I**) only the first channel is used. For the non-interleaved formats, all the channels in the input format are used. For example, if the user has interleaved the RGB image of size `W x H`, stored continuously, and the pointer to it is `pImage`, then `source` should be: * `source.channel[0] = pImage` * `source.pitch[0] = W*3` When the same image is stored in planar format, with image planes pointers stored continuously in the array `pImage[3]`, then `source` should be: * `source.channel[0] = pImage[0]` * `source.channel[1] = pImage[1]` * `source.channel[2] = pImage[2]` The `pitch` values for each channel in the `source` parameter should be set accordingly to the data layout. The nvJPEG library will perform the color transformation to the YCbCr, and will compress the result. ### 3.1.4. Retrieving the Compressed Stream[](#retrieving-the-compressed-stream "Permalink to this headline") Often it is not feasible to accurately predict the final compressed data size of the final JPEG stream for any input data and parameters. The nvJPEG library, while encoding, will calculate the size of the final stream, allocate temporary buffer in the encoder state and save the compressed data in the encoding state’s buffer. In order to get final compressed JPEG stream, the user should provide the memory buffer large enough to store this compressed data. There are two options for how to do this: 1. Use the upper bound on compressed JPEG stream size for the given parameters and image dimensions: 1. Use the `nvjpegEncodeRetrieveBitstream()` function to retrieve the maximum possible JPEG stream size at any given time. 2. Allocate the memory buffer at any given time. 3. Encode the image using one of the encoding functions. 4. Retrieve the compressed JPEG stream from the encoder state after successful encoding, using the `nvjpegEncodeRetrieveBitstream()` and the allocated buffer. 2. Wait for the encoding to complete, and retrieve the exact size of required buffer, as below: 1. Encode the image using one of the encoding functions. 2. Use the `nvjpegEncodeRetrieveBitstream()` function to retrieve the size in bytes of the compressed JPEG stream. 3. Allocate the memory buffer of at least this size. 4. Use the `nvjpegEncodeRetrieveBitstream()` function to populate your buffer with the compressed JPEG stream. Note As the same encoding image state can be reused to compress a series of images, the `nvjpegEncodeRetrieveBitstream()` function will return the result for the last compressed image. ### 3.1.5. JPEG Encoding Example[](#jpeg-encoding-example "Permalink to this headline") See below the example code, and the block diagram shown in Figure 1 , for encoding with nvJPEG Encoder. [![JPEG Encoding Using nvJPEG Encoder](_images/nvjpeg-encoding-flow.png)](_images/nvjpeg-encoding-flow.png) JPEG Encoding Using nvJPEG Encoder[](#nvjpeg-encode-examples-fig-nvjpeg-encode-example "Permalink to this image") nvjpegHandle\_t nv\_handle; nvjpegEncoderState\_t nv\_enc\_state; nvjpegEncoderParams\_t nv\_enc\_params; cudaStream\_t stream; // initialize nvjpeg structures nvjpegCreateSimple(&nv\_handle); nvjpegEncoderStateCreate(nv\_handle, &nv\_enc\_state, stream); nvjpegEncoderParamsCreate(nv\_handle, &nv\_enc\_params, stream); nvjpegImage\_t nv\_image; // Fill nv\_image with image data, let's say 640x480 image in RGB format // Compress image nvjpegEncodeImage(nv\_handle, nv\_enc\_state, nv\_enc\_params, &nv\_image, NVJPEG\_INPUT\_RGB, 640, 480, stream); // get compressed stream size size\_t length; nvjpegEncodeRetrieveBitstream(nv\_handle, nv\_enc\_state, NULL, &length, stream); // get stream itself cudaStreamSynchronize(stream); std::vector jpeg(length); nvjpegEncodeRetrieveBitstream(nv\_handle, nv\_enc\_state, jpeg.data(), &length, 0); // write stream to file cudaStreamSynchronize(stream); std::ofstream output\_file("test.jpg", std::ios::out | std::ios::binary); output\_file.write(jpeg.data(), length); output\_file.close(); 3.2. nvJPEG Encoder Type Declarations[](#nvjpeg-encoder-type-declarations "Permalink to this headline") --------------------------------------------------------------------------------------------------------- This section describes the nvJPEG Encoder Type Declarations. ### 3.2.1. nvjpegInputFormat\_t[](#nvjpeginputformat-t "Permalink to this headline") typedef enum { NVJPEG\_INPUT\_RGB \= 3, NVJPEG\_INPUT\_BGR \= 4, NVJPEG\_INPUT\_RGBI \= 5, NVJPEG\_INPUT\_BGRI \= 6 } nvjpegInputFormat\_t; The `nvjpegInputFormat_t` enum is used to select the color model and pixel format of the input image. It is used for conversion to YCbCr during encoding. | | | | --- | --- | | **Member** | **Description** | | NVJPEG\_INPUT\_RGB | Input image is in RGB color model. Pixel format is RGB. | | NVJPEG\_INPUT\_BGR | Input image is in RGB color model. Pixel format is BGR. | | NVJPEG\_INPUT\_RGBI | Input image is in RGB color model. Pixel format is interleaved RGB. | | NVJPEG\_INPUT\_BGRI | Input image is in RGB color model. Pixel format is interleaved BGR. | ### 3.2.2. nvjpegEncoderState\_t[](#nvjpegencoderstate-t "Permalink to this headline") The `nvjpegEncoderState_t` structure stores intermediate buffers and variables used for compression. ### 3.2.3. nvjpegEncoderParams\_t[](#nvjpegencoderparams-t "Permalink to this headline") The `nvjpegEncoderParams_t` structure stores JPEG encode parameters. 3.3. nvJPEG Encoder Helper API Reference[](#nvjpeg-encoder-helper-api-reference "Permalink to this headline") --------------------------------------------------------------------------------------------------------------- The nvJPEG Encoder helper functions are used for initializing. ### 3.3.1. nvjpegEncoderStateCreate()[](#nvjpegencoderstatecreate "Permalink to this headline") Creates encoder state that stores intermediate buffers used in compression. **Signature:** nvjpegStatus\_t nvjpegEncoderStateCreate( nvjpegHandle\_t handle, nvjpegEncoderState\_t \*encoder\_state, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle | | `encoder_state` | Output | Host | Pointer to the encoder state structure, where the new state will be placed. | | `stream` | Inputt | Host | CUDA stream where all the required device operations will be placed. | ### 3.3.2. nvjpegEncoderStateDestroy()[](#nvjpegencoderstatedestroy "Permalink to this headline") Destroys the encoder state. **Signature:** nvjpegStatus\_t nvjpegEncoderStateDestroy( nvjpegEncoderState\_t encoder\_state); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_state` | Input/Output | Host | Encoder state structure that will be released. | ### 3.3.3. nvjpegEncoderParamsCreate()[](#nvjpegencoderparamscreate "Permalink to this headline") Creates the structure that holds the compression parameters. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsCreate( nvjpegHandle\_t handle, nvjpegEncoderParams\_t \*encoder\_params, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_params` | Output | Host | Pointer to the location where the new parameters structure will be placed. | | `stream` | Inputt | Host | CUDA stream where all the required device operations will be placed. | ### 3.3.4. nvjpegEncoderParamsDestroy()[](#nvjpegencoderparamsdestroy "Permalink to this headline") Destroys the encoder parameters structure. **Signature:** nvjpegEncoderParamsDestroy( nvjpegEncoderParams\_t encoder\_params); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_params` | Input/Output | Host | Encoder params structure that will be released. | ### 3.3.5. nvjpegEncoderParamsSetEncoding()[](#nvjpegencoderparamssetencoding "Permalink to this headline") Sets the parameter quality in the encoder parameters structure. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsSetEncoding( nvjpegEncoderParams\_t encoder\_params, nvjpegJpegEncoding\_t etype, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_params` | Input/Output | Host | Encoder parameters structure handle. | | `etype` | Input | Host | Encoding type selection (Baseline/Progressive). Default is Baseline. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.3.6. nvjpegEncoderParamsSetQuality()[](#nvjpegencoderparamssetquality "Permalink to this headline") Sets the parameter quality in the encoder parameters structure. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsSetQuality( nvjpegEncoderParams\_t encoder\_params, const int quality, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_params` | Input/Output | Host | Encoder parameterss structure handle. | | `quality` | Input | Host | Integer value of quality between 1 and 100, where 100 is the highest quality. Default value is 70. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.3.7. nvjpegEncoderParamsSetOptimizedHuffman()[](#nvjpegencoderparamssetoptimizedhuffman "Permalink to this headline") Sets whether or not to use optimized Huffman. Using optimized Huffman produces smaller JPEG bitstream sizes with the same quality, but with slower performance. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsSetOptimizedHuffman( nvjpegEncoderParams\_t encoder\_params, const int optimized, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_params` | Input/Output | Host | Encoder parameters structure handle. | | `optimized` | Input | Host | If this value is 0 then non-optimized Huffman will be used. Otherwise optimized version will be used. Default value is 0. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.3.8. nvjpegEncoderParamsSetSamplingFactors()[](#nvjpegencoderparamssetsamplingfactors "Permalink to this headline") Sets which chroma subsampling will be used for JPEG compression. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsSetSamplingFactors( nvjpegEncoderParams\_t encoder\_params, const nvjpegChromaSubsampling\_t chroma\_subsampling, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_params` | Input/Output | Host | Encoder parameters structure handle. | | `chroma_subsampling` | Input | Host | Chroma subsampling that will be used for JPEG compression. If the input is in YUV color model and `chroma_subsampling` is different from the subsampling factors of source image, then the NVJPEG library will convert subsampling to the value of `chroma_subsampling`. Default value is 4:4:4. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | 3.4. nvJPEG Encoder API Reference[](#nvjpeg-encoder-api-reference "Permalink to this headline") ------------------------------------------------------------------------------------------------- This section describes the nvJPEG Encoder API. ### 3.4.1. nvjpegEncodeGetBufferSize()[](#nvjpegencodegetbuffersize "Permalink to this headline") Returns the maximum possible buffer size that is needed to store the compressed JPEG stream, for the given input parameters. **Signature:** nvjpegStatus\_t nvjpegEncodeGetBufferSize( nvjpegHandle\_t handle, const nvjpegEncoderParams\_t encoder\_params, int image\_width, int image\_height, size\_t \*max\_stream\_length); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_params` | Input/Output | Host | Encoder parameters structure handle. | | `image_width` | Input | Host | Input image width. | | `image_height` | Input | Host | Input image height. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.4.2. nvjpegEncodeYUV()[](#nvjpeg-encode-yuv "Permalink to this headline") Compresses the image in YUV colorspace to JPEG stream using the provided parameters, and stores it in the state structure. **Signature:** nvjpegStatus\_t nvjpegEncodeYUV( nvjpegHandle\_t handle, nvjpegEncoderState\_t encoder\_state, const nvjpegEncoderParams\_t encoder\_params, const nvjpegImage\_t \*source, nvjpegChromaSubsampling\_t chroma\_subsampling, int image\_width, int image\_height, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_state` | Input/Output | Host | Internal structure that holds the temporary buffers required for the compression and also stores the final compressed JPEG stream. | | `encoder_params` | Input | Host | Encoder parameters structure handle. | | `source` | Input | Host | Pointer to the `nvjpeg` structure that holds the device pointers to the `Y, U(Cb) and V(Cr)` image planes and the respective strides. | | `chroma_subsampling` | Input | Host | Chroma subsampling of the input data. | | `image_width` | Input | Host | Input image width. | | `image_height` | Input | Host | Input image height. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.4.3. nvjpegEncodeImage()[](#nvjpeg-encode-image "Permalink to this headline") Compresses the image in the provided format to the JPEG stream using the provided parameters, and stores it in the state structure. **Signature:** nvjpegStatus\_t nvjpegEncodeImage( nvjpegHandle\_t handle, nvjpegEncoderState\_t encoder\_state, const nvjpegEncoderParams\_t encoder\_params, const nvjpegImage\_t \*source, nvjpegInputFormat\_t input\_format, int image\_width, int image\_height, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_state` | Input/Output | Host | Internal structure that holds the temporary buffers required for the compression and also stores the final compressed JPEG stream. | | `encoder_params` | Input | Host | Encoder parameters structure handle. | | `source` | Input | Host | Pointer to the `nvjpeg` structure that holds the device pointers to the `Y, U(Cb) and V(Cr)` image planes and the respective strides. | | `input_format` | Input | Host | Value of `nvjpegInputFormat_t` type that describes the input data. | | `image_width` | Input | Host | Input image width. | | `image_height` | Input | Host | Input image height. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.4.4. nvjpegEncodeRetrieveBitstream()[](#nvjpegencoderetrievebitstream "Permalink to this headline") Retrieves the compressed stream from the encoder state that was previously used in one of the encoder functions. * If `data` parameter is NULL then the encoder will return compressed stream size in the `length` parameter. * If `data` is not NULL then the provided `length` parameter should contain the `data` buffer size. * If the provided `length` is less than compressed stream size, then an error will be returned. Otherwise the compressed stream will be stored in the `data` buffer and the actual compressed buffer size will be stored in the `length` parameter. **Signature:** nvjpegStatus\_t nvjpegEncodeRetrieveBitstream( nvjpegHandle\_t handle, nvjpegEncoderState\_t encoder\_state, unsigned char \*data, size\_t \*length, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_state` | Input/Output | Host | The `encoder_state` that was previously used in one of the encoder functions. | | `data` | Input/Output | Host | Pointer to the buffer in the host memory where the compressed stream will be stored. Can be NULL (see description). | | `length` | Input/Output | Host | Pointer to the input buffer size. On return the NVJPEG library will store the actual compressed stream size in this parameter. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | ### 3.4.5. nvjpegEncodeRetrieveBitstreamDevice()[](#nvjpegencoderetrievebitstreamdevice "Permalink to this headline") Retrieves the compressed stream from the encoder state that was previously used in one of the encoder functions. * `data` parameter should be on device memory * If `data` parameter is NULL then the encoder will return compressed stream size in the `length` parameter. * If `data` is not NULL then the provided `length` parameter should contain the `data` buffer size. * If the provided `length` is less than compressed stream size, then an error will be returned. Otherwise the compressed stream will be stored in the `data` buffer and the actual compressed buffer size will be stored in the `length` parameter. **Signature:** nvjpegStatus\_t nvjpegEncodeRetrieveBitstreamDevice( nvjpegHandle\_t handle, nvjpegEncoderState\_t encoder\_state, unsigned char \*data, size\_t \*length, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `handle` | Input | Host | Library handle. | | `encoder_state` | Input/Output | Host | The `encoder_state` that was previously used in one of the encoder functions. | | `data` | Input/Output | Device | Pointer to the buffer in the device memory where the compressed stream will be stored. Can be NULL (see description). | | `length` | Input/Output | Host | Pointer to the input buffer size. On return the NVJPEG library will store the actual compressed stream size in this parameter. | | `stream` | Input | Host | CUDA stream where all the required device operations will be placed. | 4\. JPEG Transcoding[](#jpeg-transcoding "Permalink to this headline") ======================================================================== This section describes the transcoding functions of the nvJPEG Library. 4.1. nvJPEG Transcoder Helper API Reference[](#nvjpeg-transcoder-helper-api-reference "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------- This section describes the nvJPEG Transcoder helper API. ### 4.1.1. nvjpegEncoderParamsCopyMetadata()[](#nvjpegencoderparamscopymetadata "Permalink to this headline") Copies the metadata (JFIF, APP, EXT, and COM markers) from the parsed stream. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsCopyMetadata( nvjpegEncoderState\_t encoder\_state, nvjpegEncoderParams\_t encode\_params, nvjpegJpegStream\_t jpeg\_stream, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_state` | In/Out | Host | Internal structure that stores the temporary buffers required for the compression. | | `encode_params` | Out | Host | Encoder parameters that will be used for compression. | | `jpeg_stream` | In | Host | Input parsed stream. | | `stream` | In | Host | CUDA stream where all the required device operations will be placed. | ### 4.1.2. nvjpegEncoderParamsCopyQuantizationTables()[](#nvjpegencoderparamscopyquantizationtables "Permalink to this headline") Copies the quantization tables from the parsed stream. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsCopyQuantizationTables( nvjpegEncoderParams\_t encode\_params, nvjpegJpegStream\_t jpeg\_stream, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encode_params` | Out | Host | Encoder parameters that will be used for compression. | | `jpeg_stream` | In | Host | Input parsed stream. | | `stream` | In | Host | CUDA stream where all the required device operations will be placed. | ### 4.1.3. nvjpegEncoderParamsCopyHuffmanTables() \[Deprecated\][](#nvjpegencoderparamscopyhuffmantables-deprecated "Permalink to this headline") nvjpegEncoderParamsCopyHuffmanTables() is now deprecated. Due to precision differences in the JPEG encode/decode process, the input huffman tables may no longer be valid for the image being encoded and may result in corrupt bitstream. **Signature:** nvjpegStatus\_t nvjpegEncoderParamsCopyHuffmanTables( nvjpegEncoderState\_t encoder\_state, nvjpegEncoderParams\_t encode\_params, nvjpegJpegStream\_t jpeg\_stream, cudaStream\_t stream); **Parameters:** | | | | | | --- | --- | --- | --- | | **Parameter** | **Input / Output** | **Memory** | **Description** | | `encoder_state` | In/Out | Host | Internal structure that stores the temporary buffers required for the compression. | | `encode_params` | Out | Host | Encoder parameters that will be used for compression. | | `jpeg_stream` | In | Host | Input parsed stream. | | `stream` | In | Host | CUDA stream where all the required device operations will be placed. | 4.2. JPEG Transcoding Example[](#jpeg-transcoding-example "Permalink to this headline") ----------------------------------------------------------------------------------------- See below the example code. cudaStream\_t stream; // create library handle nvjpegHandle\_t handle; nvjpegCreateSimple(&handle); /////////////////////////////////// nvJPEG decoding //////////////////////////////////////// // create bitstream object nvjpegJpegStream\_t jpeg\_stream; nvjpegJpegStreamCreate(handle, &jpeg\_stream); // parse jpeg stream nvjpegJpegStreamParse(handle, data\_ptr, data\_size, 1, // save metadata in the jpegStream structure 0, jpeg\_stream); // create decoder and decoder state nvjpegJpegDecoder\_t decoder; nvjpegJpegState\_t decoder\_state; nvjpegDecoderCreate(handle, NVJPEG\_BACKEND\_DEFAULT, &decoder); nvjpegDecoderStateCreate(handle, decoder, &decoder\_state); // create and set up decoder parameters nvjpegDecodeParams\_t decode\_params; nvjpegDecodeParamsCreate(handle, &decode\_params); nvjpegDecodeParamsSetOutputFormat(decode\_params, NVJPEG\_OUTPUT\_RGBI); // decode image nvjpegImage\_t output\_image; nvjpegDecodeJpeg(handle, decoder, decode\_params, jpeg\_stream, decoder\_state, &output\_image, stream); /////////////////////////////////// nvJPEG Transcode and encode API /////////////////////////////////// nvjpegEncoderState\_t encoder\_state; nvjpegEncoderParams\_t encode\_params; // get encoding from the jpeg stream and copy it to the encode parameters nvjpegJpegEncoding\_t jpeg\_encoding; nvjpegJpegStreamGetJpegEncoding(jpeg\_stream, &jpeg\_encoding); nvjpegEncoderParamsSetEncoding(encode\_params, jpeg\_encoding); // copies according data to the encode parameters nvjpegEncoderParamsCopyMetadata(encode\_params, jpeg\_stream, stream); nvjpegEncoderParamsCopyQuantizationTables(encode\_params, jpeg\_stream, stream); nvjpegEncoderParamsCopyHuffmanTables(encode\_params, jpeg\_stream, stream); // retrieve frame dimensions unsigned width, height; nvjpegJpegStreamGetFrameDimensions(jpeg\_stream, &width, &height); // encode using encode parameters nvjpegEncodeImage(nvjpeg\_handle, encoder\_state, encode\_params, &output\_image, input\_format, width, height, stream); // get compressed stream size size\_t length; nvjpegEncodeRetrieveBitstream(nvjpeg\_handle, encoder\_state, NULL, &length, stream); // get stream itself cudaStreamSynchronize(stream); std::vector jpeg(length); nvjpegEncodeRetrieveBitstream(nvjpeg\_handle, encoder\_state, jpeg.data(), &length, 0); 5\. List of Dropped APIs[](#list-of-dropped-apis "Permalink to this headline") ================================================================================ The following APIs are dropped starting CUDA 11.0: nvjpegStatus\_t nvjpegDecodePhaseOne( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*data, size\_t length, nvjpegOutputFormat\_t output\_format, cudaStream\_t stream); nvjpegStatus\_t nvjpegDecodePhaseTwo( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, cudaStream\_t stream); nvjpegStatus\_t nvjpegDecodePhaseThree( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, nvjpegImage\_t \*destination, cudaStream\_t stream); nvjpegStatus\_t nvjpegDecodeBatchedPhaseOne( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, const unsigned char \*data, size\_t length, int image\_idx, int thread\_idx, cudaStream\_t stream); nvjpegStatus\_t nvjpegDecodeBatchedPhaseTwo( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, cudaStream\_t stream); nvjpegStatus\_t nvjpegDecodeBatchedPhaseThree( nvjpegHandle\_t handle, nvjpegJpegState\_t jpeg\_handle, nvjpegImage\_t \*destinations, cudaStream\_t stream); 6\. Known Issues[](#known-issues "Permalink to this headline") ================================================================ Decoupled APIs, when initialized with `NVJPEG_BACKEND_GPU_HYBRID`, may not be able to correctly decode jpeg bitstreams which have out of bound run length codes. 7\. Notices[](#notices "Permalink to this headline") ====================================================== 7.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 7.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 7.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # 1. Introduction — cuFFT 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [PDF](../pdf/CUFFT_Library.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * cuFFT API Reference The API reference guide for cuFFT, the CUDA Fast Fourier Transform library. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ **cuFFT Release Notes**: [CUDA Toolkit Release Notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cufft-library) **cuFFT GitHub Samples**: [CUDA Library Samples](https://github.com/NVIDIA/CUDALibrarySamples/tree/master/cuFFT) **Nvidia Developer Forum**: [GPU-Accelerated Libraries](https://forums.developer.nvidia.com/c/accelerated-computing/gpu-accelerated-libraries/) **Provide Feedback**: [Math-Libs-Feedback@nvidia.com](mailto:Math-Libs-Feedback%40nvidia.com?subject=cuFFT-Feedback) **Related FFT Libraries**: * [cuFFTMP](https://docs.nvidia.com/hpc-sdk/cufftmp/) * [cuFFTDx](https://docs.nvidia.com/cuda/cufftdx/index.html) * [cuFFT LTO EA (DEPRECATED)](https://docs.nvidia.com/cuda/cufft/ltoea/index.html) * [NVPL FFT](https://docs.nvidia.com/nvpl/_static/fft/index.html) **Relevant cuFFT Blog Posts and GTC presentations**: * [Accelerating GPU Applications with NVIDIA Math Libraries](https://developer.nvidia.com/blog/accelerating-gpu-applications-with-nvidia-math-libraries/) * [Multinode Multi-GPU: Using NVIDIA cuFFTMp FFTs at Scale](https://developer.nvidia.com/blog/multinode-multi-gpu-using-nvidia-cufftmp-ffts-at-scale/) * [New Asynchronous Programming Model Library Now Available with NVIDIA HPC SDK v22.11](https://developer.nvidia.com/blog/building-high-performance-applications-in-the-era-of-accelerated-computing/) * [Just-In-Time Link-Time Optimization Adoption in cuSPARSE/cuFFT: Use Case Overview](https://www.nvidia.com/en-us/on-demand/session/gtcfall21-a31155/?playlistId=playList-ead11304-9931-4e91-9d5a-fb0e1ef27014) This document describes cuFFT, the NVIDIA® CUDA® Fast Fourier Transform (FFT) product. It consists of two separate libraries: cuFFT and cuFFTW. The cuFFT library is designed to provide high performance on NVIDIA GPUs. The cuFFTW library is provided as a porting tool to enable users of FFTW to start using NVIDIA GPUs with a minimum amount of effort. The FFT is a divide-and-conquer algorithm for efficiently computing discrete Fourier transforms of complex or real-valued data sets. It is one of the most important and widely used numerical algorithms in computational physics and general signal processing. The cuFFT library provides a simple interface for computing FFTs on an NVIDIA GPU, which allows users to quickly leverage the floating-point power and parallelism of the GPU in a highly optimized and tested FFT library. The cuFFT product supports a wide range of FFT inputs and options efficiently on NVIDIA GPUs. This version of the cuFFT library supports the following features: * Algorithms highly optimized for input sizes that can be written in the form \\(2^{a} \\times 3^{b} \\times 5^{c} \\times 7^{d}\\). In general the smaller the prime factor, the better the performance, i.e., powers of two are fastest. * An \\(O\\left( n\\log n \\right)\\) algorithm for every input data size * Half-precision (16-bit floating point), single-precision (32-bit floating point) and double-precision (64-bit floating point). Transforms of lower precision have higher performance. * Complex and real-valued input and output. Real valued input or output require less computations and data than complex values and often have faster time to solution. Types supported are: * C2C - Complex input to complex output * R2C - Real input to complex output * C2R - Symmetric complex input to real output * 1D, 2D and 3D transforms * Execution of multiple 1D, 2D and 3D transforms simultaneously. These batched transforms have higher performance than single transforms. * In-place and out-of-place transforms * Arbitrary intra- and inter-dimension element strides (strided layout) * FFTW compatible data layout * Execution of transforms across multiple GPUs * Streamed execution, enabling asynchronous computation and data movement The cuFFTW library provides the FFTW3 API to facilitate porting of existing FFTW applications. Please note that starting from CUDA 11.0, the minimum supported GPU architecture is SM35. See [Deprecated Functionality](index.html#deprecated-functionality) . 2\. Using the cuFFT API[](#using-the-cufft-api "Permalink to this headline") ============================================================================== This chapter provides a general overview of the cuFFT library API. For more complete information on specific functions, see [cuFFT API Reference](index.html#cufft-api-reference) . Users are encouraged to read this chapter before continuing with more detailed descriptions. The Discrete Fourier transform (DFT) maps a complex-valued vector \\(x\_{k}\\) (_time domain_) into its _frequency domain representation_ given by: \\(X\_{k} = \\sum\\limits\_{n = 0}^{N - 1}x\_{n}e^{-2\\pi i\\frac{kn}{N}}\\) where \\(X\_{k}\\) is a complex-valued vector of the same size. This is known as a _forward_ DFT. If the sign on the exponent of e is changed to be positive, the transform is an _inverse_ transform. Depending on \\(N\\), different algorithms are deployed for the best performance. The cuFFT API is modeled after [FFTW](http://www.fftw.org) , which is one of the most popular and efficient CPU-based FFT libraries. cuFFT provides a simple configuration mechanism called a _plan_ that uses internal building blocks to optimize the transform for the given configuration and the particular GPU hardware selected. Then, when the _execution_ function is called, the actual transform takes place following the plan of execution. The advantage of this approach is that once the user creates a plan, the library retains whatever state is needed to execute the plan multiple times without recalculation of the configuration. This model works well for cuFFT because different kinds of FFTs require different thread configurations and GPU resources, and the plan interface provides a simple way of reusing configurations. Computing a number `BATCH` of one-dimensional DFTs of size `NX` using cuFFT will typically look like this: #define NX 256 #define BATCH 10 #define RANK 1 ... { cufftHandle plan; cufftComplex \*data; ... cudaMalloc((void\*\*)&data, sizeof(cufftComplex)\*NX\*BATCH); cufftPlanMany(&plan, RANK, NX, &iembed, istride, idist, &oembed, ostride, odist, CUFFT\_C2C, BATCH); ... cufftExecC2C(plan, data, data, CUFFT\_FORWARD); cudaDeviceSynchronize(); ... cufftDestroy(plan); cudaFree(data); } 2.1. Accessing cuFFT[](#accessing-cufft "Permalink to this headline") ----------------------------------------------------------------------- The cuFFT and cuFFTW libraries are available as shared libraries. They consist of compiled programs ready for users to incorporate into applications with the compiler and linker. cuFFT can be downloaded from [https://developer.nvidia.com/cufft](https://developer.nvidia.com/cufft) . By selecting **Download CUDA Production Release** users are all able to install the package containing the CUDA Toolkit, SDK code samples and development drivers. The CUDA Toolkit contains cuFFT and the samples include `simplecuFFT`. The Linux release for `simplecuFFT` assumes that the root install directory is `/usr/local/cuda` and that the locations of the products are contained there as follows. Modify the Makefile as appropriate for your system. | Product | Location and name | Include file | | --- | --- | --- | | `nvcc` compiler | `/bin/nvcc` | | | `cuFFT` library | `{lib, lib64}/libcufft.so` | `inc/cufft.h` | | `cuFFT` library with Xt functionality | `{lib, lib64}/libcufft.so` | `inc/cufftXt.h` | | `cuFFTW` library | `{lib, lib64}/libcufftw.so` | `inc/cufftw.h` | The most common case is for developers to modify an existing CUDA routine (for example, `filename.cu`) to call cuFFT routines. In this case the include file `cufft.h` or `cufftXt.h` should be inserted into `filename.cu` file and the library included in the link line. A single compile and link line might appear as * `/usr/local/cuda/bin/nvcc [options] filename.cu … -I/usr/local/cuda/inc -L/usr/local/cuda/lib -lcufft` Of course there will typically be many compile lines and the compiler `g++` may be used for linking so long as the library path is set correctly. Users of the FFTW interface (see [FFTW Interface to cuFFT](index.html#fftw-supported-interface) ) should include `cufftw.h` and link with both cuFFT and cuFFTW libraries. Functions in the cuFFT and cuFFTW library assume that the data is in GPU visible memory. This means any memory allocated by `cudaMalloc`, `cudaMallocHost` and `cudaMallocManaged` or registered with `cudaHostRegister` can be used as input, output or plan work area with cuFFT and cuFFTW functions. For the best performance input data, output data and plan work area should reside in device memory. cuFFTW library also supports input data and output data that is not GPU visible. 2.2. Fourier Transform Setup[](#fourier-transform-setup "Permalink to this headline") --------------------------------------------------------------------------------------- The first step in using the cuFFT Library is to create a plan using one of the following: * `cufftPlan1D() / cufftPlan2D() / cufftPlan3D()` - Create a simple plan for a 1D/2D/3D transform respectively. * `cufftPlanMany()` - Creates a plan supporting batched input and strided data layouts. * `cufftXtMakePlanMany()` - Creates a plan supporting batched input and strided data layouts for any supported precision. Among the plan creation functions, `cufftPlanMany()` allows use of more complicated data layouts and batched executions. Execution of a transform of a particular size and type may take several stages of processing. When a plan for the transform is generated, cuFFT derives the internal steps that need to be taken. These steps may include multiple kernel launches, memory copies, and so on. In addition, all the intermediate buffer allocations (on CPU/GPU memory) take place during planning. These buffers are released when the plan is destroyed. In the worst case, the cuFFT Library allocates space for `8*batch*n[0]*..*n[rank-1] cufftComplex` or `cufftDoubleComplex` elements (where `batch` denotes the number of transforms that will be executed in parallel, `rank` is the number of dimensions of the input data (see [Multidimensional Transforms](index.html#multi-dimensional) ) and `n[]` is the array of transform dimensions) for single and double-precision transforms respectively. Depending on the configuration of the plan, less memory may be used. In some specific cases, the temporary space allocations can be as low as `1*batch*n[0]*..*n[rank-1] cufftComplex` or `cufftDoubleComplex` elements. This temporary space is allocated separately for each individual plan when it is created (i.e., temporary space is not shared between the plans). The next step in using the library is to call an execution function such as `cufftExecC2C()` (see [Parameter cufftType](index.html#cufft-transform-types) ) which will perform the transform with the specifications defined at planning. One can create a cuFFT plan and perform multiple transforms on different data sets by providing different input and output pointers. Once the plan is no longer needed, the `cufftDestroy()` function should be called to release the resources allocated for the plan. ### 2.2.1. Free Memory Requirement[](#free-memory-requirement "Permalink to this headline") The first program call to any cuFFT function causes the initialization of the cuFFT kernels. This can fail if there is not enough free memory on the GPU. It is advisable to initialize cufft first (e.g. by creating a plan) and then allocating memory. ### 2.2.2. Plan Initialization Time[](#plan-initialization-time "Permalink to this headline") During plan initialization, cuFFT conducts a series of steps, including heuristics to determine which kernels to be used as well as kernel module loads. Starting from CUDA 12.0, cuFFT delivers a larger portion of kernels using the CUDA Parallel Thread eXecution assembly form (PTX code), instead of the binary form (cubin object). The PTX code of cuFFT kernels are loaded and compiled further to the binary code by the CUDA device driver at runtime when a cuFFT plan is initialized. This is called [just-in-time (JIT) compilation](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#just-in-time-compilation) . JIT compilation slightly increases cuFFT plan initialization time, depending on the transform size and the speed of the host CPU (see [Module load driver API](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#module) ) . But the JIT overhead occurs only when a binary code is generated for the first time during plan initialization using one of the [plan creation functions](https://docs.nvidia.com/cuda/cufft/index.html#cufft-setup) . The device driver automatically caches a copy of the generated binary code to avoid repeating the compilation in subsequent invocations. If necessary, `CUDA_CACHE_PATH` or `CUDA_CACHE_MAXSIZE` can be customized to set the cache folder and max size (see detail in [CUDA Environmental Variables](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars) ), but the default settings are fine in general. 2.3. Fourier Transform Types[](#fourier-transform-types "Permalink to this headline") --------------------------------------------------------------------------------------- Apart from the general complex-to-complex (C2C) transform, cuFFT implements efficiently two other types: real-to-complex (R2C) and complex-to-real (C2R). In many practical applications the input vector is real-valued. It can be easily shown that in this case the output satisfies Hermitian symmetry ( \\(X\_{k} = X\_{N - k}^{\\ast}\\), where the star denotes complex conjugation). The converse is also true: for complex-Hermitian input the inverse transform will be purely real-valued. cuFFT takes advantage of this redundancy and works only on the first half of the Hermitian vector. Transform execution functions for single and double-precision are defined separately as: * `cufftExecC2C() / cufftExecZ2Z()` - complex-to-complex transforms for single/double precision. * `cufftExecR2C() / cufftExecD2Z()` - real-to-complex forward transform for single/double precision. * `cufftExecC2R() / cufftExecZ2D()` - complex-to-real inverse transform for single/double precision. Each of those functions demands different input data layout (see [Data Layout](index.html#data-layout) for details). Note Complex-to-real (C2R) transforms accept complex-Hermitian input. For one-dimensional signals, this requires the 0th element (and the \\(\\frac{N}{2}\\)th input if N is even) to be real-valued, i.e. its imaginary part should be zero. For d-dimension signals, this means \\(x\_{(n\_{1},n\_{2},\\ldots,n\_{d})} = x\_{(N\_{1} - n\_{1},N\_{2} - n\_{2},\\ldots,N\_{d} - n\_{d})}^{\\ast}\\). Otherwise, the behavior of the transform is undefined. Also see [Multidimensional Transforms](index.html#multidimensional-transforms) . Functions `cufftXtExec()` and `cufftXtExecDescriptor()` can perform transforms on any of the supported types. ### 2.3.1. Half-precision cuFFT Transforms[](#half-precision-cufft-transforms "Permalink to this headline") Half-precision transforms have the following limitations: * Minimum GPU architecture is SM\_53 * Sizes are restricted to powers of two only * Strides on the real part of real-to-complex and complex-to-real transforms are not supported * More than one GPU is not supported * Transforms spanning more than 4 billion elements are not supported Please refer to `cufftXtMakePlanMany` function for plan creation details. The CUDA Toolkit provides the `cuda_fp16.h` header with types and intrinsic functions for handling half-precision arithmetic. ### 2.3.2. Bfloat16-precision cuFFT Transforms[](#bfloat16-precision-cufft-transforms "Permalink to this headline") cuFFT supports bfloat16 precision using the `nv_bfloat16` data type. Please note that cuFFT utilizes a combination of single- and bfloat16-precision arithmetic operations when computing the FFT in bfloat16 precision. Bfloat16-precision transforms have similar limitations to half-precision transforms: * Minimum GPU architecture is SM\_80 * Sizes are restricted to powers of two only * Strides on the real part of real-to-complex and complex-to-real transforms are not supported * More than one GPU is not supported * Transforms spanning more than 4 billion elements are not supported Please refer to `cufftXtMakePlanMany` function for plan creation details. The CUDA Toolkit provides the `cuda_bf16.h` header with types and intrinsic functions for handling bfloat16-precision arithmetic. 2.4. Data Layout[](#data-layout "Permalink to this headline") --------------------------------------------------------------- In the cuFFT Library, data layout depends strictly on the configuration and the transform type. In the case of general complex-to-complex transform both the input and output data shall be a `cufftComplex`/`cufftDoubleComplex` array in single- and double-precision modes respectively. In C2R mode an input array \\((x\_{1},x\_{2},\\ldots,x\_{\\lfloor\\frac{N}{2}\\rfloor + 1})\\) of only non-redundant complex elements is required. The output array \\((X\_{1},X\_{2},\\ldots,X\_{N})\\) consists of `cufftReal`/`cufftDouble` elements in this mode. Finally, R2C demands an input array \\((X\_{1},X\_{2},\\ldots,X\_{N})\\) of real values and returns an array \\((x\_{1},x\_{2},\\ldots,x\_{\\lfloor\\frac{N}{2}\\rfloor + 1})\\) of non-redundant complex elements. In real-to-complex and complex-to-real transforms the size of input data and the size of output data differ. For out-of-place transforms a separate array of appropriate size is created. For in-place transforms the user should use `padded` data layout. This layout is FFTW compatibile. In the `padded` layout output signals begin at the same memory addresses as the input data. Therefore input data for real-to-complex and output data for complex-to-real must be padded. Expected sizes of input/output data for 1-d transforms are summarized in the table below: | FFT type | input data size | output data size | | --- | --- | --- | | C2C | \\(x\\)`cufftComplex` | \\(x\\)`cufftComplex` | | C2R | \\(\\left\\lfloor \\frac{x}{2} \\right\\rfloor + 1\\)`cufftComplex` | \\(x\\)`cufftReal` | | R2C\* | \\(x\\)`cufftReal` | \\(\\left\\lfloor \\frac{x}{2} \\right\\rfloor + 1\\)`cufftComplex` | The real-to-complex transform is implicitly a forward transform. For an in-place real-to-complex transform where FFTW compatible output is desired, the input size must be padded to \\(\\left( {\\lfloor\\frac{N}{2}\\rfloor + 1} \\right)\\) complex elements. For out-of-place transforms, input and output sizes match the logical transform size \\(N\\) and the non-redundant size \\(\\lfloor\\frac{N}{2}\\rfloor + 1\\), respectively. The complex-to-real transform is implicitly inverse. For in-place complex-to-real FFTs where FFTW compatible output is selected (default padding mode), the input size is assumed to be \\(\\lfloor\\frac{N}{2}\\rfloor + 1\\)`cufftComplex` elements. Note that in-place complex-to-real FFTs may **overwrite** arbitrary imaginary input point values when non-unit input and output strides are chosen. Out-of-place complex-to-real FFT will always **overwrite** input buffer. For out-of-place transforms, input and output sizes match the logical transform non-redundant size \\(\\lfloor\\frac{N}{2}\\rfloor + 1\\) and size \\(N\\), respectively. 2.5. Multidimensional Transforms[](#multidimensional-transforms "Permalink to this headline") ----------------------------------------------------------------------------------------------- Multidimensional DFT map a \\(d\\)\-dimensional array \\(x\_{\\mathbf{n}}\\), where \\(\\mathbf{n} = (n\_{1},n\_{2},\\ldots,n\_{d})\\) into its frequency domain array given by: \\(X\_{\\mathbf{k}} = \\sum\\limits\_{n = 0}^{N - 1}x\_{\\mathbf{n}}e^{-2\\pi i\\frac{\\mathbf{k}\\mathbf{n}}{\\mathbf{N}}}\\) where \\(\\frac{\\mathbf{n}}{\\mathbf{N}} = (\\frac{n\_{1}}{N\_{1}},\\frac{n\_{2}}{N\_{2}},\\ldots,\\frac{n\_{d}}{N\_{d}})\\), and the summation denotes the set of nested summations \\(\\sum\\limits\_{n\_{1} = 0}^{N\_{1} - 1}\\sum\\limits\_{n\_{2} = 0}^{N\_{2} - 1}\\ldots\\sum\\limits\_{n\_{d} = 0}^{N\_{d} - 1}\\) cuFFT supports one-dimensional, two-dimensional and three-dimensional transforms, which can all be called by the same `cufftExec*` functions (see [Fourier Transform Types](index.html#fft-types) ). Similar to the one-dimensional case, the frequency domain representation of real-valued input data satisfies Hermitian symmetry, defined as: \\(x\_{(n\_{1},n\_{2},\\ldots,n\_{d})} = x\_{(N\_{1} - n\_{1},N\_{2} - n\_{2},\\ldots,N\_{d} - n\_{d})}^{\\ast}\\). C2R and R2C algorithms take advantage of this fact by operating only on half of the elements of signal array, namely on: \\(x\_{\\mathbf{n}}\\) for \\(\\mathbf{n} \\in \\{ 1,\\ldots,N\_{1}\\} \\times \\ldots \\times \\{ 1,\\ldots,N\_{d - 1}\\} \\times \\{ 1,\\ldots,\\lfloor\\frac{N\_{d}}{2}\\rfloor + 1\\}\\). The general rules of data alignment described in [Data Layout](index.html#data-layout) apply to higher-dimensional transforms. The following table summarizes input and output data sizes for multidimensional DFTs: | Dims | FFT type | Input data size | Output data size | | --- | --- | --- | --- | | 1D | C2C | \\(\\mathbf{N}\_{1}\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\)`cufftComplex` | | 1D | C2R | \\(\\lfloor\\frac{\\mathbf{N}\_{1}}{2}\\rfloor + 1\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\)`cufftReal` | | 1D | R2C | \\(\\mathbf{N}\_{1}\\)`cufftReal` | \\(\\lfloor\\frac{\\mathbf{N}\_{1}}{2}\\rfloor + 1\\)`cufftComplex` | | 2D | C2C | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\)`cufftComplex` | | 2D | C2R | \\(\\mathbf{N}\_{1}(\\lfloor\\frac{\\mathbf{N}\_{2}}{2}\\rfloor + 1)\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\)`cufftReal` | | 2D | R2C | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\)`cufftReal` | \\(\\mathbf{N}\_{1}(\\lfloor\\frac{\\mathbf{N}\_{2}}{2}\\rfloor + 1)\\)`cufftComplex` | | 3D | C2C | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\mathbf{N}\_{3}\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\mathbf{N}\_{3}\\)`cufftComplex` | | 3D | C2R | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}(\\lfloor\\frac{\\mathbf{N}\_{3}}{2}\\rfloor + 1)\\)`cufftComplex` | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\mathbf{N}\_{3}\\)`cufftReal` | | 3D | R2C | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}\\mathbf{N}\_{3}\\)`cufftReal` | \\(\\mathbf{N}\_{1}\\mathbf{N}\_{2}(\\lfloor\\frac{\\mathbf{N}\_{3}}{2}\\rfloor + 1)\\)`cufftComplex` | For example, static declaration of a three-dimensional array for the output of an out-of-place real-to-complex transform will look like this: cufftComplex odata\[N1\]\[N2\]\[N3/2+1\]; 2.6. Advanced Data Layout[](#advanced-data-layout "Permalink to this headline") --------------------------------------------------------------------------------- The advanced data layout feature allows transforming only a subset of an input array, or outputting to only a portion of a larger data structure. It can be set by calling function: cufftResult cufftPlanMany(cufftHandle \*plan, int rank, int \*n, int \*inembed, int istride, int idist, int \*onembed, int ostride, int odist, cufftType type, int batch); Passing `inembed` or `onembed` set to `NULL` is a special case and is equivalent to passing `n` for each. This is same as the basic data layout and other advanced parameters such as `istride` are ignored. If the advanced parameters are to be used, then all of the advanced interface parameters must be specified correctly. Advanced parameters are defined in units of the relevant data type (`cufftReal`, `cufftDoubleReal`, `cufftComplex`, or `cufftDoubleComplex`). Advanced layout can be perceived as an additional layer of abstraction above the access to input/output data arrays. An element of coordinates `[z][y][x]` in signal number `b` in the batch will be associated with the following addresses in the memory: * 1D `input[ b * idist + x * istride ]` `output[ b * odist + x * ostride ]` * 2D ``input[ b * idist` + (x * inembed[1] + y) * istride ]`` `output[ b * odist + (x * onembed[1] + y) * ostride ]` * 3D `input[ b * idist + ((x * inembed[1] + y) * inembed[2] + z) * istride ]` `output[ b * odist + ((x * onembed[1] + y) * onembed[2] + z) * ostride ]` The `istride` and `ostride` parameters denote the distance between two successive input and output elements in the least significant (that is, the innermost) dimension respectively. In a single 1D transform, if every input element is to be used in the transform, `istride` should be set to \\(1\\); if every other input element is to be used in the transform, then `istride` should be set to \\(2\\). Similarly, in a single 1D transform, if it is desired to output final elements one after another compactly, `ostride` should be set to \\(1\\); if spacing is desired between the least significant dimension output data, `ostride` should be set to the distance between the elements. The `inembed` and `onembed` parameters define the number of elements in each dimension in the input array and the output array respectively. The `inembed[rank-1]` contains the number of elements in the least significant (innermost) dimension of the input data excluding the `istride` elements; the number of total elements in the least significant dimension of the input array is then `istride*inembed[rank-1]`. The `inembed[0]` or `onembed[0]` corresponds to the most significant (that is, the outermost) dimension and is effectively ignored since the `idist` or `odist` parameter provides this information instead. Note that the size of each dimension of the transform should be less than or equal to the `inembed` and `onembed` values for the corresponding dimension, that is `n[i]` ≤ `inembed[i]`, `n[i]` ≤ `onembed[i]`, where \\(i \\in \\{ 0,\\ldots,rank - 1\\}\\). The `idist` and `odist` parameters indicate the distance between the first element of two consecutive batches in the input and output data. 2.7. Streamed cuFFT Transforms[](#streamed-cufft-transforms "Permalink to this headline") ------------------------------------------------------------------------------------------- Every cuFFT plan may be associated with a CUDA stream. Once so associated, all launches of the internal stages of that plan take place through the specified stream. Streaming of cuFFT execution allows for potential overlap between transforms and memory copies. (See the _NVIDIA CUDA Programming Guide_ for more information on streams.) If no stream is associated with a plan, launches take place in `stream(0)`, the default CUDA stream. Note that many plan executions require multiple kernel launches. cuFFT uses private streams internally to sort operations, including event syncrhonization. cuFFT does not guarantee ordering of internal operations, and the order is only preserved with respect to the streams set by the user. As of CUDA 11.2 (cuFFT 10.4.0), `cufftSetStream()` is supported in multiple GPU cases. However, calls to `cufftXtMemcpy()` are still synchronous across multiple GPUs when using streams. In previous versions of cuFFT, `cufftSetStream()` returns an error in the multiple GPU case. Likewise, calling certain multi-GPU functions such as `cufftXtSetCallback()` after setting a stream with `cufftSetStream()` will result in an error (see API functions for more details). Please note that in order to overlap plans using single plan handle user needs to manage work area buffers. Each concurrent plan execution needs it’s exclusive work area. Work area can be set by `cufftSetWorkArea` function. 2.8. Multiple GPU cuFFT Transforms[](#multiple-gpu-cufft-transforms "Permalink to this headline") --------------------------------------------------------------------------------------------------- cuFFT supports using up to sixteen GPUs connected to a CPU to perform Fourier Transforms whose calculations are distributed across the GPUs. An API has been defined to allow users to write new code or modify existing code to use this functionality. Some existing functions such as the creation of a plan using `cufftCreate()` also apply in the multiple GPU case. Multiple GPU routines contain `Xt` in their name. The memory on the GPUs is managed by helper functions `cufftXtMalloc()/cufftXtFree()` and `cufftXtMemcpy()` using the `cudaLibXtDesc` descriptor. Performance is a function of the bandwidth between the GPUs, the computational ability of the individual GPUs, and the type and number of FFT to be performed. The highest performance is obtained using NVLink interconnect ([https://www.nvidia.com/object/nvlink.html](https://www.nvidia.com/object/nvlink.html) ). The second best option is using PCI Express 3.0 between the GPUs and ensuring that both GPUs are on the same switch. Note that multiple GPU execution is not guaranteed to solve a given size problem in a shorter time than single GPU execution. The multiple GPU extensions to cuFFT are built on the extensible cuFFT API. The general steps in defining and executing a transform with this API are: * `cufftCreate()` - create an empty plan, as in the single GPU case * `cufftXtSetGPUs()` - define which GPUs are to be used * Optional: `cufftEstimate{1d,2d,3d,Many}()` - estimate the sizes of the work areas required. These are the same functions used in the single GPU case although the definition of the argument `workSize` reflects the number of GPUs used. * `cufftMakePlan{1d,2d,3d,Many}()` - create the plan. These are the same functions used in the single GPU case although the definition of the argument `workSize` reflects the number of GPUs used. * Optional: `cufftGetSize{1d,2d,3d,Many}()` - refined estimate of the sizes of the work areas required. These are the same functions used in the single GPU case although the definition of the argument `workSize` reflects the number of GPUs used. * Optional: `cufftGetSize()` - check workspace size. This is the same function used in the single GPU case although the definition of the argument `workSize` reflects the number of GPUs used. * Optional: `cufftXtSetWorkArea()` - do your own workspace allocation. * `cufftXtMalloc()` - allocate descriptor and data on the GPUs * `cufftXtMemcpy()` - copy data to the GPUs * `cufftXtExecDescriptorC2C()/cufftXtExecDescriptorZ2Z()` - execute the plan * `cufftXtMemcpy()` - copy data from the GPUs * `cufftXtFree()` - free any memory allocated with `cufftXtMalloc()` * `cufftDestroy()` - free cuFFT plan resources ### 2.8.1. Plan Specification and Work Areas[](#plan-specification-and-work-areas "Permalink to this headline") In the single GPU case a plan is created by a call to `cufftCreate()` followed by a call to `cufftMakePlan*()`. For multiple GPUs, the GPUs to use for execution are identified by a call to `cufftXtSetGPUs()` and this must occur after the call to `cufftCreate()` and prior to the call to `cufftMakePlan*()`. Note that when `cufftMakePlan*()` is called for a single GPU, the work area is on that GPU. In a multiple GPU plan, the returned work area has multiple entries; one value per GPU. That is `workSize` points to a `size_t` array, one entry per GPU. Also the strides and batches apply to the entire plan across all GPUs associated with the plan. Once a plan is locked by a call to `cufftMakePlan*()`, different descriptors may be specified in calls to `cufftXtExecDescriptor*()` to execute the plan on different data sets, but the new descriptors must use the same GPUs in the same order. As in the single GPU case, `cufftEstimateSize{Many,1d,2d,3d}()` and `cufftGetSize{Many,1d,2d,3d}()` give estimates of the work area sizes required for a multiple GPU plan and in this case `workSize` points to a `size_t` array, one entry per GPU. Similarly the actual work size returned by `cufftGetSize()` is a `size_t` array, one entry per GPU in the multiple GPU case. ### 2.8.2. Helper Functions[](#helper-functions "Permalink to this headline") Multiple GPU cuFFT execution functions assume a certain data layout in terms of what input data has been copied to which GPUs prior to execution, and what output data resides in which GPUs post execution. cuFFT provides functions to assist users in manipulating data on multiple GPUs. These must be called after the call to `cufftMakePlan*()`. On a single GPU users may call `cudaMalloc()` and `cudaFree()` to allocate and free GPU memory. To provide similar functionality in the multiple GPU case, cuFFT includes `cufftXtMalloc()` and `cufftXtFree()` functions. The function `cufftXtMalloc()` returns a descriptor which specifies the location of these memories. On a single GPU users may call `cudaMemcpy()` to transfer data between host and GPU memory. To provide similar functionality in the multiple GPU case, cuFFT includes `cufftXtMemcpy()` which allows users to copy between host and multiple GPU memories or even between the GPU memories. All single GPU cuFFT FFTs return output the data in natural order, that is the ordering of the result is the same as if a DFT had been performed on the data. Some Fast Fourier Transforms produce intermediate results where the data is left in a permutation of the natural output. When batch is one, data is left in the GPU memory in a permutation of the natural output. When `cufftXtMemcpy()` is used to copy data from GPU memory back to host memory, the results are in natural order regardless of whether the data on the GPUs is in natural order or permuted. Using `CUFFT_COPY_DEVICE_TO_DEVICE` allows users to copy data from the permuted data format produced after a single transform to the natural order on GPUs. ### 2.8.3. Multiple GPU 2D and 3D Transforms on Permuted Input[](#multiple-gpu-2d-and-3d-transforms-on-permuted-input "Permalink to this headline") For single 2D or 3D transforms on multiple GPUs, when `cufftXtMemcpy()` distributes the data to the GPUs, the array is divided on the X axis. E.G. for two GPUs half of the X dimenson points, for all Y (and Z) values, are copied to each of the GPUs. When the transform is computed, the data are permuted such that they are divided on the Y axis. I.E. half of the Y dimension points, for all X (and Z) values are on each of the GPUs. When cuFFT creates a 2D or 3D plan for a single transform on multiple GPUs, it actually creates two plans. One plan expects input to be divided on the X axis. The other plan expects data to be divided on the Y axis. This is done because many algorithms compute a forward FFT, then perform some point-wise operation on the result, and then compute the inverse FFT. A memory copy to restore the data to the original order would be expensive. To avoid this, `cufftXtMemcpy` and `cufftXtExecDescriptor()` keep track of the data ordering so that the correct operation is used. The ability of cuFFT to process data in either order makes the following sequence possible. * `cufftCreate()` - create an empty plan, as in the single GPU case * `cufftXtSetGPUs()` - define which GPUs are to be used * `cufftMakePlan{1d,2d,3d,Many}()` - create the plan. * `cufftXtMalloc()` - allocate descriptor and data on the GPUs * `cufftXtMemcpy()` - copy data to the GPUs * `cufftXtExecDescriptorC2C()/cufftXtExecDescriptorZ2Z()` - compute the forward FFT * `userFunction()` - modify the data in the frequency domain * `cufftXtExecDescriptorC2C()/cufftXtExecDescriptorZ2Z()` - compute the inverse FFT * Note that it was not necessary to copy/permute the data between execute calls * `cufftXtMemcpy()` - copy data to the host * `cufftXtFree()` - free any memory allocated with `cufftXtMalloc()` * `cufftDestroy()` - free cuFFT plan resources ### 2.8.4. Supported Functionality[](#supported-functionality "Permalink to this headline") Starting with cuFFT version 7.0, a subset of single GPU functionality is supported for multiple GPU execution. Requirements and limitations: * All GPUs must have the same CUDA architecture level and support Unified Virtual Address Space. * On Windows, the GPU boards must be operating in Tesla Compute Cluster (TCC) mode. * For an application that uses the CUDA Driver API, running cuFFT on multiple GPUs is only compatible with applications using the primary context on each GPU. * Strided input and output are not supported. * Running cuFFT on more than 8 GPUs (16 GPUs is max) is supported on machines with NVLink only. While transforms with batch count greater than one do not impose additional constraints, those with a single batch have some restrictions. Single-batch FFTs support only in-place mode, and have additional constraints depending on the FFT type. This behavior is summarized in the following table: | batch=1 | 1D | 2D | 3D | | --- | --- | --- | --- | | `C2C`/`Z2Z` | * 2,4,8,16 GPUs

* power of 2 sizes only

* Minimum size for 2-4 GPUs is 64

* Minimum size for 8 GPUs is 128

* Minimum size for 16 GPUs is 1024 | * 2-16 GPUs

* One of the following conditions is met for each dimension:

* Dimension must factor into primes less than or equal to 127

* Maximum dimension size is 4096 for single precision

* Maximum dimension size is 2048 for double precision

* Minimum size is 32

* No LTO callback support | | | `R2C`/`D2Z` | not supported | * 2-16 GPUs

* One of the following conditions is met for each dimension:

* Dimension must factor into primes less than or equal to 127

* Maximum dimension size is 4096 for single precision

* Maximum dimension size is 2048 for double precision

* Minimum size is 32

* Fastest changing dimension size needs to be even

* Supports only `CUFFT_XT_FORMAT_INPLACE` input descriptor format

* No legacy callback / LTO callback support | | | `C2R`/`Z2D` | not supported | * 2-16 GPUs

* One of the following conditions is met for each dimension:

* Dimension must factor into primes less than or equal to 127

* Maximum dimension size is 4096 for single precision

* Maximum dimension size is 2048 for double precision

* Minimum size is 32

* Fastest changing dimension size needs to be even

* Supports only `CUFFT_XT_FORMAT_INPLACE_SHUFFLED` input descriptor format

* No legacy callback / LTO callback support | | General guidelines are: * Parameter `whichGPUs` of `cufftXtSetGPUs()` function determines ordering of the GPUs with respect to data decomposition (first data chunk is placed on GPU denoted by first element of `whichGPUs`) * The data for the entire transform must fit within the memory of the GPUs assigned to it. * For batch size `m` on `n` GPUs : * The first `m % n` GPUs execute \\(\\left\\lfloor \\frac{m}{n} \\right\\rfloor+\\ 1\\) transforms. * The remaining GPUs execute \\(\\left\\lfloor \\frac{m}{n} \\right\\rfloor\\) transforms. Batch size output differences: Single GPU cuFFT results are always returned in natural order. When multiple GPUs are used to perform more than one transform, the results are also returned in natural order. When multiple GPUs are used to perform a single transform the results are returned in a permutation of the normal results to reduce communication time. This behavior is summarized in the following table: | Number of GPUs | Number of transforms | Output Order on GPUs | | --- | --- | --- | | One | One or multiple transforms | Natural order | | Multiple | One | Permuted results | | Multiple | Multiple | Natural order | To produce natural order results in GPU memory for multi-GPU runs in the 1D single transform case, requires calling `cufftXtMemcpy()` with `CUFFT_COPY_DEVICE_TO_DEVICE`. 2D and 3D multi-GPU transforms support execution of a transform given permuted order results as input. After execution in this case, the output will be in natural order. It is also possible to use `cufftXtMemcpy()` with `CUFFT_COPY_DEVICE_TO_DEVICE` to return 2D or 3D data to natural order. See the cuFFT Code Examples section for single GPU and multiple GPU examples. 2.9. cuFFT Callback Routines[](#cufft-callback-routines "Permalink to this headline") --------------------------------------------------------------------------------------- Callback routines are user-supplied kernel routines that cuFFT will call when loading or storing data. They allow the user to do data pre- or post- processing without additional kernel calls. Note In CUDA 12.6 Update 2, we introduced support for Link-Time Optimized (LTO) callbacks as a replacement for the deprecated (legacy) callbacks. See more in [LTO Load and Store Callback Routines](index.html#lto-load-and-store-callback-routines) . Starting from CUDA 11.4, support for callback functionality using separately compiled device code (i.e. legacy callbacks) is deprecated on all GPU architectures. Callback functionality will continue to be supported for all GPU architectures. ### 2.9.1. Overview of the cuFFT Callback Routine Feature[](#overview-of-the-cufft-callback-routine-feature "Permalink to this headline") cuFFT provides a set of APIs that allow the cuFFT user to provide CUDA functions that re-direct or manipulate the data as it is loaded prior to processing the FFT, or stored once the FFT has been done. For the load callback, cuFFT calls the callback routine the address of the input data and the offset to the value to be loaded from device memory, and the callback routine returns the value it wishes cuFFT to use instead. For the store callback, cuFFT calls the callback routine the value it has computed, along with the address of the output data and the offset to the value to be written to device memory, and the callback routine modifies the value and stores the modified result. In order to provide a callback to cuFFT, a plan is created using the extensible plan APIs. After the call to `cufftCreate`, the user may associate a load callback routine, or a store callback routine, or both, with the plan, by: * Calling `cufftXtSetJITCallback` before `cufftMakePlan`, for LTO callbacks * Calling `cufftXtSetCallback` after `cufftMakePlan`, for legacy callbacks The caller also has the option to specify a device pointer to an opaque structure they wish to associate with the plan. This pointer will be passed to the callback routine by the cuFFT library. The caller may use this structure to remember plan dimensions and strides, or have a pointer to auxiliary data, etc. With some restrictions, the callback routine is allowed to request shared memory for its own use. If the requested amount of shared memory is available, cufft will pass a pointer to it when it calls the callback routine. CUFFT allows for 8 types of callback routines, one for each possible combination of: load or store, real or complex, single precision or double: * For LTO callbacks, the user must provide an LTO routine that matches the function prototype for the type of routine specified. Otherwise, the planning function `cufftMakePlan` **will fail**. * For legacy callbacks, it is the caller’s responsibility to provide a routine that matches the function prototype for the type of routine specified. If there is already a callback of the specified type associated with the plan handle, the set callback functions will replace it with the new one. The callback routine extensions to cuFFT are built on the extensible cuFFT API. The general steps in defining and executing a transform with callbacks are: * `cufftCreate()` - create an empty plan, as in the single GPU case. * (For **LTO** callbacks) `cufftXtSetJITCallback()` - set a load and/or store LTO callback for this plan. * `cufftMakePlan{1d,2d,3d,Many}()` - create the plan. These are the same functions used in the single GPU case. * (For **legacy** callbacks) `cufftXtSetCallback()` - set a load and/or store legacy callback for this plan. * `cufftExecC2C() etc.` - execute the plan. * `cufftDestroy()` - free cuFFT plan resources. Callback functions are not supported on transforms with a dimension size that does not factor into primes smaller than 127. Callback functions on plans whose dimensions’ prime factors are limited to 2, 3, 5, and 7 can safely call `__syncthreads()`. On other plans, results are not defined. Note The LTO callback API is available in the dynamic and static cuFFT libraries on 64 bit Windows and LINUX operating systems. The LTO callback API requires compatible nvJitLink and NVRTC libraries present in the dynamic library path. See more details in [LTO Load and Store Callback Routines](index.html#lto-load-and-store-callback-routines) . The legacy callback API is available only in the static cuFFT library on 64 bit LINUX operating systems. ### 2.9.2. LTO Load and Store Callback Routines[](#lto-load-and-store-callback-routines "Permalink to this headline") LTO callbacks in cuFFT for a given toolkit version require using the [nvJitLink library](https://docs.nvidia.com/cuda/nvjitlink/index.html) from the same toolkit or greater, but within the same toolkit major. Additionally, in order to specify custom names for the LTO callback routines, cuFFT requires using the [NVRTC library](https://docs.nvidia.com/cuda/nvrtc/index.html) . cuFFT uses NVRTC to compile a minimal wrapper around the user callback with custom symbol name. The custom symbol name provided to the cuFFT API must be a valid, null-terminated C-string containing the unmangled name; currently, keywords that alter the scope of the symbol name (such as `namespace`) or the mangling (such as `extern "C"`) are not supported. The NVRTC library used must be from a toolkit that is either the same version or older than the nvJitLink library, and both must be from the same toolkit major. For example, in toolkit version 12.6 cuFFT requires nvJitLink to be from toolkit version 12.X, where `X >= 6`, and NVRTC to be from toolkit version 12.Y, where `0 <= Y <= X`. Both the nvJitLink and the NVRTC libraries are loaded dynamically, and should be present in the system’s dynamic linking path (e.g. `LD_LIBRARY_PATH` on Unix systems, or `PATH` on Windows systems). Code samples for LTO callbacks are available in the public [CUDA Library Samples github repository](https://github.com/NVIDIA/CUDALibrarySamples/tree/master/cuFFT) . #### 2.9.2.1. Specifying LTO Load and Store Callback Routines[](#specifying-lto-load-and-store-callback-routines "Permalink to this headline") Usage of LTO callbacks in cuFFT is divided in two parts: * Generating the LTO callback (i.e. compiling the callback routine to LTO-IR). * Associating the LTO callback with the cuFFT plan. **To generate** the LTO callback, users can compile the callback device function to LTO-IR using nvcc with any of the supported flags (such as `-dlto` or `-gencode=arch=compute_XX,code=lto_XX`, with `XX` indicating the target GPU architecture); alternatively, users can generate the LTO callback using NVRTC to do runtime compilation via the `-dlto` flag. Notice that PTX JIT is part of the JIT LTO kernel finalization trajectory, so architectures older than the current system architecture are supported; users can compile their callback function to LTO-IR for target arch `XX` and and execute plans which use the callback functions on GPUs with arch `YY`, where `XX <= YY`. Please see [Compiler Support for Runtime LTO Using nvJitLink Library](https://developer.nvidia.com/blog/cuda-12-0-compiler-support-for-runtime-lto-using-nvjitlink-library/) and [Just-in-Time (JIT) Compilation](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#just-in-time-compilation) for more details. As an example, if a user wants to specify a load callback for a R2C transform, they could write the following code \_\_device\_\_ cufftReal myOwnLTOCallback(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPtr) { cufftReal ret; // use offset, dataIn, and optionally callerInfo to // compute the return value return ret; } To compile the callback to LTO-IR, the user could do \# Compile the code to SM60 LTO-IR into a fatbin file nvcc \-gencode\=arch\=compute\_60,code\=lto\_60 \-dc \-fatbin callback.cu \-o callback.fatbin #Turn the fatbin data into a C array inside a header, for easy inclusion in host code bin2c \--name my\_lto\_callback\_fatbin \--type longlong callback.fatbin \> callback\_fatbin.h **To associate** the LTO callback with the cuFFT plan, users can leverage the new API call `cufftXtSetJITCallback()`, which works similarly to `cufftXtSetCallback()`, with a few caveats. First, `cufftXtSetJITCallback()` must be called after plan creation with `cufftCreate()`, and before calling the plan initialization function with `cufftMakePlan*()` and similar routines. Second, removing the LTO callback from the plan (using `cufftXtClearCallback()`) is currently not supported. A new plan must be created. #include #include "callback\_fatbin.h" int main() { cufftResult status; cufftHandle fft\_plan; ... status \= cufftCreate(&fft\_plan); // NOTE: LTO callbacks must be set before plan creation and cannot be unset (yet) size\_t lto\_callback\_fatbin\_size \= sizeof(my\_lto\_callback\_fatbin); status \= cufftXtSetJITCallback(fft\_plan, "myOwnLTOCallback", (void\*)my\_lto\_callback\_fatbin, lto\_callback\_fatbin\_size, CUFFT\_CB\_LD\_REAL, (void \*\*)&device\_params)); status \= cufftMakePlan1d(fft\_plan, signal\_size, CUFFT\_C2R, batches, &work\_size); ... } #### 2.9.2.2. LTO Callback Routine Function Details[](#lto-callback-routine-function-details "Permalink to this headline") Below are the function prototypes for the user-supplied LTO callback routines that cuFFT calls to load data prior to the transform. typedef cufftComplex (\*cufftJITCallbackLoadC)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleComplex (\*cufftJITCallbackLoadZ)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftReal (\*cufftJITCallbackLoadR)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleReal (\*cufftJITCallbackLoadD)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); Parameters for all of the LTO load callbacks are defined as below: * `offset`: offset of the input element from the start of input data. This is not a byte offset, rather it is the number of elements from start of data. * `dataIn`: device pointer to the start of the input array that was passed in the `cufftExecute` call. * `callerInfo`: device pointer to the optional caller specified data passed in the `cufftXtSetCallback` call. * `sharedPointer`: pointer to shared memory, valid only if the user has called `cufftXtSetCallbackSharedSize()`. Below are the function prototypes, and typedefs for pointers to the user supplied LTO callback routines that cuFFT calls to store data after completion of the transform. Note that the store callback functions do not return a value. This is because a store callback function is responsible not only for transforming the data as desired, but also for writing the data to the desired location. This allows the store callback to rearrange the data, for example to shift the zero frequency result to the center of the ouput. typedef void (\*cufftJITCallbackStoreC)(void \*dataOut, unsigned long long offset, cufftComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreZ)(void \*dataOut, unsigned long long offset, cufftDoubleComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreR)(void \*dataOut, unsigned long long offset, cufftReal element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreD)(void \*dataOut, unsigned long long offset, cufftDoubleReal element, void \*callerInfo, void \*sharedPointer); Parameters for all of the LTO store callbacks are defined as below: * `offset`: offset of the output element from the start of output data. This is not a byte offset, rather it is the number of elements from start of data. * `dataOut`: device pointer to the start of the output array that was passed in the `cufftExecute` call. * `element`: the real or complex result computed by CUFFT for the element specified by the offset argument. * `callerInfo`: device pointer to the optional caller specified data passed in the `cufftXtSetCallback` call. * `sharedPointer`: pointer to shared memory, valid only if the user has called `cufftXtSetCallbackSharedSize()`. ### 2.9.3. Legacy Load and Store Callback Routines[](#legacy-load-and-store-callback-routines "Permalink to this headline") #### 2.9.3.1. Specifying Legacy Load and Store Callback Routines[](#specifying-legacy-load-and-store-callback-routines "Permalink to this headline") In order to associate a legacy callback routine with a plan, it is necessary to obtain a device pointer to the callback routine. As an example, if the user wants to specify a load callback for an R2C transform, they would write the device code for the callback function, and define a global device variable that contains a pointer to the function: \_\_device\_\_ cufftReal myOwnCallback(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPtr) { cufftReal ret; // use offset, dataIn, and optionally callerInfo to // compute the return value return ret; } \_\_device\_\_ cufftCallbackLoadR myOwnCallbackPtr \= myOwnCallback; From the host side, the user then has to get the address of the legacy callback routine, which is stored in `myOwnCallbackPtr`. This is done with `cudaMemcpyFromSymbol`, as follows: cufftCallbackLoadR hostCopyOfCallbackPtr; cudaMemcpyFromSymbol(&hostCopyOfCallbackPtr, myOwnCallbackPtr, sizeof(hostCopyOfCallbackPtr)); `hostCopyOfCallbackPtr` then contains the device address of the callback routine, that should be passed to `cufftXtSetCallback`. Note that, for multi-GPU transforms, `hostCopyOfCallbackPtr` will need to be an array of pointers, and the `cudaMemcpyFromSymbol` will have to be invoked for each GPU. Please note that `__managed__` variables are not suitable to pass to `cufftSetCallback` due to restrictions on variable usage (See the _NVIDIA CUDA Programming Guide_ for more information about `__managed__` variables). #### 2.9.3.2. Legacy Callback Routine Function Details[](#legacy-callback-routine-function-details "Permalink to this headline") Below are the function prototypes, and typedefs for pointers to the user supplied legacy callback routines that cuFFT calls to load data prior to the transform. typedef cufftComplex (\*cufftCallbackLoadC)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleComplex (\*cufftCallbackLoadZ)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftReal (\*cufftCallbackLoadR)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleReal (\*cufftCallbackLoadD)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); Parameters for all of the legacy load callbacks are defined as below: * `offset`: offset of the input element from the start of input data. This is not a byte offset, rather it is the number of elements from start of data. * `dataIn`: device pointer to the start of the input array that was passed in the `cufftExecute` call. * `callerInfo`: device pointer to the optional caller specified data passed in the `cufftXtSetCallback` call. * `sharedPointer`: pointer to shared memory, valid only if the user has called `cufftXtSetCallbackSharedSize()`. Below are the function prototypes, and typedefs for pointers to the user supplied legacy callback routines that cuFFT calls to store data after completion of the transform. Note that the store callback functions do not return a value. This is because a store callback function is responsible not only for transforming the data as desired, but also for writing the data to the desired location. This allows the store callback to rearrange the data, for example to shift the zero frequency result to the center of the ouput. typedef void (\*cufftCallbackStoreC)(void \*dataOut, size\_t offset, cufftComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreZ)(void \*dataOut, size\_t offset, cufftDoubleComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreR)(void \*dataOut, size\_t offset, cufftReal element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreD)(void \*dataOut, size\_t offset, cufftDoubleReal element, void \*callerInfo, void \*sharedPointer); Parameters for all of the legacy store callbacks are defined as below: * `offset`: offset of the output element from the start of output data. This is not a byte offset, rather it is the number of elements from start of data. * `dataOut`: device pointer to the start of the output array that was passed in the `cufftExecute` call. * `element`: the real or complex result computed by CUFFT for the element specified by the offset argument. * `callerInfo`: device pointer to the optional caller specified data passed in the `cufftXtSetCallback` call. * `sharedPointer`: pointer to shared memory, valid only if the user has called `cufftXtSetCallbackSharedSize()`. ### 2.9.4. Coding Considerations for the cuFFT Callback Routine Feature[](#coding-considerations-for-the-cufft-callback-routine-feature "Permalink to this headline") cuFFT supports callbacks on all types of transforms, dimension, batch, or stride between elements. Callbacks are supported for transforms of single and double precision. cuFFT supports a wide range of parameters, and based on those for a given plan, it attempts to optimize performance. The number of kernels launched, and for each of those, the number of blocks launched and the number of threads per block, will vary depending on how cuFFT decomposes the transform. For some configurations, cuFFT will load or store (and process) multiple inputs or outputs per thread. For some configurations, threads may load or store inputs or outputs in any order, and cuFFT does not guarantee that the inputs or outputs handled by a given thread will be contiguous. These characteristics may vary with transform size, transform type (e.g. C2C vs C2R), number of dimensions, and GPU architecture. These variations may also change from one library version to the next. When more than one kernel are used to implement a transform, the thread and block structure of the first kernel (the one that does the load) is often different from the thread and block structure of the last kernel (the one that does the store). One common use of callbacks is to reduce the amount of data read or written to memory, either by selective filtering or via type conversions. When more than one kernel are used to implement a transform, cuFFT alternates using the workspace and the output buffer to write intermediate results. This means that the output buffer must always be large enough to accommodate the entire transform. For transforms whose dimensions can be factored into powers of 2, 3, 5, or 7, cuFFT guarantees that it will call the load and store callback routines from points in the kernel where it is safe to call the `__syncthreads` function from within the callback routine. The caller is responsible for guaranteeing that the callback routine is at a point where the callback code has converged, to avoid deadlock. For plans whose dimensions are factored into higher primes, results of a callback routine calling `__syncthreads` are not defined. Note that there are no guarantees on the relative order of execution of blocks within a grid. As such, callbacks should not rely on any particular ordering within a kernel. For instance, reordering data (such as an FFT-shift) could rely on the order of execution of the blocks. Results in this case would be undefined. #### 2.9.4.1. Coding Considerations for LTO Callback Routines[](#coding-considerations-for-lto-callback-routines "Permalink to this headline") cuFFT will call the LTO load callback routine, for each point in the input, once and only once for real-to-complex (`R2C`, `D2Z`) and complex-to-complex (`C2C`, `Z2Z`) transforms. Unlike with legacy callbacks, **LTO load callbacks may be called more than once per element for complex-to-real** (`C2R`, `Z2D`) transforms. The input value will not be updated twice (i.e. the transformed value will be stored in register and not memory, even for in-place transforms), but users should not rely on the amount of calls per element in their callback device functions. Similarly to legacy callbacks, LTO store callbacks will be called once and only once for each point in the output. If the transform is being done in-place (i.e. the input and output data are in the same memory location) the store callback for a given element cannot overwrite other elements. It can either overwrite the given element, or write in a completely distinct output buffer. cuFFT does not support LTO callbacks for multi-GPU transforms (yet). #### 2.9.4.2. Coding Considerations for Legacy Callback Routines[](#coding-considerations-for-legacy-callback-routines "Permalink to this headline") cuFFT supports legacy callbacks on any number of GPUs. cuFFT will call the load callback routine, for each point in the input, once and only once. Similarly it will call the store callback routine, for each point in the output, once and only once. If the transform is being done in-place (i.e. the input and output data are in the same memory location) the store callback for a given element cannot overwrite other elements. It can either overwrite the given element, or write in a completely distinct output buffer. For multi-GPU transforms, the index passed to the callback routine is the element index from the start of data _on that GPU_, not from the start of the entire input or output data array. 2.10. Thread Safety[](#thread-safety "Permalink to this headline") -------------------------------------------------------------------- cuFFT APIs are thread safe as long as different host threads execute FFTs using different plans and the output data are disjoint. 2.11. CUDA Graphs Support[](#cuda-graphs-support "Permalink to this headline") -------------------------------------------------------------------------------- Using [CUDA Graphs](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs) with cuFFT is supported on single GPU plans. It is also supported on multiple GPU plans starting with cuFFT version 10.4.0. The stream associated with a cuFFT plan must meet the requirements stated in [Creating a Graph Using Stream Capture](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#creating-a-graph-using-stream-capture) . Note Starting from CUDA 11.8 (including CUDA 12.0 onward), CUDA Graphs are no longer supported for legacy callback routines that load data in out-of-place mode transforms. Starting from CUDA 12.6 Update 2, LTO callbacks can be used as a replacement for legacy callbacks without this limitation. cuFFT deprecated callback functionality based on separate compiled device code (legacy callbacks) in cuFFT 11.4. 2.12. Static Library and Callback Support[](#static-library-and-callback-support "Permalink to this headline") ---------------------------------------------------------------------------------------------------------------- Starting with release 6.5, the cuFFT libraries are also delivered in a static form as libcufft\_static.a and libcufftw\_static.a on Linux and Mac. Static libraries are not supported on Windows. The static cufft and cufftw libraries depend on thread abstraction layer library `libculibos.a`. For example, on linux, to compile a small application using cuFFT against the dynamic library, the following command can be used: nvcc mCufftApp.c -lcufft -o myCufftApp For cufftw on Linux, to compile a small application against the dynamic library, the following command can be used: nvcc mCufftwApp.c -lcufftw -lcufft -o myCufftwApp Whereas to compile against the static cuFFT library, extra steps need to be taken. The library needs to be device linked. It may happen during building and linking of a simple program, or as a separate step. The entire process is described in [Using Separarate Compilation in CUDA](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#using-separate-compilation-in-cuda) . For cuFFT and cufftw in version 9.0 or later any supported architecture can be used to do the device linking: Static cuFFT compilation command: nvcc mCufftApp.c -lcufft\_static -lculibos -o myCufftApp Static cufftw compilation command: nvcc mCufftwApp.c -lcufftw\_static -lcufft\_static -lculibos -o myCufftwApp Prior to version 9.0 proper linking required specifying a subset of supported architectures, as shown in the following commands: Static cuFFT compilation command: nvcc mCufftApp.c -lcufft\_static -lculibos -o myCufftApp\\ -gencode arch=compute\_20,\\"code=sm\_20\\"\\ -gencode arch=compute\_30,\\"code=sm\_30\\"\\ -gencode arch=compute\_35,\\"code=sm\_35\\"\\ -gencode arch=compute\_50,\\"code=sm\_50\\"\\ -gencode arch=compute\_60,\\"code=sm\_60\\"\\ -gencode arch=compute\_60,\\"code=compute\_60\\" Static cufftw compilation command: nvcc mCufftwApp.c -lcufftw\_static -lcufft\_static -lculibos -o myCufftwApp\\ -gencode arch=compute\_20,\\"code=sm\_20\\"\\ -gencode arch=compute\_30,\\"code=sm\_30\\"\\ -gencode arch=compute\_35,\\"code=sm\_35\\"\\ -gencode arch=compute\_50,\\"code=sm\_50\\"\\ -gencode arch=compute\_60,\\"code=sm\_60\\"\\ -gencode arch=compute\_60,\\"code=compute\_60\\" Please note that the cuFFT library might not contain code for certain architectures as long as there is code for a lower architecture that is binary compatibile (e.g. SM52, SM61). This is reflected in link commands above and significant when using versions prior r9.0. To determine if a specific SM is included in the cuFFT library, one may use `cuobjdump` utility. For example, if you wish to know if SM\_50 is included, the command to run is `cuobjdump -arch sm_50 libcufft_static.a`. Some kernels are built only on select architectures (e.g. kernels with half precision arithmetics are present only for SM53 and above). This can cause warnings at link time that architectures are missing from these kernels. These warnings can be safely ignored. It is also possible to use the native Host C++ compiler and perform device link as a separate step. Please consult NVCC documentation for more details. Depending on the Host Operating system, some additional libraries like `pthread` or `dl` might be needed on the linking line. Note that in this case, the library `cuda` is not needed. The CUDA Runtime will try to open explicitly the `cuda` library if needed. In the case of a system which does not have the CUDA driver installed, this allows the application to gracefully manage this issue and potentially run if a CPU-only path is available. The cuFFT static library supports user supplied legacy callback routines. The legacy callback routines are CUDA device code, and must be separately compiled with NVCC and linked with the cuFFT library. Please refer to the NVCC documentation regarding separate compilation for details. If you specify an SM when compiling your callback functions, you must specify one of the SM’s cuFFT includes. ### 2.12.1. Static library without legacy callback support[](#static-library-without-legacy-callback-support "Permalink to this headline") Starting with cuFFT version 9.2, a new variant of the cuFTT static library, `libcufft_static_nocallback.a`, was added. This new version does not contain legacy callback functionality and can be linked using the host compiler only. Note Starting from CUDA 12.8, support for the cuFFT binary with no callback support, `libcufft_static_nocallback.a`, is deprecated and will be removed in a future release. Users are encouraged to update their build to use `libcufft_static.a` instead, as the two binaries are nearly identical after the release of LTO callbacks as part of cuFFT in CUDA Toolkit 12.6 Update 2. 2.13. Accuracy and Performance[](#accuracy-and-performance "Permalink to this headline") ------------------------------------------------------------------------------------------ A DFT can be implemented as a matrix vector multiplication that requires \\(O(N^{2})\\) operations. However, the cuFFT Library employs the [Cooley-Tukey algorithm](http://en.wikipedia.org/wiki/Cooley-Tukey_FFT_algorithm) to reduce the number of required operations to optimize the performance of particular transform sizes. This algorithm expresses the DFT matrix as a product of sparse building block matrices. The cuFFT Library implements the following building blocks: radix-2, radix-3, radix-5, and radix-7. Hence the performance of any transform size that can be factored as \\(2^{a} \\times 3^{b} \\times 5^{c} \\times 7^{d}\\) (where _a_, _b_, _c_, and _d_ are non-negative integers) is optimized in the cuFFT library. There are also radix-m building blocks for other primes, m, whose value is < 128. When the length cannot be decomposed as multiples of powers of primes from 2 to 127, [Bluestein’s algorithm](http://en.wikipedia.org/wiki/Bluestein's_FFT_algorithm) is used. Since the Bluestein implementation requires more computations per output point than the Cooley-Tukey implementation, the accuracy of the Cooley-Tukey algorithm is better. The pure Cooley-Tukey implementation has excellent accuracy, with the relative error growing proportionally to \\(\\log\_{2}(N)\\) , where \\(N\\) is the transform size in points. For sizes handled by the Cooley-Tukey code path, the most efficient implementation is obtained by applying the following constraints (listed in order from the most generic to the most specialized constraint, with each subsequent constraint providing the potential of an additional performance improvement). Half precision transforms might not be suitable for all kinds of problems due to limited range represented by half precision floating point arithmetics. Please note that the first element of FFT result is the sum of all input elements and it is likely to overflow for certain inputs. Results produced by the cuFFT library are deterministic (ie, bitwise reproducible) as long as the following are kept constant between runs: plan input parameters, cuFFT version, and GPU model. cuFFT batched plans require that input data includes valid signal for all batches. Performance optimizations in batched mode can combine signal from different batches for processing. Optimizations used in cuFFT can vary from version to version. | Applies to | Recommendation | Comment | | --- | --- | --- | | All | Use single precision transforms. | Single precision transforms require less bandwidth per computation than double precision transforms. | | All | Restrict the size along all dimensions to be representable as \\(2^{a} \\times 3^{b} \\times 5^{c} \\times 7^{d}\\). | The cuFFT library has highly optimized kernels for transforms whose dimensions have these prime factors. In general the best performance occurs when using powers of 2, followed by powers of 3, then 5, 7. | | All | Restrict the size along each dimension to use fewer distinct prime factors. | A transform of size \\(2^{n}\\) or \\(3^{n}\\) will usually be faster than one of size \\(2^{i} \\times 3^{j}\\) even if the latter is slightly smaller, due to the composition of specialized paths. | | All | Restrict the data to be contiguous in memory when performing a single transform. When performing multiple transforms make the individual datasets contiguous | The cuFFT library has been optimized for this data layout. | | All | Perform multiple (i.e., batched) transforms. | Additional optimizations are performed in batched mode. | | real-to-complex transforms or complex-to-real transforms | Ensure problem size of x dimension is a multiple of 4. | This scheme uses more efficient kernels to implement conjugate symmetry property. | | real-to-complex transforms or complex-to-real transforms | Use `out-of-place` mode. | This scheme uses more efficient kernels than `in-place` mode. | | Multiple GPU transforms | Use PCI Express 3.0 between GPUs and ensure the GPUs are on the same switch. | The faster the interconnect between the GPUs, the faster the performance. | 2.14. Caller Allocated Work Area Support[](#caller-allocated-work-area-support "Permalink to this headline") -------------------------------------------------------------------------------------------------------------- cuFFT plans may use additional memory to store intermediate results. The cuFFT library offers several functions to manage this temporary memory utilization behavior: * `cufftSetAutoAllocation` * `cufftEstimate1d`, `cufftEstimate2d`, `cufftEstimate3d` and `cufftEstimateMany` * `cufftGetSize` * `cufftXtSetWorkAreaPolicy` The first two functions manage allocation and ownership of temporary memory. By default cuFFT always allocates its own work area in GPU memory. Each cuFFT handle allocates data separately. If multiple cuFFT plans are to be launched sequentially it is possible to assign the same memory chunk as work area to all those plans and reduce memory overhead. The memory assigned as work area needs to be GPU visible. In addition to the regular memory acquired with `cudaMalloc`, usage of CUDA Unified Virtual Addressing enables cuFFT to use the following types of memory as work area memory: pinned host memory, managed memory, memory on GPU other than the one performing the calculations. While this provides flexibility, it comes with a performance penalty whose magnitude depends on the available memory bandwidth. The `cufftEstimateNd`, `cufftEstimateMany`, and `cufftGetSize` functions provide information about the required memory size for cases where the user is allocating the work space buffer. In version 9.2 cuFFT also introduced the `cufftXtSetWorkAreaPolicy` function. This function allows fine tuning of work area memory usage. cuFFT 9.2 version supports only the `CUFFT_WORKAREA_MINIMAL` policy, which instructs cuFFT to re-plan the existing plan without the need to use work area memory. Also as of cuFFT 9.2, supported FFT transforms that allow for `CUFFT_WORKAREA_MINIMAL` policy are as follows: * Transforms of type `C2C` are supported with sizes up to 4096 in any dimension. * Transforms of type `Z2Z` are supported with sizes up to 2048 in any dimension. * Only single GPU transforms are supported. Depending on the FFT transform size, a different FFT algorithm may be used when the `CUFFT_WORKAREA_MINIMAL` policy is set. 2.15. cuFFT Link-Time Optimized Kernels[](#cufft-link-time-optimized-kernels "Permalink to this headline") ------------------------------------------------------------------------------------------------------------ Starting from CUDA 12.4, cuFFT ships Link-Time Optimized (LTO) kernels. These kernels are linked and finalized at runtime as part of the cuFFT planning routines. This enables the cuFFT library to generate kernels optimized for the underlying architecture and the specific problem to solve. The current LTO kernel coverage includes: * Kernels for 64-bit addressing (with FFTs spanning addresses greater than 2^(32)-1 elements). * Some single- and double-precision R2C and C2R sizes. The number and coverage of LTO kernels will grow with future releases of cuFFT. We encourage our users to test whether LTO kernels improve the performance for their use case. Users can opt-in into LTO kernels by setting the `NVFFT_PLAN_PROPERTY_INT64_PATIENT_JIT` plan property using the `cufftSetPlanProperty` routine. In order to finalize LTO kernels, cuFFT relies on the nvJitLink library that ships as part of the CUDA Toolkit. Finalizing the kernels at runtime can cause an **increase in planning time** (which could be in the order of hundreds of milliseconds, depending on the cuFFT plan and hardware characteristics of the host system), in exchange for faster execution time of the optimized kernels. Note that nvJitLink caches kernels linked at runtime to speed-up subsequent kernel finalizations in repeated planning routines. If for any reason the runtime linking of the kernel fails, cuFFT will fall back to offline-compiled kernels to compute the FFT. Note cuFFT LTO kernels for a given toolkit version require using the nvJitLink library from the same toolkit or greater, but within the same toolkit major. For example, cuFFT in 12.4 requires nvJitLink to be from a CUDA Toolkit 12.X, with `X >= 4`. The nvJitLink library is loaded dynamically, and should be present in the system’s dynamic linking path (e.g. `LD_LIBRARY_PATH` on Unix systems, or `PATH` on Windows systems). 3\. cuFFT API Reference[](#cufft-api-reference "Permalink to this headline") ============================================================================== This chapter specifies the behavior of the cuFFT library functions by describing their input/output parameters, data types, and error codes. The cuFFT library is initialized upon the first invocation of an API function, and cuFFT shuts down automatically when all user-created FFT plans are destroyed. 3.1. Return value cufftResult[](#return-value-cufftresult "Permalink to this headline") ----------------------------------------------------------------------------------------- All cuFFT Library return values except for `CUFFT_SUCCESS` indicate that the current API call failed and the user should reconfigure to correct the problem. The possible return values are defined as follows: typedef enum cufftResult\_t { CUFFT\_SUCCESS \= 0, // The cuFFT operation was successful CUFFT\_INVALID\_PLAN \= 1, // cuFFT was passed an invalid plan handle CUFFT\_ALLOC\_FAILED \= 2, // cuFFT failed to allocate GPU or CPU memory CUFFT\_INVALID\_TYPE \= 3, // No longer used CUFFT\_INVALID\_VALUE \= 4, // User specified an invalid pointer or parameter CUFFT\_INTERNAL\_ERROR \= 5, // Driver or internal cuFFT library error CUFFT\_EXEC\_FAILED \= 6, // Failed to execute an FFT on the GPU CUFFT\_SETUP\_FAILED \= 7, // The cuFFT library failed to initialize CUFFT\_INVALID\_SIZE \= 8, // User specified an invalid transform size CUFFT\_UNALIGNED\_DATA \= 9, // No longer used CUFFT\_INCOMPLETE\_PARAMETER\_LIST \= 10, // Missing parameters in call CUFFT\_INVALID\_DEVICE \= 11, // Execution of a plan was on different GPU than plan creation CUFFT\_PARSE\_ERROR \= 12, // Internal plan database error CUFFT\_NO\_WORKSPACE \= 13 // No workspace has been provided prior to plan execution CUFFT\_NOT\_IMPLEMENTED \= 14, // Function does not implement functionality for parameters given. CUFFT\_LICENSE\_ERROR \= 15, // Used in previous versions. CUFFT\_NOT\_SUPPORTED \= 16 // Operation is not supported for parameters given. } cufftResult; Users are encouraged to check return values from cuFFT functions for errors as shown in [cuFFT Code Examples](index.html#cufft-code-examples) . 3.2. cuFFT Basic Plans[](#cufft-basic-plans "Permalink to this headline") --------------------------------------------------------------------------- These API routines take care of initializing the cufftHandle. Any already-initialized handle attributes passed to the planning functions will be ignored. ### 3.2.1. cufftPlan1d()[](#cufftplan1d "Permalink to this headline") cufftResult cufftPlan1d([cufftHandle](#c.cufftHandle "cufftHandle") \*plan, int nx, cufftType type, int batch);[](#c.cufftPlan1d "Permalink to this definition") Creates a 1D FFT plan configuration for a specified signal size and data type. The `batch` input parameter tells cuFFT how many 1D transforms to configure. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. Parameters * **plan\[In\]** – Pointer to an uninitialized `cufftHandle` object. * **nx\[In\]** – The transform size (e.g. 256 for a 256-point FFT). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2C` for single precision complex to complex). * **batch\[In\]** – Number of transforms of size `nx`. Please consider using `cufftPlanMany` for multiple transforms. * **plan\[Out\]** – Contains a cuFFT 1D plan handle value. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – The `nx` or `batch` parameter is not a supported size. ### 3.2.2. cufftPlan2d()[](#cufftplan2d "Permalink to this headline") cufftResult cufftPlan2d([cufftHandle](#c.cufftHandle "cufftHandle") \*plan, int nx, int ny, cufftType type);[](#c.cufftPlan2d "Permalink to this definition") Creates a 2D FFT plan configuration according to specified signal sizes and data type. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. Parameters * **plan\[In\]** – Pointer to an uninitialized `cufftHandle` object. * **nx\[In\]** – The transform size in the _x_ dimension This is slowest changing dimension of a transform (strided in memory). * **ny\[In\]** – The transform size in the _y_ dimension. This is fastest changing dimension of a transform (contiguous in memory). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2R` for single precision complex to real). * **plan\[Out\]** – Contains a cuFFT 2D plan handle value. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – Either or both of the `nx` or `ny` parameters is not a supported size. ### 3.2.3. cufftPlan3d()[](#cufftplan3d "Permalink to this headline") cufftResult cufftPlan3d([cufftHandle](#c.cufftHandle "cufftHandle") \*plan, int nx, int ny, int nz, cufftType type);[](#c.cufftPlan3d "Permalink to this definition") Creates a 3D FFT plan configuration according to specified signal sizes and data type. This function is the same as `cufftPlan2d()` except that it takes a third size parameter `nz`. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. Parameters * **plan\[In\]** – Pointer to an uninitialized `cufftHandle` object. * **nx\[In\]** – The transform size in the _x_ dimension. This is slowest changing dimension of a transform (strided in memory). * **ny\[In\]** – The transform size in the _y_ dimension. * **nz\[In\]** – The transform size in the _z_ dimension. This is fastest changing dimension of a transform (contiguous in memory). * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **plan\[Out\]** – Contains a cuFFT 3D plan handle value. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the `nx`, `ny`, or `nz` parameters is not a supported size. ### 3.2.4. cufftPlanMany()[](#cufftplanmany "Permalink to this headline") cufftResult cufftPlanMany([cufftHandle](#c.cufftHandle "cufftHandle") \*plan, int rank, int \*n, int \*inembed, int istride, int idist, int \*onembed, int ostride, int odist, cufftType type, int batch);[](#c.cufftPlanMany "Permalink to this definition") Creates a FFT plan configuration of dimension `rank`, with sizes specified in the array `n`. The `batch` input parameter tells cuFFT how many transforms to configure. With this function, batched plans of 1, 2, or 3 dimensions may be created. The `cufftPlanMany()` API supports more complicated input and output data layouts via the advanced data layout parameters: `inembed`, `istride`, `idist`, `onembed`, `ostride`, and `odist`. If `inembed` and `onembed` are set to `NULL`, all other stride information is ignored, and default strides are used. The default assumes contiguous data arrays. All arrays are assumed to be in CPU memory. Please note that behavior of `cufftPlanMany` function when `inembed` and `onembed` is `NULL` is different than corresponding function in FFTW library `fftw_plan_many_dft`. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. Parameters * **plan\[In\]** – Pointer to an uninitialized `cufftHandle` object. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension, `n[0]` being the size of the outermost and `n[rank-1]` innermost (contiguous) dimension of a transform. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **batch\[In\]** – Batch size for this transform. * **plan\[Out\]** – Contains a cuFFT plan handle. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. 3.3. cuFFT Extensible Plans[](#cufft-extensible-plans "Permalink to this headline") ------------------------------------------------------------------------------------- These API routines separates handle creation from plan generation. This makes it possible to change plan settings, which may alter the outcome of the plan generation phase, before the plan is actually generated. ### 3.3.1. cufftCreate()[](#cufftcreate "Permalink to this headline") cufftResult cufftCreate([cufftHandle](#c.cufftHandle "cufftHandle") \*plan)[](#c.cufftCreate "Permalink to this definition") Creates only an opaque handle, and allocates small data structures on the host. The `cufftMakePlan*()` calls actually do the plan generation. Parameters * **plan\[In\]** – Pointer to a `cufftHandle` object. * **plan\[Out\]** – Contains a cuFFT plan handle value. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_ALLOC\_FAILED** – The allocation of resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.3.2. cufftDestroy()[](#cufftdestroy "Permalink to this headline") cufftResult cufftDestroy([cufftHandle](#c.cufftHandle "cufftHandle") plan)[](#c.cufftDestroy "Permalink to this definition") Frees all GPU resources associated with a cuFFT plan and destroys the internal plan data structure. This function should be called once a plan is no longer needed, to avoid wasting GPU memory. In the case of multi-GPU plans, the plan created first should be destroyed last. Parameters * **plan\[In\]** – The `cufftHandle` object of the plan to be destroyed. Return values * **CUFFT\_SUCCESS** – cuFFT successfully destroyed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. ### 3.3.3. cufftMakePlan1d()[](#cufftmakeplan1d "Permalink to this headline") cufftResult cufftMakePlan1d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, cufftType type, int batch, size\_t \*workSize);[](#c.cufftMakePlan1d "Permalink to this definition") Following a call to `cufftCreate()` makes a 1D FFT plan configuration for a specified signal size and data type. The `batch` input parameter tells cuFFT how many 1D transforms to configure. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size (e.g. 256 for a 256-point FFT). For multiple GPUs, this must be a power of 2. * **type\[In\]** – The transform data type (e.g., `CUFFT_C2C` for single precision complex to complex). For multiple GPUs this must be a complex to complex transform. * **batch\[In\]** – Number of transforms of size `nx`. Please consider using `cufftMakePlanMany` for multiple transforms. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size(s) of the work areas. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked or multi-GPU restrictions are not met. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED\`** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – The `nx` or `batch` parameter is not a supported size. ### 3.3.4. cufftMakePlan2d()[](#cufftmakeplan2d "Permalink to this headline") cufftResult cufftMakePlan2d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, int ny, cufftType type, size\_t \*workSize);[](#c.cufftMakePlan2d "Permalink to this definition") Following a call to `cufftCreate()` makes a 2D FFT plan configuration according to specified signal sizes and data type. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size in the _x_ dimension. This is slowest changing dimension of a transform (strided in memory). For multiple GPUs, this must be factorable into primes less than or equal to 127. * **ny\[In\]** – The transform size in the _y_ dimension. This is fastest changing dimension of a transform (contiguous in memory). For 2 GPUs, this must be factorable into primes less than or equal to 127. * **type\[In\]** – The transform data type (e.g., `CUFFT_C2R` for single precision complex to real). * **workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size(s) of the work areas. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – Either or both of the `nx` or `ny` parameters is not a supported size. ### 3.3.5. cufftMakePlan3d()[](#cufftmakeplan3d "Permalink to this headline") cufftResult cufftMakePlan3d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, int ny, int nz, cufftType type, size\_t \*workSize);[](#c.cufftMakePlan3d "Permalink to this definition") Following a call to `cufftCreate()` makes a 3D FFT plan configuration according to specified signal sizes and data type. This function is the same as `cufftPlan2d()` except that it takes a third size parameter `nz`. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size in the _x_ dimension. This is slowest changing dimension of a transform (strided in memory). For multiple GPUs, this must be factorable into primes less than or equal to 127. * **ny\[In\]** – The transform size in the _y_ dimension. For multiple GPUs, this must be factorable into primes less than or equal to 127. * **nz\[In\]** – The transform size in the _z_ dimension. This is fastest changing dimension of a transform (contiguous in memory). For multiple GPUs, this must be factorable into primes less than or equal to 127. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size(s) of the work area(s). Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the `nx`, `ny`, or `nz` parameters is not a supported size. ### 3.3.6. cufftMakePlanMany()[](#cufftmakeplanmany "Permalink to this headline") cufftResult cufftMakePlanMany([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, int \*n, int \*inembed, int istride, int idist, int \*onembed, int ostride, int odist, cufftType type, int batch, size\_t \*workSize);[](#c.cufftMakePlanMany "Permalink to this definition") Following a call to `cufftCreate()` makes a FFT plan configuration of dimension `rank`, with sizes specified in the array `n`. The `batch` input parameter tells cuFFT how many transforms to configure. With this function, batched plans of 1, 2, or 3 dimensions may be created. The `cufftPlanMany()` API supports more complicated input and output data layouts via the advanced data layout parameters: `inembed`, `istride`, `idist`, `onembed`, `ostride`, and `odist`. If `inembed` and `onembed` are set to `NULL`, all other stride information is ignored, and default strides are used. The default assumes contiguous data arrays. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. All arrays are assumed to be in CPU memory. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3) * **n\[In\]** – Array of size `rank`, describing the size of each dimension, `n[0]` being the size of the outermost and `n[rank-1]` innermost (contiguous) dimension of a transform. For multiple GPUs and rank equal to 1, the sizes must be a power of 2. For multiple GPUs and rank equal to 2 or 3, the sizes must be factorable into primes less than or equal to 127. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory, `inembed[0]` being the storage dimension of the outermost dimension. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory, `onembed[0]` being the storage dimension of the outermost dimension. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). For 2 GPUs this must be a complex to complex transform. * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size(s) of the work areas. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked or multi-GPU restrictions are not met. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. ### 3.3.7. cufftMakePlanMany64()[](#cufftmakeplanmany64 "Permalink to this headline") cufftResult cufftMakePlanMany64([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, long long int \*n, long long int \*inembed, long long int istride, long long int idist, long long int \*onembed, long long int ostride, long long int odist, cufftType type, long long int batch, size\_t \*workSize);[](#c.cufftMakePlanMany64 "Permalink to this definition") Following a call to `cufftCreate()` makes a FFT plan configuration of dimension `rank`, with sizes specified in the array `n`. The `batch` input parameter tells cuFFT how many transforms to configure. With this function, batched plans of 1, 2, or 3 dimensions may be created. This API is identical to `cufftMakePlanMany` except that the arguments specifying sizes and strides are 64 bit integers. This API makes very large transforms possible. cuFFT includes kernels that use 32 bit indexes, and kernels that use 64 bit indexes. cuFFT planning selects 32 bit kernels whenever possible to avoid any overhead due to 64 bit arithmetic. All sizes and types of transform are supported by this interface, with two exceptions. For transforms whose size exceeds 4G elements, the dimensions specified in the array `n` must be factorable into primes that are less than or equal to 127. For real to complex and complex to real transforms whose size exceeds 4G elements, the fastest changing dimension must be even. The `cufftPlanMany64()` API supports more complicated input and output data layouts via the advanced data layout parameters: `inembed`, `istride`, `idist`, `onembed`, `ostride`, and `odist`. If `inembed` and `onembed` are set to `NULL`, all other stride information is ignored, and default strides are used. The default assumes contiguous data arrays. This call can only be used once for a given handle. It will fail and return `CUFFT_INVALID_PLAN` if the plan is locked, i.e. the handle was previously used with a different `cufftPlan` or `cufftMakePlan` call. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. All arrays are assumed to be in CPU memory. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension. For multiple GPUs and rank equal to 1, the sizes must be a power of 2. For multiple GPUs and rank equal to 2 or 3, the sizes must be factorable into primes less than or equal to 127. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). For 2 GPUs this must be a complex to complex transform. * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size(s) of the work areas. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when the plan is locked or multi-GPU restrictions are not met. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. ### 3.3.8. cufftXtMakePlanMany()[](#cufftxtmakeplanmany "Permalink to this headline") cufftResult cufftXtMakePlanMany([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, long long int \*n, long long int \*inembed, long long int istride, long long int idist, cudaDataType inputtype, long long int \*onembed, long long int ostride, long long int odist, cudaDataType outputtype, long long int batch, size\_t \*workSize, cudaDataType executiontype);[](#c.cufftXtMakePlanMany "Permalink to this definition") Following a call to `cufftCreate()` makes an FFT plan configuration of dimension `rank`, with sizes specified in the array `n`. The `batch` input parameter tells cuFFT how many transforms to configure. With this function, batched plans of 1, 2, or 3 dimensions may be created. Type specifiers `inputtype`, `outputtype` and `executiontype` dictate type and precision of transform to be performed. Not all combinations of parameters are supported. Currently all three parameters need to match precision. Parameters `inputtype` and `outputtype` need to match transform type complex-to-complex, real-to-complex or complex-to-real. Parameter `executiontype` needs to match precision and be of a complex type. Example: for a half-precision real-to-complex transform, parameters `inputtype`, `outputtype` and `executiontype` would have values of `CUDA_R_16F`, `CUDA_C_16F` and `CUDA_C_16F` respectively. Similarly, a bfloat16 complex-to-real transform would use `CUDA_C_16BF` for `inputtype` and `executiontype`, and `CUDA_R_16BF` for `outputtype`. The `cufftXtMakePlanMany()` API supports more complicated input and output data layouts via the advanced data layout parameters: `inembed`, `istride`, `idist`, `onembed`, `ostride`, and `odist`. If `inembed` and `onembed` are set to `NULL`, all other stride information is ignored, and default strides are used. The default assumes contiguous data arrays. If `cufftXtSetGPUs()` was called prior to this call with multiple GPUs, then `workSize` will contain multiple sizes. See sections on multiple GPUs for more details. All arrays are assumed to be in CPU memory. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension, `n[0]` being the size of the outermost and `n[rank-1]` innermost (contiguous) dimension of a transform. For multiple GPUs and rank equal to 1, the sizes must be a power of 2. For multiple GPUs and rank equal to 2 or 3, the sizes must be factorable into primes less than or equal to 127. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory, `inembed[0]` being the storage dimension of the outermost dimension. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **inputtype\[In\]** – Type of input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory, `onembed[0]` being the storage dimension of the outermost dimension. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **outputtype\[In\]** – Type of output data. * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **executiontype\[In\]** – Type of data to be used for computations. * **\*workSize\[Out\]** – Pointer to the size(s) of the work areas. Return values * **CUFFT\_SUCCESS** – cuFFT successfully created the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. Handle is not valid when multi-GPU restrictions are not met. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. 3.4. cuFFT Plan Properties[](#cufft-plan-properties "Permalink to this headline") ----------------------------------------------------------------------------------- Users can further customize cuFFT plans using plan properties. These properties can be set, queried and reset on a per-plan basis as needed, using the routines listed in this section. The current supported properties are listed below: | Property | Underlying Type | Description | Behavior | | --- | --- | --- | --- | | `NVFFT_PLAN_PROPERTY_INT64_PATIENT_JIT` | long long int | * Runtime LTO kernels are enabled when set to not-zero value. See [Link-Time Optimized Kernels](index.html#cufft-link-time-optimized-kernels)

* Runtime LTO kernles are disabled when set to zero (default) | * Can be set / reset before planning

* Cannot be set / reset after planning | ### 3.4.1. cufftSetPlanPropertyInt64()[](#cufftsetplanpropertyint64 "Permalink to this headline") cufftResult cufftSetPlanPropertyInt64([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftProperty property, const long long int propertyValueInt64);[](#c.cufftSetPlanPropertyInt64 "Permalink to this definition") Associates a cuFFT plan with a property identified by the key `property`. The value for the property is given by value `propertyValueInt64`, which is a signed long long integer. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **property\[In\]** – The property identifier, of type `cufftPlanProperty`. * **propertyValueInt64\[In\]** – Value to set for the property, a long long signed integer. Return values * **CUFFT\_SUCCESS** – cuFFT successfully set the property. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_NOT\_SUPPORTED** – The property is not supported, or it cannot be set at the time (e.g. some properties cannot be set after calling a planning routine for the plan, see [cuFFT Plan Properties](index.html#cufft-plan-properties) ). * **CUFFT\_INVALID\_VALUE** – Invalid property or value with which to set the property ### 3.4.2. cufftGetPlanPropertyInt64()[](#cufftgetplanpropertyint64 "Permalink to this headline") cufftResult cufftGetPlanPropertyInt64([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftProperty property, long long int \*propertyValueInt64);[](#c.cufftGetPlanPropertyInt64 "Permalink to this definition") Retrieves the property value identified by the key `property` associated with the cuFFT plan `plan`. The value for the property, which is a signed long long integer, is set in the address space pointed by `propertyValueInt64`. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **property\[In\]** – The property identifier, of type `cufftPlanProperty`. * **propertyValueInt64\[In\]** – Pointer to the value to be set with the value of the property. Return values * **CUFFT\_SUCCESS** – cuFFT successfully retrieved the property value. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_NOT\_SUPPORTED** – The property is not supported. * **CUFFT\_INVALID\_VALUE** – Invalid property, or pointer `propertyValueInt64` is null ### 3.4.3. cufftResetPlanProperty()[](#cufftresetplanproperty "Permalink to this headline") cufftResult cufftResetPlanProperty([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftProperty property);[](#c.cufftResetPlanProperty "Permalink to this definition") Resets the value of the property identified by the key `property`, associated with the cuFFT plan `plan`, to its default value. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **property\[In\]** – The property identifier, of type `cufftPlanProperty`. Return values * **CUFFT\_SUCCESS** – cuFFT successfully reset the property value. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_NOT\_SUPPORTED** – The property is not supported for `plan`, or cannot be reset at present time (see Behavior column on [cuFFT Plan Properties](index.html#cufft-plan-properties) ). * **CUFFT\_INVALID\_VALUE** – Invalid property 3.5. cuFFT Estimated Size of Work Area[](#cufft-estimated-size-of-work-area "Permalink to this headline") ----------------------------------------------------------------------------------------------------------- During plan execution, cuFFT requires a work area for temporary storage of intermediate results. The `cufftEstimate*()` calls return an estimate for the size of the work area required, given the specified parameters, and assuming default plan settings. Some problem sizes require much more storage than others. In particular powers of 2 are very efficient in terms of temporary storage. Large prime numbers, however, use different algorithms and may need up to the eight times that of a similarly sized power of 2. These routines return estimated `workSize` values which may still be smaller than the actual values needed especially for values of `n` that are not multiples of powers of 2, 3, 5 and 7. More refined values are given by the `cufftGetSize*()` routines, but these values may still be conservative. ### 3.5.1. cufftEstimate1d()[](#cufftestimate1d "Permalink to this headline") cufftResult cufftEstimate1d(int nx, cufftType type, int batch, size\_t \*workSize);[](#c.cufftEstimate1d "Permalink to this definition") During plan execution, cuFFT requires a work area for temporary storage of intermediate results. This call returns an estimate for the size of the work area required, given the specified parameters, and assuming default plan settings. Parameters * **nx\[In\]** – The transform size (e.g. 256 for a 256-point FFT). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2C` for single precision complex to complex). * **batch\[In\]** – Number of transforms of size `nx`. Please consider using `cufftEstimateMany` for multiple transforms. * **\*workSize\[In\]** – Pointer to the size, in bytes, of the work space. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – The `nx` parameter is not a supported size. ### 3.5.2. cufftEstimate2d()[](#cufftestimate2d "Permalink to this headline") cufftResult cufftEstimate2d(int nx, int ny, cufftType type, size\_t \*workSize);[](#c.cufftEstimate2d "Permalink to this definition") During plan execution, cuFFT requires a work area for temporary storage of intermediate results. This call returns an estimate for the size of the work area required, given the specified parameters, and assuming default plan settings. Parameters * **nx\[In\]** – The transform size in the _x_ dimension (number of rows). * **ny\[In\]** – The transform size in the _y_ dimension (number of columns). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2R` for single precision complex to real). * **\*workSize\[In\]** – Pointer to the size, in bytes, of the work space. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – Either or both of the `nx` or `ny` parameters is not a supported size. ### 3.5.3. cufftEstimate3d()[](#cufftestimate3d "Permalink to this headline") cufftResult cufftEstimate3d(int nx, int ny, int nz, cufftType type, size\_t \*workSize);[](#c.cufftEstimate3d "Permalink to this definition") During plan execution, cuFFT requires a work area for temporary storage of intermediate results. This call returns an estimate for the size of the work area required, given the specified parameters, and assuming default plan settings. Parameters * **nx\[In\]** – The transform size in the _x_ dimension. * **ny\[In\]** – The transform size in the _y_ dimension. * **nz\[In\]** – The transform size in the _z_ dimension. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **\*workSize\[In\]** – Pointer to the size, in bytes, of the work space. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the `nx`, `ny`, or `nz` parameters is not a supported size. ### 3.5.4. cufftEstimateMany()[](#cufftestimatemany "Permalink to this headline") cufftResult cufftEstimateMany(int rank, int \*n, int \*inembed, int istride, int idist, int \*onembed, int ostride, int odist, cufftType type, int batch, size\_t \*workSize);[](#c.cufftEstimateMany "Permalink to this definition") During plan execution, cuFFT requires a work area for temporary storage of intermediate results. This call returns an estimate for the size of the work area required, given the specified parameters, and assuming default plan settings. The `cufftEstimateMany()` API supports more complicated input and output data layouts via the advanced data layout parameters: `inembed`, `istride`, `idist`, `onembed`, `ostride`, and `odist`. All arrays are assumed to be in CPU memory. Parameters * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size, in bytes, of the work space. * **\*workSize\[Out\]** – Pointer to the size of the work space Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. 3.6. cuFFT Refined Estimated Size of Work Area[](#cufft-refined-estimated-size-of-work-area "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------- The `cufftGetSize*()` routines give a more accurate estimate of the work area size required for a plan than the `cufftEstimate*()` routines as they take into account any plan settings that may have been made. As discussed in the section [cuFFT Estimated Size of Work Area](index.html#work-estimate) , the `workSize` value(s) returned may be conservative especially for values of `n` that are not multiples of powers of 2, 3, 5 and 7. ### 3.6.1. cufftGetSize1d()[](#cufftgetsize1d "Permalink to this headline") cufftResult cufftGetSize1d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, cufftType type, int batch, size\_t \*workSize);[](#c.cufftGetSize1d "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimate1d()`, given the specified parameters, and taking into account any plan settings that may have been made. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size (e.g. 256 for a 256-point FFT). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2C` for single precision complex to complex). * **batch\[In\]** – Number of transforms of size `nx`. Please consider using `cufftGetSizeMany` for multiple transforms. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – The `nx` parameter is not a supported size. ### 3.6.2. cufftGetSize2d()[](#cufftgetsize2d "Permalink to this headline") cufftResult cufftGetSize2d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, int ny, cufftType type, size\_t \*workSize);[](#c.cufftGetSize2d "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimate2d()`, given the specified parameters, and taking into account any plan settings that may have been made. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size in the _x_ dimension (number of rows). * **ny\[In\]** – The transform size in the _y_ dimension (number of columns). * **type\[In\]** – The transform data type (e.g., `CUFFT_C2R` for single precision complex to real). * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – Either or both of the `nx` or `ny` parameters is not a supported size. ### 3.6.3. cufftGetSize3d()[](#cufftgetsize3d "Permalink to this headline") cufftResult cufftGetSize3d([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nx, int ny, int nz, cufftType type, size\_t \*workSize);[](#c.cufftGetSize3d "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimate3d()`, given the specified parameters, and taking into account any plan settings that may have been made. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nx\[In\]** – The transform size in the _x_ dimension. * **ny\[In\]** – The transform size in the _y_ dimension. * **nz\[In\]** – The transform size in the _z_ dimension. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work space. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the `nx`, `ny`, or `nz` parameters is not a supported size. ### 3.6.4. cufftGetSizeMany()[](#cufftgetsizemany "Permalink to this headline") cufftResult cufftGetSizeMany([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, int \*n, int \*inembed, int istride, int idist, int \*onembed, int ostride, int odist, cufftType type, int batch, size\_t \*workSize);[](#c.cufftGetSizeMany "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimateSizeMany()`, given the specified parameters, and taking into account any plan settings that may have been made. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work area. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. ### 3.6.5. cufftGetSizeMany64()[](#cufftgetsizemany64 "Permalink to this headline") cufftResult cufftGetSizeMany64([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, long long int \*n, long long int \*inembed, long long int istride, long long int idist, long long int \*onembed, long long int ostride, long long int odist, cufftType type, long long int batch, size\_t \*workSize);[](#c.cufftGetSizeMany64 "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimateSizeMany()`, given the specified parameters, and taking into account any plan settings that may have been made. This API is identical to `cufftMakePlanMany` except that the arguments specifying sizes and strides are 64 bit integers. This API makes very large transforms possible. cuFFT includes kernels that use 32 bit indexes, and kernels that use 64 bit indexes. cuFFT planning selects 32 bit kernels whenever possible to avoid any overhead due to 64 bit arithmetic. All sizes and types of transform are supported by this interface, with two exceptions. For transforms whose total size exceeds 4G elements, the dimensions specified in the array `n` must be factorable into primes that are less than or equal to 127. For real to complex and complex to real transforms whose total size exceeds 4G elements, the fastest changing dimension must be even. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **type\[In\]** – The transform data type (e.g., `CUFFT_R2C` for single precision real to complex). * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work area. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. ### 3.6.6. cufftXtGetSizeMany()[](#cufftxtgetsizemany "Permalink to this headline") cufftResult cufftXtGetSizeMany([cufftHandle](#c.cufftHandle "cufftHandle") plan, int rank, long long int \*n, long long int \*inembed, long long int istride, long long int idist, cudaDataType inputtype, long long int \*onembed, long long int ostride, long long int odist, cudaDataType outputtype, long long int batch, size\_t \*workSize, cudaDataType executiontype);[](#c.cufftXtGetSizeMany "Permalink to this definition") This call gives a more accurate estimate of the work area size required for a plan than `cufftEstimateSizeMany()`, given the specified parameters that match signature of `cufftXtMakePlanMany` function, and taking into account any plan settings that may have been made. For more information about valid combinations of `inputtype`, `outputtype` and `executiontype` parameters please refer to documentation of `cufftXtMakePlanMany` function. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **rank\[In\]** – Dimensionality of the transform (1, 2, or 3). * **n\[In\]** – Array of size `rank`, describing the size of each dimension. * **inembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the input data in memory. If set to NULL all other advanced data layout parameters are ignored. * **istride\[In\]** – Indicates the distance between two successive input elements in the least significant (i.e., innermost) dimension. * **idist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the input data. * **inputtype\[In\]** (cudaDataType) – Type of input data. * **onembed\[In\]** – Pointer of size `rank` that indicates the storage dimensions of the output data in memory. If set to NULL all other advanced data layout parameters are ignored. * **ostride\[In\]** – Indicates the distance between two successive output elements in the output array in the least significant (i.e., innermost) dimension. * **odist\[In\]** – Indicates the distance between the first element of two consecutive signals in a batch of the output data. * **outputtype\[In\]** (cudaDataType) – Type of output data. * **batch\[In\]** – Batch size for this transform. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **executiontype\[In\]** (cudaDataType) – Type of data to be used for computations. * **\*workSize\[Out\]** – Pointer to the size of the work area. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_SIZE** – One or more of the parameters is not a supported size. 3.7. cufftGetSize()[](#cufftgetsize "Permalink to this headline") ------------------------------------------------------------------- cufftResult cufftGetSize([cufftHandle](#c.cufftHandle "cufftHandle") plan, size\_t \*workSize);[](#c.cufftGetSize "Permalink to this definition") Once plan generation has been done, either with the original API or the extensible API, this call returns the actual size of the work area required to support the plan. Callers who choose to manage work area allocation within their application must use this call after plan generation, and after any `cufftSet*()` calls subsequent to plan generation, if those calls might alter the required work space size. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*workSize\[In\]** – Pointer to the size(s), in bytes, of the work areas. For example for two GPUs worksize must be declared to have two elements. * **\*workSize\[Out\]** – Pointer to the size of the work area. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. 3.8. cuFFT Caller Allocated Work Area Support[](#cufft-caller-allocated-work-area-support "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------- ### 3.8.1. cufftSetAutoAllocation()[](#cufftsetautoallocation "Permalink to this headline") cufftResult cufftSetAutoAllocation([cufftHandle](#c.cufftHandle "cufftHandle") plan, int autoAllocate);[](#c.cufftSetAutoAllocation "Permalink to this definition") `cufftSetAutoAllocation()` indicates that the caller intends to allocate and manage work areas for plans that have been generated. cuFFT default behavior is to allocate the work area at plan generation time. If `cufftSetAutoAllocation()` has been called with autoAllocate set to 0 (“false”) prior to one of the `cufftMakePlan*()` calls, cuFFT does not allocate the work area. This is the preferred sequence for callers wishing to manage work area allocation. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **autoAllocate\[In\]** – Indicates whether to allocate work area. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. ### 3.8.2. cufftSetWorkArea()[](#cufftsetworkarea "Permalink to this headline") cufftResult cufftSetWorkArea([cufftHandle](#c.cufftHandle "cufftHandle") plan, void \*workArea);[](#c.cufftSetWorkArea "Permalink to this definition") `cufftSetWorkArea()` overrides the work area pointer associated with a plan. If the work area was auto-allocated, cuFFT frees the auto-allocated space. The `cufftExecute*()` calls assume that the work area pointer is valid and that it points to a contiguous region in device memory that does not overlap with any other work area. If this is not the case, results are indeterminate. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*workArea\[In\]** – Pointer to `workArea`. For multiple GPUs, multiple work area pointers must be given. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.8.3. cufftXtSetWorkAreaPolicy()[](#cufftxtsetworkareapolicy "Permalink to this headline") cufftResult cufftXtSetWorkAreaPolicy([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftXtWorkAreaPolicy policy, size\_t \*workSize);[](#c.cufftXtSetWorkAreaPolicy "Permalink to this definition") `cufftXtSetWorkAreaPolicy()` indicates that the caller intends to change work area size for a given plan handle. cuFFT’s default behavior is to allocate the work area at plan generation time with a default size that depends on the plan type and other parameters. If `cufftXtSetWorkAreaPolicy()` has been called with the `policy` parameter set to `CUFFT_WORKAREA_MINIMAL`, cuFFT will attempt to re-plan the handle to use zero bytes of work area memory. If the `cufftXtSetWorkAreaPolicy()` call is successful the auto-allocated work area memory is released. Currently the policies `CUFFT_WORKAREA_PERFORMANCE`, `CUFFT_WORKAREA_USER` and the `workSize` parameter are not supported and reserved for use in future cuFFT releases. This function can be called once per lifetime of a plan handle. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **policy\[In\]** – Type of work area policy to apply. * **\*workSize\[In\]** – Reserved for future use. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_SIZE** – FFT size does not allow use of the selected policy. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. 3.9. cuFFT Execution[](#cufft-execution "Permalink to this headline") ----------------------------------------------------------------------- ### 3.9.1. cufftExecC2C() and cufftExecZ2Z()[](#cufftexecc2c-and-cufftexecz2z "Permalink to this headline") cufftResult cufftExecC2C([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftComplex \*idata, cufftComplex \*odata, int direction);[](#c.cufftExecC2C "Permalink to this definition") cufftResult cufftExecZ2Z([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftDoubleComplex \*idata, cufftDoubleComplex \*odata, int direction);[](#c.cufftExecZ2Z "Permalink to this definition") `cufftExecC2C()` (`cufftExecZ2Z()`) executes a single-precision (double-precision) complex-to-complex transform plan in the transform direction as specified by `direction` parameter. cuFFT uses the GPU memory pointed to by the `idata` parameter as input data. This function stores the Fourier coefficients in the `odata` array. If `idata` and `odata` are the same, this method does an in-place transform. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **idata\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **odata\[In\]** – Pointer to the complex output data (in GPU memory). * **direction\[In\]** – The transform direction: `CUFFT_FORWARD` or `CUFFT_INVERSE`. * **odata\[Out\]** – ontains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `idata`, `odata`, and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.9.2. cufftExecR2C() and cufftExecD2Z()[](#cufftexecr2c-and-cufftexecd2z "Permalink to this headline") cufftResult cufftExecR2C([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftReal \*idata, cufftComplex \*odata);[](#c.cufftExecR2C "Permalink to this definition") cufftResult cufftExecD2Z([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftDoubleReal \*idata, cufftDoubleComplex \*odata);[](#c.cufftExecD2Z "Permalink to this definition") `cufftExecR2C()` (`cufftExecD2Z()`) executes a single-precision (double-precision) real-to-complex, implicitly forward, cuFFT transform plan. cuFFT uses as input data the GPU memory pointed to by the `idata` parameter. This function stores the nonredundant Fourier coefficients in the `odata` array. Pointers to `idata` and `odata` are both required to be aligned to `cufftComplex` data type in single-precision transforms and `cufftDoubleComplex` data type in double-precision transforms. If `idata` and `odata` are the same, this method does an in-place transform. Note the data layout differences between in-place and out-of-place transforms as described in [Parameter cufftType](index.html#cufft-transform-types) . Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **idata\[In\]** – Pointer to the real input data (in GPU memory) to transform. * **odata\[In\]** – Pointer to the complex output data (in GPU memory). * **odata\[Out\]** – Contains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully returned the size of the work space. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `idata` and `odata` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.9.3. cufftExecC2R() and cufftExecZ2D()[](#cufftexecc2r-and-cufftexecz2d "Permalink to this headline") cufftResult cufftExecC2R([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftComplex \*idata, cufftReal \*odata);[](#c.cufftExecC2R "Permalink to this definition") cufftResult cufftExecZ2D([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftDoubleComplex \*idata, cufftDoubleReal \*odata);[](#c.cufftExecZ2D "Permalink to this definition") `cufftExecC2R()` (`cufftExecZ2D()`) executes a single-precision (double-precision) complex-to-real, implicitly inverse, cuFFT transform plan. cuFFT uses as input data the GPU memory pointed to by the `idata` parameter. The input array holds only the nonredundant complex Fourier coefficients. This function stores the real output values in the `odata` array. and pointers are both required to be aligned to `cufftComplex` data type in single-precision transforms and `cufftDoubleComplex` type in double-precision transforms. If `idata` and `odata` are the same, this method does an in-place transform. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **idata\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **odata\[In\]** – Pointer to the real output data (in GPU memory). * **odata\[Out\]** – Contains the real output data. Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `idata` and `odata` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.9.4. cufftXtExec()[](#cufftxtexec "Permalink to this headline") cufftResult cufftXtExec([cufftHandle](#c.cufftHandle "cufftHandle") plan, void \*input, void \*output, int direction);[](#c.cufftXtExec "Permalink to this definition") Function `cufftXtExec` executes any cuFFT transform regardless of precision and type. In case of complex-to-real and real-to-complex transforms `direction` parameter is ignored. cuFFT uses the GPU memory pointed to by the `input` parameter as input data. This function stores the Fourier coefficients in the `output` array. If `input` and `output` are the same, this method does an in-place transform. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **input\[In\]** – Pointer to the input data (in GPU memory) to transform. * **output\[In\]** – Pointer to the output data (in GPU memory). * **direction\[In\]** – The transform direction: `CUFFT_FORWARD` or `CUFFT_INVERSE`. Ignored for complex-to-real and real-to-complex transforms. * **output\[Out\]** – Contains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `idata`, `odata`, and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.9.5. cufftXtExecDescriptor()[](#cufftxtexecdescriptor "Permalink to this headline") cufftResult cufftXtExecDescriptor([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output, int direction);[](#c.cufftXtExecDescriptor "Permalink to this definition") Function `cufftXtExecDescriptor()` executes any cuFFT transform regardless of precision and type. In case of complex-to-real and real-to-complex transforms `direction` parameter is ignored. cuFFT uses the GPU memory pointed to by `cudaLibXtDesc                                         *input` descriptor as input data and `cudaLibXtDesc *output` as output data. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **input\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **output\[In\]** – Pointer to the complex output data (in GPU memory). * **direction\[In\]** – The transform direction: `CUFFT_FORWARD` or `CUFFT_INVERSE`. Ignored for complex-to-real and real-to-complex transforms. * **idata\[Out\]** – Contains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `idata` and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in a descriptor. 3.10. cuFFT and Multiple GPUs[](#cufft-and-multiple-gpus "Permalink to this headline") ---------------------------------------------------------------------------------------- ### 3.10.1. cufftXtSetGPUs()[](#cufftxtsetgpus "Permalink to this headline") cufftResult cufftXtSetGPUs([cufftHandle](#c.cufftHandle "cufftHandle") plan, int nGPUs, int \*whichGPUs);[](#c.cufftXtSetGPUs "Permalink to this definition") `cufftXtSetGPUs()` identifies which GPUs are to be used with the plan. As in the single GPU case `cufftCreate()` creates a plan and `cufftMakePlan*()` does the plan generation. In cuFFT prior to 10.4.0, this call will return an error if a non-default stream has been associated with the plan. Note that the call to `cufftXtSetGPUs()` must occur after the call to `cufftCreate()` and prior to the call to `cufftMakePlan*()`. Parameter `whichGPUs` of `cufftXtSetGPUs()` function determines ordering of the GPUs with respect to data decomposition (first data chunk is placed on GPU denoted by first element of `whichGPUs`). Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **nGPUs\[In\]** – Number of GPUs to use. * **whichGPUs\[In\]** – The GPUs to use. Return values * **CUFFT\_SUCCESS** – cuFFT successfully set the GPUs to use. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle, or a [non-default stream has been associated with the plan in cuFFT prior to 10.4.0](index.html#streamed-cufft-transforms) . * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_VALUE** – The requested number of GPUs was less than 2 or more than 8. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified. * **CUFFT\_INVALID\_SIZE** – Transform size that `plan` was created for does not meet minimum size criteria. ### 3.10.2. cufftXtSetWorkArea()[](#cufftxtsetworkarea "Permalink to this headline") cufftResult cufftXtSetWorkArea([cufftHandle](#c.cufftHandle "cufftHandle") plan, void \*\*workArea);[](#c.cufftXtSetWorkArea "Permalink to this definition") `cufftXtSetWorkArea()` overrides the work areas associated with a plan. If the work area was auto-allocated, cuFFT frees the auto-allocated space. The `cufftXtExec*()` calls assume that the work area is valid and that it points to a contiguous region in each device memory that does not overlap with any other work area. If this is not the case, results are indeterminate. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **workArea\[In\]** – Pointer to the pointers to workArea. Return values * **CUFFT\_SUCCESS** – cuFFT successfully set the GPUs to use. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – A GPU associated with the plan could not be selected. ### 3.10.3. cuFFT Multiple GPU Execution[](#cufft-multiple-gpu-execution "Permalink to this headline") #### 3.10.3.1. cufftXtExecDescriptorC2C() and cufftXtExecDescriptorZ2Z()[](#cufftxtexecdescriptorc2c-and-cufftxtexecdescriptorz2z "Permalink to this headline") cufftResult cufftXtExecDescriptorC2C([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output, int direction);[](#c.cufftXtExecDescriptorC2C "Permalink to this definition") cufftResult cufftXtExecDescriptorZ2Z([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output, int direction);[](#c.cufftXtExecDescriptorZ2Z "Permalink to this definition") `cufftXtExecDescriptorC2C()` (`cufftXtExecDescriptorZ2Z()`) executes a single-precision (double-precision) complex-to-complex transform plan in the transform direction as specified by `direction` parameter. cuFFT uses the GPU memory pointed to by `cudaLibXtDesc *input` as input data. Since only in-place multiple GPU functionality is supported, this function also stores the result in the `cudaLibXtDesc *input` arrays. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*input\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **\*output\[In\]** – Pointer to the complex output data (in GPU memory). * **direction\[In\]** – The transform direction: `CUFFT_FORWARD` or `CUFFT_INVERSE`. * **input\[Out\]** – Contains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `input` and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in a descriptor. #### 3.10.3.2. cufftXtExecDescriptorR2C() and cufftXtExecDescriptorD2Z()[](#cufftxtexecdescriptorr2c-and-cufftxtexecdescriptord2z "Permalink to this headline") cufftResult cufftXtExecDescriptorR2C([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output);[](#c.cufftXtExecDescriptorR2C "Permalink to this definition") cufftResult cufftXtExecDescriptorD2Z([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output);[](#c.cufftXtExecDescriptorD2Z "Permalink to this definition") `cufftXtExecDescriptorR2C()` (`cufftXtExecDescriptorD2Z()`) executes a single-precision (double-precision) real-to-complex transform plan. cuFFT uses the GPU memory pointed to by `cudaLibXtDesc *input` as input data. Since only in-place multiple GPU functionality is supported, this function also stores the result in the `cudaLibXtDesc *input` arrays. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*input\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **\*output\[In\]** – Pointer to the complex output data (in GPU memory). * **input\[Out\]** – Contains the complex Fourier coefficients Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `input` and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in a descriptor. #### 3.10.3.3. cufftXtExecDescriptorC2R() and cufftXtExecDescriptorZ2D()[](#cufftxtexecdescriptorc2r-and-cufftxtexecdescriptorz2d "Permalink to this headline") cufftResult cufftXtExecDescriptorC2R([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output);[](#c.cufftXtExecDescriptorC2R "Permalink to this definition") cufftResult cufftXtExecDescriptorZ2D([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*input, cudaLibXtDesc \*output);[](#c.cufftXtExecDescriptorZ2D "Permalink to this definition") `cufftXtExecDescriptorC2R()` (`cufftXtExecDescriptorZ2D()`) executes a single-precision (double-precision) complex-to-real transform plan in the transform direction as specified by `direction` parameter. cuFFT uses the GPU memory pointed to by `cudaLibXtDesc *input` as input data. Since only in-place multiple GPU functionality is supported, this function also stores the result in the `cudaLibXtDesc *input` arrays. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*input\[In\]** – Pointer to the complex input data (in GPU memory) to transform. * **\*output\[In\]** – Pointer to the complex output data (in GPU memory). * **input\[Out\]** – Contains the complex Fourier coefficients. Return values * **CUFFT\_SUCCESS** – cuFFT successfully executed the FFT plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – At least one of the parameters `input` and `direction` is not valid. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_EXEC\_FAILED** – cuFFT failed to execute the transform on the GPU. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in a descriptor. ### 3.10.4. Memory Allocation and Data Movement Functions[](#memory-allocation-and-data-movement-functions "Permalink to this headline") Multiple GPU cuFFT execution functions assume a certain data layout in terms of what input data has been copied to which GPUs prior to execution, and what output data resides in which GPUs post execution. The following functions assist in allocation, setup and retrieval of the data. They must be called after the call to `cufftMakePlan*()`. #### 3.10.4.1. cufftXtMalloc()[](#cufftxtmalloc "Permalink to this headline") cufftResult cufftXtMalloc([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaLibXtDesc \*\*descriptor, cufftXtSubFormat format);[](#c.cufftXtMalloc "Permalink to this definition") `cufftXtMalloc()` allocates a descriptor, and all memory for data in GPUs associated with the plan, and returns a pointer to the descriptor. Note the descriptor contains an array of device pointers so that the application may preprocess or postprocess the data on the GPUs. The enumerated parameter `cufftXtSubFormat_t` indicates if the buffer will be used for input or output. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **\*\*descriptor\[In\]** – Pointer to a pointer to a `cudaLibXtDesc` object. * **format\[In\]** – cufftXtSubFormat\`\` value. * **\*\*descriptor\[Out\]** – Pointer to a pointer to a `cudaLibXtDesc` object. Return values * **CUFFT\_SUCCESS** – cuFFT successfully allows user to allocate descriptor and GPU memory. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle or it is not a multiple GPU `plan`. * **CUFFT\_ALLOC\_FAILED** – The allocation of GPU resources for the plan failed. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in the descriptor. ##### 3.10.4.1.1. Parameter cufftXtSubFormat[](#parameter-cufftxtsubformat "Permalink to this headline") `cufftXtSubFormat_t` is an enumerated type that indicates if the buffer will be used for input or output and the ordering of the data. typedef enum cufftXtSubFormat\_t { CUFFT\_XT\_FORMAT\_INPUT, //by default input is in linear order across GPUs CUFFT\_XT\_FORMAT\_OUTPUT, //by default output is in scrambled order depending on transform CUFFT\_XT\_FORMAT\_INPLACE, //by default inplace is input order, which is linear across GPUs CUFFT\_XT\_FORMAT\_INPLACE\_SHUFFLED, //shuffled output order after execution of the transform CUFFT\_FORMAT\_UNDEFINED } cufftXtSubFormat; #### 3.10.4.2. cufftXtFree()[](#cufftxtfree "Permalink to this headline") cufftResult cufftXtFree(cudaLibXtDesc \*descriptor);[](#c.cufftXtFree "Permalink to this definition") `cufftXtFree()` frees the descriptor and all memory associated with it. The descriptor and memory must have been returned by a previous call to `cufftXtMalloc()`. Parameters * **\*descriptor\[In\]** – Pointer to a `cudaLibXtDesc` object. Return values * **CUFFT\_SUCCESS** – cuFFT successfully allows user to free descriptor and associated GPU memory. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. #### 3.10.4.3. cufftXtMemcpy()[](#cufftxtmemcpy "Permalink to this headline") cufftResult cufftXtMemcpy([cufftHandle](#c.cufftHandle "cufftHandle") plan, void \*dstPointer, void \*srcPointer, cufftXtCopyType type);[](#c.cufftXtMemcpy "Permalink to this definition") `cufftXtMemcpy()` copies data between buffers on the host and GPUs or between GPUs. The enumerated parameter `cufftXtCopyType_t` indicates the type and direction of transfer. Calling `cufftXtMemcpy` function for multi-GPU batched FFT plans with `CUFFT_COPY_DEVICE_TO_DEVICE` transfer type is not supported. Note that starting from CUDA 11.2 (cuFFT 10.4.0), `cufftSetStream()` is supported on multi-GPU plans. When associating a stream with a plan, `cufftXtMemcpy()` remains synchronous across the multiple GPUs. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **dstPointer\[In\]** – Pointer to the destination address(es). * **srcPointer\[In\]** – Pointer to the source address(es). * **type\[In\]** – `cufftXtCopyType`value. Return values * **CUFFT\_SUCCESS** – cuFFT successfully allows user to copy memory between host and GPUs or between GPUs. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle. * **CUFFT\_INVALID\_VALUE** – One or more invalid parameters were passed to the API. * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. * **CUFFT\_INVALID\_DEVICE** – An invalid GPU index was specified in a descriptor. ##### 3.10.4.3.1. Parameter cufftXtCopyType[](#parameter-cufftxtcopytype "Permalink to this headline") `cufftXtCopyType_t` is an enumerated type for multiple GPU functions that specifies the type of copy for `cufftXtMemcpy()`. `CUFFT_COPY_HOST_TO_DEVICE` copies data from a contiguous host buffer to multiple device buffers, in the layout cuFFT requires for input data. `dstPointer` must point to a `cudaLibXtDesc` structure, and `srcPointer` must point to a host memory buffer. `CUFFT_COPY_DEVICE_TO_HOST` copies data from multiple device buffers, in the layout cuFFT produces for output data, to a contiguous host buffer. `dstPointer` must point to a host memory buffer, and `srcPointer` must point to a `cudaLibXtDesc` structure. `CUFFT_COPY_DEVICE_TO_DEVICE` copies data from multiple device buffers, in the layout cuFFT produces for output data, to multiple device buffers, in the layout cuFFT requires for input data. `dstPointer` and `srcPointer` must point to different `cudaLibXtDesc` structures (and therefore memory locations). That is, the copy cannot be in-place. Note that device\_to\_device `cufftXtMemcpy()` for 2D and 3D data is not currently supported. typedef enum cufftXtCopyType\_t { CUFFT\_COPY\_HOST\_TO\_DEVICE, CUFFT\_COPY\_DEVICE\_TO\_HOST, CUFFT\_COPY\_DEVICE\_TO\_DEVICE } cufftXtCopyType; ### 3.10.5. General Multiple GPU Descriptor Types[](#general-multiple-gpu-descriptor-types "Permalink to this headline") #### 3.10.5.1. cudaXtDesc[](#cudaxtdesc "Permalink to this headline") A descriptor type used in multiple GPU routines that contains information about the GPUs and their memory locations. struct cudaXtDesc\_t{ int version; //descriptor version int nGPUs; //number of GPUs int GPUs\[MAX\_CUDA\_DESCRIPTOR\_GPUS\]; //array of device IDs void \*data\[MAX\_CUDA\_DESCRIPTOR\_GPUS\]; //array of pointers to data, one per GPU size\_t size\[MAX\_CUDA\_DESCRIPTOR\_GPUS\]; //array of data sizes, one per GPU void \*cudaXtState; //opaque CUDA utility structure }; typedef struct cudaXtDesc\_t cudaXtDesc; #### 3.10.5.2. cudaLibXtDesc[](#cudalibxtdesc "Permalink to this headline") A descriptor type used in multiple GPU routines that contains information about the library used. struct cudaLibXtDesc\_t{ int version; //descriptor version cudaXtDesc \*descriptor; //multi-GPU memory descriptor libFormat library; //which library recognizes the format int subFormat; //library specific enumerator of sub formats void \*libDescriptor; //library specific descriptor e.g. FFT transform plan object }; typedef struct cudaLibXtDesc\_t cudaLibXtDesc; 3.11. cuFFT Callbacks[](#cufft-callbacks "Permalink to this headline") ------------------------------------------------------------------------ ### 3.11.1. cufftXtSetJITCallback()[](#cufftxtsetjitcallback "Permalink to this headline") cufftResult cufftXtSetJITCallback([cufftHandle](#c.cufftHandle "cufftHandle") plan, const char \*callbackSymbolName, const void \*callbackFatbin, size\_t callbackFatbinSize, cufftXtCallbackType type, void \*\*caller\_info)[](#c.cufftXtSetJITCallback "Permalink to this definition") `cufftXtSetJITCallback()` specifies a load or store LTO callback to be used with the plan. This call is valid only after a call to `cufftCreate()`, but before calling `cufftMakePlan*()`, which does the plan generation. If there was already an LTO callback of this type associated with the plan, this new callback routine replaces it. If the new callback requires shared memory, you must call `cufftXtSetCallbackSharedSize` with the amount of shared memory the callback function needs. cuFFT will not retain the amount of shared memory associated with the previous callback if the callback function is changed. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **callbackSymbolName\[In\]** – null-terminated C string containing the (unmangled) callback symbol name (i.e. the name of the LTO callback routine). This symbol name will be runtime-compiled, and modifiers such as `extern "C"` or `namespace` are not supported. * **callbackFatbin\[In\]** – Pointer to the location in host memory where the callback device function is located, after being compiled into LTO-IR with nvcc or NVRTC. * **callbackFatbinSize\[In\]** – Size in bytes of the data pointed at by `callbackFatbin`. * **type\[In\]** – Type of callback routine. * **callerInfo\[In\]** – Optional array of device pointers to caller specific information, one per GPU. Return values * **CUFFT\_SUCCESS** – cuFFT successfully associated the callback function with the plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not valid (e.g. the handle was already used to make a plan). * **CUFFT\_INVALID\_TYPE** – The callback type is not valid. * **CUFFT\_INVALID\_VALUE** – The pointer to the callback device function is invalid or the size is `0`. * **CUFFT\_NOT\_SUPPORTED** – The functionality is not supported yet (e.g. multi-GPU with LTO callbacks). * **CUFFT\_INTERNAL\_ERROR** – cuFFT encountered an unexpected error, likely in the runtime linking process; error codes will be expanded in a future release. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.11.2. cufftXtSetCallback()[](#cufftxtsetcallback "Permalink to this headline") cufftResult cufftXtSetCallback([cufftHandle](#c.cufftHandle "cufftHandle") plan, void \*\*callbackRoutine, cufftXtCallbackType type, void \*\*callerInfo)[](#c.cufftXtSetCallback "Permalink to this definition") `cufftXtSetCallback()` specifies a load or store legacy callback to be used with the plan. This call is valid only after a call to `cufftMakePlan*()`, which does the plan generation. If there was already a legacy callback of this type associated with the plan, this new callback routine replaces it. If the new callback requires shared memory, you must call `cufftXtSetCallbackSharedSize` with the amount of shared memory it needs. cuFFT will not retain the amount of shared memory associated with the previous callback. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **callbackRoutine\[In\]** – Array of callback routine pointers, one per GPU. * **type\[In\]** – Type of callback routine. * **callerInfo\[In\]** – Optional array of device pointers to caller specific information, one per GPU. Return values * **CUFFT\_SUCCESS** – cuFFT successfully associated the callback function with the plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle, or a [non-default stream has been associated with the plan in cuFFT prior to 10.4.0](index.html#streamed-cufft-transforms) . * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_SETUP\_FAILED** – The cuFFT library failed to initialize. ### 3.11.3. cufftXtClearCallback()[](#cufftxtclearcallback "Permalink to this headline") cufftResult cufftXtClearCallback([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftXtCallbackType type)[](#c.cufftXtClearCallback "Permalink to this definition") `cufftXtClearCallback()` instructs cuFFT to stop invoking the specified legacy callback type when executing the plan. Only the specified callback is cleared. If no callback of this type had been specified, the return code is `CUFFT_SUCCESS`. Note that this method **does not work** with LTO callbacks. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **type\[In\]** – Type of callback routine. Return values * **CUFFT\_SUCCESS** – cuFFT successfully disassociated the callback function with the plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle, or a [non-default stream has been associated with the plan in cuFFT prior to 10.4.0](index.html#streamed-cufft-transforms) . * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. ### 3.11.4. cufftXtSetCallbackSharedSize()[](#cufftxtsetcallbacksharedsize "Permalink to this headline") cufftResult cufftXtSetCallbackSharedSize([cufftHandle](#c.cufftHandle "cufftHandle") plan, cufftXtCallbackType type, size\_t sharedSize)[](#c.cufftXtSetCallbackSharedSize "Permalink to this definition") `cufftXtSetCallbackSharedSize()` instructs cuFFT to dynamically allocate shared memory at launch time, for use by the callback. The maximum allowable amount of shared memory is 16K bytes. cuFFT passes a pointer to this shared memory to the callback routine at execution time. This shared memory is only valid for the life of the load or store callback operation. During execution, cuFFT may overwrite shared memory for its own purposes. Parameters * **plan\[In\]** – `cufftHandle` returned by `cufftCreate`. * **type\[In\]** – Type of callback routine. * **sharedSize\[In\]** – Amount of shared memory requested. Return values * **CUFFT\_SUCCESS** – cuFFT will invoke the callback routine with a pointer to the requested amount of shared memory. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle, or a [non-default stream has been associated with the plan in cuFFT prior to 10.4.0](index.html#streamed-cufft-transforms) . * **CUFFT\_INTERNAL\_ERROR** – An internal driver error was detected. * **CUFFT\_ALLOC\_FAILED** – cuFFT will not be able to allocate the requested amount of shared memory. 3.12. cufftSetStream()[](#cufftsetstream "Permalink to this headline") ------------------------------------------------------------------------ cufftResult cufftSetStream([cufftHandle](#c.cufftHandle "cufftHandle") plan, cudaStream\_t stream);[](#c.cufftSetStream "Permalink to this definition") Associates a CUDA stream with a cuFFT plan. All kernel launches made during plan execution are now done through the associated stream, enabling overlap with activity in other streams (e.g. data copying). The association remains until the plan is destroyed or the stream is changed with another call to `cufftSetStream()`. Note that starting from CUDA 11.2 (cuFFT 10.4.0), `cufftSetStream()` is supported on multi-GPU plans. When associating a stream with a plan, `cufftXtMemcpy()` remains synchronous across the multiple GPUs. For previous versions of cuFFT, `cufftSetStream()` will return an error in multiple GPU plans. Note that starting from CUDA 12.2 (cuFFT 11.0.8), on multi-GPU plans, `stream` can be associated with any context on any GPU. However, repeated calls to `cufftSetStream()` with streams from different contexts incur a small time penalty. Optimal performance is obtained when repeated calls to `cufftSetStream` use streams from the same CUDA context. Parameters * **plan\[In\]** – The `cufftHandle` object to associate with the stream. * **stream\[In\]** – A valid CUDA stream created with `cudaStreamCreate()`; `0` for the default stream. Return values * **CUFFT\_SUCCESS** – The stream was associated with the plan. * **CUFFT\_INVALID\_PLAN** – The `plan` parameter is not a valid handle, or plan is multi-gpu in cuFFT version prior to 10.4.0. 3.13. cufftGetVersion()[](#cufftgetversion "Permalink to this headline") -------------------------------------------------------------------------- cufftResult cufftGetVersion(int \*version);[](#c.cufftGetVersion "Permalink to this definition") Returns the version number of cuFFT. Parameters * **\*version\[In\]** – Pointer to the version number. * **\*version\[Out\]** – Contains the version number. Return values **CUFFT\_SUCCESS** – cuFFT successfully returned the version number. 3.14. cufftGetProperty()[](#cufftgetproperty "Permalink to this headline") ---------------------------------------------------------------------------- cufftResult cufftGetProperty(libraryPropertyType type, int \*value);[](#c.cufftGetProperty "Permalink to this definition") Return in `*value` the number for the property described by `type` of the dynamically linked CUFFT library. Parameters * **type\[In\]** – CUDA library property. * **value\[Out\]** – Contains the integer value for the requested property. Return values * **CUFFT\_SUCCESS** – The property value was successfully returned. * **CUFFT\_INVALID\_TYPE** – The property type is not recognized. * **CUFFT\_INVALID\_VALUE** – `value` is `NULL`. 3.15. cuFFT Types[](#cufft-types "Permalink to this headline") ---------------------------------------------------------------- ### 3.15.1. Parameter cufftType[](#parameter-cuffttype "Permalink to this headline") The cuFFT library supports complex- and real-data transforms. The `cufftType` data type is an enumeration of the types of transform data supported by cuFFT. typedef enum cufftType\_t { CUFFT\_R2C \= 0x2a, // Real to complex (interleaved) CUFFT\_C2R \= 0x2c, // Complex (interleaved) to real CUFFT\_C2C \= 0x29, // Complex to complex (interleaved) CUFFT\_D2Z \= 0x6a, // Double to double-complex (interleaved) CUFFT\_Z2D \= 0x6c, // Double-complex (interleaved) to double CUFFT\_Z2Z \= 0x69 // Double-complex to double-complex (interleaved) } cufftType; ### 3.15.2. Parameters for Transform Direction[](#parameters-for-transform-direction "Permalink to this headline") The cuFFT library defines forward and inverse Fast Fourier Transforms according to the sign of the complex exponential term. #define CUFFT\_FORWARD -1 #define CUFFT\_INVERSE 1 cuFFT performs un-normalized FFTs; that is, performing a forward FFT on an input data set followed by an inverse FFT on the resulting set yields data that is equal to the input, scaled by the number of elements. Scaling either transform by the reciprocal of the size of the data set is left for the user to perform as seen fit. ### 3.15.3. Type definitions for callbacks[](#type-definitions-for-callbacks "Permalink to this headline") The cuFFT library supports callback funtions for all combinations of single or double precision, real or complex data, load or store. These are enumerated in the parameter `cufftXtCallbackType`. typedef enum cufftXtCallbackType\_t { CUFFT\_CB\_LD\_COMPLEX \= 0x0, CUFFT\_CB\_LD\_COMPLEX\_DOUBLE \= 0x1, CUFFT\_CB\_LD\_REAL \= 0x2, CUFFT\_CB\_LD\_REAL\_DOUBLE \= 0x3, CUFFT\_CB\_ST\_COMPLEX \= 0x4, CUFFT\_CB\_ST\_COMPLEX\_DOUBLE \= 0x5, CUFFT\_CB\_ST\_REAL \= 0x6, CUFFT\_CB\_ST\_REAL\_DOUBLE \= 0x7, CUFFT\_CB\_UNDEFINED \= 0x8 } cufftXtCallbackType; #### 3.15.3.1. Type definitions for LTO callbacks[](#type-definitions-for-lto-callbacks "Permalink to this headline") The LTO callback function prototypes and pointer type definitions are as follows: typedef cufftComplex (\*cufftJITCallbackLoadC)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleComplex (\*cufftJITCallbackLoadZ)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftReal (\*cufftJITCallbackLoadR)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleReal(\*cufftJITCallbackLoadD)(void \*dataIn, unsigned long long offset, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreC)(void \*dataOut, unsigned long long offset, cufftComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreZ)(void \*dataOut, unsigned long long offset, cufftDoubleComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreR)(void \*dataOut, unsigned long long offset, cufftReal element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftJITCallbackStoreD)(void \*dataOut, unsigned long long offset, cufftDoubleReal element, void \*callerInfo, void \*sharedPointer); Notice the difference in the type of the `offset` parameter (`unsigned long long`) vs. legacy callbacks (which use `size_t`). #### 3.15.3.2. Type definitions for legacy callbacks[](#type-definitions-for-legacy-callbacks "Permalink to this headline") The legacy callback function prototypes and pointer type definitions are as follows: typedef cufftComplex (\*cufftCallbackLoadC)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleComplex (\*cufftCallbackLoadZ)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftReal (\*cufftCallbackLoadR)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef cufftDoubleReal(\*cufftCallbackLoadD)(void \*dataIn, size\_t offset, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreC)(void \*dataOut, size\_t offset, cufftComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreZ)(void \*dataOut, size\_t offset, cufftDoubleComplex element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreR)(void \*dataOut, size\_t offset, cufftReal element, void \*callerInfo, void \*sharedPointer); typedef void (\*cufftCallbackStoreD)(void \*dataOut, size\_t offset, cufftDoubleReal element, void \*callerInfo, void \*sharedPointer); ### 3.15.4. Other cuFFT Types[](#other-cufft-types "Permalink to this headline") #### 3.15.4.1. cufftHandle[](#cuffthandle "Permalink to this headline") type cufftHandle[](#c.cufftHandle "Permalink to this definition") A handle type used to store and access cuFFT plans. The user receives a handle after creating a cuFFT plan and uses this handle to execute the plan. typedef unsigned int cufftHandle; #### 3.15.4.2. cufftReal[](#cufftreal "Permalink to this headline") A single-precision, floating-point real data type. typedef float cufftReal; #### 3.15.4.3. cufftDoubleReal[](#cufftdoublereal "Permalink to this headline") A double-precision, floating-point real data type. typedef double cufftDoubleReal; #### 3.15.4.4. cufftComplex[](#cufftcomplex "Permalink to this headline") A single-precision, floating-point complex data type that consists of interleaved real and imaginary components. typedef cuComplex cufftComplex; #### 3.15.4.5. cufftDoubleComplex[](#cufftdoublecomplex "Permalink to this headline") A double-precision, floating-point complex data type that consists of interleaved real and imaginary components. typedef cuDoubleComplex cufftDoubleComplex; 3.16. Common types[](#common-types "Permalink to this headline") ------------------------------------------------------------------ ### 3.16.1. cudaDataType[](#cudadatatype "Permalink to this headline") The `cudaDataType` data type is an enumeration of the types supported by CUDA libraries. typedef enum cudaDataType\_t { CUDA\_R\_16F\= 2, // 16 bit real CUDA\_C\_16F\= 6, // 16 bit complex CUDA\_R\_32F\= 0, // 32 bit real CUDA\_C\_32F\= 4, // 32 bit complex CUDA\_R\_64F\= 1, // 64 bit real CUDA\_C\_64F\= 5, // 64 bit complex CUDA\_R\_8I\= 3, // 8 bit real as a signed integer CUDA\_C\_8I\= 7, // 8 bit complex as a pair of signed integers CUDA\_R\_8U\= 8, // 8 bit real as an unsigned integer CUDA\_C\_8U\= 9 // 8 bit complex as a pair of unsigned integers } cudaDataType; ### 3.16.2. libraryPropertyType[](#librarypropertytype "Permalink to this headline") The `libraryPropertyType` data type is an enumeration of library property types. (ie. CUDA version X.Y.Z would yield `MAJOR_VERSION=X`, `MINOR_VERSION=Y`, `PATCH_LEVEL=Z`) typedef enum libraryPropertyType\_t { MAJOR\_VERSION, MINOR\_VERSION, PATCH\_LEVEL } libraryPropertyType; 4\. Multiple GPU Data Organization[](#multiple-gpu-data-organization "Permalink to this headline") ==================================================================================================== This chapter explains how data are distributed between the GPUs, before and after a multiple GPU transform. For simplicity, it is assumed in this chapter that the caller has specified GPU 0 and GPU 1 to perform the transform. 4.1. Multiple GPU Data Organization for Batched Transforms[](#multiple-gpu-data-organization-for-batched-transforms "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------- For batches of transforms, each individual transform is executed on a single GPU. If possible the batches are evenly distributed among the GPUs. For a batch of size `m` performed on `n` GPUs, where `m` is not divisible by `n`, the first `m % n` GPUs will perform \\(\\left\\lfloor \\frac{m}{n} \\right\\rfloor+\\ 1\\) transforms. The remaining GPUs will perform \\(\\left\\lfloor \\frac{m}{n} \\right\\rfloor\\) transforms. For example, in a batch of 15 transforms performed on 4 GPUs, the first three GPUs would perform 4 transforms, and the last GPU would perform 3 transforms. This approach removes the need for data exchange between the GPUs, and results in nearly perfect scaling for cases where the batch size is divisible by the number of GPUs. 4.2. Multiple GPU Data Organization for Single 2D and 3D Transforms[](#multiple-gpu-data-organization-for-single-2d-and-3d-transforms "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Single transforms performed on multiple GPUs require the data to be divided between the GPUs. Then execution takes place in phases. For example with 2 GPUs, for 2D and 3D transforms with even sized dimensions, each GPU does half of the transform in (rank - 1) dimensions. Then data are exchanged between the GPUs so that the final dimension can be processed. Since 2D and 3D transforms support sizes other than powers of 2, it is possible that the data can not be evenly distributed among the GPUs. In general for the case of `n` GPUs, a dimension of size `m` that is not a multiple of `n` would be distributed such that the first `m % n` GPUs would get one extra row for 2D transforms, one extra plane for 3D transforms. Take for example, a 2D transform on 4 GPUs, using an array declared in C as `data[x][y]`, where `x` is 65 and `y` is 99. The surface is distributed prior to the transform such that GPU 0 receives a surface with dimensions `[17][99]`, and GPUs 1…3 receive surfaces with dimensions `[16][99]`. After the transform, each GPU again has a portion of the surface, but divided in the y dimension. GPUs 0…2 have surfaces with dimensions `[65][25]`. GPU 3 has a surface with dimensions `[65][24]` For a 3D transform on 4 GPUs consider an array declared in C as `data[x][y][z]`, where `x` is 103, `y` is 122, and `z` is 64. The volume is distributed prior to the transform such that each GPUs 0…2 receive volumes with dimensions `[26][122][64]`, and GPU 3 receives a volume with dimensions `[25][122][64]`. After the transform, each GPU again has a portion of the surface, but divided in the y dimension. GPUs 0 and 1 have a volumes with dimensions `[103][31][64]`, and GPUs 2 and 3 have volumes with dimensions `[103][30][64]`. 4.3. Multiple-GPU Data Organization for Single 1D Transforms[](#multiple-gpu-data-organization-for-single-1d-transforms "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------------------------------- By default for 1D transforms, the initial distribution of data to the GPUs is similar to the 2D and 3D cases. For a transform of dimension x on two GPUs, GPU 0 receives data ranging from 0…(x/2-1). GPU 1 receives data ranging from (x/2)…(x-1). Similarly, with 4 GPUs, the data are evenly distributed among all 4 GPUs. Before computation can begin, data are redistributed among the GPUs. It is possible to perform this redistribution in the copy from host memory, in cases where the application does not need to pre-process the data prior to the transform. To do this, the application can create the data descriptor with `cufftXtMalloc` using the sub-format `CUFFT_XT_FORMAT_1D_INPUT_SHUFFLED`. This can significantly reduce the time it takes to execute the transform. cuFFT performs multiple GPU 1D transforms by decomposing the transform size into factors `Factor1` and `Factor2`, and treating the data as a grid of size `Factor1` x `Factor2`. The four steps done to calculate the 1D FFT are: `Factor1` transforms of size `Factor2`, data exchange between the GPUs, a pointwise twiddle multiplication, and `Factor2` transforms of size `Factor1`. To gain efficiency by overlapping computation with data exchange, cuFFT breaks the whole transform into independent segments or strings, which can be processed while others are in flight. A side effect of this algorithm is that the output of the transform is not in linear order. The output in GPU memory is in strings, each of which is composed of `Factor2` substrings of equal size. Each substring contains contiguous results starting `Factor1` elements subsequent to start of the previous substring. Each string starts substring size elements after the start of the previous string. The strings appear in order, the first half on GPU 0, and the second half on GPU 1. See the example below: transform size \= 1024 number of strings \= 8 Factor1 \= 64 Factor2 \= 16 substrings per string for output layout is Factor2 (16) string size \= 1024/8 \= 128 substring size \= 128/16 \= 8 stride between substrings \= 1024/16 \= Factor1 (64) On GPU 0: string 0 has substrings with indices 0...7 64...71 128...135 ... 960...967 string 1 has substrings with indices 8...15 72...79 136...143 ... 968...975 ... On GPU 1: string 4 has substrings with indices 32...39 96...103 160...167 ... 992...999 ... string 7 has substrings with indices 56...63 120...127 184...191 ... 1016...1023 The cufftXtQueryPlan API allows the caller to retrieve a structure containing the number of strings, the decomposition factors, and (in the case of power of 2 size) some useful mask and shift elements. The example below shows how cufftXtQueryPlan is invoked. It also shows how to translate from an index in the host input array to the corresponding index on the device, and vice versa. /\* \* These routines demonstrate the use of cufftXtQueryPlan to get the 1D \* factorization and convert between permuted and linear indexes. \*/ /\* \* Set up a 1D plan that will execute on GPU 0 and GPU1, and query \* the decomposition factors \*/ int main(int argc, char \*\*argv){ cufftHandle plan; cufftResult stat; int whichGPUs\[2\] \= { 0, 1 }; cufftXt1dFactors factors; stat \= cufftCreate( &plan ); if (stat != CUFFT\_SUCCESS) { printf("Create error %d\\n",stat); return 1; } stat \= cufftXtSetGPUs( plan, 2, whichGPUs ); if (stat != CUFFT\_SUCCESS) { printf("SetGPU error %d\\n",stat); return 1; } stat \= cufftMakePlan1d( plan, size, CUFFT\_C2C, 1, workSizes ); if (stat != CUFFT\_SUCCESS) { printf("MakePlan error %d\\n",stat); return 1; } stat \= cufftXtQueryPlan( plan, (void \*) &factors, CUFFT\_QUERY\_1D\_FACTORS ); if (stat != CUFFT\_SUCCESS) { printf("QueryPlan error %d\\n",stat); return 1; } printf("Factor 1 %zd, Factor2 %zd\\n",factors.factor1,factors.factor2); cufftDestroy(plan); return 0; } /\* \* Given an index into a permuted array, and the GPU index return the \* corresponding linear index from the beginning of the input buffer. \* \* Parameters: \* factors input: pointer to cufftXt1dFactors as returned by \* cufftXtQueryPlan \* permutedIx input: index of the desired element in the device output \* array \* linearIx output: index of the corresponding input element in the \* host array \* GPUix input: index of the GPU containing the desired element \*/ cufftResult permuted2Linear( cufftXt1dFactors \* factors, size\_t permutedIx, size\_t \*linearIx, int GPUIx ) { size\_t indexInSubstring; size\_t whichString; size\_t whichSubstring; // the low order bits of the permuted index match those of the linear index indexInSubstring \= permutedIx & factors\->substringMask; // the next higher bits are the substring index whichSubstring \= (permutedIx \>> factors\->substringShift) & factors\->factor2Mask; // the next higher bits are the string index on this GPU whichString \= (permutedIx \>> factors\->stringShift) & factors\->stringMask; // now adjust the index for the second GPU if (GPUIx) { whichString += factors\->stringCount/2; } // linear index low order bits are the same // next higher linear index bits are the string index \*linearIx \= indexInSubstring + ( whichString << factors\->substringShift ); // next higher bits of linear address are the substring index \*linearIx += whichSubstring << factors\->factor1Shift; return CUFFT\_SUCCESS; } /\* \* Given a linear index into a 1D array, return the GPU containing the permuted \* result, and index from the start of the data buffer for that element. \* \* Parameters: \* factors input: pointer to cufftXt1dFactors as returned by \* cufftXtQueryPlan \* linearIx input: index of the desired element in the host input \* array \* permutedIx output: index of the corresponding result in the device \* output array \* GPUix output: index of the GPU containing the result \*/ cufftResult linear2Permuted( cufftXt1dFactors \* factors, size\_t linearIx, size\_t \*permutedIx, int \*GPUIx ) { size\_t indexInSubstring; size\_t whichString; size\_t whichSubstring; size\_t whichStringMask; int whichStringShift; if (linearIx \>= factors\->size) { return CUFFT\_INVALID\_VALUE; } // get a useful additional mask and shift count whichStringMask \= factors\->stringCount \-1; whichStringShift \= (factors\->factor1Shift + factors\->factor2Shift) \- factors\->stringShift ; // the low order bits identify the index within the substring indexInSubstring \= linearIx & factors\->substringMask; // first determine which string has our linear index. // the low order bits indentify the index within the substring. // the next higher order bits identify which string. whichString \= (linearIx \>> factors\->substringShift) & whichStringMask; // the first stringCount/2 strings are in the first GPU, // the rest are in the second. \*GPUIx \= whichString/(factors\->stringCount/2); // next determine which substring within the string has our index // the substring index is in the next higher order bits of the index whichSubstring \= (linearIx \>>(factors\->substringShift + whichStringShift)) & factors\->factor2Mask; // now we can re-assemble the index \*permutedIx \= indexInSubstring; \*permutedIx += whichSubstring << factors\->substringShift; if ( !\*GPUIx ) { \*permutedIx += whichString << factors\->stringShift; } else { \*permutedIx += (whichString \- (factors\->stringCount/2) ) << factors\->stringShift; } return CUFFT\_SUCCESS; } 5\. FFTW Conversion Guide[](#fftw-conversion-guide "Permalink to this headline") ================================================================================== cuFFT differs from FFTW in that FFTW has many plans and a single execute function while cuFFT has fewer plans, but multiple execute functions. The cuFFT execute functions determine the precision (single or double) and whether the input is complex or real valued. The following table shows the relationship between the two interfaces. | FFTW function | cuFFT function | | --- | --- | | `fftw_plan_dft_1d(), fftw_plan_dft_r2c_1d(), fftw_plan_dft_c2r_1d()` | `cufftPlan1d()` | | `fftw_plan_dft_2d(), fftw_plan_dft_r2c_2d(), fftw_plan_dft_c2r_2d()` | `cufftPlan2d()` | | `fftw_plan_dft_3d(), fftw_plan_dft_r2c_3d(), fftw_plan_dft_c2r_3d()` | `cufftPlan3d()` | | `fftw_plan_dft(), fftw_plan_dft_r2c(), fftw_plan_dft_c2r()` | `cufftPlanMany()` | | `fftw_plan_many_dft(), fftw_plan_many_dft_r2c(), fftw_plan_many_dft_c2r()` | `cufftPlanMany()` | | `fftw_execute()` | `cufftExecC2C(), cufftExecZ2Z(), cufftExecR2C(), cufftExecD2Z(), cufftExecC2R(), cufftExecZ2D()` | | `fftw_destroy_plan()` | `cufftDestroy()` | 6\. FFTW Interface to cuFFT[](#fftw-interface-to-cufft "Permalink to this headline") ====================================================================================== NVIDIA provides FFTW3 interfaces to the cuFFT library. This allows applications using FFTW to use NVIDIA GPUs with minimal modifications to program source code. To use the interface first do the following two steps * It is recommended that you replace the include file `fftw3.h` with `cufftw.h` * Instead of linking with the double/single precision libraries such as `fftw3/fftw3f` libraries, link with both the cuFFT and cuFFTW libraries * Ensure the search path includes the directory containing `cuda_runtime_api.h` After an application is working using the FFTW3 interface, users may want to modify their code to move data to and from the GPU and use the routines documented in the [FFTW Conversion Guide](index.html#fftw-conversion-guide) for the best performance. The following tables show which components and functions of FFTW3 are supported in cuFFT. | Section in FFTW manual | Supported | Unsupported | | --- | --- | --- | | Complex numbers | `fftw_complex, fftwf_complex` types | | | Precision | double `fftw3`, single `fftwf3` | long double `fftw3l`, quad precision `fftw3q` are not supported since CUDA functions operate on double and single precision floating-point quantities | | Memory Allocation | | `fftw_malloc(), fftw_free(), fftw_alloc_real(), fftw_alloc_complex(), fftwf_alloc_real(), fftwf_alloc_complex()` | | Multi-threaded FFTW | | `fftw3_threads, fftw3_omp` are not supported | | Distributed-memory FFTW with MPI | | `fftw3_mpi,fftw3f_mpi` are not supported | Note that for each of the double precision functions below there is a corresponding single precision version with the letters `fftw` replaced by `fftwf`. | Section in FFTW manual | Supported | Unsupported | | --- | --- | --- | | Using Plans | `fftw_execute(), fftw_destroy_plan(), fftw_cleanup()` | `fftw_print_plan(), fftw_cost(), fftw_flops()` exist but are not functional | | **Basic Interface** | | | | Complex DFTs | `fftw_plan_dft_1d(), fftw_plan_dft_2d(), fftw_plan_dft_3d(), fftw_plan_dft()` | | | Planner Flags | | Planner flags are ignored and the same plan is returned regardless | | Real-data DFTs | `fftw_plan_dft_r2c_1d(), fftw_plan_dft_r2c_2d(), fftw_plan_dft_r2c_3d(), fftw_plan_dft_r2c(), fftw_plan_dft_c2r_1d(), fftw_plan_dft_c2r_2d(), fftw_plan_dft_c2r_3d(), fftw_plan_dft_c2r()` | | | Read-data DFT Array Format | | Not supported | | Read-to-Real Transform | | Not supported | | Read-to-Real Transform Kinds | | Not supported | | **Advanced Interface** | | | | Advanced Complex DFTs | `fftw_plan_many_dft()` with multiple 1D, 2D, 3D transforms | `fftw_plan_many_dft()` with 4D or higher transforms or a 2D or higher batch of embedded transforms | | Advanced Real-data DFTs | `fftw_plan_many_dft_r2c(), fftw_plan_many_dft_c2r()` with multiple 1D, 2D, 3D transforms | `fftw_plan_many_dft_r2c(), fftw_plan_many_dft_c2r()` with 4D or higher transforms or a 2D or higher batch of embedded transforms | | Advanced Real-to-Real Transforms | | Not supported | | **Guru Interface** | | | | Interleaved and split arrays | Interleaved format | Split format | | Guru vector and transform sizes | `fftw_iodim` struct | | | Guru Complex DFTs | `fftw_plan_guru_dft(), fftw_plan_guru_dft_r2c(), fftw_plan_guru_dft_c2r()` with multiple 1D, 2D, 3D transforms | `fftw_plan_guru_dft(), fftw_plan_guru_dft_r2c(), fftw_plan_guru_dft_c2r()` with 4D or higher transforms or a 2D or higher batch of transforms | | Guru Real-data DFTs | | Not supported | | Guru Real-to-real Transforms | | Not supported | | 64-bit Guru Interface | `fftw_plan_guru64_dft(), fftw_plan_guru64_dft_r2c(), fftw_plan_guru64_dft_c2r()` with multiple 1D, 2D, 3D transforms | `fftw_plan_guru64_dft(), fftw_plan_guru64_dft_r2c(), fftw_plan_guru64_dft_c2r()` with 4D or higher transforms or a 2D or higher batch of transforms | | New-array Execute Functions | `fftw_execute_dft(), fftw_execute_dft_r2c(), fftw_execute_dft_c2r()` with interleaved format | Split format and real-to-real functions | | Wisdom | | `fftw_export_wisdom_to_file(), fftw_import_wisdom_from_file()` exist but are not functional. Other wisdom functions do not have entry points in the library. | 7\. Deprecated Functionality[](#deprecated-functionality "Permalink to this headline") ======================================================================================== Starting from CUDA 12.8: * The cuFFT binary `libcufft_static_nocallback.a` is deprecated and will be removed in a future release. `libcufft_static.a` can be used as a replacement. Starting from CUDA 12.0: * GPU architectures SM35 and SM37 are no longer supported. The minimum required architecture is SM50. Starting from CUDA 11.8: * CUDA Graphs capture is no longer supported for legacy callback routines that load data in out-of-place mode transforms. Starting from CUDA 12.6 Update 2, LTO callbacks can be used as a replacement for legacy callbacks without this limitation. Starting from CUDA 11.4: * Support for callback functionality using separately compiled device code (legacy callbacks) is deprecated on all GPU architectures. Callback functionality will continue to be supported for all GPU architectures. Starting from CUDA 11.0: * GPU architecture SM30 is no longer supported. The minimum required architecture is SM35. * Support for GPU architectures SM35, SM37 (Kepler), and SM50, SM52 (Maxwell) is deprecated. Function `cufftSetCompatibilityMode` was removed in version 9.1. 8\. Notices[](#notices "Permalink to this headline") ====================================================== 8.1. Notice[](#notice "Permalink to this headline") ----------------------------------------------------- This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 8.2. OpenCL[](#opencl "Permalink to this headline") ----------------------------------------------------- OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 8.3. Trademarks[](#trademarks "Permalink to this headline") ------------------------------------------------------------- NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. --- # cuDLA API :: CUDA Toolkit Documentation cuDLA API ([PDF](../pdf/cuDLA_API.pdf) ) - v12.8.1 ([older](https://developer.nvidia.com/cuda-toolkit-archive) ) - Last updated March 04, 2025 - [Send Feedback](mailto:CUDAIssues@nvidia.com?subject=CUDA%20Toolkit%20Documentation%20Feedback:%20cuDLA%20API) 1. Modules ---------- Here is a list of all modules: * [Data types used by cuDLA driver](index.html#group__CUDLA__TYPES) * [cuDLA API](index.html#group__CUDLA__API) ### 1.1. Data types used by cuDLA driver #### Classes struct  [CudlaFence](index.html#structCudlaFence) [](index.html#structCudlaFence) union  [cudlaDevAttribute](index.html#unioncudlaDevAttribute) [](index.html#unioncudlaDevAttribute) struct  [cudlaExternalMemoryHandleDesc\_t](index.html#structcudlaExternalMemoryHandleDesc__t) [](index.html#structcudlaExternalMemoryHandleDesc__t) struct  [cudlaExternalSemaphoreHandleDesc\_t](index.html#structcudlaExternalSemaphoreHandleDesc__t) [](index.html#structcudlaExternalSemaphoreHandleDesc__t) union  [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) [](index.html#unioncudlaModuleAttribute) struct  [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) [](index.html#structcudlaModuleTensorDescriptor) struct  [cudlaSignalEvents](index.html#structcudlaSignalEvents) [](index.html#structcudlaSignalEvents) struct  [cudlaTask](index.html#structcudlaTask) [](index.html#structcudlaTask) struct  [cudlaWaitEvents](index.html#structcudlaWaitEvents) [](index.html#structcudlaWaitEvents) #### Typedefs typedef cudlaDevHandle\_t \*  [cudlaDevHandle](#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63) typedef cudlaModule\_t \*  [cudlaModule](#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5) #### Enumerations enum [cudlaAccessPermissionFlags](#group__CUDLA__TYPES_1g631e2c17faee6b825b83577efb55ed75) enum [cudlaDevAttributeType](#group__CUDLA__TYPES_1gae3e445aeb1fe1c992b9357cdc4c913a) enum [cudlaFenceType](#group__CUDLA__TYPES_1gfd5fd195c04a78364ab8b66904354bb8) enum [cudlaMode](#group__CUDLA__TYPES_1g74ca7d641e2873189059953d3ed65cb2) enum [cudlaModuleAttributeType](#group__CUDLA__TYPES_1ga2b041ca59fb0103b62272b83a3b2ba2) enum [cudlaModuleLoadFlags](#group__CUDLA__TYPES_1g5f665d89a741263fbb6fed8c343861de) enum [cudlaNvSciSyncAttributes](#group__CUDLA__TYPES_1g3efeaae42e362ebb197cb69170941c6e) enum [cudlaStatus](#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) enum [cudlaSubmissionFlags](#group__CUDLA__TYPES_1gbe4b7a209e3dec6e3e3ef1ad082e1f0d) #### Typedefs typedef cudlaDevHandle\_t \* cudlaDevHandle cuDLA Device Handle typedef cudlaModule\_t \* cudlaModule cuDLA Module Handle #### Enumerations enum cudlaAccessPermissionFlags Access permission flags for importing NvSciBuffers ###### Values CUDLA\_READ\_WRITE\_PERM = 0 Flag to import memory with read-write permission CUDLA\_READ\_ONLY\_PERM = 1 Flag to import memory with read-only permission CUDLA\_TASK\_STATISTICS = 1<<1 Flag to indicate buffer as layerwise statistics buffer. enum cudlaDevAttributeType Device attribute type. ###### Values CUDLA\_UNIFIED\_ADDRESSING = 0 Flag to check for support for UVA. CUDLA\_DEVICE\_VERSION = 1 Flag to check for DLA HW version. enum cudlaFenceType Supported fence types. ###### Values CUDLA\_NVSCISYNC\_FENCE = 1 NvSciSync fence type for EOF. CUDLA\_NVSCISYNC\_FENCE\_SOF = 2 enum cudlaMode Device creation modes. ###### Values CUDLA\_CUDA\_DLA = 0 Hyrbid mode. CUDLA\_STANDALONE = 1 Standalone mode. enum cudlaModuleAttributeType Module attribute types. ###### Values CUDLA\_NUM\_INPUT\_TENSORS = 0 Flag to retrieve number of input tensors. CUDLA\_NUM\_OUTPUT\_TENSORS = 1 Flag to retrieve number of output tensors. CUDLA\_INPUT\_TENSOR\_DESCRIPTORS = 2 Flag to retrieve all the input tensor descriptors. CUDLA\_OUTPUT\_TENSOR\_DESCRIPTORS = 3 Flag to retrieve all the output tensor descriptors. CUDLA\_NUM\_OUTPUT\_TASK\_STATISTICS = 4 Flag to retrieve total number of output task statistics buffer. CUDLA\_OUTPUT\_TASK\_STATISTICS\_DESCRIPTORS = 5 Flag to retrieve all the output task statistics descriptors. enum cudlaModuleLoadFlags Module load flags for [cudlaModuleLoadFromMemory](index.html#group__CUDLA__API_1gcd725924569cec1a3214fd09cb38601d "Load a DLA module.") . ###### Values CUDLA\_MODULE\_DEFAULT = 0 Default flag. CUDLA\_MODULE\_ENABLE\_FAULT\_DIAGNOSTICS = 1 Flag to load a module that is used to perform permanent fault diagnostics for DLA HW. enum cudlaNvSciSyncAttributes cuDLA NvSciSync attributes. ###### Values CUDLA\_NVSCISYNC\_ATTR\_WAIT = 1 Wait attribute. CUDLA\_NVSCISYNC\_ATTR\_SIGNAL = 2 Signal attribute. enum cudlaStatus Error codes. ###### Values cudlaSuccess = 0 The API call returned with no errors. cudlaErrorInvalidParam = 1 This indicates that one or more parameters passed to the API is/are incorrect. cudlaErrorOutOfResources = 2 This indicates that the API call failed due to lack of underlying resources. cudlaErrorCreationFailed = 3 This indicates that an internal error occurred during creation of device handle. cudlaErrorInvalidAddress = 4 This indicates that the memory object being passed in the API call has not been registered before. cudlaErrorOs = 5 This indicates that an OS error occurred. cudlaErrorCuda = 6 This indicates that there was an error in a CUDA operation as part of the API call. cudlaErrorUmd = 7 This indicates that there was an error in the DLA runtime for the API call. cudlaErrorInvalidDevice = 8 This indicates that the device handle passed to the API call is invalid. cudlaErrorInvalidAttribute = 9 This indicates that an invalid attribute is being requested. cudlaErrorIncompatibleDlaSWVersion = 10 This indicates that the underlying DLA runtime is incompatible with the current cuDLA version. cudlaErrorMemoryRegistered = 11 This indicates that the memory object is already registered. cudlaErrorInvalidModule = 12 This indicates that the module being passed is invalid. cudlaErrorUnsupportedOperation = 13 This indicates that the operation being requested by the API call is unsupported. cudlaErrorNvSci = 14 This indicates that the NvSci operation requested by the API call failed. cudlaErrorDlaErrInvalidInput = 0x40000001 DLA HW Error. cudlaErrorDlaErrInvalidPreAction = 0x40000002 DLA HW Error. cudlaErrorDlaErrNoMem = 0x40000003 DLA HW Error. cudlaErrorDlaErrProcessorBusy = 0x40000004 DLA HW Error. cudlaErrorDlaErrTaskStatusMismatch = 0x40000005 DLA HW Error. cudlaErrorDlaErrEngineTimeout = 0x40000006 DLA HW Error. cudlaErrorDlaErrDataMismatch = 0x40000007 DLA HW Error. cudlaErrorUnknown = 0x7fffffff This indicates that an unknown error has occurred. enum cudlaSubmissionFlags Task submission flags for [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") . ###### Values CUDLA\_SUBMIT\_NOOP = 1 Flag to specify that the submitted task must be bypassed for execution. CUDLA\_SUBMIT\_SKIP\_LOCK\_ACQUIRE = 1<<1 Flag to specify that the global lock acquire must be skipped. CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK = 1<<2 Flag to specify that the submitted task is to run permanent fault diagnostics for DLA HW. ### 1.2. cuDLA API This section describes the application programming interface of the cuDLA driver. #### Functions [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaCreateDevice](#group__CUDLA__API_1gf9d00e3a93dfd31814736144764ae478) ( const uint64\_t  device, const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63) \* devHandle, const uint32\_t  flags ) Create a device handle. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaDestroyDevice](#group__CUDLA__API_1g030f50a094bad8a99e00d30d2759762e) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle ) Destroy device handle. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaDeviceGetAttribute](#group__CUDLA__API_1g9d7b0d59e0a9537a25e42e650831b89a) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const [cudlaDevAttributeType](index.html#group__CUDLA__TYPES_1gae3e445aeb1fe1c992b9357cdc4c913a)  attrib, const [cudlaDevAttribute](index.html#unioncudlaDevAttribute) \* pAttribute ) Get cuDLA device attributes. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaDeviceGetCount](#group__CUDLA__API_1g582475cdf72dc5b1dffea8ab2d10295a) ( const uint64\_t\* pNumDevices ) Get device count. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaGetLastError](#group__CUDLA__API_1ge95c8c295bb1abbd9ed1cc553a91f282) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle ) Gets the last asynchronous error in task execution. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaGetNvSciSyncAttributes](#group__CUDLA__API_1gf24b21ae42b495d3c69fbe44c513a319) ( uint64\_t\* attrList, const uint32\_t  flags ) Get cuDLA's NvSciSync attributes. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaGetVersion](#group__CUDLA__API_1g86f4d620e1004325e32cf957e038ffc4) ( const uint64\_t\* version ) Returns the version number of the library. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaImportExternalMemory](#group__CUDLA__API_1gca69cd7ac008500693ffeedb18d7a9c8) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const cudlaExternalMemoryHandleDesc\* desc, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Imports external memory into cuDLA. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaImportExternalSemaphore](#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const cudlaExternalSemaphoreHandleDesc\* desc, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Imports external semaphore into cuDLA. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaMemRegister](#group__CUDLA__API_1ge07f8bb22373163a0117fc5738a23be0) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint64\_t\* ptr, const size\_t  size, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Registers the CUDA memory to DLA engine. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaMemUnregister](#group__CUDLA__API_1g886f9d28ad1fe869c06a7d7ad108b1c9) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint64\_t\* devPtr ) Unregisters the input memory from DLA engine. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaModuleGetAttributes](#group__CUDLA__API_1g7c7e68c05dbc5a7f7ea011c8e3285a7a) ( const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5)  hModule, const [cudlaModuleAttributeType](index.html#group__CUDLA__TYPES_1ga2b041ca59fb0103b62272b83a3b2ba2)  attrType, const [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) \* attribute ) Get DLA module attributes. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaModuleLoadFromMemory](#group__CUDLA__API_1gcd725924569cec1a3214fd09cb38601d) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint8\_t\* pModule, const size\_t  moduleSize, const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5) \* hModule, const uint32\_t  flags ) Load a DLA module. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaModuleUnload](#group__CUDLA__API_1g8b9de473e03b716fcbe4b689b172a15a) ( const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5)  hModule, const uint32\_t  flags ) Unload a DLA module. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaSetTaskTimeoutInMs](#group__CUDLA__API_1g6a308eb6d7b30ded8250b61f0508de33) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint32\_t  timeout ) Set task timeout in millisecond. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) [cudlaSubmitTask](#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb) ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const [cudlaTask](index.html#structcudlaTask) \* ptrToTasks, const uint32\_t  numTasks, const void\* stream, const uint32\_t  flags ) Submits the inference operation on DLA. #### Functions [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaCreateDevice ( const uint64\_t  device, const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63) \* devHandle, const uint32\_t  flags ) Create a device handle. ###### Parameters device \- Device number (can be 0 or 1). devHandle \- Pointer to hold the created cuDLA device handle. flags \- Flags controlling device creation. Valid values for flags are: * [CUDLA\_CUDA\_DLA](index.html#group__CUDLA__TYPES_1gg74ca7d641e2873189059953d3ed65cb21dffbcf6879c77d2c7a1efc6012f72c6) - In this mode, cuDLA serves as a programming model extension of CUDA wherein DLA work can be submitted using CUDA constructs. * [CUDLA\_STANDALONE](index.html#group__CUDLA__TYPES_1gg74ca7d641e2873189059953d3ed65cb27789b9d73cb032d268d84463a5c26f91) - In this mode, cuDLA works standalone without any interaction with CUDA. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorOutOfResources](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a05b9d884f3dd0210f0cfa56e064c29dcb) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorIncompatibleDlaSWVersion](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a02c136741dbd4e0fa0d3c8c3643d9b03e) , [cudlaErrorCreationFailed](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c848145627b7a723ee09b29aaa41a834) , [cudlaErrorCuda](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0727b16e07c9c63a5f1379826a31eb983) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) ###### Description Creates an instance of a cuDLA device which can be used to submit DLA operations. The application can create the handle in hybrid or standalone mode. In hybrid mode, the current set GPU device is used by this API to decide the association of the created DLA device handle. This function returns [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) if the current set GPU device is a dGPU as cuDLA is not supported on dGPU presently. cuDLA supports 16 cuDLA device handles per DLA HW instance. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaDestroyDevice ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle ) Destroy device handle. ###### Parameters devHandle \- A valid device handle. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorCuda](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0727b16e07c9c63a5f1379826a31eb983) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) ###### Description Destroys the instance of the cuDLA device which was created with cudlaCreateDevice. Before destroying the handle, it is important to ensure that all the tasks submitted previously to the device are completed. Failure to do so can lead to application crashes. In hybrid mode, cuDLA internally performs memory allocations with CUDA using the primary context. As a result, before destroying or resetting a CUDA primary context, it is mandatory that all cuDLA device initializations are destroyed. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaDeviceGetAttribute ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const [cudlaDevAttributeType](index.html#group__CUDLA__TYPES_1gae3e445aeb1fe1c992b9357cdc4c913a)  attrib, const [cudlaDevAttribute](index.html#unioncudlaDevAttribute) \* pAttribute ) Get cuDLA device attributes. ###### Parameters devHandle \- The input cuDLA device handle. attrib \- The attribute that is being requested. pAttribute \- The output pointer where the attribute will be available. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) ###### Description UVA addressing between CUDA and DLA requires special support in the underlying kernel mode drivers. Applications are expected to query the cuDLA runtime to check if the current version of cuDLA supports UVA addressing. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaDeviceGetCount ( const uint64\_t\* pNumDevices ) Get device count. ###### Parameters pNumDevices \- The number of DLA devices will be available in this variable upon successful completion. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorIncompatibleDlaSWVersion](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a02c136741dbd4e0fa0d3c8c3643d9b03e) ###### Description Get number of DLA devices available to use. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaGetLastError ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle ) Gets the last asynchronous error in task execution. ###### Parameters devHandle \- A valid device handle. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorDlaErrInvalidInput](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a01ddae2b987abee60fb416abc9540a8cb) , [cudlaErrorDlaErrInvalidPreAction](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a05e8ecc197526e1367f51951c9bbf36e4) , [cudlaErrorDlaErrNoMem](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a089328cd138b285d0c00f26ed4115f88e) , [cudlaErrorDlaErrProcessorBusy](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04367e9544354f75e5c7556d60bfaf149) , [cudlaErrorDlaErrTaskStatusMismatch](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a007a6137f3875744e2e4b2cf54d5fe7d4) , [cudlaErrorDlaErrEngineTimeout](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0b57798c888d3515eceff4ff78d7ec140) , [cudlaErrorDlaErrDataMismatch](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a07c3a9a91cde8c19b111cd633ae6251f4) , [cudlaErrorUnknown](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0026f9875981d619043bad450ca90e5ed) ###### Description The DLA tasks execute asynchronously on the DLA HW. As a result, the status of the task execution is not known at the time of task submission. The status of the task executed by the DLA HW most recently for the particular device handle can be queried using this interface. Note that a return code of [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) from this function does not necessarily imply that most recent task executed successfully. Since this function returns immediately, it can only report the status of the tasks at the snapshot of time when it is called. To be guaranteed of task completion, applications must synchronize on the submitted tasks in hybrid or standalone modes and then call this API to check for errors. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaGetNvSciSyncAttributes ( uint64\_t\* attrList, const uint32\_t  flags ) Get cuDLA's NvSciSync attributes. ###### Parameters attrList \- Attribute list created by the application. flags \- Applications can use this flag to specify how they intend to use the NvSciSync object created from the attrList. The valid values of flags can be one of the following (or an OR of these values): * [CUDLA\_NVSCISYNC\_ATTR\_WAIT](index.html#group__CUDLA__TYPES_1gg3efeaae42e362ebb197cb69170941c6e2d049b652955f2bf109b4e1e41660d01) , specifies that the application intend to use the NvSciSync object created using this attribute list as a waiter in cuDLA and therefore needs cuDLA to fill waiter specific NvSciSyncAttr. * [CUDLA\_NVSCISYNC\_ATTR\_SIGNAL](index.html#group__CUDLA__TYPES_1gg3efeaae42e362ebb197cb69170941c6ed31eff6290cfb76e793883564ba0fce6) , specifies that the application intend to use the NvSciSync object created using this attribute list as a signaler in cuDLA and therefore needs cuDLA to fill signaler specific NvSciSyncAttr. ###### Returns * [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , The API call returned with no errors. * [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , This API call failed because invalid parameter attrList was passed. * [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) , This error code indicates that the API call failed because the operation is not supported in hybrid mode. * [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) , The API call failed as parameter attrList has invalid values. * [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) , This error code indicates error in the NvSci operation as part of the API call. * cudlaErrorNotPermittedOperation, This error code indicates that the API call is not permitted when DRIVE OS is in Operational state. * [cudlaErrorUnknown](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0026f9875981d619043bad450ca90e5ed) , This error code indicates that an unknown error has occurred. ###### Description Gets the NvSciSync's attributes in the attribute list created by the application. cuDLA supports two types of NvSciSync object primitives - * Sync point * Deterministic semaphore cuDLA prioritizes sync point primitive over deterministic semaphore primitive by default and sets these priorities in the NvSciSync attribute list. For Deterministic semaphore, NvSciSync attribute list used to create the NvSciSync object must have value of NvSciSyncAttrKey\_RequireDeterministicFences key set to true. cuDLA also supports Timestamp feature on NvSciSync objects. Waiter can request for this by setting NvSciSync attribute "NvSciSyncAttrKey\_WaiterRequireTimestamps" as true. In the event of failed NvSci initialization this function would return [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) . This function can return [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) or [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) in certain cases when the underlying NvSci operation fails. This API updates the input nvSciSyncAttrList with values equivalent to the following public attribute key-values: NvSciSyncAttrKey\_RequiredPerm is set to * NvSciSyncAccessPerm\_SignalOnly if value of flag is set to CUDLA\_NVSCISYNC\_ATTR\_WAIT. * NvSciSyncAccessPerm\_WaitOnly if value of flag is set to CUDLA\_NVSCISYNC\_ATTR\_SIGNAL. * NvSciSyncAccessPerm\_WaitSignal if value of flag is set to CUDLA\_NVSCISYNC\_ATTR\_SIGNAL | CUDLA\_NVSCISYNC\_ATTR\_WAIT. As NvSciSyncAttrKey\_RequiredPerm is internally set by cuDLA, setting this value by the application is disallowed. Note: Users of cuDLA can only append attributes to output attrList using NvSci API, modifying already populated values of the output attrList can result in undefined behavior. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaGetVersion ( const uint64\_t\* version ) Returns the version number of the library. ###### Parameters version \- cuDLA library version will be available in this variable upon successful execution. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) ###### Description cuDLA is semantically versioned. This function will return the version as 1000000\*major + 1000\*minor + patch. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaImportExternalMemory ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const cudlaExternalMemoryHandleDesc\* desc, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Imports external memory into cuDLA. ###### Parameters devHandle \- A valid device handle. desc \- Contains description about allocated external memory. devPtr \- The output pointer where the mapping will be available. flags \- Application can use this flag to specify the memory access permissions of the memory that needs to be registered with DLA. The valid values of flags can be one of the following: * [CUDLA\_READ\_WRITE\_PERM](index.html#group__CUDLA__TYPES_1gg631e2c17faee6b825b83577efb55ed75748d993f14cb963da3fa1a806234645d) , specifies that the external memory needs to be registered with DLA as read-write memory. * [CUDLA\_READ\_ONLY\_PERM](index.html#group__CUDLA__TYPES_1gg631e2c17faee6b825b83577efb55ed75d9ed991984f3164d4eef1806a598db0c) , specifies that the external memory needs to be registered with DLA as read-only memory. * [CUDLA\_TASK\_STATISTICS](index.html#group__CUDLA__TYPES_1gg631e2c17faee6b825b83577efb55ed751237111e1003c0f81c4fc3bf62bf6001) , specifies that the external memory needs to be registered with DLA for layerwise statistics. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) , [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) , [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) , [cudlaErrorMemoryRegistered](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0e335fc20958745722f1f7a22f22c383d) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) ###### Description Imports the allocated external memory by registering it with DLA. After successful registration, the returned pointer can be used in a task submit. On Tegra, cuDLA supports importing NvSciBuf objects in standalone mode only. In the event of failed NvSci initialization (either due to usage of this API in hybrid mode or an issue in the NvSci library initialization), this function would return [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) . This function can return [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) or [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) in certain cases when the underlying NvSci operation fails. Note: cuDLA only supports importing NvSciBuf objects of type NvSciBufType\_RawBuffer or NvSciBufType\_Tensor. Importing NvSciBuf object of any other type can result in an undefined behaviour. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaImportExternalSemaphore ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const cudlaExternalSemaphoreHandleDesc\* desc, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Imports external semaphore into cuDLA. ###### Parameters devHandle \- A valid device handle. desc \- Contains sempahore object. devPtr \- The output pointer where the mapping will be available. flags \- Reserved for future. Must be set to 0. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) , [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) , [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) , [cudlaErrorMemoryRegistered](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0e335fc20958745722f1f7a22f22c383d) ###### Description Imports the allocated external semaphore by registering it with DLA. After successful registration, the returned pointer can be used in a task submission to signal synchronization objects. On Tegra, cuDLA supports importing NvSciSync objects in standalone mode only. NvSciSync object primitives that cuDLA supports are sync point and deterministic semaphore. cuDLA also supports Timestamp feature on NvSciSync objects, using which the user can get a snapshot of DLA clock at which a particular fence is signaled. At any point in time there are only 512 valid timestamp buffers that can be associated with fences. For example, If User has created 513 fences from a single NvSciSync object with timestamp enabled then the timestamp buffer associated with 1st fence is same as with 513th fence. In the event of failed NvSci initialization (either due to usage of this API in hybrid mode or an issue in the NvSci library initialization), this function would return [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) . This function can return [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) or [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) in certain cases when the underlying NvSci operation fails. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaMemRegister ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint64\_t\* ptr, const size\_t  size, const uint64\_t\*\* devPtr, const uint32\_t  flags ) Registers the CUDA memory to DLA engine. ###### Parameters devHandle \- A valid cuDLA device handle create by a previous call to [cudlaCreateDevice](index.html#group__CUDLA__API_1gf9d00e3a93dfd31814736144764ae478 "Create a device handle.") . ptr \- The CUDA pointer to be registered. size \- The size of the mapping i.e the number of bytes from ptr that must be mapped. devPtr \- The output pointer where the mapping will be available. flags \- Applications can use this flag to control several aspects of the registration process. The valid values of flags can be one of the following (or an OR of these values): * 0, default * [CUDLA\_TASK\_STATISTICS](index.html#group__CUDLA__TYPES_1gg631e2c17faee6b825b83577efb55ed751237111e1003c0f81c4fc3bf62bf6001) , specifies that the external memory needs to be registered with DLA for layerwise statistics. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidAddress](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03959fb7cceee17b5c54b60cdffa40e7f) , [cudlaErrorCuda](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0727b16e07c9c63a5f1379826a31eb983) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorOutOfResources](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a05b9d884f3dd0210f0cfa56e064c29dcb) , [cudlaErrorMemoryRegistered](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0e335fc20958745722f1f7a22f22c383d) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) ###### Description As part of registration, a system mapping is created whereby the DLA HW can access the underlying CUDA memory. The resultant mapping is available in devPtr and applications must use this mapping while referring this memory in submit operations. This function will return [cudlaErrorInvalidAddress](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03959fb7cceee17b5c54b60cdffa40e7f) if the pointer or size to be registered is invalid. In addition, if the input pointer was already registered, then this function will return [cudlaErrorMemoryRegistered](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0e335fc20958745722f1f7a22f22c383d) . Attempting to re-register memory does not cause any irrecoverable error in cuDLA and applications can continue to use cuDLA APIs even after this error has occurred. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaMemUnregister ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint64\_t\* devPtr ) Unregisters the input memory from DLA engine. ###### Parameters devHandle \- A valid cuDLA device handle create by a previous call to [cudlaCreateDevice](index.html#group__CUDLA__API_1gf9d00e3a93dfd31814736144764ae478 "Create a device handle.") . devPtr \- The pointer to be unregistered. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorInvalidAddress](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03959fb7cceee17b5c54b60cdffa40e7f) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) ###### Description The system mapping that enables the DLA HW to access the memory is removed. This mapping could have been created by a previous call to [cudlaMemRegister](index.html#group__CUDLA__API_1ge07f8bb22373163a0117fc5738a23be0 "Registers the CUDA memory to DLA engine.") , [cudlaImportExternalMemory](index.html#group__CUDLA__API_1gca69cd7ac008500693ffeedb18d7a9c8 "Imports external memory into cuDLA.") or [cudlaImportExternalSemaphore](index.html#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34 "Imports external semaphore into cuDLA.") . Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaModuleGetAttributes ( const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5)  hModule, const [cudlaModuleAttributeType](index.html#group__CUDLA__TYPES_1ga2b041ca59fb0103b62272b83a3b2ba2)  attrType, const [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) \* attribute ) Get DLA module attributes. ###### Parameters hModule \- The input DLA module. attrType \- The attribute type that is being requested. attribute \- The output pointer where the attribute will be available. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidModule](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d2b4370207a0a2926eda404f880989d6) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) ###### Description Get module attributes from the loaded module. This API returns [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) if the module is not loaded in any device. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaModuleLoadFromMemory ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint8\_t\* pModule, const size\_t  moduleSize, const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5) \* hModule, const uint32\_t  flags ) Load a DLA module. ###### Parameters devHandle \- The input cuDLA device handle. The module will be loaded in the context of this handle. pModule \- A pointer to an in-memory module. moduleSize \- The size of the module. hModule \- The address in which the loaded module handle will be available upon successful execution. flags \- Applications can use this flag to specify how the module is going to be used. The valid values of flags can be one of the following: * [CUDLA\_MODULE\_DEFAULT](index.html#group__CUDLA__TYPES_1gg5f665d89a741263fbb6fed8c343861de9bdbf383d6e0b00b4e6d02f1ad460e86) , Default value which is 0. * [CUDLA\_MODULE\_ENABLE\_FAULT\_DIAGNOSTICS](index.html#group__CUDLA__TYPES_1gg5f665d89a741263fbb6fed8c343861de3c6cabb1b33ab42a535f5660597d2858) , Application can specify this flag to load a module that is used for performing fault diagnostics for DLA HW. With this flag set, the pModule and moduleSize parameters shall be NULL and 0 as the diagnostics module is loaded internally. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorOutOfResources](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a05b9d884f3dd0210f0cfa56e064c29dcb) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) ###### Description Loads the module into the current device handle. * Multiple loadables are not allowed to load onto single cuDLA device handle. * A Loadable can only be loaded once in cuDLA device handle lifecycle. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaModuleUnload ( const [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5)  hModule, const uint32\_t  flags ) Unload a DLA module. ###### Parameters hModule \- Handle to the loaded module. flags \- Reserved for future. Must be set to 0. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorInvalidModule](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d2b4370207a0a2926eda404f880989d6) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) ###### Description Unload the module from the device handle that it was loaded into. This API returns [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) if the module is not loaded into a valid device. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaSetTaskTimeoutInMs ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const uint32\_t  timeout ) Set task timeout in millisecond. ###### Parameters devHandle \- A valid device handle. timeout \- task timeout value in ms. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) ###### Description Set task timeout in ms for each device handle. cuDLA sets 30 seconds as default timeout value if user doesn't explicitly set the timeout. In case , device handle is invalid or timeout is 0 or timeout is greater than 1000 sec, this function would return cudlaErrorInvalidParam otherwise cudlaSuccess. Note: This API can return task execution errors from previous DLA task submissions. [cudlaStatus](index.html#group__CUDLA__TYPES_1gc95f8bcde047e255b11ea1b3b80598a0) cudlaSubmitTask ( const [cudlaDevHandle](index.html#group__CUDLA__TYPES_1gc52f41cd392913019a800d5f850a9b63)  devHandle, const [cudlaTask](index.html#structcudlaTask) \* ptrToTasks, const uint32\_t  numTasks, const void\* stream, const uint32\_t  flags ) Submits the inference operation on DLA. ###### Parameters devHandle \- A valid cuDLA device handle. ptrToTasks \- A list of inferencing tasks. numTasks \- The number of tasks. stream \- The stream on which the DLA task has to be submitted. flags \- Applications can use this flag to control several aspects of the submission process. The valid values of flags can be one of the following (or an OR of these values): * 0, default * [CUDLA\_SUBMIT\_NOOP](index.html#group__CUDLA__TYPES_1ggbe4b7a209e3dec6e3e3ef1ad082e1f0d3b7e226f4ea9ad2336d6c00855cb7764) , specifies that the submitted task must be skipped during execution on the DLA. However, all the waitEvents and signalEvents dependencies must be satisfied. This flag is ignored when NULL data submissions are being done as in that case only the wait and signal events are internally stored for the next task submission. * [CUDLA\_SUBMIT\_SKIP\_LOCK\_ACQUIRE](index.html#group__CUDLA__TYPES_1ggbe4b7a209e3dec6e3e3ef1ad082e1f0dddca43caf261dc44569bd4b2a2c3a455) , specifies that the submitted task is being enqueued in a device handle and that no other task is being enqueued in that device handle at that time in any other thread. This is a flag that apps can use as an optimization. Ordinarily, the cuDLA APIs acquire a global lock internally to guarantee thread safety. However, this lock causes unwanted serialization in cases where the the applications are submitting tasks to different device handles. If an application was submitting one or more tasks in multiple threads and if these submissions are to different device handles and if there is no shared data being provided as part of the task information in the respective submissions then applications can specify this flag during submission so that the internal lock acquire is skipped. Shared data also includes the input stream in hybrid mode operation. Therefore, if the same stream is being used to submit two different tasks and even if the two device handles are different, the usage of this flag is invalid. * [CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK](index.html#group__CUDLA__TYPES_1ggbe4b7a209e3dec6e3e3ef1ad082e1f0d344e7c37d198b239bdad09cca8801b3b) , specifies that the submitted task is to run permanent fault diagnostics for DLA HW. User can use this task to probe the state of DLA HW. With this flag set, in standalone mode user is not allowed to do event only submissions, where tensor information is NULL and only events (wait/signal or both) are present in task. This is because the task always runs on a internally loaded diagnostic module. This diagnostic module does not expect any input tensors and so input tensor memory, however user is expected to query no. of output tensors, allocate the output tensor memory and pass the same while using the submit task. ###### Returns [cudlaSuccess](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0af0a9db4b79be4325d9bb957b05314f1) , [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) , [cudlaErrorInvalidDevice](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04b8ca897368458ceb53b537b8979218b) , [cudlaErrorInvalidModule](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d2b4370207a0a2926eda404f880989d6) , [cudlaErrorCuda](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0727b16e07c9c63a5f1379826a31eb983) , [cudlaErrorUmd](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d22f87ac0d722cd49a74217f472df994) , [cudlaErrorOutOfResources](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a05b9d884f3dd0210f0cfa56e064c29dcb) , [cudlaErrorInvalidAddress](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03959fb7cceee17b5c54b60cdffa40e7f) , [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) , [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) , [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) [cudlaErrorOs](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04576f8875559f34aeb3865f75d409931) ###### Description This operation takes in a sequence of tasks and submits them to the DLA HW for execution in the same sequence as they appear in the input task array. The input and output tensors (and statistics buffer if used) are assumed to be pre-registered using [cudlaMemRegister](index.html#group__CUDLA__API_1ge07f8bb22373163a0117fc5738a23be0 "Registers the CUDA memory to DLA engine.") (in hybrid mode) or [cudlaImportExternalMemory](index.html#group__CUDLA__API_1gca69cd7ac008500693ffeedb18d7a9c8 "Imports external memory into cuDLA.") (in standalone mode). Failure to do so can result in this function returning [cudlaErrorInvalidAddress](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03959fb7cceee17b5c54b60cdffa40e7f) . The stream parameter must be specified as the CUDA stream on which the DLA task is submitted for execution in hybrid mode. In standalone mode, this parameter must be passed as NULL and failure to do so will result in this function returning [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) . The [cudlaTask](index.html#structcudlaTask) structure has a provision to specify wait and signal events that cuDLA must wait on and signal respectively as part of [cudlaSubmitTask()](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") . Each submitted task will wait for all its wait events to be signaled before beginning execution and will provide a signal event (if one is requested for during [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") ) that the application (or any other entity) can wait on to ensure that the submitted task has completed execution. In cuDLA 1.0, only NvSciSync fences are supported as part of wait events. Furthermore, only NvSciSync objects (registered as part of [cudlaImportExternalSemaphore](index.html#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34 "Imports external semaphore into cuDLA.") ) can be signaled as part of signal events and the fence corresponding to the signaled event is returned as part of [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") . In standalone mode, if inputTensor and outputTensor fields are set to NULL inside the [cudlaTask](index.html#structcudlaTask) structure, the task submission is interpreted as an enqueue of wait and signal events that must be considered for subsequent task submission. No actual task submission is done. Multiple such subsequent task submissions with NULL fields in the input/outputTensor fields will overwrite the list of wait and signal events to be considered. In other words, the latest non-null wait events and/or latest non-null signal events before a non-null submission are considered for subsequent actual task submission. During an actual task submit in standalone mode, the effective wait events and signal events that will be considered are what the application sets using NULL data submissions and what is set for that particular task submission in the waitEvents and signalEvents fields. The wait events set as part of NULL data submission are considered as dependencies for only the first task and the signal events set as part of NULL data submission are signaled when the last task of task list is complete. All constraints that apply to waitEvents and signalEvents individually (as described below) are also applicable to the combined list. cuDLA supports 3 kinds of fences - preFence, SOF fence and EOF fence. * preFence is the type of fence that DLA waits on to start the task execution. Use cudlaFenceType as CUDLA\_NVSCISYNC\_FENCE to mark a fence as preFence. * SOF(Start Of Frame) fence is the type of fence which is signaled before the task execution on DLA starts. Use cudlaFenceType as CUDLA\_NVSCISYNC\_FENCE\_SOF to mark a fence as SOF fence. * EOF(End Of Frame) fence is the type of fence which is signaled after the task execution on DLA is complete. Use cudlaFenceType as CUDLA\_NVSCISYNC\_FENCE to mark a fence as EOF fence. For wait events, applications are expected to * register their synchronization objects using [cudlaImportExternalSemaphore](index.html#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34 "Imports external semaphore into cuDLA.") . * create the required number of preFence placeholders using [CudlaFence](index.html#structCudlaFence) . * fill in the placeholders with the relevant fences from the application. * list out all the fences in [cudlaWaitEvents](index.html#structcudlaWaitEvents) . For signal events, applications are expected to * register their synchronization objects using [cudlaImportExternalSemaphore](index.html#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34 "Imports external semaphore into cuDLA.") . * create the required number of SOF and EOF fence placeholder fences using [CudlaFence](index.html#structCudlaFence) . * place the registered objects and the corresponding fences in [cudlaSignalEvents](index.html#structcudlaSignalEvents) . In case ofdeterministic semaphore, fence is not required to be passed in [cudlaSignalEvents](index.html#structcudlaSignalEvents) . When [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") returns successfully, the fences present in [cudlaSignalEvents](index.html#structcudlaSignalEvents) can be used to wait for the particular task to be completed. cuDLA supports 1 sync point and any number of semaphores as part of [cudlaSignalEvents](index.html#structcudlaSignalEvents) . If more than 1 sync point is specified, [cudlaErrorInvalidParam](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0c80900e54c2a9c02f23935a69f214428) is returned. cuDLA adheres to DLA's restriction to support 29 preFences and SOF fences combined together and 29 EOF fences per DLA Task. During submission, users have an option to enable layerwise statistics profiling for the individual layers of the network. This option needs to be exercised by specifying additional output buffers that would contain the profiling information. Specifically, * "cudlaTask::numOutputTensors" should be the sum of value returned by cudlaModuleGetAttributes(...,CUDLA\_NUM\_OUTPUT\_TENSORS,...) and cudlaModuleGetAttributes(...,CUDLA\_NUM\_OUTPUT\_TASK\_STATISTICS,...) * "cudlaTask::outputTensor" should contain the array of output tensors appended with array of statistics output buffer. This function can return [cudlaErrorUnsupportedOperation](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a03ba9ef84a3f15d6d1145047e11d1804a) if * stream being used in hybrid mode is in capturing state. * application attempts to use NvSci functionalities in hybrid mode. * loading of NvSci libraries failed for a particular platform. * fence type other than [CUDLA\_NVSCISYNC\_FENCE](index.html#group__CUDLA__TYPES_1ggfd5fd195c04a78364ab8b66904354bb80c71b2c385910641adc43fa2c987fc68) is specified. * waitEvents or signaEvents is not NULL in hybrid mode. * inputTensor or outputTensor is NULL in hybrid mode and the flags are not CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK. * inputTensor is NULL and outputTensor is not NULL and vice versa in standalone mode and the flags are not CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK. * inputTensor and outputTensor is NULL and number of tasks is not equal to 1 in standalone mode and the flags are not CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK. * inputTensor is not NULL or output tensor is NULL and the flags are CUDLA\_SUBMIT\_DIAGNOSTICS\_TASK. * the effective signal events list has multiple sync points to signal. * if layerwise feature is unsupported. * if preFences, SOF fences and EOF fences limit per task is not met. This function can return [cudlaErrorNvSci](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0d3dfc3fc5ad64fbb4b03c3861502667e) or [cudlaErrorInvalidAttribute](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a0df7b517d9f6d1ca10675c1e988e5101b) in certain cases when the underlying NvSci operation fails. This function can return [cudlaErrorOs](index.html#group__CUDLA__TYPES_1ggc95f8bcde047e255b11ea1b3b80598a04576f8875559f34aeb3865f75d409931) if an internal system operation fails. Note: This API can return task execution errors from previous DLA task submissions. 2. Data Structures ------------------ Here are the data structures with brief descriptions: [cudlaDevAttribute](index.html#unioncudlaDevAttribute) [cudlaExternalMemoryHandleDesc](index.html#structcudlaExternalMemoryHandleDesc__t) [cudlaExternalSemaphoreHandleDesc](index.html#structcudlaExternalSemaphoreHandleDesc__t) [CudlaFence](index.html#structCudlaFence) [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) [cudlaSignalEvents](index.html#structcudlaSignalEvents) [cudlaTask](index.html#structcudlaTask) [cudlaWaitEvents](index.html#structcudlaWaitEvents) ### 2.1. cudlaDevAttribute Union Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Device attribute. #### Public Variables uint32\_t  [deviceVersion](#unioncudlaDevAttribute_133aed5d6be4a007d2eff7dfd942aac55) uint8\_t  [unifiedAddressingSupported](#unioncudlaDevAttribute_164d9b42c7cd37db2213b1c57592cc208) #### Variables uint32\_t [cudlaDevAttribute](index.html#unioncudlaDevAttribute) ::[deviceVersion](index.html#unioncudlaDevAttribute_133aed5d6be4a007d2eff7dfd942aac55) \[inherited\] DLA device version. Xavier has 1.0 and Orin has 2.0. uint8\_t [cudlaDevAttribute](index.html#unioncudlaDevAttribute) ::[unifiedAddressingSupported](index.html#unioncudlaDevAttribute_164d9b42c7cd37db2213b1c57592cc208) \[inherited\] Returns 0 if unified addressing is not supported. ### 2.2. cudlaExternalMemoryHandleDesc\_t Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] External memory handle descriptor. #### Public Variables const void \* [extBufObject](#structcudlaExternalMemoryHandleDesc__t_1a6d1e29a22e55ffdf323bc52e52c3836) unsigned long long  [size](#structcudlaExternalMemoryHandleDesc__t_102606c9f4cf2dbe1d0b17da4b29de3b0) #### Variables const void \* [cudlaExternalMemoryHandleDesc\_t](index.html#structcudlaExternalMemoryHandleDesc__t) ::[extBufObject](index.html#structcudlaExternalMemoryHandleDesc__t_1a6d1e29a22e55ffdf323bc52e52c3836) \[inherited\] A handle representing an external memory object. unsigned long long [cudlaExternalMemoryHandleDesc\_t](index.html#structcudlaExternalMemoryHandleDesc__t) ::[size](index.html#structcudlaExternalMemoryHandleDesc__t_102606c9f4cf2dbe1d0b17da4b29de3b0) \[inherited\] Size of the memory allocation ### 2.3. cudlaExternalSemaphoreHandleDesc\_t Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] External semaphore handle descriptor. #### Public Variables const void \* [extSyncObject](#structcudlaExternalSemaphoreHandleDesc__t_17f57bcbdb8b0e8bf0655e14b9965c9aa) #### Variables const void \* [cudlaExternalSemaphoreHandleDesc\_t](index.html#structcudlaExternalSemaphoreHandleDesc__t) ::[extSyncObject](index.html#structcudlaExternalSemaphoreHandleDesc__t_17f57bcbdb8b0e8bf0655e14b9965c9aa) \[inherited\] A handle representing an external synchronization object. ### 2.4. CudlaFence Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Fence description. #### Public Variables void \* [fence](#structCudlaFence_12776eab9e902b8db4863097634b35a15) [cudlaFenceType](index.html#group__CUDLA__TYPES_1gfd5fd195c04a78364ab8b66904354bb8) [type](#structCudlaFence_1c7cf9a34d20dea7e789be4e5b9ef21fe) #### Variables void \* [CudlaFence](index.html#structCudlaFence) ::[fence](index.html#structCudlaFence_12776eab9e902b8db4863097634b35a15) \[inherited\] Fence. [cudlaFenceType](index.html#group__CUDLA__TYPES_1gfd5fd195c04a78364ab8b66904354bb8) [CudlaFence](index.html#structCudlaFence) ::[type](index.html#structCudlaFence_1c7cf9a34d20dea7e789be4e5b9ef21fe) \[inherited\] Fence type. ### 2.5. cudlaModuleAttribute Union Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Module attribute. #### Public Variables [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) \* [inputTensorDesc](#unioncudlaModuleAttribute_13c7a3e737a6b176e6e9c70988022d5ec) uint32\_t  [numInputTensors](#unioncudlaModuleAttribute_1589db412b98fdf2da4fd6c30391dda4a) uint32\_t  [numOutputTensors](#unioncudlaModuleAttribute_13fb1292746718ce5f18cec46a9e63b86) [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) \* [outputTensorDesc](#unioncudlaModuleAttribute_12eaa15b61ffd7e6e68c8889305c5826f) #### Variables [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) \* [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) ::[inputTensorDesc](index.html#unioncudlaModuleAttribute_13c7a3e737a6b176e6e9c70988022d5ec) \[inherited\] Returns an array of input tensor descriptors. uint32\_t [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) ::[numInputTensors](index.html#unioncudlaModuleAttribute_1589db412b98fdf2da4fd6c30391dda4a) \[inherited\] Returns the number of input tensors. uint32\_t [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) ::[numOutputTensors](index.html#unioncudlaModuleAttribute_13fb1292746718ce5f18cec46a9e63b86) \[inherited\] Returns the number of output tensors. [cudlaModuleTensorDescriptor](index.html#structcudlaModuleTensorDescriptor) \* [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) ::[outputTensorDesc](index.html#unioncudlaModuleAttribute_12eaa15b61ffd7e6e68c8889305c5826f) \[inherited\] Returns an array of output tensor descriptors. ### 2.6. cudlaModuleTensorDescriptor Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Tensor descriptor. ### 2.7. cudlaSignalEvents Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Signal events for [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") #### Public Variables const \* [devPtrs](#structcudlaSignalEvents_1754b5a36ca8b858395c02fadc6492ce1) [CudlaFence](index.html#structCudlaFence) \* [eofFences](#structcudlaSignalEvents_1be21907c65bc41726c111405c376c3ac) uint32\_t  [numEvents](#structcudlaSignalEvents_1723456f4fef95443670e308dcfc94845) #### Variables const \* [cudlaSignalEvents](index.html#structcudlaSignalEvents) ::[devPtrs](index.html#structcudlaSignalEvents_1754b5a36ca8b858395c02fadc6492ce1) \[inherited\] Array of registered synchronization objects (via [cudlaImportExternalSemaphore](index.html#group__CUDLA__API_1g12751fbcc295349c16ad3aea0e8bda34 "Imports external semaphore into cuDLA.") ). [CudlaFence](index.html#structCudlaFence) \* [cudlaSignalEvents](index.html#structcudlaSignalEvents) ::[eofFences](index.html#structcudlaSignalEvents_1be21907c65bc41726c111405c376c3ac) \[inherited\] Array of fences pointers for all the signal events corresponding to the synchronization objects. uint32\_t [cudlaSignalEvents](index.html#structcudlaSignalEvents) ::[numEvents](index.html#structcudlaSignalEvents_1723456f4fef95443670e308dcfc94845) \[inherited\] Total number of signal events. ### 2.8. cudlaTask Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Structure of Task. #### Public Variables const \* [inputTensor](#structcudlaTask_1dd4cb87ea94c16385134477170c40aef) [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5) [moduleHandle](#structcudlaTask_1f61768f306610f76fe18ca2ef0bafd78) uint32\_t  [numInputTensors](#structcudlaTask_17003a7c700c41d5919b947f0bc4caf33) uint32\_t  [numOutputTensors](#structcudlaTask_1f05bb162c3f14bb8b012d0f9f60ce9ac) const \* [outputTensor](#structcudlaTask_192e99490ea0198e094233cf758a96e47) [cudlaSignalEvents](index.html#structcudlaSignalEvents) \* [signalEvents](#structcudlaTask_1c8f145341906339b3cffdeda9db30d3c) const [cudlaWaitEvents](index.html#structcudlaWaitEvents) \* [waitEvents](#structcudlaTask_1d1b971c2701c6292b7fe894d251d5848) #### Variables const \* [cudlaTask](index.html#structcudlaTask) ::[inputTensor](index.html#structcudlaTask_1dd4cb87ea94c16385134477170c40aef) \[inherited\] Array of input tensors. [cudlaModule](index.html#group__CUDLA__TYPES_1ge3ecb829f32791357568b7d005d107a5) [cudlaTask](index.html#structcudlaTask) ::[moduleHandle](index.html#structcudlaTask_1f61768f306610f76fe18ca2ef0bafd78) \[inherited\] cuDLA module handle. uint32\_t [cudlaTask](index.html#structcudlaTask) ::[numInputTensors](index.html#structcudlaTask_17003a7c700c41d5919b947f0bc4caf33) \[inherited\] Number of input tensors. uint32\_t [cudlaTask](index.html#structcudlaTask) ::[numOutputTensors](index.html#structcudlaTask_1f05bb162c3f14bb8b012d0f9f60ce9ac) \[inherited\] Number of output tensors. const \* [cudlaTask](index.html#structcudlaTask) ::[outputTensor](index.html#structcudlaTask_192e99490ea0198e094233cf758a96e47) \[inherited\] Array of output tensors. [cudlaSignalEvents](index.html#structcudlaSignalEvents) \* [cudlaTask](index.html#structcudlaTask) ::[signalEvents](index.html#structcudlaTask_1c8f145341906339b3cffdeda9db30d3c) \[inherited\] Signal events. const [cudlaWaitEvents](index.html#structcudlaWaitEvents) \* [cudlaTask](index.html#structcudlaTask) ::[waitEvents](index.html#structcudlaTask_1d1b971c2701c6292b7fe894d251d5848) \[inherited\] Wait events. ### 2.9. cudlaWaitEvents Struct Reference ### \[[Data types used by cuDLA driver](index.html#group__CUDLA__TYPES)\ \] Wait events for [cudlaSubmitTask](index.html#group__CUDLA__API_1gc560a614b388d50216bd161c0b3d88cb "Submits the inference operation on DLA.") . #### Public Variables uint32\_t  [numEvents](#structcudlaWaitEvents_1103e821b8ab32dfb82d20818a8467882) const [CudlaFence](index.html#structCudlaFence) \* [preFences](#structcudlaWaitEvents_1fe868337134eb67e12a2a30679976f0c) #### Variables uint32\_t [cudlaWaitEvents](index.html#structcudlaWaitEvents) ::[numEvents](index.html#structcudlaWaitEvents_1103e821b8ab32dfb82d20818a8467882) \[inherited\] Total number of wait events. const [CudlaFence](index.html#structCudlaFence) \* [cudlaWaitEvents](index.html#structcudlaWaitEvents) ::[preFences](index.html#structcudlaWaitEvents_1fe868337134eb67e12a2a30679976f0c) \[inherited\] Array of fence pointers for all the wait events. 3. Data Fields -------------- Here is a list of all documented struct and union fields with links to the struct/union documentation for each field: deviceVersion [cudlaDevAttribute](index.html#unioncudlaDevAttribute) devPtrs [cudlaSignalEvents](index.html#structcudlaSignalEvents) eofFences [cudlaSignalEvents](index.html#structcudlaSignalEvents) extBufObject [cudlaExternalMemoryHandleDesc](index.html#structcudlaExternalMemoryHandleDesc__t) extSyncObject [cudlaExternalSemaphoreHandleDesc](index.html#structcudlaExternalSemaphoreHandleDesc__t) fence [CudlaFence](index.html#structCudlaFence) inputTensor [cudlaTask](index.html#structcudlaTask) inputTensorDesc [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) moduleHandle [cudlaTask](index.html#structcudlaTask) numEvents [cudlaWaitEvents](index.html#structcudlaWaitEvents) [cudlaSignalEvents](index.html#structcudlaSignalEvents) numInputTensors [cudlaTask](index.html#structcudlaTask) [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) numOutputTensors [cudlaTask](index.html#structcudlaTask) [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) outputTensor [cudlaTask](index.html#structcudlaTask) outputTensorDesc [cudlaModuleAttribute](index.html#unioncudlaModuleAttribute) preFences [cudlaWaitEvents](index.html#structcudlaWaitEvents) signalEvents [cudlaTask](index.html#structcudlaTask) size [cudlaExternalMemoryHandleDesc](index.html#structcudlaExternalMemoryHandleDesc__t) type [CudlaFence](index.html#structCudlaFence) unifiedAddressingSupported [cudlaDevAttribute](index.html#unioncudlaDevAttribute) waitEvents [cudlaTask](index.html#structcudlaTask) Notices ------- ### Notice This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. ### OpenCL OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. ### Trademarks NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. ### Copyright © 2021\-2025 NVIDIA Corporation & affiliates. All rights reserved. This product includes software developed by the Syncro Soft SRL (http://www.sync.ro/). * * * --- # 1. GDS cuFile API Reference — GDS cuFile API Reference [Skip to main content](#main-content) Back to top Ctrl+K [![GDS cuFile API Reference - Home](_static/nvidia-logo-horiz-rgb-blk-for-screen.svg)\ \ GDS cuFile API Reference](contents.html) * v1.13 | * [PDF](../pdf/api-reference.pdf) 1\. GDS cuFile API Reference[#](#gds-cufile-api-reference "Link to this heading") ================================================================================== The NVIDIA® GPUDirect® Storage cuFile API Reference Guide provides information about the cuFile API reference that is used in applications and frameworks to leverage GDS technology and describes the intent, context, and operation of those APIs, which are part of the GDS technology. 2\. Introduction[#](#introduction "Link to this heading") ========================================================== NVIDIA® Magnum IO GPUDirect® Storage (GDS) is part of the GPUDirect family. GDS enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU. This direct path increases system bandwidth and decreases the latency and utilization load on the CPU. This document provides information about the cuFile APIs that are used in applications and frameworks to leverage GDS technology and describes the intent, context, and operation of those APIs which are part of the GDS technology. Note The APIs and descriptions are subject to change without notice. 3\. Usage[#](#usage "Link to this heading") ============================================ This section describes the operation of the cuFile APIs. Because the functionality is part of the CUDA Driver C API, the APIs use the `cuFile` prefix and camel case motif of the CUDA Driver. * All APIs are thread-safe. * The fork system call should not be used after the library is initialized. The behavior of the APIs after the fork system call is undefined in the child process. * The APIs with GPU buffers should be called in a valid CUDA context and stream if applicable. * All APIs are issued from the CPU, not the GPU. Note Starting from CUDA toolkit 12.2 (GDS version 1.7.x) release cuFile APIs support memory allocated on GPU device as well as host memory. peer to peer transfer using GPUDirect is supported to and from device memory on supported file system and hardware configurations. The APIs will refer to this memory address as buffer pointer unless the API specifically applies to a particular type of memory. 3.1. Dynamic Interactions[#](#dynamic-interactions "Link to this heading") --------------------------------------------------------------------------- The following describes the dynamic interactions between the cuFile APIs. Some of the cuFile APIs are optional. If they are not called proactively, their actions will occur reactively: If `cuFile{DriverOpen, HandleRegister, BufRegister} `` is called on a driver, file, or buffer, respectively that has been opened or registered by a previous ``cuFile`\* API call, this will result in an error. Calling `cuFile{BufDeregister, HandleDeregister, DriverClose}` on a buffer, file, or driver, respectively that has never been opened or registered by a previous `cuFile`\* API call results in an error. For these errors, the output parameters of the APIs are left in an undefined state, and there are no other side effects. * `cuFileDriverOpen` explicitly causes driver initialization. Its use is optional. If it is not used, driver initialization happens implicitly at the first use of the `cuFile{HandleRegister, Read, Write, BufRegister}` APIs. * (Mandatory) `cuFileHandleRegister` turns an OS-specific file descriptor into a `CUfileHandle_t` and performs checking on the GDS supportability based on the mount point and the way that the file was opened. * `cuFileBufRegister` explicitly registers a memory buffer. If this API is not called, an internal registered memory is used if required on the first time the buffer is used, for example, in `cuFile{Read, Write}`. * `cuFile{BufDeregister, HandleDeregister}` explicitly frees a buffer and file resources, respectively. If this API is not called, the buffer and resources are implicitly freed when the driver is closed using `cuFileDriverClose`. * `cuFileDriverClose` explicitly frees driver resources. If this API is not called, the driver resources are implicitly freed when `dlclose()` is performed on the library handle or when the process is terminated. 3.2. Driver, File, and Buffer Management[#](#driver-file-and-buffer-management "Link to this heading") ------------------------------------------------------------------------------------------------------- This section describes the overall workflow to manage the driver, the file, and buffer management: 1. Call `cuFileDriverOpen()` to initialize the state of the critical performance path. 2. Allocate GPU memory with cudaMalloc, `cudaMallocManaged`, `cuMem*` APIs or host memory using `cudaMallocHost`, `malloc` or `mmap`. 3. To register the buffer, call `cuFileBufRegister` to initialize the buffer state of the critical performance path. 4. Complete the following IO workflow: 1. For Linux, open a file with POSIX open. 2. Call `cuFileHandleRegister` to wrap an existing file descriptor in an OS-agnostic `CUfileHandle_t`. This step evaluates the suitability of the file state and the file mount for GDS and initializes the file state of the critical performance path. 3. Call IO APIs such as `cuFileRead`/`cuFileWrite` on an existing cuFile handle and existing buffer. * If the `cuFileBufRegister` has not been previously called on the buffer pointer, `cuFileRead/cuFileWrite` will use internal registered buffers when required. * Not using cuFileBufRegister might not be performant for small IO sizes. * Refer to the [GPUDirect Best Practices Guide](https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html) for more information. 4. Unless an error condition is returned, the IO is performed successfully. 5. Call `cuFileBufDeregister` to free the buffer-specific cuFile state. 6. Call `cuFileHandleDeregister` to free the file-specific cuFile state. 7. Call `cuFileDriverClose` to free up the cuFile state. Note Not using the `cuFileDeregister` and `cuFileDriverClose` APIs (steps 5, 6, and 7) might unnecessarily consume resources, as shown by tools such as valgrind. The best practice is to always call these APIs in the application cleanup paths. 3.3. cuFile Compatibility Mode[#](#cufile-compatibility-mode "Link to this heading") ------------------------------------------------------------------------------------- **Use Cases** cuFile APIs can be used in different scenarios: * Developers building GPUDirect Storage applications with cuFile APIs, but don’t have the supported hardware configurations. * Developers building applications running on GPU cards that have CUDA compute capability > 6, but don’t have BAR space exposed. * Deployments where `nvidia-fs.ko` is not loaded or cannot be loaded. * Deployments where the Linux distribution does not support GPUDirect Storage. * Deployments where the filesystem may be not supported with GPUDirect Storage. * Deployments where the network links are not enabled with RDMA support. * Deployment where the configuration is not optimal for GPUDirect Storage. **Behavior** The cuFile library provides a mechanism for cuFile reads and writes to use compatibility mode using POSIX `pread`, `pwrite`, and `aio_submit` APIS respectively to host memory and copying to GPU memory when applicable. The behavior of compatibility mode with cuFile APIs is determined by the following configuration parameters. | Configuration Option (default) | cuFile IO Behavior | | --- | --- | | “allow\_compat\_mode”: true | If `true`, falls back to using compatibility mode when the library detects that the buffer file descriptor opened cannot use GPUDirect Storage. | | “force\_compat\_mode”: false | If `true`, this option can be used to force all IO to use compatibility mode. Alternatively the admin can unload the nvidia\_fs.ko or not expose the character devices in the docker container environment. | | “gds\_rdma\_write\_support”: true | If `false`, forces compatibility mode to be used for writes even when the underlying file system is capable of performing GPUDirect Storage writes. **Note:** If the option is “false”, this option will override and disable any filesystem-specific option to enable RDMA writes. | | “posix\_unaligned\_writes” : false | If `true`, forces compatibility mode to be used for writes where the file offset and/or IO size is not aligned to Page Boundary (4KB). | | “lustre:posix\_gds\_min\_kb” : 0 | For a lustre filesystem, if greater than `0`, compatibility mode is used for IO sizes between \[1 - `posix_gds_min_kb`\] specified in kB.

**Note:** This option will force posix mode even if `allow_compat_mode` is set to `false`. | | “weka:rdma\_write\_support” : false | If this option is `false`, all writes to WekaFS will use compatibility mode.

**Note:** If the option is set to `false`, cuFile library will use the posix path even if the `allow_compat_mode` option is `true` or `false`. | | “gpfs:gds\_write\_support” : false | If this option is false, all writes to IBM Spectrum Scale will use compatibility mode.

**Note:** If the option is set to `false`, cuFile library will use the posix path even if the `allow_compat_mode` option is true or false. | | “rdma\_dynamic\_routing”: false,

“rdma\_dynamic\_routing\_order”: \[ “ “SYS\_MEM” \] | If `rdma_dynamic_routing` is set to `true` and `rdma_dynamic_routing_order` is set to `[SYS_MEM]`, then all IO for DFS will use compatibility mode. | In addition to the above configuration options, compatibility mode will be used as a fallback option for following use cases. | Use Case | cuFile IO Behavior | | --- | --- | | No BAR1 memory in GPU. | Use compatibility mode. | | For wekaFS or IBM Spectrum Scale mounts: If there are no `rdma_dev_addr_list` specified, or failure to register MR with ib device. | Use compatibility mode. | | Bounce buffers cannot be allocated in GPU memory. | Use compatibility mode. | | For WekaFS and IBM Spectrum Scale: If the kernel returns `-ENOTSUP` for GPUDirect Storage read/write. | Retry the IO operation internally using compatibility mode. | | cuFile Stream and cuFile Batch APIs on IBM Spectrum Scale or WekaFS | All Async and batch operations will internally use compatibility mode IO. | | The `nvidia_fs.ko` driver is not loaded. | All IO operations will use compatibility mode. | **Limitations** * Compatible mode does not work in cases where the GPUs have CUDA compute capability less than 6. * GDS Compat mode has been tested and works with GDS enabled file systems and environments. It has not been tested to work on all other filesystems. 4\. cuFile API Specification[#](#cufile-api-specification "Link to this heading") ================================================================================== This section provides information about the cuFile APIs that are used from the CPU to enable applications and frameworks. 4.1. Data Types[#](#data-types "Link to this heading") ------------------------------------------------------- ### 4.1.1. Declarations and Definitions[#](#declarations-and-definitions "Link to this heading") Here are the relevant cuFile enums and their descriptions. typedef struct CUfileError { CUfileOpError err; // cufile error enum CUresult cu\_err; // for CUDA-specific errors } CUfileError\_t; /\*\* \* error macros to inspect error status of type CUfileOpError \*/ #define IS\_CUFILE\_ERR(err) \\ (abs((err)) > CUFILEOP\_BASE\_ERR) #define CUFILE\_ERRSTR(err) \\ cufileop\_status\_error(static\_cast(abs((err)))) #define IS\_CUDA\_ERR(status) \\ ((status).err == CU\_FILE\_CUDA\_DRIVER\_ERROR) #define CU\_FILE\_CUDA\_ERR(status) ((status).cu\_ The following enum and two structures enable broader cross-OS support: enum CUfileFileHandleType { CU\_FILE\_HANDLE\_TYPE\_OPAQUE\_FD \= 1, /\* linux based fd \*/ CU\_FILE\_HANDLE\_TYPE\_OPAQUE\_WIN32 \= 2, /\* windows based handle \*/ CU\_FILE\_HANDLE\_TYPE\_USERSPACE\_FS \= 3, /\* userspace based FS \*/ }; typedef struct CUfileDescr\_t { CUfileFileHandleType type; /\* type of file being registered \*/ union { int fd; /\* Linux \*/ void \*handle; /\* Windows \*/ } handle; const CUfileFSOps\_t \*fs\_ops; /\* file system operation table \*/ } CUfileDescr\_t; /\* cuFile handle type \*/ typedef void\* CUfileHandle\_t; typedef struct cufileRDMAInfo { int version; int desc\_len; const char \*desc\_str; } cufileRDMAInfo\_t; typedef struct CUfileFSOps { /\* NULL means discover using fstat \*/ const char\* (\*fs\_type) (void \*handle); /\* list of host addresses to use, NULL means no restriction \*/ int (\*getRDMADeviceList)(void \*handle, sockaddr\_t \*\*hostaddrs); /\* -1 no pref \*/ int (\*getRDMADevicePriority)(void \*handle, char\*, size\_t, loff\_t, sockaddr\_t\* hostaddr); /\* NULL means try VFS \*/ ssize\_t (\*read) (void \*handle, char\*, size\_t, loff\_t, cufileRDMAInfo\_t\*); ssize\_t (\*write) (void \*handle, const char \*, size\_t, loff\_t , cufileRDMAInfo\_t\*); } CUfileFSOps\_t; typedef enum CUfileDriverStatusFlags { CU\_FILE\_LUSTRE\_SUPPORTED \= 0, /\*!< Support for DDN LUSTRE \*/ CU\_FILE\_WEKAFS\_SUPPORTED \= 1, /\*!< Support for WEKAFS \*/ CU\_FILE\_NFS\_SUPPORTED \= 2, /\*!< Support for NFS \*/ CU\_FILE\_GPFS\_SUPPORTED \= 3, /\*! < Support for GPFS \*/ CU\_FILE\_NVME\_SUPPORTED \= 4, /\*!< Support for NVMe \*/ CU\_FILE\_NVMEOF\_SUPPORTED \= 5, /\*!< Support for NVMeOF \*/ CU\_FILE\_SCSI\_SUPPORTED \= 6, /\*!< Support for SCSI \*/ CU\_FILE\_SCALEFLUX\_CSD\_SUPPORTED \= 7, /\*!< Support for Scaleflux CSD\*/ CU\_FILE\_NVMESH\_SUPPORTED \= 8, /\*!< Support for NVMesh Block Dev\*/ CU\_FILE\_BEEGFS\_SUPPORTED \= 9, /\*!< Support for BeeGFS \*/ } CUfileDriverStatusFlags\_t; enum CUfileDriverControlFlags { CU\_FILE\_USE\_POLL\_MODE \= 0, /\*!< use POLL mode. properties.use\_poll\_mode\*/ CU\_FILE\_ALLOW\_COMPAT\_MODE \= 1 /\*!< allow COMPATIBILITY mode. properties.allow\_compat\_mode\*/ }; typedef enum CUfileFeatureFlags { CU\_FILE\_DYN\_ROUTING\_SUPPORTED \=0, CU\_FILE\_BATCH\_IO\_SUPPORTED \= 1, CU\_FILE\_STREAMS\_SUPPORTED \= 2 } CUfileFeatureFlags\_t;; /\* cuFileDriverGetProperties describes this structure's members \*/ typedef struct CUfileDrvProps { struct { unsigned int major\_version; unsigned int minor\_version; size\_t poll\_thresh\_size; size\_t max\_direct\_io\_size; unsigned int dstatusflags; unsigned int dcontrolflags; } nvfs; CUfileFeatureFlags\_t fflags; unsigned int max\_device\_cache\_size; unsigned int per\_buffer\_cache\_size; unsigned int max\_pinned\_memory\_size; unsigned int max\_batch\_io\_timeout\_msecs; } CUfileDrvProps\_t; /\* Parameter block for async cuFile IO \*/ /\* Batch APIs use an array of these \*/ /\* Status must be CU\_FILE\_WAITING when submitted, and is updated when enqueued and when complete, so this user-allocated structure is live until the operation completes. \*/ typedef enum CUFILEStatus\_enum { CUFILE\_WAITING \= 0x000001, /\* required value prior to submission \*/ CUFILE\_PENDING \= 0x000002, /\* once enqueued \*/ CUFILE\_INVALID \= 0x000004, /\* request was ill-formed or could not be enqueued \*/ CUFILE\_CANCELED \= 0x000008, /\* request successfully canceled \*/ CUFILE\_COMPLETE \= 0x0000010, /\* request successfully completed \*/ CUFILE\_TIMEOUT \= 0x0000020, /\* request timed out \*/ CUFILE\_FAILED \= 0x0000040 /\* unable to complete \*/ }CUfileStatus\_t; typedef enum cufileBatchMode { CUFILE\_BATCH \= 1, } CUfileBatchMode\_t; typedef struct CUfileIOParams { CUfileBatchMode\_t mode; // Must be the very first field. union { struct { void \*devPtr\_base; off\_t file\_offset; off\_t devPtr\_offset; size\_t size; }batch; }u; CUfileHandle\_t fh; CUfileOpcode\_t opcode; void \*cookie; } CUfileIOParams\_t; typedef struct CUfileIOEvents { void \*cookie; CUfileStatus\_t status; /\* status of the operation \*/ size\_t ret; /\* -ve error or amount of I/O done. \*/ } CUfileIOEvents\_t; ### 4.1.2. Typedefs[#](#typedefs "Link to this heading") cuFile typedefs: typedef struct CUfileDescr CUfileDesr\_t typedef struct CUfileError CUfileError\_t typedef struct CUfileDrvProps CUfileDrvProps\_t typedef enum CUfileFeatureFlags CUfileFeatureFlags\_t typedef enum CUfileDriverStatusFlags\_enum CUfileDriverStatusFlags\_t typedef enum CUfileDriverControlFlags\_enum CUfileDriverControlFlags\_t typedef struct CUfileIOParams CUfileIOParams\_t typedef enum CUfileBatchOpcode CUfileBatchOpcode\_t ### 4.1.3. Enumerations[#](#enumerations "Link to this heading") cuFile enums: * `enum CUfileOpcode_enum` This is the cuFile operation code for batch mode. | OpCode | Value | Description | | --- | --- | --- | | `CU_FILE_READ` | 0 | Batch Read | | `CU_FILE_WRITE` | 1 | Batch Write | > /\* cuFile Batch IO operation kind \*/ > enum CUfileOpcode { > CU\_FILE\_READ, > CU\_FILE\_WRITE, > }; * `enum CUfileStatus` The cuFile Status codes for batch mode. | Status | Value | Description | | --- | --- | --- | | `CUFILE_WAITING` | 0x01 | The initial value. | | `CUFILE_PENDING` | 0x02 | Set once enqueued into the driver. | | `CUFILE_INVALID` | 0x04 | Invalid parameters. | | `CUFILE_CANCELED` | 0x08 | Request successfully canceled. | | `CUFILE_COMPLETE` | 0x10 | Successfully completed. | | `CUFILE_TIMEOUT` | 0x20 | The operation has timed out. | | `CUFILE_FAILED` | 0x40 | IO has failed. | * `enum CUfileOpError` * The cuFile Operation error types. * All error code values, other than `CU_FILE_SUCCESS`, are considered failures that might leave the output and input parameter values of APIs in an undefined state. These values cannot have any side effects on the file system, the application process, and the larger system. Note cuFile-specific errors will be greater than `CUFILEOP_BASE_ERR` to enable users to distinguish between POSIX errors and cuFile errors. #define CUFILEOP\_BASE\_ERR 5000 | Error Code | Value | Description | | --- | --- | --- | | `CU_FILE_SUCCESS` | 0 | The cufile is successful. | | `CU_FILE_DRIVER_NOT_INITIALIZED` | 5001 | The nvidia-fs driver is not loaded. | | `CU_FILE_DRIVER_INVALID_PROPS` | 5002 | An invalid property. | | `CU_FILE_DRIVER_UNSUPPORTED_LIMIT` | 5003 | A property range error. | | `CU_FILE_DRIVER_VERSION_MISMATCH` | 5004 | An nvidia-fs driver version mismatch. | | `CU_FILE_DRIVER_VERSION_READ_ERROR` | 5005 | An nvidia-fs driver version read error. | | `CU_FILE_DRIVER_CLOSING` | 5006 | Driver shutdown in progress. | | `CU_FILE_PLATFORM_NOT_SUPPORTED` | 500 | GDS is not supported on the current platform. | | `CU_FILE_IO_NOT_SUPPORTED` | 5008 | GDS is not supported on the current file. | | `CU_FILE_DEVICE_NOT_SUPPORTED` | 5009 | GDS is not supported on the current GPU. | | `CU_FILE_NVFS_DRIVER_ERROR` | 5010 | An nvidia-fs driver ioctl error. | | `CU_FILE_CUDA_DRIVER_ERROR` | 5011 | A CUDA Driver API error.

This error indicates a CUDA driver-api error. If this is set, a CUDA-specific error code is set in the cu\_err field for cuFileError. | | `CU_FILE_CUDA_POINTER_INVALID` | 5012 | An invalid device pointer. | | `CU_FILE_CUDA_MEMORY_TYPE_INVALID` | 5013 | An invalid pointer memory type. | | `CU_FILE_CUDA_POINTER_RANGE_ERROR` | 5014 | The pointer range exceeds the allocated address range. | | `CU_FILE_CUDA_CONTEXT_MISMATCH` | 5015 | A CUDA context mismatch. | | `CU_FILE_INVALID_MAPPING_SIZE` | 5016 | Access beyond the maximum pinned memory size. | | `CU_FILE_INVALID_MAPPING_RANGE` | 5017 | Access beyond the mapped size. | | `CU_FILE_INVALID_FILE_TYPE` | 5018 | An unsupported file type. | | `CU_FILE_INVALID_FILE_OPEN_FLAG` | 5019 | Unsupported file open flags. | | `CU_FILE_DIO_NOT_SET` | 5020 | The fd direct IO is not set. | | `CU_FILE_INVALID_VALUE` | 5022 | Invalid API arguments. | | `CU_FILE_MEMORY_ALREADY_REGISTERED` | 5023 | Device pointer is already registered. | | `CU_FILE_MEMORY_NOT_REGISTERED` | 5024 | A device pointer lookup failure has occurred. | | `CU_FILE_PERMISSION_DENIED` | 5025 | A driver or file access error. | | `CU_FILE_DRIVER_ALREADY_OPEN` | 5026 | The driver is already open. | | `CU_FILE_HANDLE_NOT_REGISTERED` | 5027 | The file descriptor is not registered. | | `CU_FILE_HANDLE_ALREADY_REGISTERED` | 5028 | The file descriptor is already registered. | | `CU_FILE_DEVICE_NOT_FOUND` | 5029 | The GPU device cannot be not found. | | `CU_FILE_INTERNAL_ERROR` | 5030 | An internal error has occurred. Refer to `cufile.log` for more details. | | `CU_FILE_GETNEWFD_FAILED` | 5031 | Failed to obtain a new file descriptor. | | `CU_FILE_NVFS_SETUP_ERROR` | 5033 | An NVFS driver initialization error has occurred. | | `CU_FILE_IO_DISABLED` | 5034 | GDS is disabled by config on the current file. | | `CU_FILE_BATCH_SUBMIT_FAILED` | 5035 | Failed to submit a batch operation. | | `CU_FILE_GPU_MEMORY_PINNING_FAILED` | 5036 | Failed to allocate pinned GPU memory. | | `CU_FILE_BATCH_FULL` | 5037 | Queue full for batch operation. | | `CU_FILE_ASYNC_NOT_SUPPORTED` | 5038 | cuFile stream operation is not supported. | Note Data path errors are captured via standard error codes by using errno. The APIs will return -1 on error. 4.2. cuFile Driver APIs[#](#cufile-driver-apis "Link to this heading") ----------------------------------------------------------------------- The following cuFile APIs that are used to initialize, finalize, query, and tune settings for the cuFile system. /\* Initialize the cuFile infrastructure \*/ CUfileError\_t cuFileDriverOpen(); /\* Finalize the cuFile system \*/ CUfileError\_t cuFileDriverClose(); /\* Query capabilities based on current versions, installed functionality \*/ CUfileError\_t cuFileGetDriverProperties(CUfileDrvProps\_t \*props); /\*API to set whether the Read/Write APIs use polling to do IO operations \*/ CUfileError\_t cuFileDriverSetPollMode(bool poll, size\_t poll\_threshold\_size); /\*API to set max IO size(KB) used by the library to talk to nvidia-fs driver \*/ CUfileError\_t cuFileDriverSetMaxDirectIOSize(size\_t max\_direct\_io\_size); /\* API to set maximum GPU memory reserved per device by the library for internal buffering \*/ CUfileError\_t cuFileDriverSetMaxCacheSize(size\_t max\_cache\_size); /\* Sets maximum buffer space that is pinned in KB for use by cuFileBufRegister CUfileError\_t cuFileDriverSetMaxPinnedMemSize(size\_t max\_pinned\_memory\_size); /\* Retrieves the cuFile library version. \*/ CUfileError\_t cuFileGetVersion(int \*version); Note Refer to [sample\_007](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_007.cc) for usage. 4.3. cuFile Synchronous IO APIs[#](#cufile-synchronous-io-apis "Link to this heading") --------------------------------------------------------------------------------------- The core of the cuFile IO APIs are the read and write functions. ssize\_t cuFileRead(CUFileHandle\_t fh, void \*bufPtr\_base, size\_t size, off\_t file\_offset, off\_t devPtr\_offset); ssize\_t cuFileWrite(CUFileHandle\_t fh, const void \*bufPtr\_base, size\_t size, off\_t file\_offset, off\_t devPtr\_offset); The starting offset of the buffer on the device or host is determined by a base (`bufPtr_base`) and offset (`bufPtr_offset`). This offset is distinct from the offset in the file. Note To use the registered buffer, the bufPtr\_base must be the buffer pointer used to register during `cuFileBufRegister`. Otherwise `cuFileRead` and `cuFileWrite` APIs may use internal memory buffers for GPUDirect Storage peer to peer operations. Note The default behavior for all paths where GDS is not supported is for the cuFile IO API to attempt IO using file system supported posix mode APIs when `properties.allow_compat_mode` is set to true. In order to disable cuFile APIs falling back to posix APIs for unsupported GDS paths, `properties.allow_compat_mode` in the `/etc/cufile.json` file should be set to false. Note Refer to sample [sample\_003](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_003.cc) for usage. 4.4. cuFile File Handle APIs[#](#cufile-file-handle-apis "Link to this heading") --------------------------------------------------------------------------------- Here is some information about the cuFile Handle APIs. The `cuFileHandleRegister` API makes a file descriptor or handle that is known to the cuFile subsystem by using an OS-agnostic interface. The API returns an opaque handle that is owned by the cuFile subsystem. To conserve memory, the `cuFileHandleDeregister` API is used to release cuFile-related memory objects. Using only the POSIX close will not clean up resources that were used by cuFile. Additionally, the clean up of cuFile objects associated with the files that were operated on in the cuFile context will occur at `cuFileDriverClose`. CUfileError\_t cuFileHandleRegister(CUFileHandle\_t \*fh, CUFileDescr\_t \*descr); void cuFileHandleDeregister(CUFileHandle\_t fh); Note Refer to [sample\_003](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_003.cc) for usage. 4.5. cuFile Buffer APIs[#](#cufile-buffer-apis "Link to this heading") ----------------------------------------------------------------------- The `cuFileBufRegister` API incurs a significant performance cost, so registration costs should be amortized where possible. Developers must ensure that buffers are registered up front and off the critical path. The `cuFileBufRegister` API is optional. If this is not used, instead of pinning the user’s memory, cuFile-managed and internally pinned buffers are used. The `cuFileBufDeregister` API is used to optimally clean up cuFile-related memory objects, but CUDA currently has no analog to `cuFileBufDeregister`. The cleaning up of objects associated with the buffers operated on in the cuFile context occurs at `cuFileDriverClose`. If explicit APIs are used, the incurred errors are reported immediately, but if the operations of these explicit APIs are performed implicitly, error reporting and handling are less clear. CUfileError\_t cuFileBufRegister(const void \*devPtr\_base, size\_t size, int flags); CUfileError\_t cuFileBufDeregister(const void \*devPtr\_base); Note Refer to [sample\_005](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_005.cc) for usage. 4.6. cuFile Stream APIs[#](#cufile-stream-apis "Link to this heading") ----------------------------------------------------------------------- Operations that are enqueued with cuFile Stream APIs are FIFO ordered with respect to other work on the stream and must be completed before continuing with the next action in the stream. CUfileError\_t cuFileReadAsync(CUFileHandle\_t fh, void \*bufPtr\_base, size\_t \*size\_p, off\_t \*file\_offset\_p, off\_t \*bufPtr\_offset\_p, ssize\_t \*bytes\_read\_p, CUStream stream); CUfileError\_t cuFileWriteAsync(CUFileHandle\_t fh, void \*bufPtr\_base, size\_t \*size\_p, off\_t \*file\_offset\_p, off\_t \*bufPtr\_offse\_pt, ssize\_t \*bytes\_written\_p, CUstream stream); Note Refer to samples [sample\_031](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_031.cc) , [sample\_032](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_032.cc) , [sample\_033](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_033.cc) , and [sample\_034](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_034.cc) for usage. 4.7. cuFile Batch APIs[#](#cufile-batch-apis "Link to this heading") --------------------------------------------------------------------- Batch APIs are submitted synchronously, but executed asynchronously with respect to host thread. These operations can be submitted on different files, different locations in the same file, or a mix. Completion of IO can be checked asynchronously using a status API in the same host thread or in a different thread. The `cuFileBatchIOGetStatus` API takes an array of `CUfileIOEvents_t` and minimum number of elements to poll for, which describes the IO action, status, errors, and bytes transacted for each instance. The bytes transacted field is valid only when the status indicates a successful completion. Note Refer to samples [sample\_019](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_019.cc) , [sample\_020](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_020.cc) , [sample\_021](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_021.cc) , and [sample\_022](https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_022.cc) for usage. 5\. cuFile API Functional Specification[#](#cufile-api-functional-specification "Link to this heading") ======================================================================================================== This section provides information about the cuFile API functional specification. See the [GPUDirect Storage Overview Guide](https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html) for a high-level analysis of the set of functions and their relation to each other. We anticipate adding additional return codes for some of these functions. All cuFile APIs are called from the host code. 5.1. cuFileDriver API Functional Specification[#](#cufiledriver-api-functional-specification "Link to this heading") --------------------------------------------------------------------------------------------------------------------- This section provides information about the cuFileDriver API functional specification. ### 5.1.1. cuFileDriverOpen[#](#cufiledriveropen "Link to this heading") CUfileError\_t cuFileDriverOpen(); Opens the Driver session to support GDS IO operations. **Parameters** None **Returns** * `CU_FILE_SUCCESS` on a successful open, or if the driver is already open. * `CU_FILE_DRIVER_NOT_INITIALIZED` on a failure to open the driver. * `CU_FILE_PERMISSION_DENIED` on a failure to open. This can happen when the character device (`/dev/nvidia_fs[0-15]`) is restricted to certain users by an administrator, for example, admin, where `/dev` is not exposed with read permissions in the container. * `CU_FILE_DRIVER_VERSION_MISMATCH`, when there is a mismatch between the cuFile library and its kernel driver. * `CU_FILE_CUDA_DRIVER_ERROR` if the CUDA driver failed to initialize. `CU_FILE_PLATFORM_NOT_SUPPORTED` if the current platform is not supported by GDS. * `CU_FILE_NVFS_SETUP_ERROR` for a cuFile-specific internal error. Refer to the `cufile.log` file for more information. **Description** * This API opens the session with the NVFS kernel driver to communicate from userspace to kernel space and calls the GDS driver to set up the resources required to support GDS IO operations. * The API checks whether the current platform supports GDS and initializes the cuFile library. * This API loads the cuFile settings from a JSON configuration file in `/etc/cufile.JSON`. If the JSON configuration file does not exist, the API loads the default library settings. To modify this default config file, administrative privileges are needed. The administrator can modify it to grant cuFile access to the specified devices and mount paths and also tune IO parameters (in KB, 4K aligned) that are based on the type of workload. Refer to the [default config file](https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html#gds-parameters) (`/etc/cufile.json`) for more information. ### 5.1.2. cuFileDriverClose[#](#cufiledriverclose "Link to this heading") CUfileError\_t cuFileDriverClose(); * Closes the driver session and frees any associated resources for GDS. * This happens implicitly upon process exit. * The driver can be reopened once it is closed. **Parameters** None **Returns** * `CU_FILE_SUCCESS` on a successful close. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure. **Description** * Close the GDS session and any associated memory resources. If there are buffers registered by using `cuFileBufRegister`, which are not unregistered, a `cuFileDriverClose` implicitly unregisters those buffers. Any in-flight IO when `cuFileDriverClose` is in-progress will receive an error. ### 5.1.3. cuFileDriverGetProperties[#](#cufiledrivergetproperties "Link to this heading") The `cuFileDrvProps_t` structure can be queried with `cuFileDriverGetProperties` and selectively modified with `cuFileDriverSetProperties`. The structure is self-describing, and its fields are consistent with the major and minor API version parameters. CUfileError\_t cuFileDriverGetProperties(cuFileDrvProps\_t \*props); * Gets the Driver session properties for GDS functionality. **Parameters** `props` > Pointer to the cuFile Driver properties. **Returns** * `CU_FILE_SUCCESS` on a successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure. * `CU_FILE_DRIVER_VERSION_MISMATCH` on a driver version mismatch. * `CU_FILE_INVALID_VALUE` if input is invalid. **Description** This API is used to get current GDS properties and nvidia-fs driver properties and functionality, such as support for SCSI, NVMe, and NVMe-OF. This API is used to get the current `nvidia-fs` drivers-specific properties such as the following: * `major_version`: the cuFile major version * `minor_version`: the cuFile minor version * `props.nvfs.dstatusflags`, which are bit flags that indicate support for the following driver features: * `CU_FILE_EXASCALER_SUPPORTED`, a bit to check whether the DDN EXAScaler parallel filesystem solutions (based on the Lustre filesystem) client supports GDS. * `CU_FILE_WEKAFS_SUPPORTED`, a bit to check whether WekaFS supports GDS. * `Props.nvfs.dcontrolflags`, which are bit flags that indicate the current activation for driver features: * `CU_FILE_USE_POLL_MODE`, when bit is set, IO uses polling mode. * `CU_FILE_ALLOW_COMPAT_MODE`, if the value is 1 compatible mode is set. > Otherwise, the compatible mode is disabled. * `Props.fflags`, which are bit flags that indicate whether the following library features are supported: * `CU_FILE_STREAMS_SUPPORTED`, an attribute that checks whether CUDA-streams are supported. * `CU_FILE_DYN_ROUTING_SUPPORTED`, an attribute that checks whether dynamic routing feature is supported. * `Props.nvfs.poll_thresh_size`, a maximum IO size, in KB and must be 4K-aligned, that is used for the POLLING mode. * `Props.nvfs.max_direct_io_size`, a maximum GDS IO size, in KB and must be 4K-aligned, that is requested by the nvidia-fs driver to the underlying filesystem. * `Props.max_device_cache_size`, a maximum GPU buffer space per device, in KB and must be 4K-aligned. Used internally, for example, to handle unaligned IO and optimal IO path routing. This value might be rounded down to the nearest GPU page size. * `Props.max_device_pinned_mem_size`, a maximum buffer space, in KB and must be 4K-aligned, that is pinned and mapped to the GPU BAR space. This might be rounded down to the nearest GPU page size. * `Props.per_buffer_cache_size`, a GPU bounce buffer size, in KB, used for internal pools. **Additional Information** See the following for more information: * [cuFileDriverSetPollMode(bool poll, size\_t poll\_threshold\_size)](#cufiledriversetpollmode) * [cuFileDriverSetMaxDirectIOSize(size\_t max\_direct\_io\_size)](#cufiledriversetmaxdirectiosize) * [(size\_t max\_cache\_size)](#cufiledriversetmaxcachesize) * [cuFileDriverSetMaxPinnedMemSize(size\_t max\_pinned\_memory\_size)](#cufiledriversetmaxpinnedmemsize) ### 5.1.4. cuFileDriverSetPollMode(bool poll, size\_t poll\_threshold\_size)[#](#cufiledriversetpollmode-bool-poll-size-t-poll-threshold-size "Link to this heading") `cuFileDriverSetPollMode(bool poll, size_t poll_threshold_size)` API CUfileError\_t cuFileDriverSetPollMode(bool poll, size\_t poll\_threshold\_size); * Sets whether the Read/Write APIs use polling to complete IO operations. If poll mode is enabled, an IO size less than or equal to the threshold value is used for polling. * The `poll_threshold_size` must be 4K aligned. **Parameters** `poll` > Boolean to indicate whether to use the poll mode. `poll_threshold_size` > IO size to use for POLLING mode in KB. The default value is 4KB. **Returns** * `CU_FILE_SUCCESS` on a successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure to load the driver. * `CU_FILE_DRIVER_UNSUPPORTED_LIMIT` on failure to set with valid threshold size. **Description** This API is used in conjunction with `cuFileGetDriverProperties`. This API is used to set whether the library should use polling and the maximum IO threshold size less than or equal to which it will poll. This API overrides the default value that may be set through the JSON configuration file using the config keys `properties.poll_mode` and `properties.poll_max_size_kb` for the current process. Refer to the following for more information: [cuFileDriverGetProperties](#cufiledrivergetproperties) ### 5.1.5. cuFileDriverSetMaxDirectIOSize(size\_t max\_direct\_io\_size)[#](#cufiledriversetmaxdirectiosize-size-t-max-direct-io-size "Link to this heading") CUfileError\_t cuFileDriverSetMaxDirectIOSize(size\_t max\_direct\_io\_size); * Sets the max IO size, in KB. This parameter is used by the nvidia-fs driver as the maximum IO chunk size in which IO is issued to the underlying filesystem. In compatible mode, this is the maximum IO chunk size that the library uses to issue POSIX read/writes. * The max direct IO size must be 4K aligned. **Parameters** `max_direct_io_size` > The maximum allowed direct IO size in KB. The default value is 16384KB. This is because typically parallel-file systems perform better with bulk read/writes. **Returns** * `CU_FILE_SUCCESS` on successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure to load the driver. * `CU_FILE_DRIVER_UNSUPPORTED_LIMIT` on failure to set with valid size. **Description** This API is used with `cuFileGetDriverProperties` and is used to set the maximum direct IO size used by the library to specify the nvidia-fs kernel driver the maximum chunk size in which the latter can issue IO to the underlying filesystem. In compatible mode, this is the maximum IO chunk size which the library uses for issuing POSIX read/writes. This parameter is dependent on the underlying GPU hardware and system memory. This API overrides the default value that might be set through the JSON configuration file by using the `properties.max_direct_io_size_kb` config key for the current process. Refer to the following for more information: * [cuFileDriverGetProperties](#cufiledrivergetproperties) ### 5.1.6. (size\_t max\_cache\_size)[#](#size-t-max-cache-size "Link to this heading") CUfileError\_t cuFileDriverSetMaxCacheSize(size\_t max\_cache\_size); * Sets the maximum GPU buffer space, in KB, per device and is used for internal use, for example, to handle unaligned IO and optimal IO path routing. This value might be rounded down to the nearest GPU page size. * The max cache size must be 4K aligned. * This API overrides the default value that might be set through the JSON configuration file using the `properties.max_device_cache_size_kb` config key for the current process. **Parameters** `max_cache_size` > The maximum GPU buffer space, in KB, per device used for internal use, for example, to handle unaligned IO and optimal IO path routing. This value might be rounded down to the nearest GPU page size. > > The default value is 131072KB. **Returns** * `CU_FILE_SUCCESS` on successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure to load the driver. * `CU_FILE_DRIVER_UNSUPPORTED_LIMIT` on failure to set with valid IO size **Description** This API is used with `cuFileGetDriverProperties` and is used to set the upper limit on the cache size per device for internal use by the library. Refer to [cuFileDriverGetProperties](#cufiledrivergetproperties) for more information. ### 5.1.7. cuFileDriverSetMaxPinnedMemSize(size\_t max\_pinned\_memory\_size)[#](#cufiledriversetmaxpinnedmemsize-size-t-max-pinned-memory-size "Link to this heading") CUfileError\_t cuFileDriverSetMaxPinnedMemSize(size\_t max\_pinned\_mem\_size); * Sets the maximum GPU buffer space, in KB, that is pinned and mapped. This value might be rounded down to the nearest GPU page size. * The max pinned size must be 4K aligned. * The default value corresponds to the maximum `PinnedMemory` or the physical memory size of the device. * This API overrides the default value that may be set by the `properties.max_device_pinned_mem_size_kb` JSON config key for the current process. **Parameters** `max_pinned_memory_size` > The maximum buffer space, in KB, that is pinned and mapped to the GPU BAR space. This value might be rounded down to the nearest GPU page size. The maximum limit may be set to UINT64\_MAX, which is equivalent to no enforced limit. It may be set to something smaller than the size of the GPU’s physical memory. **Returns** * `CU_FILE_SUCCESS` on successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure to load driver. * `CU_FILE_DRIVER_UNSUPPORTED_LIMIT` on failure to set with valid size. **Description** This API is used with `cuFileGetDriverProperties` and is used to set an upper limit on the maximum size of GPU memory that can be pinned and mapped and is dependent on the underlying GPU hardware and system memory. This API is related to `cuFileBufRegister`, which is used to register GPU device memory. See [cuFileDriverGetProperties](#cufiledrivergetproperties) for more information. ### 5.1.8. cuFileGetVersion(int \*version)[#](#cufilegetversion-int-version "Link to this heading") CUfileError\_t cuFileGetVersion(int \*version); * Retrieves the cuFile library version. * The version is returned as (1000 \* major + 10 \* minor). * For example, cuFile 1.7.0 would be represented by 1070. **Parameters** `version` > Output argument which would contain the version number in the above format upon successful completion. **Returns** * `CU_FILE_SUCCESS` on successful completion. * `CU_FILE_INVALID_VALUE` if the version parameter is null. * `CU_FILE_DRIVER_VERSION_READ_ERROR` if the version is not available. **Description** This API is used to obtain the current version of the cuFile library. It may be useful sometimes for an application to expect based on the version if any specific GDS feature is present or not. 5.2. cuFile IO API Functional Specification[#](#cufile-io-api-functional-specification "Link to this heading") --------------------------------------------------------------------------------------------------------------- This section provides information about the cuFile IO API function specification. The device pointer addresses referred to in these APIs pertain to the current context for the caller. Unlike the non-async version of `cuMemcpy`, the `cuFileHandleRegister`, `cuFileHandleDeregister`, `cuFileRead`, and `cuFileWrite` APIs do not have the semantic of being ordered with respect to other work in the null stream. ### 5.2.1. cuFileHandleRegister[#](#cufilehandleregister "Link to this heading") CUfileError\_t cuFileHandleRegister(CUFileHandle\_t \*fh, CUfileDescr\_t \*descr); * Register an open file. * `cuFileHandleRegister` is required and performs extra checking that is memoized to provide increased performance on later cuFile operations. * This API is OS agnostic. > Note > > CUDA toolkit 12.2 (GDS version 1.7.x) supports non O\_DIRECT open flags as well as O\_DIRECT. Application is allowed to open a file in non O\_DIRECT mode in compat mode and also with nvidia-fs.ko installed. In the latter case, an O\_DIRECT path between GPU and Storage will be used if such a path exists. **Parameters** `fh` > Valid pointer to the OS-neutral cuFile handle structure supplied by the user but populated and maintained by the cuFile runtime. `desc` > Valid pointer to the OS-neutral file descriptor supplied by the user carrying details regarding the file to be opened such as `fd` for Linux-based files. **Returns** * `CU_FILE_SUCCESS` on successful completion. * `CU_FILE_DRIVER_NOT_INITIALIZED` on failure to load the driver. * `CU_FILE_IO_NOT_SUPPORTED`, if the filesystem is not supported. * `CU_FILE_INVALID_VALUE` if there are null or bad API arguments. * `CU_FILE_INVALID_FILE_OPEN_FLAG`, if the file is opened with unsupported modes such as no `O_APPEND`, `O_NOCTTY`, `O_NONBLOCK`, `O_DIRECTORY`, `O_NOFOLLOW`, `O_NOATIME`, and `O_TMPFILE`. * `CU_FILE_INVALID_FILE_TYPE`, if the file path is not valid, not a regular file, not a symbolic link, or not a device file. * `CU_FILE_HANDLE_ALREADY_REGISTERED` if the file is already registered using the same file-descriptor. **Description** * Given a file-descriptor will populate and return the `CUfileHandle_t` needed for issuing IO with cuFile APIs. * A return value of anything other than CU\_FILE\_SUCCESS leaves fh in an undefined state but has no other side effects. * By default this API accepts whether the file descriptor is opened with O\_DIRECT mode or non O\_DIRECT mode. Refer to the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) * [cuFileWriteAsync](#cufilewriteasync) * [cuFileHandleDeregister](#cufilehandlederegister) ### 5.2.2. cuFileHandleDeregister[#](#cufilehandlederegister "Link to this heading") CUfileError\_t cuFileHandleDeregister(CUFileHandle\_t \*fh); **Parameters** `fh` > The file handle obtained from `cuFileHandleRegister`. **Returns** None Note This API only logs an ERROR level message in the `cufile.log` file for valid inputs. **Description** * The API is used to release resources that are claimed by `cuFileHandleRegister`. This API should be invoked only after the application ensures there are no outstanding IO operations with the handle. If `cuFileHandleDeregister` is called while IO on the file is in progress might result in undefined behavior. * The user is still expected to close the file descriptor outside the cuFile subsystem after calling this API using `close` system call. Closing a file handle without calling `cuFileHandleDeregister` does not release the resources that are held in the cuFile library. If this API is not called, the cuFile subsystem releases the resources lazily or when the application exits. Refer to the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileHandleRegister](#cufilehandleregister) ### 5.2.3. cuFileRead[#](#cufileread "Link to this heading") ssize\_t cuFileRead(CUfileHandle\_tfh, void \*bufPtr\_base, size\_t size, off\_t file\_offset, off\_t bufPtr\_offset); * Reads specified bytes from the file descriptor into the device memory or the host memory. **Parameters** `fh` > File descriptor for the file. `bufPtr_base` > Base address of buffer in device memory or host memory. For registered buffers, `bufPtr_base` must remain set to the base address used in the `cuFileBufRegister` call. `size` > Size in bytes to read. `file_offset` > Offset in the file to read from. `bufPtr_offset` > Offset relative to the `bufPtr_base` pointer to read into. This parameter should be used only with registered buffers. **Returns** * Size of bytes that were successfully read. * \-1 on an error, so errno is set to indicate filesystem errors. * All other errors return a negative integer value of the `CUfileOpError` enum value. **Description** This API reads the data from a specified file handle at a specified offset and size bytes into the GPU memory by using GDS functionality or into the host memory based on the type of memory pointer. The API works correctly for unaligned offsets and any data size, although the performance might not match the performance of aligned reads. This is a synchronous call and blocks until the IO is complete. Note For the `bufPtr_offset`, if data will be read starting exactly from the `bufPtr_base` that is registered with `cuFileBufRegister`, `bufPtr_offset` should be set to 0. To read starting from an offset in the registered buffer range, the relative offset should be specified in the `bufPtr_offset,` and the `bufPtr_base` must remain set to the base address that was used in the `cuFileBufRegister` call. See the following for more information: * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) * [cuFileWriteAsync](#cufilewriteasync) ### 5.2.4. cuFileWrite[#](#cufilewrite "Link to this heading") ssize\_t cuFileWrite(CUfileHandle\_t fh, const void \*bufPtr\_base, size\_t size, off\_t file\_offset, off\_t bufPtr\_offset); * Writes specified bytes from the device memory into the file descriptor using GDS. **Parameters** `fh` > File descriptor for the file `bufPtr_base` > Base address of buffer in device memory or host memory. For registered buffers, `bufPtr_base` must remain set to the base address used in the `cuFileBufRegister` call. `size` > Size in bytes to which to write. `file_offset` > Offset in the file to which to write. `bufPtr_offset` > Offset relative to the `bufPtr_base` pointer from which to write. This parameter should be used only with registered buffers. **Returns** * Size of bytes that were successfully written. * \-1 on an error, so errno is set to indicate filesystem errors. * All other errors return a negative integer value of the `CUfileOpError` enum value. **Description** This API writes the data from the GPU memory or the host memory to a file specified by the file handle at a specified offset and size bytes by using GDS functionality. The API works correctly for unaligned offset and data sizes, although the performance is not on-par with aligned writes.This is a synchronous call and will block until the IO is complete. Note GDS functionality modified the standard file system metadata in SysMem. However, GDS functionality does not take any special responsibility for writing that metadata back to permanent storage. The data is not guaranteed to be present after a system crash unless the application uses an explicit `fsync(2)` call. If the file is opened with an `O_SYNC` flag, the metadata will be written to the disk before the call is complete. Refer to the note in [cuFileRead](#cufileread) for more information about `bufPtr_offset:`. Refer to the following for more information: * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) * [cuFileWriteAsync](#cufilewriteasync) 5.3. cuFile Memory Management Functional Specification[#](#cufile-memory-management-functional-specification "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------- The device pointer addresses that are mentioned in the APIs in this section pertain to the current context for the caller. cuFile relies on users to complete their own allocation before using the `cuFileBufRegister` API and free after using the `cuFileBufDeregister` API. ### 5.3.1. cuFileBufRegister[#](#cufilebufregister "Link to this heading") CUfileError\_t cuFileBufRegister(const void \*bufPtr\_base, size\_t size, int flags); * Based on the memory type, this API registers existing `cuMemAlloc`’d (pinned) memory for GDS IO operations or host memory for IO operations. **Parameters** `bufPtr_base` > Address of device pointer. `cuFileRead` and `cuFileWrite` **must** use this `bufPtr_base` as the base address. `size` > Size in bytes from the start of memory to map. `flags` > Reserved for future use; must be 0. **Returns** * `CU_FILE_SUCCESS` on a successful registration. * `CU_FILE_NVFS_DRIVER_ERROR` if the nvidia-fs driver cannot handle the request. * `CU_FILE_INVALID_VALUE` on a failure. * `CU_FILE_CUDA_DRIVER_ERROR` on CUDA-specific errors. CUresult code can be obtained using `CU_FILE_CUDA_ERR` (err). * `CU_FILE_MEMORY_ALREADY_REGISTERED`, if memory is already registered. * `CU_FILE_INTERNAL_ERROR`, an internal library-specific error. * `CU_FILE_CUDA_MEMORY_TYPE_INVALID`, for device memory that is not allocated via `cudaMalloc` or `cuMemAlloc`. * `CU_FILE_CUDA_POINTER_RANGE_ERROR`, if the size exceeds the bounds of the allocated memory. * `CU_FILE_INVALID_MAPPING_SIZE`, if the size exceeds the GPU resource limits. * `CU_FILE_GPU_MEMORY_PINNING_FAILED`, if not enough pinned memory is available. **Description** Based on the memory type, this API either registers the specified GPU address or host memory address and size for use with the `cuFileRead` and `cuFileWrite` operations. The user must call `cuFileBufDeregister` to release the pinned memory mappings for GPU memory if needed. See the following for more information: * [cuFileBufDeregister](#cufilebufderegister) ### 5.3.2. cuFileBufDeregister[#](#cufilebufderegister "Link to this heading") CUfileError\_t cuFileBufDeregister(const void \*bufPtr\_base); * Based on the memory type, this API either deregisters CUDA memory or the host memory registered using the `cuFileBufRegister` API. **Parameters** `bufPtr_base` > Address of device pointer to release the mappings that were provided to `cuFileBufRegister` **Returns** * `CU_FILE_SUCCESS` on a successful deregistration. * `CU_FILE_MEMORY_NOT_REGISTERED`, if `bufPtr_base` was not registered. * `CU_FILE_ERROR_INVALID_VALUE` on failure to find the registration for the specified memory. * `CU_FILE_INTERNAL_ERROR`, an internal library-specific error. **Description** This API deregisters memory mappings that were registered by `cuFileBufRegister`. Refer to [cuFileBufRegister](#cufilebufregister) for more information. 5.4. cuFile Stream API Functional Specification[#](#cufile-stream-api-functional-specification "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- This section provides information about the cuFile stream API functional specification. The stream APIs are similar to Read and Write, but they take a stream parameter to support asynchronous operations and execute in the CUDA stream order. ### 5.4.1. cuFileStreamRegister[#](#cufilestreamregister "Link to this heading") CUfileError\_t cuFileStreamRegister(CUStream\_t stream, unsigned flags); * Defines the input behavior for stream I/O APIs. **Parameters** `stream` > CUDA stream in which to enqueue the operation. If NULL, make this operation in the default CUDA stream. `flags` > The following are valid values: | Value | Description | | --- | --- | | 0x0 | All the I/O parameters are valid only at the time of execution. | | 0x1 | Buffer offset value is valid at submission time. | | 0x2 | File offset value is valid at submission time. | | 0x4 | Size is valid at submission time. | | 0x8 | All inputs i.e. buffer offset, file offset and size are 4K aligned. | | 0xf | All inputs are aligned and known at submission time. | Note Using the flag `0XF` will perform best as the workflow can be optimized during submission time. **Description** This optional API registers the stream with the cuFile subsystem. This API will allocate resources to handle stream operations for cuFile. The API will synchronize on the stream before allocating resources. The stream pointer is expected to be a valid pointer. **Returns** * `CU_FILE_SUCCESS` on a successful submission. * `CU_FILE_ERROR_INVALID_VALUE` on a invalid stream specification. * `CU_FILE_DRIVER_ERROR` if the NVIDIA-fs driver cannot handle the request. * `CU_FILE_PLATFORM_NOT_SUPPORTED` on unsupported platforms. ### 5.4.2. cuFileStreamDeregister[#](#cufilestreamderegister "Link to this heading") CUfileError\_t cuFileStreamDeregister(CUStream\_t stream); **Parameters** `stream` > CUDA stream in which to enqueue the operation. If NULL, make this operation in the default CUDA stream. `flags` > Reserved for future use. **Description** This optional API deregisters the stream with the cuFile subsystem. This API will free allocated cuFile resources associated with the stream. The API will synchronize on the stream before releasing resources. The stream pointer is expected to be a valid pointer. The stream will be automatically deregistered as part of `cuFileDriverClose`. **Returns** * `CU_FILE_SUCCESS` on a successful submission. * `CU_FILE_ERROR_INVALID_VALUE` on a invalid stream specification. * `CU_FILE_PLATFORM_NOT_SUPPORTED` on unsupported platforms. ### 5.4.3. cuFileReadAsync[#](#cufilereadasync "Link to this heading") CUfileError\_t cuFileReadAsync(CUFileHandle\_t fh, void \*bufPtr\_base, size\_t \*size\_p, off\_t \*file\_offset\_p, off\_t \*bufPtr\_offset\_p, int \*bytes\_read\_p, CUstream stream); * Enqueues a read operation for the specified bytes from the cuFile handle into the device memory by using GDS functionality or to the host memory based on the type of memory pointer. * If non-NULL, the action is ordered in the stream. * The current context of the caller is assumed. **Parameters** `fh` > The cuFile handle for the file. `bufPtr_base` > * The base address of the buffer in the memory into which to read. > > * The buffer can be allocated using either `cudaMemory`, `cudaMallocHost`, `malloc`, or `mmap`. > > * For registered buffers, `bufPtr_base` must remain set to the base address used in `cuFileBufRegister` call. > `size_p` > Pointer to size in bytes to read. If the exact size is not known at the time of I/O submission, then yo must set it to the maximum possible I/O size for that stream I/O. `file_offset_p` > Pointer to offset in the file from which to read. Unless otherwise set using `cuFileStreamRegister` API, this value will not be evaluated until execution time. `bufPtr_offset_p` > Pointer to the offset relative to the `bufPtr_base` pointer from which to write. Unless otherwise set using `cuFileStreamRegister` API, this value will not be evaluated until execution time. `bytes_read_p` > Pointer to the bytes read from the specified filehandle. This pointer should be a non NULL value and `*bytes_read_p` set to 0. After successful execution of the operation in the stream, the value `*bytes_read_p` will contain either: > > * The number of bytes successfully read. > > * \-1 on IO errors. > > * All other errors return a negative integer value of the `CUfileOpError` enum value. > `stream` > * CUDA stream in which to enqueue the operation. > > * If NULL, make this operation synchronous. > **Returns** * `CU_FILE_SUCCESS` on a successful submission. * `CU_FILE_DRIVER_ERROR`, if the nvidia-fs driver cannot handle the request. * `CU_FILE_ERROR_INVALID_VALUE` on an input failure. * `CU_FILE_CUDA_ERROR` on CUDA-specific errors. CUresult code can be obtained by using `CU_FILE_CUDA_ERR(err)`. **Description** * This API reads the data from the specified file handle at the specified offset and size bytes into the GPU memory using GDS functionality. This is an asynchronous call and enqueues the operation into the specified CUDA stream and will not block the host thread for IO completion. The operation can be waited upon using `cuStreamSynchronize(stream)`. * The `bytes_read_p` memory should be allocated with `cuMemHostAlloc/malloc/mmap` or registered with `cuMemHostRegister`. The pointer to access that memory from the device can be obtained by using `cuMemHostGetDevicePointer`. * Operations that are enqueued with cuFile Stream APIs are FIFO ordered with respect to other work on the stream and must be completed before continuing to the next action in the stream. * Unless otherwise specified through `cuFileStreamRegister` API, file offset, buffer offset or size parameter will not be evaluated until execution time. In these scenarios, size parameters should be set to the maximum possible I/O size at the time of submission and can be set to the actual size prior to the stream I/O execution. Refer to the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileWriteAsync](#cufilewriteasync) ### 5.4.4. cuFileWriteAsync[#](#cufilewriteasync "Link to this heading") CUfileError\_t cuFileWriteAsync(CUFileHandle\_t fh, void \*bufPtr\_base, size\_t \*size\_p, off\_t file\_offset\_p, off\_t bufPtr\_offset\_p, int \*bytes\_written\_p, CUstream\_t stream); * Queues Write operation for the specified bytes from the device memory into the cuFile handle by using GDS. **Parameters** `fh` > The cuFile handle for the file. `bufPtr_base` > The base address of the buffer in the memory from which to write. The buffer can be allocated using either `cudaMemory/cudaMallocHost/malloc/mmap`. For registered buffers, `bufPtr_base` must remain set to the base address used in the `cuFileBufRegister` call. `size_p` > Pointer to the size in bytes to write. If the exact size is not known at the time of I/O submission, then you must set it to the maximum possible I/O size for that stream I/O. `file_offset_p` > Pointer to the offset in the file from which to write. Unless otherwise set using `cuFileStreamRegister` API, this value will not be evaluated until execution time. `bufPtr_offset_p` > Pointer to the offset relative to the `bufPtr_base` pointer from which to write. Unless otherwise set using cuFileStreamRegister API, this value will not be evaluated until execution time. `bytes_written_p` > Pointer to the bytes written to the specified filehandle.This pointer should be a non NULL value and `*bytes_written_p` set to 0. After successful execution of the operation in the stream, the value `*bytes_written_p` will contain either: > > * The number of bytes successfully written. > > * \-1 on IO errors. > > * All other errors will return a negative integer value of the `CUfileOpError` enum value. > `stream` > The CUDA stream to enqueue the operation. **Returns** * `CU_FILE_SUCCESS` on a successful submission. * `CU_FILE_DRIVER_ERROR`, if the nvidia-fs driver cannot handle the request. * `CU_FILE_ERROR_INVALID_VALUE` on an input failure. * `CU_FILE_CUDA_ERROR` on CUDA-specific errors. The CUresult code can be obtained by using `CU_FILE_CUDA_ERR(err)`. **Description** * This API writes the data from the GPU memory to a file specified by the file handle at a specified offset and size bytes by using GDS functionality. This is an asynchronous call and enqueues the operation into the specified CUDA stream and will not block the host thread for IO completion. The operation can be waited upon by using `cuStreamSynchronize(stream)`. * The `bytes_written` pointer should be allocated with `cuMemHostAlloc` or registered with `cuMemHostRegister`, and the pointer to access that memory from the device can be obtained by using `cuMemHostGetDevicePointer`. * Operations that are enqueued with cuFile Stream APIs are FIFO ordered with respect to other work on the stream and must be completed before continuing to the next action in the stream. * Unless otherwise specified through `cuFileStreamRegister` API, file offset, buffer offset or size parameter will not be evaluated until execution time. In these scenarios, size parameters should be set to the maximum possible I/O size at the time of submission and can be set to the actual size prior to the stream I/O execution. See the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) 5.5. cuFile Batch API Functional Specification[#](#cufile-batch-api-functional-specification "Link to this heading") --------------------------------------------------------------------------------------------------------------------- ### 5.5.1. cuFileBatchIOSetUp[#](#cufilebatchiosetup "Link to this heading") CUfileError\_t cuFileBatchIOSetUp(CUfileBatchHandle\_t \*batch\_idp, int max\_nr); **Parameters** `max_nr` > (Input) The maximum number of events this batch will hold. > > Note > > The number should be between 1 - `properties.io_batch_size` `batch_idp` > (Output) Will be used in subsequent batch IO calls. **Returns** * `CU_FILE_SUCCESS` on success. * `CU_FILE_INTERNAL_ERROR` on on any failures. **Description** This interface should be the first call in the sequence of batch I/O operation. This takes the maximum number of batch entries the caller intends to use and returns a `CUFileBatchHandle_t` which should be used by the caller for subsequent batch I/O calls. Refer to the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) * [cuFileWriteAsync](#cufilewriteasync) * [cuFileBatchIOGetStatus](#cufilebatchiogetstatus) * [cuFileBatchIOCancel](#cufilebatchiocancel) * [cuFileBatchIODestroy](#cufilebatchiodestroy) ### 5.5.2. cuFileBatchIOSubmit[#](#cufilebatchiosubmit "Link to this heading") CUfileError\_t cuFileBatchIOSubmit(CUfileBatchHandle\_t batch\_idp, unsigned nr, CUfileIOParams\_t \*iocbp, unsigned int flags) **Parameters** `batch_idp` > The address of the output parameter for the newly created batch ID, which was obtained from a `cuFileBatchSetup` call. `nr` > * The number of requests for the batch request. > > * The value must be greater than 0 and less than or equal to `max_nr` specified in `cuFileBatchIOSetup`. > `iocbp` > The pointer contains the `CUfileIOParams_t` array structures of the length `nr` array. `flags` > Reserved for future use. Should be set to 0. **Returns** * `CU_FILE_SUCCESS` on success. * `CU_FILE_INTERNAL_ERROR` on any failures. **Description** * This API will need to be used to submit a read/write operation on an array of GPU/CPU data pointers from their respective file handle, offset, and size bytes. Based on the type of memory pointer, the data is transferred to/from the GPU memory by using GDS or the data is transferred to/from the CPU memory. * This is an asynchronous call and will enqueue the operation on a `batch_id` provided by the `cuFileIOSetup` API. The operation can be monitored when using this `batch_id` through `cuFileBatchIOGetStatus`. * The operation can be canceled by calling `cuFileBatchIOCancel` or destroyed by `cuFileBatchIODestroy`. * The entries in the `CUfileIOParams_t` array describe individual IOs. The bytes transacted field is valid only when the status indicates a completion. * Operations that are enqueued with cuFile Batch APIs are FIFO ordered with respect to other work on the stream and must be completed before continuing to the next action in the stream. Operations in each batch might be reordered with respect to each other. * The status field of individual IO operations via `CUfileIOParams_t` entries will have undefined values before the entire batch is complete. This definition is subject to change. Refer to the following for more information: * [cuFileRead](#cufileread) * [cuFileWrite](#cufilewrite) * [cuFileReadAsync](#cufilereadasync) * [cuFileWriteAsync](#cufilewriteasync) * [cuFileBatchIOGetStatus](#cufilebatchiogetstatus) * [cuFileBatchIOCancel](#cufilebatchiocancel) * [cuFileBatchIODestroy](#cufilebatchiodestroy) ### 5.5.3. cuFileBatchIOGetStatus[#](#cufilebatchiogetstatus "Link to this heading") CUfileError\_t cuFileBatchIOGetStatus(CUfileBatchHandle\_t batch\_idp, unsigned min\_nr, unsigned \*nr, CUfileIOEvents\_t \*iocbp, struct timespec\* timeout)); **Parameters** `batch_idp` > Obtained during setup. `min_nr` > The minimum number of IO entries for which status is requested. The `min_nr` should be greater than or equal to zero and less than or equal to `*nr`. `nr` > This is a pointer to max requested IO entries to poll for completion and is used as an Input/Output parameter. As an input `*nr` must be set to pass the maximum number of IO requests to poll for. As an output, `*nr` returns the number of completed I/Os. `iocbp` > `CUFileIOEvents_t` array containing the status of completed I/Os in that batch. `timeout` > This parameter is used to specify the amount of time to wait for in this API, even if the minimum number of requests have not completed. If the timeout hits, it is possible that the number of returned IOs can be less than `min_nr`. **Returns** * `CU_FILE_SUCCESS` on success. The success here refers to the completion of the API. Individual IO status and error can be obtained by examining the returned status and error in the array iocbp. * `CU_FILE_ERROR_INVALID_VALUE` for an invalid batch ID. **Description** * This is a batch API to monitor the status of batch IO operations by using the `batch_id` that was returned by `cuFileBatchIOSubmit`. The operation will be canceled automatically if `cuFileBatchIOCancel` is called and the status will reflect `CU_FILE_CANCELED` for all canceled IO operations. * The status of each member of the batch is queried, which would not be possible with one `CUEvent`. The status field of individual IO operations via `CUfileIOParams_t` entries will have undefined values before the entire batch is completed. This definition is subject to change. Refer to the following for more information: * [cuFileBatchIOSubmit](#cufilebatchiosubmit) * [cuFileBatchIODestroy](#cufilebatchiodestroy) ### 5.5.4. cuFileBatchIOCancel[#](#cufilebatchiocancel "Link to this heading") CUfileError\_t cuFileBatchIOCancel(CUfileBatchHandle\_t batch\_idp) **Parameters** `batch_idp` > The batch ID to cancel. **Returns** * `CU_FILE_SUCCESS` on success. * `CU_FILE_ERROR_INVALID_VALUE` on any failures. **Description** * This is a batch API to cancel an ongoing IO batch operation by using the `batch_id` that was returned by `cuFileBatchIOSubmit`. This API tries to cancel an individual IO operation in the batch if possible and provides no guarantee about canceling an ongoing operation. Refer to the following for more information: * [cuFileBatchIOGetStatus](#cufilebatchiogetstatus) * [cuFileBatchIOSubmit](#cufilebatchiosubmit) * [cuFileBatchIODestroy](#cufilebatchiodestroy) ### 5.5.5. cuFileBatchIODestroy[#](#cufilebatchiodestroy "Link to this heading") void cuFileBatchIODestroy(CUfileBatchHandle\_t batch\_idp) **Parameters** `batch_idp` > The batch handle to be destroyed. **Returns** void **Description** This is a batch API that destroys a batch context and the resources that are allocated with `cuFileBatchIOSetup`. Refer to the following for more information: * [cuFileBatchIOGetStatus](#cufilebatchiogetstatus) * [cuFileBatchIOSubmit](#cufilebatchiosubmit) * [cuFileBatchIOCancel](#cufilebatchiocancel) 6\. Sample Program with cuFile APIs[#](#sample-program-with-cufile-apis "Link to this heading") ================================================================================================ The following sample program uses the cuFile APIs: // To compile this sample code: // // nvcc gds\_helloworld.cxx -o gds\_helloworld -lcufile // // Set the environment variable TESTFILE // to specify the name of the file on a GDS enabled filesystem // // Ex: TESTFILE=/mnt/gds/gds\_test ./gds\_helloworld // // #include #include #include #include #include #include #include #include "cufile.h" //#include "cufile\_sample\_utils.h" using namespace std; int main(void) { int fd; ssize\_t ret; void \*devPtr\_base; off\_t file\_offset \= 0x2000; off\_t devPtr\_offset \= 0x1000; ssize\_t IO\_size \= 1UL << 24; size\_t buff\_size \= IO\_size + 0x1000; CUfileError\_t status; // CUResult cuda\_result; int cuda\_result; CUfileDescr\_t cf\_descr; CUfileHandle\_t cf\_handle; char \*testfn; testfn\=getenv("TESTFILE"); if (testfn\==NULL) { std::cerr << "No testfile defined via TESTFILE. Exiting." << std::endl; return \-1; } cout << std::endl; cout << "Opening File " << testfn << std::endl; fd \= open(testfn, O\_CREAT|O\_WRONLY|O\_DIRECT, 0644); if(fd < 0) { std::cerr << "file open " << testfn << "errno " << errno << std::endl; return \-1; } // the above fd could also have been opened without O\_DIRECT starting CUDA toolkit 12.2 // (gds 1.7.x version) as follows // fd = open(testfn, O\_CREAT|O\_WRONLY, 0644); cout << "Opening cuFileDriver." << std::endl; status \= cuFileDriverOpen(); if (status.err != CU\_FILE\_SUCCESS) { std::cerr << " cuFile driver failed to open " << std::endl; close(fd); return \-1; } cout << "Registering cuFile handle to " << testfn << "." << std::endl; memset((void \*)&cf\_descr, 0, sizeof(CUfileDescr\_t)); cf\_descr.handle.fd \= fd; cf\_descr.type \= CU\_FILE\_HANDLE\_TYPE\_OPAQUE\_FD; status \= cuFileHandleRegister(&cf\_handle, &cf\_descr); if (status.err != CU\_FILE\_SUCCESS) { std::cerr << "cuFileHandleRegister fd " << fd << " status " << status.err << std::endl; close(fd); return \-1; } cout << "Allocating CUDA buffer of " << buff\_size << " bytes." << std::endl; cuda\_result \= cudaMalloc(&devPtr\_base, buff\_size); if (cuda\_result != CUDA\_SUCCESS) { std::cerr << "buffer allocation failed " << cuda\_result << std::endl; cuFileHandleDeregister(cf\_handle); close(fd); return \-1; } cout << "Registering Buffer of " << buff\_size << " bytes." << std::endl; status \= cuFileBufRegister(devPtr\_base, buff\_size, 0); if (status.err != CU\_FILE\_SUCCESS) { std::cerr << "buffer registration failed " << status.err << std::endl; cuFileHandleDeregister(cf\_handle); close(fd); cudaFree(devPtr\_base); return \-1; } // fill a pattern cout << "Filling memory." << std::endl; cudaMemset((void \*) devPtr\_base, 0xab, buff\_size); cuStreamSynchronize(0); // perform write operation directly from GPU mem to file cout << "Writing buffer to file." << std::endl; ret \= cuFileWrite(cf\_handle, devPtr\_base, IO\_size, file\_offset, devPtr\_offset); if (ret < 0 || ret != IO\_size) { std::cerr << "cuFileWrite failed " << ret << std::endl; } // release the GPU memory pinning cout << "Releasing cuFile buffer." << std::endl; status \= cuFileBufDeregister(devPtr\_base); if (status.err != CU\_FILE\_SUCCESS) { std::cerr << "buffer deregister failed" << std::endl; cudaFree(devPtr\_base); cuFileHandleDeregister(cf\_handle); close(fd); return \-1; } cout << "Freeing CUDA buffer." << std::endl; cudaFree(devPtr\_base); // deregister the handle from cuFile cout << "Releasing file handle. " << std::endl; (void) cuFileHandleDeregister(cf\_handle); close(fd); // release all cuFile resources cout << "Closing File Driver." << std::endl; (void) cuFileDriverClose(); cout << std::endl; return 0; } 7\. Known Limitations of cuFile Batch APIs[#](#known-limitations-of-cufile-batch-apis "Link to this heading") ============================================================================================================== This section provides information about the known limitations of cuFile Batch APIs in this release of GDS. * Batch I/Os will be supported mainly by either the local file systems which are hosted on NVMe or NVMeOF devices or by the native file system that supports Linux AIO. Following table provides an overview of the cuFile batch API support with respect to different file systems. The following table provides an overview of cuFile batch API support with respect to distributed file systems: | File System | GDS Batch Mode | Comments | | --- | --- | --- | | Ext4/XFS | Read/Write support | | | DDN EXAScaler | Read/Write support | | | NFS | Read/Write support | | | IBM Spectrum Scale | Not available | Will work in compat mode | | Weka | Not available | Will work in compat mode | | BeeGFS | Not available | Will work in compat mode | 8\. Notice[#](#notice "Link to this heading") ============================================== This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. 9\. OpenCL[#](#opencl "Link to this heading") ============================================== OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. 10\. Trademarks[#](#trademarks "Link to this heading") ======================================================= NVIDIA, the NVIDIA logo, CUDA, DGX, DGX-1, DGX-2, DGX-A100, Tesla, and Quadro are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. On this page --- # cuRAND :: CUDA Toolkit Documentation cuRAND ([PDF](../pdf/CURAND_Library.pdf) ) - v12.8.1 ([older](https://developer.nvidia.com/cuda-toolkit-archive) ) - Last updated March 04, 2025 - [Send Feedback](mailto:CUDAIssues@nvidia.com?subject=CUDA%20Toolkit%20Documentation%20Feedback:%20cuRAND) cuRAND ------ The API reference guide for cuRAND, the CUDA random number generation library. Table of Contents ================= * [Introduction](introduction.html#introduction) * [1. Compatibility and Versioning](compatibility-and-versioning.html#compatibility-and-versioning) * [2. Host API Overview](host-api-overview.html#host-api-overview) * [2.1. Generator Types](host-api-overview.html#generator-types) * [2.2. Generator Options](host-api-overview.html#generator-options) * [2.2.1. Seed](host-api-overview.html#seed) * [2.2.2. Offset](host-api-overview.html#offset) * [2.2.3. Order](host-api-overview.html#order) * [2.3. Return Values](host-api-overview.html#return-values) * [2.4. Generation Functions](host-api-overview.html#generation-functions) * [2.5. Host API Example](host-api-overview.html#host-api-example) * [2.6. Static Library support](host-api-overview.html#static-library) * [2.7. Performance Notes](host-api-overview.html#performance-notes2) * [2.8. Thread Safety](host-api-overview.html#thread-safety) * [3. Device API Overview](device-api-overview.html#device-api-overview) * [3.1. Pseudorandom Sequences](device-api-overview.html#pseudorandom-sequences) * [3.1.1. Bit Generation with XORWOW and MRG32k3a generators](device-api-overview.html#bit-generation-1) * [3.1.2. Bit Generation with the MTGP32 generator](device-api-overview.html#bit-generation-2) * [3.1.3. Bit Generation with Philox\_4x32\_10 generator](device-api-overview.html#bit-generation-3) * [3.1.4. Distributions](device-api-overview.html#distributions) * [3.2. Quasirandom Sequences](device-api-overview.html#quasirandom-sequences) * [3.3. Skip-Ahead](device-api-overview.html#skip-ahead) * [3.4. Device API for discrete distributions](device-api-overview.html#device-api-for-discrete-distributions) * [3.5. Performance Notes](device-api-overview.html#performance-notes) * [3.6. Device API Examples](device-api-overview.html#device-api-example) * [3.7. Thrust and cuRAND Example](device-api-overview.html#thrust-and-curand-example) * [3.8. Poisson API Example](device-api-overview.html#poisson-api-example) * [4. Testing](testing.html#testing) * [5. Modules](modules.html#modules) * [5.1. Host API](group__HOST.html#group__HOST) * [5.2. Device API](group__DEVICE.html#group__DEVICE) * [A. Bibliography](bibliography.html#bibliography) * [B. Acknowledgements](acknowledgements.html#acknowledgements) * * * --- # NVIDIA 2D Image and Signal Processing Performance Primitives (NPP) — npp 12.8 documentation * [](../index.html) » * NVIDIA 2D Image and Signal Processing Performance Primitives (NPP) * v23.05 | [PDF](../pdf/NPP_Library.pdf) | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * NVIDIA 2D Image and Signal Processing Performance Primitives (NPP)[](#nvidia-2d-image-and-signal-processing-performance-primitives-npp "Permalink to this headline") ====================================================================================================================================================================== Indices and Search[](#indices-and-search "Permalink to this headline") ======================================================================== * [Index](genindex.html) * [Search Page](search.html) Contents: * [What is NPP ?](introduction.html) * [Files](introduction.html#introduction_lb_1Files) * [Header Files](introduction.html#introduction_lb_1header_files) * [Library Files](introduction.html#introduction_lb_1library_files) * [Library Organization](introduction.html#introduction_lb_1NPP) * [Supported NVIDIA Hardware](introduction.html#introduction_lb_1supported_hardware) * [General Conventions](introduction.html#general-conventions) * [Memory Management](introduction.html#general_conventions_lb_1memory_management_lb) * [Scratch Buffer and Host Pointer](introduction.html#general_conventions_lb_1general_scratch_buffer) * [Function Naming](introduction.html#general_conventions_lb_1npp_naming) * [Integer Result Scaling](introduction.html#general_conventions_lb_1integer_result_scaling) * [Rounding Modes](introduction.html#general_conventions_lb_1rounding_modes) * [Rounding Mode Parameter](introduction.html#general_conventions_lb_1rounding_mode_parameter) * [Image Processing Conventions](introduction.html#image-processing-conventions) * [Function Naming](introduction.html#nppi_conventions_lb_1nppi_naming) * [Image Data](introduction.html#nppi_conventions_lb_1passing_image_data) * [Line Step](introduction.html#nppi_conventions_lb_1line_step) * [Parameter Names for Image Data](introduction.html#nppi_conventions_lb_1image_data_parameter_names) * [Passing Source-Image Data](introduction.html#nppi_conventions_lb_1source_image) * [Source-Image Pointer](introduction.html#nppi_conventions_lb_1source_image_pointer) * [Source-Batch-Images Pointer](introduction.html#nppi_conventions_lb_1source_batch_images_pointer) * [Source-Planar-Image Pointer Array](introduction.html#nppi_conventions_lb_1source_planar_image_pointer_array) * [Source-Planar-Image Pointer](introduction.html#nppi_conventions_lb_1source_planar_image_pointer) * [Source-Image Line Step](introduction.html#nppi_conventions_lb_1source_image_line_step) * [Source-Planar-Image Line Step Array](introduction.html#nppi_conventions_lb_1source_planar_image_line_step_array) * [Source-Planar-Image Line Step](introduction.html#nppi_conventions_lb_1source_planar_image_line_step) * [Passing Destination-Image Data](introduction.html#nppi_conventions_lb_1destination_image) * [Destination-Image Pointer](introduction.html#nppi_conventions_lb_1destination_image_pointer) * [Destination-Batch-Images Pointer](introduction.html#nppi_conventions_lb_1destination_batch_images_pointer) * [Destination-Planar-Image Pointer Array](introduction.html#nppi_conventions_lb_1destination_planar_image_pointer_array) * [Destination-Planar-Image Pointer](introduction.html#nppi_conventions_lb_1destination_planar_image_pointer) * [Destination-Image Line Step](introduction.html#nppi_conventions_lb_1destination_image_line_step) * [Destination-Planar-Image Line Step](introduction.html#nppi_conventions_lb_1destination_planar_image_line_step) * [Passing In-Place Image Data](introduction.html#nppi_conventions_lb_1in_place_image) * [In-Place Image Pointer](introduction.html#nppi_conventions_lb_1in_place_image_pointer) * [In-Place-Image Line Step](introduction.html#nppi_conventions_lb_1in_place_image_line_step) * [Passing Mask-Image Data](introduction.html#nppi_conventions_lb_1mask_image) * [Mask-Image Pointer](introduction.html#nppi_conventions_lb_1mask_image_pointer) * [Mask-Image Line Step](introduction.html#nppi_conventions_lb_1mask_image_line_step) * [Passing Channel-of-Interest Data](introduction.html#nppi_conventions_lb_1channel_of_interest_section) * [Channel\_of\_Interest Number](introduction.html#nppi_conventions_lb_1channel_of_interest_number) * [Image Data Alignment Requirements](introduction.html#nppi_conventions_lb_1image_data_alignment) * [Image Data Related Error Codes](introduction.html#nppi_conventions_lb_1image_data_error_codes) * [Region-Of-Interest (ROI)](introduction.html#nppi_conventions_lb_1roi_specification) * [ROI Related Error Codes](introduction.html#nppi_conventions_lb_1roi_error_codes) * [Masked Operation](introduction.html#nppi_conventions_lb_1masked_operation) * [Channel-of-Interest API](introduction.html#nppi_conventions_lb_1channel_of_interest) * [Select-Channel Source-Image Pointer](introduction.html#nppi_conventions_lb_1select_source_pointer) * [Select-Channel Source-Image](introduction.html#nppi_conventions_lb_1select_source_channel) * [Select-Channel Destination-Image Pointer](introduction.html#nppi_conventions_lb_1select_destination_pointer) * [Source-Image Sampling](introduction.html#nppi_conventions_lb_1source_image_sampling) * [Point-Wise Operations](introduction.html#nppi_conventions_lb_1point_wise_operations) * [Neighborhood Operations](introduction.html#nppi_conventions_lb_1neighborhood_operations) * [Mask-Size Parameter](introduction.html#nppi_conventions_lb_1mask_size_parameter) * [Anchor-Point Parameter](introduction.html#nppi_conventions_lb_1anchor_point_parameter) * [Sampling Beyond Image Boundaries](introduction.html#nppi_conventions_lb_1sampling_beyond_image_boundaries) * [Signal Processing Conventions](introduction.html#signal-processing-conventions) * [Signal Data](introduction.html#npps_conventions_lb_1passing_signal_data) * [Parameter Names for Signal Data](introduction.html#npps_conventions_lb_1signal_data_parameter_names) * [Source Signal Pointer](introduction.html#npps_conventions_lb_1source_signal_pointer) * [Destination Signal Pointer](introduction.html#npps_conventions_lb_1destination_signal_pointer) * [In-Place Signal Pointer](introduction.html#npps_conventions_lb_1in_place_signal_pointer) * [Signal Data Alignment Requirements](introduction.html#npps_conventions_lb_1signal_data_alignment) * [Signal Data Related Error Codes](introduction.html#npps_conventions_lb_1signal_data_error_codes) * [Signal Length](introduction.html#npps_conventions_lb_1length_specification) * [Length Related Error Codes](introduction.html#npps_conventions_lb_1length_error_codes) * [Data Types, Structs, Enums, and Constants](nppdefs.html) * [Core NPP Functions](core_npp.html) * [Image Arithmetic And Logical Operations](image_arithmetic_and_logical_operations.html) * [Arithmetic Operations](image_arithmetic_and_logical_operations.html#arithmetic-operations) * [Arithmetic Operations](image_arithmetic_and_logical_operations.html#group__image__arithmetic__operations_1image_arithmetic_operations) * [AddC](image_arithmetic_and_logical_operations.html#group__image__addc_1image_addc) * [MulC](image_arithmetic_and_logical_operations.html#group__image__mulc_1image_mulc) * [MulCScale](image_arithmetic_and_logical_operations.html#group__image__mulcscale_1image_mulcscale) * [SubC](image_arithmetic_and_logical_operations.html#group__image__subc_1image_subc) * [DivC](image_arithmetic_and_logical_operations.html#group__image__divc_1image_divc) * [AbsDiffC](image_arithmetic_and_logical_operations.html#group__image__absdiffc_1image_absdiffc) * [Add](image_arithmetic_and_logical_operations.html#group__image__add_1image_add) * [AddSquare](image_arithmetic_and_logical_operations.html#group__image__addsquare_1image_addsquare) * [AddProduct](image_arithmetic_and_logical_operations.html#group__image__addproduct_1image_addproduct) * [AddWeighted](image_arithmetic_and_logical_operations.html#group__image__addweighted_1image_addweighted) * [Mul](image_arithmetic_and_logical_operations.html#group__image__mul_1image_mul) * [MulScale](image_arithmetic_and_logical_operations.html#group__image__mulscale_1image_mulscale) * [Sub](image_arithmetic_and_logical_operations.html#group__image__sub_1image_sub) * [Div](image_arithmetic_and_logical_operations.html#group__image__div_1image_div) * [Div\_Round](image_arithmetic_and_logical_operations.html#group__image__divround_1image_divround) * [Abs](image_arithmetic_and_logical_operations.html#group__image__abs_1image_abs) * [AbsDiff](image_arithmetic_and_logical_operations.html#group__image__absdiff_1image_absdiff) * [Sqr](image_arithmetic_and_logical_operations.html#group__image__sqr_1image_sqr) * [Sqrt](image_arithmetic_and_logical_operations.html#group__image__sqrt_1image_sqrt) * [Ln](image_arithmetic_and_logical_operations.html#group__image__ln_1image_ln) * [Exp](image_arithmetic_and_logical_operations.html#group__image__exp_1image_exp) * [Logical Operations](image_arithmetic_and_logical_operations.html#logical-operations) * [Logical Operations](image_arithmetic_and_logical_operations.html#group__image__logical__operations_1image_logical_operations) * [AndC](image_arithmetic_and_logical_operations.html#group__image__andc_1image_andc) * [OrC](image_arithmetic_and_logical_operations.html#group__image__orc_1image_orc) * [XorC](image_arithmetic_and_logical_operations.html#group__image__xorc_1image_xorc) * [RShiftC](image_arithmetic_and_logical_operations.html#group__image__rshiftc_1image_rshiftc) * [LShiftC](image_arithmetic_and_logical_operations.html#group__image__lshiftc_1image_lshiftc) * [And](image_arithmetic_and_logical_operations.html#group__image__and_1image_and) * [Or](image_arithmetic_and_logical_operations.html#group__image__or_1image_or) * [Xor](image_arithmetic_and_logical_operations.html#group__image__xor_1image_xor) * [Not](image_arithmetic_and_logical_operations.html#group__image__not_1image_not) * [Image Alpha Composition Operations](image_arithmetic_and_logical_operations.html#image-alpha-composition-operations) * [AlphaCompC](image_arithmetic_and_logical_operations.html#group__image__alphacompc_1image_alphacompc) * [AlphaComp](image_arithmetic_and_logical_operations.html#group__image__alphacomp_1image_alphacomp) * [Image Color Conversion Functions](image_color_conversion.html) * [Color Processing Functions](image_color_conversion.html#color-processing-functions) * [Color To Gray Conversion](image_color_conversion.html#color-to-gray-conversion) * [Color Debayer](image_color_conversion.html#color-debayer) * [Color Gamma Correction](image_color_conversion.html#color-gamma-correction) * [Complement Color Key](image_color_conversion.html#complement-color-key) * [ColorTwist](image_color_conversion.html#colortwist) * [ColorTwistBatch](image_color_conversion.html#colortwistbatch) * [ColorLUT](image_color_conversion.html#colorlut) * [ColorLUTLinear](image_color_conversion.html#colorlutlinear) * [ColorLUTCubic](image_color_conversion.html#colorlutcubic) * [ColorLUTTrilinear](image_color_conversion.html#colorluttrilinear) * [ColorLUTPalette](image_color_conversion.html#colorlutpalette) * [Color Sampling Format Conversion Functions](image_color_conversion.html#color-sampling-format-conversion-functions) * [YCbCr420ToYCbCr411](image_color_conversion.html#ycbcr420toycbcr411) * [YCbCr422ToYCbCr422](image_color_conversion.html#ycbcr422toycbcr422) * [YCbCr422ToYCrCb422](image_color_conversion.html#ycbcr422toycrcb422) * [YCbCr422ToCbYCr422](image_color_conversion.html#ycbcr422tocbycr422) * [CbYCr422ToYCbCr411](image_color_conversion.html#cbycr422toycbcr411) * [YCbCr422ToYCbCr420](image_color_conversion.html#ycbcr422toycbcr420) * [YCrCb420ToYCbCr422](image_color_conversion.html#ycrcb420toycbcr422) * [YCbCr422ToYCrCb420](image_color_conversion.html#ycbcr422toycrcb420) * [YCbCr422ToYCbCr411](image_color_conversion.html#ycbcr422toycbcr411) * [YCrCb422ToYCbCr422](image_color_conversion.html#ycrcb422toycbcr422) * [YCrCb422ToYCbCr420](image_color_conversion.html#ycrcb422toycbcr420) * [YCrCb422ToYCbCr411](image_color_conversion.html#ycrcb422toycbcr411) * [CbYCr422ToYCbCr422](image_color_conversion.html#cbycr422toycbcr422) * [CbYCr422ToYCbCr420](image_color_conversion.html#cbycr422toycbcr420) * [CbYCr422ToYCrCb420](image_color_conversion.html#cbycr422toycrcb420) * [YCbCr420ToYCbCr420](image_color_conversion.html#ycbcr420toycbcr420) * [YCbCr420ToYCbCr422](image_color_conversion.html#ycbcr420toycbcr422) * [YCbCr420ToCbYCr422](image_color_conversion.html#ycbcr420tocbycr422) * [YCbCr420ToYCrCb420](image_color_conversion.html#ycbcr420toycrcb420) * [YCrCb420ToCbYCr422](image_color_conversion.html#ycrcb420tocbycr422) * [YCrCb420ToYCbYCr420](image_color_conversion.html#ycrcb420toycbycr420) * [YCrCb420ToYCbYCr411](image_color_conversion.html#ycrcb420toycbycr411) * [YCbCr411ToYCbCr411](image_color_conversion.html#ycbcr411toycbcr411) * [YCbCr411ToYCbCr422](image_color_conversion.html#ycbcr411toycbcr422) * [YCbCr411ToYCrCb422](image_color_conversion.html#ycbcr411toycrcb422) * [YCbCr411ToYCbCr420](image_color_conversion.html#ycbcr411toycbcr420) * [YCbCr411ToYCrCb420](image_color_conversion.html#ycbcr411toycrcb420) * [NV12ToYUV420](image_color_conversion.html#nv12toyuv420) * [Color Model Conversion Functions](image_color_conversion.html#color-model-conversion-functions) * [RGBToYUV](image_color_conversion.html#rgbtoyuv) * [BGRToYUV](image_color_conversion.html#bgrtoyuv) * [YUVToRGB](image_color_conversion.html#yuvtorgb) * [YUVToRGBBatch](image_color_conversion.html#yuvtorgbbatch) * [YUVToRGBBatchAdvanced](image_color_conversion.html#yuvtorgbbatchadvanced) * [YUVToBGR](image_color_conversion.html#yuvtobgr) * [YUVToBGRBatch](image_color_conversion.html#yuvtobgrbatch) * [YUVToBGRBatchAdvanced](image_color_conversion.html#yuvtobgrbatchadvanced) * [RGBToYUV422](image_color_conversion.html#rgbtoyuv422) * [YUV422ToRGB](image_color_conversion.html#yuv422torgb) * [YUV422ToRGBBatch](image_color_conversion.html#yuv422torgbbatch) * [YUV422ToRGBBatchAdvanced](image_color_conversion.html#yuv422torgbbatchadvanced) * [YUV422ToBGRBatch](image_color_conversion.html#yuv422tobgrbatch) * [YUV422ToBGRBatchAdvanced](image_color_conversion.html#yuv422tobgrbatchadvanced) * [RGBToYUV420](image_color_conversion.html#rgbtoyuv420) * [YUV420ToRGB](image_color_conversion.html#yuv420torgb) * [YUV420ToRGBBatch](image_color_conversion.html#yuv420torgbbatch) * [YUV420ToRGBBatchAdvanced](image_color_conversion.html#yuv420torgbbatchadvanced) * [NV12ToRGB](image_color_conversion.html#nv12torgb) * [NV21ToRGB](image_color_conversion.html#nv21torgb) * [BGRToYUV420](image_color_conversion.html#bgrtoyuv420) * [YUV420ToBGR](image_color_conversion.html#yuv420tobgr) * [YUV420ToBGRBatch](image_color_conversion.html#yuv420tobgrbatch) * [YUV420ToBGRBatchAdvanced](image_color_conversion.html#yuv420tobgrbatchadvanced) * [NV12ToBGR](image_color_conversion.html#nv12tobgr) * [NV21ToBGR](image_color_conversion.html#nv21tobgr) * [RGBToYCbCr](image_color_conversion.html#rgbtoycbcr) * [YCbCrToRGB](image_color_conversion.html#ycbcrtorgb) * [YCbCrToRGBBatch](image_color_conversion.html#ycbcrtorgbbatch) * [YCbCrToRGBBatchAdvanced](image_color_conversion.html#ycbcrtorgbbatchadvanced) * [YCbCrToBGR](image_color_conversion.html#ycbcrtobgr) * [YCbCrToBGRBatch](image_color_conversion.html#ycbcrtobgrbatch) * [YCbCrToBGRBatchAdvanced](image_color_conversion.html#ycbcrtobgrbatchadvanced) * [YCbCrToBGR709CSC](image_color_conversion.html#ycbcrtobgr709csc) * [RGBToYCbCr422](image_color_conversion.html#rgbtoycbcr422) * [YCbCr422ToRGB](image_color_conversion.html#ycbcr422torgb) * [YCbCr422ToRGBBatch](image_color_conversion.html#ycbcr422torgbbatch) * [YCbCr422ToRGBBatchAdvanced](image_color_conversion.html#ycbcr422torgbbatchadvanced) * [RGBToYCrCb422](image_color_conversion.html#rgbtoycrcb422) * [YCrCb422ToRGB](image_color_conversion.html#ycrcb422torgb) * [YCbCr422ToBGR](image_color_conversion.html#ycbcr422tobgr) * [YCbCr422ToBGRBatch](image_color_conversion.html#ycbcr422tobgrbatch) * [YCbCr422ToBGRBatchAdvanced](image_color_conversion.html#ycbcr422tobgrbatchadvanced) * [RGBToCbYCr422](image_color_conversion.html#rgbtocbycr422) * [CbYCr422ToRGB](image_color_conversion.html#cbycr422torgb) * [BGRToCbYCr422](image_color_conversion.html#bgrtocbycr422) * [BGRToCbYCr422 709HDTV](image_color_conversion.html#bgrtocbycr422-709hdtv) * [CbYCr422ToBGR](image_color_conversion.html#cbycr422tobgr) * [CbYCr422ToBGR 709HDTV](image_color_conversion.html#cbycr422tobgr-709hdtv) * [RGBToYCbCr420](image_color_conversion.html#rgbtoycbcr420) * [YCbCr420ToRGB](image_color_conversion.html#ycbcr420torgb) * [YCbCr420ToRGBBatch](image_color_conversion.html#ycbcr420torgbbatch) * [YCbCr420ToRGBBatchAdvanced](image_color_conversion.html#ycbcr420torgbbatchadvanced) * [RGBToYCrCb420](image_color_conversion.html#rgbtoycrcb420) * [YCrCb420ToRGB](image_color_conversion.html#ycrcb420torgb) * [BGRToYCbCr420](image_color_conversion.html#bgrtoycbcr420) * [BGRToYCbCr420 709CSC](image_color_conversion.html#bgrtoycbcr420-709csc) * [BGRToYCbCr420 709HDTV](image_color_conversion.html#bgrtoycbcr420-709hdtv) * [BGRToYCrCb420 709CSC](image_color_conversion.html#bgrtoycrcb420-709csc) * [YCbCr420ToBGR](image_color_conversion.html#ycbcr420tobgr) * [YCbCr420ToBGRBatch](image_color_conversion.html#ycbcr420tobgrbatch) * [YCbCr420ToBGRBatchAdvanced](image_color_conversion.html#ycbcr420tobgrbatchadvanced) * [YCbCr420ToBGR 709CSC](image_color_conversion.html#ycbcr420tobgr-709csc) * [YCbCr420ToBGR 709HDTV](image_color_conversion.html#ycbcr420tobgr-709hdtv) * [BGRToYCrCb420](image_color_conversion.html#bgrtoycrcb420) * [BGRToYCbCr411](image_color_conversion.html#bgrtoycbcr411) * [BGRToYCbCr](image_color_conversion.html#bgrtoycbcr) * [YCbCr411ToBGR](image_color_conversion.html#ycbcr411tobgr) * [YCbCr411ToRGB](image_color_conversion.html#ycbcr411torgb) * [RGBToXYZ](image_color_conversion.html#rgbtoxyz) * [XYZToRGB](image_color_conversion.html#xyztorgb) * [RGBToLUV](image_color_conversion.html#rgbtoluv) * [LUVToRGB](image_color_conversion.html#luvtorgb) * [BGRToLab](image_color_conversion.html#bgrtolab) * [LabToBGR](image_color_conversion.html#labtobgr) * [RGBToYCC](image_color_conversion.html#rgbtoycc) * [YCCToRGB](image_color_conversion.html#ycctorgb) * [YCCKToCMYK\_JPEG](image_color_conversion.html#yccktocmyk-jpeg) * [CMYKOrYCCKJPEGToRGB](image_color_conversion.html#cmykorycckjpegtorgb) * [YCCKJPEGOrCMYKToBGR](image_color_conversion.html#ycckjpegorcmyktobgr) * [RGBToHLS](image_color_conversion.html#rgbtohls) * [HLSToRGB](image_color_conversion.html#hlstorgb) * [BGRToHLS](image_color_conversion.html#bgrtohls) * [HLSToBGR](image_color_conversion.html#hlstobgr) * [RBGToHSV](image_color_conversion.html#rbgtohsv) * [HSVToRGB](image_color_conversion.html#hsvtorgb) * [JPEG Color Conversion](image_color_conversion.html#jpeg-color-conversion) * [Image Data Exchange And Initialization Functions](image_data_exchange_and_initialization.html) * [Set](image_data_exchange_and_initialization.html#group__image__set_1image_set) * [Common parameters for nppiSet functions:](image_data_exchange_and_initialization.html#group__image__set_1CommonImageSetParameters) * [Masked Set](image_data_exchange_and_initialization.html#group__image__masked__set_1image_masked_set) * [Common parameters for nppiSet\_CXM functions:](image_data_exchange_and_initialization.html#group__image__masked__set_1CommonImageMaskedSetParameters) * [Channel Set](image_data_exchange_and_initialization.html#group__image__channel__set_1image_channel_set) * [Common parameters for nppiSet\_CXC functions:](image_data_exchange_and_initialization.html#group__image__channel__set_1CommonImageChannelSetParameters) * [Copy](image_data_exchange_and_initialization.html#group__image__copy_1image_copy) * [Common parameters for nppiCopy functions:](image_data_exchange_and_initialization.html#group__image__copy_1CommonImageCopyParameters) * [Masked Copy](image_data_exchange_and_initialization.html#group__image__masked__copy_1image_masked_copy) * [Common parameters for nppiCopy\_CXM functions:](image_data_exchange_and_initialization.html#group__image__masked__copy_1CommonImageMaskedCopyParameters) * [Channel Copy](image_data_exchange_and_initialization.html#group__image__channel__copy_1image_channel_copy) * [Common parameters for nppiCopy\_CXC functions:](image_data_exchange_and_initialization.html#group__image__channel__copy_1CommonImageChannelCopyParameters) * [Extract Channel Copy](image_data_exchange_and_initialization.html#group__image__extract__channel__copy_1image_extract_channel_copy) * [Common parameters for nppiCopy\_CXC1 functions:](image_data_exchange_and_initialization.html#group__image__extract__channel__copy_1CommonImageExtractChannelCopyParameters) * [Insert Channel Copy](image_data_exchange_and_initialization.html#group__image__insert__channel__copy_1image_insert_channel_copy) * [Common parameters for nppiCopy\_C1CX functions:](image_data_exchange_and_initialization.html#group__image__insert__channel__copy_1CommonImageInsertChannelCopyParameters) * [Packed To Planar Channel Copy](image_data_exchange_and_initialization.html#group__image__packed__to__planar__channel__copy_1image_packed_to_planar_channel_copy) * [Common parameters for nppiCopy\_CXPX functions:](image_data_exchange_and_initialization.html#group__image__packed__to__planar__channel__copy_1CommonImagePackedToPlanarChannelCopyParameters) * [Planar To Packed Channel Copy](image_data_exchange_and_initialization.html#group__image__planar__to__packed__channel__copy_1image_planar_to_packed_channel_copy) * [Common parameters for nppiCopy\_PXCX functions:](image_data_exchange_and_initialization.html#group__image__planar__to__packed__channel__copy_1CommonImagePlanarToPackedChannelCopyParameters) * [Copy Constant Border](image_data_exchange_and_initialization.html#group__image__copy__constant__border_1image_copy_constant_border) * [Common parameters for nppiCopyConstBorder functions:](image_data_exchange_and_initialization.html#group__image__copy__constant__border_1CommonImageCopyConstantBorderParameters) * [Copy Replicate Border](image_data_exchange_and_initialization.html#group__image__copy__replicate__border_1image_copy_replicate_border) * [Common parameters for nppiCopyReplicateBorder functions:](image_data_exchange_and_initialization.html#group__image__copy__replicate__border_1CommonImageCopyReplicateBorderParameters) * [Copy Wrap Border](image_data_exchange_and_initialization.html#group__image__copy__wrap__border_1image_copy_wrap_border) * [Common parameters for nppiCopyWrapBorder functions:](image_data_exchange_and_initialization.html#group__image__copy__wrap__border_1CommonImageCopyWrapBorderParameters) * [Copy Sub-Pixel](image_data_exchange_and_initialization.html#group__image__copy__sub__pixel_1image_copy_sub_pixel) * [Common parameters for nppiCopySubPix functions:](image_data_exchange_and_initialization.html#group__image__copy__sub__pixel_1CommonImageCopySubPixelParameters) * [Convert Bit Depth](image_data_exchange_and_initialization.html#group__image__convert_1image_convert) * [Convert To Increased Bit Depth](image_data_exchange_and_initialization.html#group__image__convert__increase_1image_convert_increase) * [Common parameters for nppiConvert to increased bit depth functions:](image_data_exchange_and_initialization.html#group__image__convert__increase_1CommonImageConvertToIncreasedBitDepthParameters) * [Convert To Decreased Bit Depth](image_data_exchange_and_initialization.html#group__image__convert__decrease_1image_convert_decrease) * [Common parameters for nppiConvert to decreased bit depth functions:](image_data_exchange_and_initialization.html#group__image__convert__decrease_1CommonImageConvertToDecreasedBitDepthParameters) * [Scale Bit Depth](image_data_exchange_and_initialization.html#group__image__scale_1image_scale) * [Scale To Higher Bit Depth](image_data_exchange_and_initialization.html#group__image__scale__to__higher__bit__depth_1image_scale_to_higher_bit_depth) * [Common parameters for nppiScale to higher bit depth functions:](image_data_exchange_and_initialization.html#group__image__scale__to__higher__bit__depth_1CommonImageScaleToHigherBitDepthParameters) * [Scale To Lower Bit Depth](image_data_exchange_and_initialization.html#group__image__scale__to__lower__bit__depth_1image_scale_to_lower_bit_depth) * [Common parameters for nppiScale to lower bit depth functions:](image_data_exchange_and_initialization.html#group__image__scale__to__lower__bit__depth_1CommonImageScaleToLowerBitDepthParameters) * [Duplicate Channel](image_data_exchange_and_initialization.html#group__image__duplicate__channel_1image_duplicate_channel) * [Common parameters for nppiDup functions:](image_data_exchange_and_initialization.html#group__image__duplicate__channel_1CommonImageDuplicateChannelParameters) * [Transpose](image_data_exchange_and_initialization.html#group__image__transpose_1image_transpose) * [Common parameters for nppiTranspose functions:](image_data_exchange_and_initialization.html#group__image__transpose_1CommonImageTransposeParameters) * [Swap Channels](image_data_exchange_and_initialization.html#group__image__swap__channels_1image_swap_channels) * [Image Filtering Functions](image_filtering_functions.html) * [Image 1D Linear Filters](image_filtering_functions.html#image-1d-linear-filters) * [1DLinearFilter](image_filtering_functions.html#group__image__1D__linear__filter_1image_1D_linear_filter) * [Image Filter Column](image_filtering_functions.html#image-filter-column) * [FilterColumn](image_filtering_functions.html#group__image__filter__column_1image_filter_column) * [Image Filter Column Border](image_filtering_functions.html#image-filter-column-border) * [FilterColumnBorder](image_filtering_functions.html#group__image__filter__column__border_1image_filter_column_border) * [Image Filter Column 32f](image_filtering_functions.html#image-filter-column-32f) * [FilterColumn32f](image_filtering_functions.html#group__image__filter__column__32f_1image_filter_column_32f) * [Image Filter Column Border 32f](image_filtering_functions.html#image-filter-column-border-32f) * [FilterColumnBorder32f](image_filtering_functions.html#group__image__filter__column__border__32f_1image_filter_column_border_32f) * [Image Filter Row](image_filtering_functions.html#image-filter-row) * [FilterRow](image_filtering_functions.html#group__image__filter__row_1image_filter_row) * [Image Filter Row Border](image_filtering_functions.html#image-filter-row-border) * [FilterRowBorder](image_filtering_functions.html#group__image__filter__row__border_1image_filter_row_border) * [Image Filter Row 32f](image_filtering_functions.html#image-filter-row-32f) * [FilterRow32f](image_filtering_functions.html#group__image__filter__row__32f_1image_filter_row_32f) * [Image Filter Row Border 32f](image_filtering_functions.html#image-filter-row-border-32f) * [FilterRowBorder32f](image_filtering_functions.html#group__image__filter__row__border__32f_1image_filter_row_border_32f) * [Image Filter 1D Window Sum](image_filtering_functions.html#image-filter-1d-window-sum) * [1D Window Sum](image_filtering_functions.html#group__image__1D__window__sum_1image_1D_window_sum) * [Image Filter 1D Window Column Sum](image_filtering_functions.html#image-filter-1d-window-column-sum) * [1D Window Column Sum](image_filtering_functions.html#group__image__filter__1D__window__column__sum_1image_filter_1D_window_column_sum) * [Image Filter 1D Window Row Sum](image_filtering_functions.html#image-filter-1d-window-row-sum) * [1D Window Row Sum](image_filtering_functions.html#group__image__filter__1D__window__row__sum_1image_filter_1D_window_row_sum) * [Image Filter 1D Window Sum Border](image_filtering_functions.html#image-filter-1d-window-sum-border) * [1D Window Sum with Border Control](image_filtering_functions.html#group__image__1D__window__sum__border_1image_1D_window_sum_border) * [Image Filter 1D Window Column Sum Border](image_filtering_functions.html#image-filter-1d-window-column-sum-border) * [1D Window Column Sum Border](image_filtering_functions.html#group__image__filter__1D__window__column__sum__border_1image_filter_1D_window_column_sum_border) * [Image Filter 1D Window Row Sum Border](image_filtering_functions.html#image-filter-1d-window-row-sum-border) * [1D Window Row Sum Border](image_filtering_functions.html#group__image__filter__1D__window__row__sum__border_1image_filter_1D_window_row_sum_border) * [Image Convolution](image_filtering_functions.html#image-convolution) * [Convolution](image_filtering_functions.html#group__image__convolution_1image_convolution) * [Image Filter](image_filtering_functions.html#image-filter) * [Filter](image_filtering_functions.html#group__image__filter_1image_filter) * [Image Filter 32f](image_filtering_functions.html#image-filter-32f) * [Filter32f](image_filtering_functions.html#group__image__filter__32f_1image_filter_32f) * [Image Filter Border](image_filtering_functions.html#image-filter-border) * [FilterBorder](image_filtering_functions.html#group__image__filter__border_1image_filter_border) * [Image Filter Border 32f](image_filtering_functions.html#image-filter-border-32f) * [FilterBorder32f](image_filtering_functions.html#group__image__filter__border__32f_1image_filter_border_32f) * [2D Fixed Linear Filters](image_filtering_functions.html#d-fixed-linear-filters) * [2D Fixed Linear Filters](image_filtering_functions.html#group__image__2D__fixed__linear__filters_1image_2D_fixed_linear_filters) * [Image Filter Box](image_filtering_functions.html#image-filter-box) * [FilterBox](image_filtering_functions.html#group__image__filter__box_1image_filter_box) * [Image Filter Box Border](image_filtering_functions.html#image-filter-box-border) * [FilterBoxBorder](image_filtering_functions.html#group__image__filter__box__border_1image_filter_box_border) * [Image Filter Box Border Advanced](image_filtering_functions.html#image-filter-box-border-advanced) * [FilterBoxBorderAdvanced](image_filtering_functions.html#group__image__filter__box__border__advanced_1image_filter_box_border_advanced) * [Image Filter Threshold Adaptive Box Border](image_filtering_functions.html#image-filter-threshold-adaptive-box-border) * [FilterThresholdAdaptiveBoxBorder](image_filtering_functions.html#group__image__filter__threshold__adaptive__box__border_1image_filter_threshold_adaptive_box_border) * [Rank Filters](image_filtering_functions.html#rank-filters) * [Rank Filters](image_filtering_functions.html#group__image__rank__filters_1image_rank_filters) * [Image Filter Max](image_filtering_functions.html#image-filter-max) * [FilterMax](image_filtering_functions.html#group__image__filter__max_1image_filter_max) * [Image Filter Max Border](image_filtering_functions.html#image-filter-max-border) * [FilterMaxBorder](image_filtering_functions.html#group__image__filter__max__border_1image_filter_max_border) * [Image Filter Min](image_filtering_functions.html#image-filter-min) * [FilterMin](image_filtering_functions.html#group__image__filter__min_1image_filter_min) * [Image Filter Min Border](image_filtering_functions.html#image-filter-min-border) * [FilterMinBorder](image_filtering_functions.html#group__image__filter__min__border_1image_filter_min_border) * [Image Filter Median](image_filtering_functions.html#image-filter-median) * [FilterMedian](image_filtering_functions.html#group__image__filter__median_1image_filter_median) * [Image Filter Median Border](image_filtering_functions.html#image-filter-median-border) * [FilterMedianBorder](image_filtering_functions.html#group__image__filter__median__border_1image_filter_median_border) * [Fixed Filters](image_filtering_functions.html#fixed-filters) * [Fixed Filters](image_filtering_functions.html#group__fixed__filters_1fixed_filters) * [Image Filter Prewitt](image_filtering_functions.html#image-filter-prewitt) * [FilterPrewitt](image_filtering_functions.html#group__image__filter__prewitt_1image_filter_prewitt) * [Image Filter Prewitt Border](image_filtering_functions.html#image-filter-prewitt-border) * [FilterPrewittBorder](image_filtering_functions.html#group__image__filter__prewitt__border_1image_filter_prewitt_border) * [Image Filter Scharr](image_filtering_functions.html#image-filter-scharr) * [FilterScharr](image_filtering_functions.html#group__image__filter__scharr_1image_filter_scharr) * [Image Filter Scharr Border](image_filtering_functions.html#image-filter-scharr-border) * [FilterScharrBorder](image_filtering_functions.html#group__image__filter__scharr__border_1image_filter_scharr_border) * [Image Filter Sobel](image_filtering_functions.html#image-filter-sobel) * [FilterSobel](image_filtering_functions.html#group__image__filter__sobel_1image_filter_sobel) * [Image Filter Sobel Border](image_filtering_functions.html#image-filter-sobel-border) * [FilterSobelBorder](image_filtering_functions.html#group__image__filter__sobel__border_1image_filter_sobel_border) * [Image Filter Roberts](image_filtering_functions.html#image-filter-roberts) * [FilterRoberts](image_filtering_functions.html#group__image__filter__roberts_1image_filter_roberts) * [Image Filter Roberts Border](image_filtering_functions.html#image-filter-roberts-border) * [FilterRobertsBorder](image_filtering_functions.html#group__image__filter__roberts__border_1image_filter_roberts_border) * [Image Filter Laplace](image_filtering_functions.html#image-filter-laplace) * [FilterLaplace](image_filtering_functions.html#group__image__filter__laplace_1image_filter_laplace) * [Image Filter Laplace Border](image_filtering_functions.html#image-filter-laplace-border) * [FilterLaplaceBorder](image_filtering_functions.html#group__image__filter__laplace__border_1image_filter_laplace_border) * [Image Filter Gauss](image_filtering_functions.html#image-filter-gauss) * [FilterGauss](image_filtering_functions.html#group__image__filter__gauss_1image_filter_gauss) * [Image Filter Gauss Advanced](image_filtering_functions.html#image-filter-gauss-advanced) * [FilterGaussAdvanced](image_filtering_functions.html#group__image__filter__gauss__advanced_1image_filter_gauss_advanced) * [Image Filter Gauss Border](image_filtering_functions.html#image-filter-gauss-border) * [FilterGaussBorder](image_filtering_functions.html#group__image__filter__gauss__border_1image_filter_gauss_border) * [Image Filter Advanced Gauss Border](image_filtering_functions.html#image-filter-advanced-gauss-border) * [FilterGaussAdvancedBorder](image_filtering_functions.html#group__image__filter__gauss__advanced__border_1image_filter_gauss_advanced_border) * [Image Filter Gauss Pyramid Layer Down Border](image_filtering_functions.html#image-filter-gauss-pyramid-layer-down-border) * [FilterGaussPyramidLayerDownBorder](image_filtering_functions.html#group__image__filter__gauss__pyramid__layer__down__border_1image_filter_gauss_pyramid_layer_down_border) * [Image Filter Gauss Pyramid Layer Up Border](image_filtering_functions.html#image-filter-gauss-pyramid-layer-up-border) * [FilterGaussPyramidLayerUpBorder](image_filtering_functions.html#group__image__filter__gauss__pyramid__layer__up__border_1image_filter_gauss_pyramid_layer_up_border) * [Image Filter Bilateral Gauss Border](image_filtering_functions.html#image-filter-bilateral-gauss-border) * [FilterBilateralGaussBorder](image_filtering_functions.html#group__image__filter__bilateral__gauss__border_1image_filter_bilateral_gauss_border) * [Image Filter High Pass](image_filtering_functions.html#image-filter-high-pass) * [FilterHighPass](image_filtering_functions.html#group__image__filter__high__pass_1image_filter_high_pass) * [Image Filter High Pass Border](image_filtering_functions.html#image-filter-high-pass-border) * [FilterHighPassBorder](image_filtering_functions.html#group__image__filter__high__pass__border_1image_filter_high_pass_border) * [Image Filter Low Pass](image_filtering_functions.html#image-filter-low-pass) * [FilterLowPass](image_filtering_functions.html#group__image__filter__low__pass_1image_filter_low_pass) * [Image Filter Low Pass Border](image_filtering_functions.html#image-filter-low-pass-border) * [FilterLowPassBorder](image_filtering_functions.html#group__image__filter__low__pass__border_1image_filter_low_pass_border) * [Image Filter Sharpen](image_filtering_functions.html#image-filter-sharpen) * [FilterSharpen](image_filtering_functions.html#group__image__filter__sharpen_1image_filter_sharpen) * [Image Filter Sharpen Border](image_filtering_functions.html#image-filter-sharpen-border) * [FilterSharpenBorder](image_filtering_functions.html#group__image__filter__sharpen__border_1image_filter_sharpen_border) * [Image Filter Unsharp Border](image_filtering_functions.html#image-filter-unsharp-border) * [FilterUnsharpBorder](image_filtering_functions.html#group__image__filter__unsharp__border_1image_filter_unsharp_border) * [Image Filter Wiener Border](image_filtering_functions.html#image-filter-wiener-border) * [FilterWienerBorder](image_filtering_functions.html#group__image__filter__wiener__border_1image_filter_wiener_border) * [Image Filter Gradient Vector Prewitt Border](image_filtering_functions.html#image-filter-gradient-vector-prewitt-border) * [GradientVectorPrewittBorder](image_filtering_functions.html#group__image__filter__gradient__vector__prewitt__border_1image_filter_gradient_vector_prewitt_border) * [Image Filter Gradient Vector Scharr Border](image_filtering_functions.html#image-filter-gradient-vector-scharr-border) * [GradientVectorScharrBorder](image_filtering_functions.html#group__image__filter__gradient__vector__scharr__border_1image_filter_gradient_vector_scharr_border) * [Image Filter Gradient Vector Sobel Border](image_filtering_functions.html#image-filter-gradient-vector-sobel-border) * [GradientVectorSobelBorder](image_filtering_functions.html#group__image__filter__gradient__vector__sobel__border_1image_filter_gradient_vector_sobel_border) * [Computer Vision Filtering Functions](image_filtering_functions.html#computer-vision-filtering-functions) * [Computer Vision](image_filtering_functions.html#group__image__computer__vision__filtering__functions_1image_computer_vision_filtering_functions) * [Image Filter Distance Transform](image_filtering_functions.html#image-filter-distance-transform) * [FilterDistanceTransform](image_filtering_functions.html#group__image__filter__distance__transform_1image_filter_distance_transform) * [Image Filter Harris Corners Border](image_filtering_functions.html#image-filter-harris-corners-border) * [FilterHarrisCornersBorder](image_filtering_functions.html#group__image__filter__harris__corners__border_1image_filter_harris_corners_border) * [Image Filter Hough Line](image_filtering_functions.html#image-filter-hough-line) * [FilterHoughLine](image_filtering_functions.html#group__image__filter__hough__line_1image_filter_hough_line) * [Image Filter Histogram Of Oriented Gradients Border](image_filtering_functions.html#image-filter-histogram-of-oriented-gradients-border) * [HistogramOfOrientedGradientsBorder](image_filtering_functions.html#group__image__filter__histogram__of__oriented__gradients__border_1image_filter_histogram_of_oriented_gradients_border) * [Image Filter Flood Fill](image_filtering_functions.html#image-filter-flood-fill) * [FloodFill](image_filtering_functions.html#group__image__filter__flood__fill_1image_filter_flood_fill) * [Flood Fill](image_filtering_functions.html#flood-fill) * [FloodFill](image_filtering_functions.html#group__flood__fill_1flood_fill) * [Flood Fill Boundary](image_filtering_functions.html#flood-fill-boundary) * [FloodFillBoundary](image_filtering_functions.html#group__flood__fill__boundary_1flood_fill_boundary) * [Flood Fill Range](image_filtering_functions.html#flood-fill-range) * [FloodFillRange](image_filtering_functions.html#group__flood__fill__range_1flood_fill_range) * [Flood Fill Range Boundary](image_filtering_functions.html#flood-fill-range-boundary) * [FloodFillRangeBoundary](image_filtering_functions.html#group__flood__fill__range__boundary_1flood_fill_range_boundary) * [Flood Fill Gradient](image_filtering_functions.html#flood-fill-gradient) * [FloodFillGradient](image_filtering_functions.html#group__flood__fill__gradient_1flood_fill_gradient) * [Flood Fill Gradient Boundary](image_filtering_functions.html#flood-fill-gradient-boundary) * [FloodFillGradientBoundary](image_filtering_functions.html#group__flood__fill__gradient__boundary_1flood_fill_gradient_boundary) * [Label Markers](image_filtering_functions.html#label-markers) * [LabelMarkers](image_filtering_functions.html#group__image__filter__label__markers_1image_filter_label_markers) * [Label MarkersUF](image_filtering_functions.html#label-markersuf) * [LabelMarkersUF](image_filtering_functions.html#group__label__markersUF_1label_markersUF) * [Label MarkersUF Batch](image_filtering_functions.html#label-markersuf-batch) * [LabelMarkersUFBatch](image_filtering_functions.html#group__label__markersUF__batch_1label_markersUF_batch) * [Label MarkersUF Batch Advanced](image_filtering_functions.html#label-markersuf-batch-advanced) * [LabelMarkersUFBatchAdvanced](image_filtering_functions.html#group__label__markersUF__batch__advanced_1label_markersUF_batch_advanced) * [Image Filter Compress Marker Labels](image_filtering_functions.html#image-filter-compress-marker-labels) * [Image Filter Compressed Marker Labels Info](image_filtering_functions.html#image-filter-compressed-marker-labels-info) * [Image Filter Contour Pixel Interpolation](image_filtering_functions.html#image-filter-contour-pixel-interpolation) * [ContourPixelInterpolation](image_filtering_functions.html#group__image__filter__contour__pixel__interpolation_1image_filter_contour_pixel_interpolation) * [Bound Segments](image_filtering_functions.html#bound-segments) * [Watershed Segmentation](image_filtering_functions.html#watershed-segmentation) * [WatershedSegmentation](image_filtering_functions.html#group__image__filter__watershed__segmentation_1image_filter_watershed_segmentation) * [Image Geometry Transforms Functions](image_geometry_transforms.html) * [Geometric Transform API Specifics](image_geometry_transforms.html#group__image__geometry__transforms_1geometric_transform_api) * [Geometric Transforms and ROIs](image_geometry_transforms.html#group__image__geometry__transforms_1geometric_transform_roi) * [Pixel Interpolation](image_geometry_transforms.html#group__image__geometry__transforms_1geometric_transforms_interpolation) * [Resize Error Codes](image_geometry_transforms.html#group__image__geometry__transforms_1resize_error_codes) * [ResizeSqrPixel](image_geometry_transforms.html#group__image__resize__square__pixel_1image_resize_square_pixel) * [Resize](image_geometry_transforms.html#group__image__resize_1image_resize) * [ResizeBatch](image_geometry_transforms.html#group__image__resize__batch_1image_resize_batch) * [Common parameters for nppiResizeBatch functions:](image_geometry_transforms.html#group__image__resize__batch_1CommonResizeBatchParameters) * [Common parameters for nppiResizeBatchAdvanced functions:](image_geometry_transforms.html#group__image__resize__batch_1CommonResizeBatchAdvancedParameters) * [Remap](image_geometry_transforms.html#group__image__remap_1image_remap) * [Error Codes](image_geometry_transforms.html#group__image__remap_1remap_error_codes) * [Common parameters for nppiRemap packed pixel functions:](image_geometry_transforms.html#group__image__remap_1CommonRemapPackedPixelParameters) * [Common parameters for nppiRemap planar pixel functions:](image_geometry_transforms.html#group__image__remap_1CommonRemapPlanarPixelParameters) * [Rotate](image_geometry_transforms.html#group__image__rotate_1image_rotate) * [Rotate Error Codes](image_geometry_transforms.html#group__image__rotate_1rotate_error_codes) * [Common parameters for nppiRotate functions:](image_geometry_transforms.html#group__image__rotate_1CommonRotateParameters) * [Rotate Utility Functions](image_geometry_transforms.html#group__rotate__utility__functions_1rotate_utility_functions) * [Mirror](image_geometry_transforms.html#group__image__mirror_1image_mirror) * [Mirror Error Codes](image_geometry_transforms.html#group__image__mirror_1mirror_error_codes) * [Common parameters for nppiMirror functions:](image_geometry_transforms.html#group__image__mirror_1CommonMirrorParameters) * [Affine Transforms](image_geometry_transforms.html#affine-transforms) * [Affine Transforms](image_geometry_transforms.html#group__image__affine__transform_1image_affine_transform) * [Affine Transform Error Codes](image_geometry_transforms.html#group__image__affine__transform_1affine_transform_error_codes) * [Affine Transform Utility Functions](image_geometry_transforms.html#group__affine__transform__utility__functions_1affine_transform_utility_functions) * [Affine Transform](image_geometry_transforms.html#group__affine__transform_1affine_transform) * [Common parameters for nppiWarpAffine packed pixel functions:](image_geometry_transforms.html#group__affine__transform_1CommonWarpAffinePackedPixelParameters) * [Common parameters for nppiWarpAffine planar pixel functions:](image_geometry_transforms.html#group__affine__transform_1CommonWarpAffinePlanarPixelParameters) * [Affine Transform Batch](image_geometry_transforms.html#group__affine__transform__batch_1affine_transform_batch) * [Codes](image_geometry_transforms.html#group__affine__transform__batch_1Error) * [Common parameters for nppiWarpAffineBatch functions:](image_geometry_transforms.html#group__affine__transform__batch_1CommonWarpAffineBatchParameters) * [Backwards Affine Transform](image_geometry_transforms.html#group__backwards__affine__transform_1backwards_affine_transform) * [Common parameters for nppiWarpAffineBack packed pixel functions:](image_geometry_transforms.html#group__backwards__affine__transform_1CommonWarpAffineBackPackedPixelParameters) * [Common parameters for nppiWarpAffineBack planar pixel functions:](image_geometry_transforms.html#group__backwards__affine__transform_1CommonWarpAffineBackPlanarPixelParameters) * [Quad-Based Affine Transform](image_geometry_transforms.html#group__quad__based__affine__transform_1quad_based_affine_transform) * [Common parameters for nppiWarpAffineQuad packed pixel functions:](image_geometry_transforms.html#group__quad__based__affine__transform_1CommonWarpAffineQuadPackedPixelParameters) * [Common parameters for nppiWarpAffineQuad planar pixel functions:](image_geometry_transforms.html#group__quad__based__affine__transform_1CommonWarpAffineQuadPlanarPixelParameters) * [Perspective Transforms](image_geometry_transforms.html#perspective-transforms) * [Perspective Transform](image_geometry_transforms.html#group__image__perspective__transforms_1image_perspective_transforms) * [Perspective Transform Error Codes](image_geometry_transforms.html#group__image__perspective__transforms_1perspective_transform_error_codes) * [Perspective Transform](image_geometry_transforms.html#group__perspective__transform_1perspective_transform) * [Common parameters for nppiWarpPerspective packed pixel functions:](image_geometry_transforms.html#group__perspective__transform_1CommonWarpPerspectivePackedPixelParameters) * [Common parameters for nppiWarpPerspective planar pixel functions:](image_geometry_transforms.html#group__perspective__transform_1CommonWarpPerspectivePlanarPixelParameters) * [Perspective Transform Batch](image_geometry_transforms.html#group__perspective__transform__batch_1perspective_transform_batch) * [Common parameters for nppiWarpPerspectiveBatch functions:](image_geometry_transforms.html#group__perspective__transform__batch_1CommonWarpPerspectiveBatchParameters) * [Backwards Perspective Transform](image_geometry_transforms.html#group__backwards__perspective__transform_1backwards_perspective_transform) * [Common parameters for nppiWarpPerspectiveBack packed pixel functions:](image_geometry_transforms.html#group__backwards__perspective__transform_1CommonWarpPerspectiveBackPackedPixelParameters) * [Common parameters for nppiWarpPerspectiveBack planar pixel functions:](image_geometry_transforms.html#group__backwards__perspective__transform_1CommonWarpPerspectiveBackPlanarPixelParameters) * [Quad-Based Perspective Transform](image_geometry_transforms.html#group__quad__based__perspective__transform_1quad_based_perspective_transform) * [Common parameters for nppiWarpPerspectiveQuad packed pixel functions:](image_geometry_transforms.html#group__quad__based__perspective__transform_1CommonWarpPerspectiveQuadPackedPixelParameters) * [Common parameters for nppiWarpPerspectiveQuad planar pixel functions:](image_geometry_transforms.html#group__quad__based__perspective__transform_1CommonWarpPerspectiveQuadPlanarPixelParameters) * [Image Linear Transforms Functions](image_linear_transforms.html) * [Fourier Transforms](image_linear_transforms.html#group__image__fourier__transforms_1image_fourier_transforms) * [Image Morphological Operations](image_morphological_operations.html) * [Dilation Functions](image_morphological_operations.html#dilation-functions) * [Image Dilate](image_morphological_operations.html#image-dilate) * [Dilation](image_morphological_operations.html#group__image__dilate_1image_dilate) * [Image Dilate Border](image_morphological_operations.html#image-dilate-border) * [Dilation with border control](image_morphological_operations.html#group__image__dilate__border_1image_dilate_border) * [Image Dilate 3x3](image_morphological_operations.html#image-dilate-3x3) * [Dilate3x3](image_morphological_operations.html#group__image__dilate__3x3_1image_dilate_3x3) * [Image Dilate 3x3 Border](image_morphological_operations.html#image-dilate-3x3-border) * [Dilate3x3Border](image_morphological_operations.html#group__image__dilate__3x3__border_1image_dilate_3x3_border) * [Erosion Functions](image_morphological_operations.html#erosion-functions) * [Image Erode](image_morphological_operations.html#image-erode) * [Erode](image_morphological_operations.html#group__image__erode_1image_erode) * [Image Erode Border](image_morphological_operations.html#image-erode-border) * [Erosion with border control](image_morphological_operations.html#group__image__erode__border_1image_erode_border) * [Image Erode 3x3](image_morphological_operations.html#image-erode-3x3) * [Erode3x3](image_morphological_operations.html#group__image__erode__3x3_1image_erode_3x3) * [Image Erode 3x3 Border](image_morphological_operations.html#image-erode-3x3-border) * [Erode3x3Border](image_morphological_operations.html#group__image__erode__3x3__border_1image_erode_3x3_border) * [Image Complex Morphphological Operations](image_morphological_operations.html#image-complex-morphphological-operations) * [Image Morph](image_morphological_operations.html#image-morph) * [ComplexImageMorphology](image_morphological_operations.html#group__image__morph_1image_morph) * [Image Morph Get Buffer Size](image_morphological_operations.html#image-morph-get-buffer-size) * [MorphGetBufferSize](image_morphological_operations.html#group__image__morph__get__buffer__size_1image_morph_get_buffer_size) * [Image Morph Close Border](image_morphological_operations.html#image-morph-close-border) * [MorphCloseBorder](image_morphological_operations.html#group__image__morph__close__border_1image_morph_close_border) * [Image Morph Open Border](image_morphological_operations.html#image-morph-open-border) * [MorphOpenBorder](image_morphological_operations.html#group__image__morph__open__border_1image_morph_open_border) * [Image Morph Top Hat Border](image_morphological_operations.html#image-morph-top-hat-border) * [MorphToHatBorder](image_morphological_operations.html#group__image__morph__top__hat__border_1image_morph_top_hat_border) * [Image Morph Black Hat Border](image_morphological_operations.html#image-morph-black-hat-border) * [MorphBlackHatBorder](image_morphological_operations.html#group__image__morph__black__hat__border_1image_morph_black_hat_border) * [Image Morph Gradient Border](image_morphological_operations.html#image-morph-gradient-border) * [MorphGradientBorder](image_morphological_operations.html#group__image__morph__gradient__border_1image_morph_gradient_border) * [Image Statistics Functions](image_statistics_functions.html) * [CommonGetBufferHostSizeParameters](image_statistics_functions.html#group__image__statistics__functions_1CommonGetBufferHostSizeParameters) * [Image Sum](image_statistics_functions.html#image-sum) * [Sum](image_statistics_functions.html#group__image__sum_1image_sum) * [Image Min](image_statistics_functions.html#image-min) * [Min](image_statistics_functions.html#group__image__min_1image_min) * [Image Min Index](image_statistics_functions.html#image-min-index) * [MinIndx](image_statistics_functions.html#group__image__min__index_1image_min_index) * [Image Max](image_statistics_functions.html#image-max) * [Max](image_statistics_functions.html#group__image__max_1image_max) * [Common parameters for nppiMax functions include:](image_statistics_functions.html#group__image__max_1CommonMaxParameters) * [Image Max Index](image_statistics_functions.html#image-max-index) * [MaxIndx](image_statistics_functions.html#group__image__max__index_1image_max_index) * [Image MinMax](image_statistics_functions.html#image-minmax) * [MinMax](image_statistics_functions.html#group__image__min__max_1image_min_max) * [Image Mean](image_statistics_functions.html#image-mean) * [Mean](image_statistics_functions.html#group__image__mean_1image_mean) * [Image Mean StdDev](image_statistics_functions.html#image-mean-stddev) * [Mean\_StdDev](image_statistics_functions.html#group__image__mean__stddev_1image_mean_stddev) * [Image Norms](image_statistics_functions.html#image-norms) * [Image Norms](image_statistics_functions.html#group__image__norm_1image_norm) * [Common parameters for nppiNorm functions include:](image_statistics_functions.html#group__image__norm_1CommonNormParameters) * [Image Norm Inf](image_statistics_functions.html#image-norm-inf) * [Norm\_Inf](image_statistics_functions.html#group__image__inf__norm_1image_inf_norm) * [Image Norm L1](image_statistics_functions.html#image-norm-l1) * [Norm\_L1](image_statistics_functions.html#group__image__L1__norm_1image_L1_norm) * [Image Norm L2](image_statistics_functions.html#image-norm-l2) * [Norm\_L2](image_statistics_functions.html#group__image__L2__norm_1image_L2_norm) * [Image NormDiff Inf](image_statistics_functions.html#image-normdiff-inf) * [NormDiff\_Inf](image_statistics_functions.html#group__image__inf__normdiff_1image_inf_normdiff) * [Image NormDiff L1](image_statistics_functions.html#image-normdiff-l1) * [NormDiff\_L1](image_statistics_functions.html#group__image__L1__normdiff_1image_L1_normdiff) * [Image NormDiff L2](image_statistics_functions.html#image-normdiff-l2) * [NormDiff\_L2](image_statistics_functions.html#group__image__L2__normdiff_1image_L2_normdiff) * [Image NormRel Inf](image_statistics_functions.html#image-normrel-inf) * [NormRel\_Inf](image_statistics_functions.html#group__image__inf__normrel_1image_inf_normrel) * [Image NormRel L1](image_statistics_functions.html#image-normrel-l1) * [NormRel\_L1](image_statistics_functions.html#group__image__L1__normrel_1image_L1_normrel) * [Image NormRel L2](image_statistics_functions.html#image-normrel-l2) * [NormRel\_L2](image_statistics_functions.html#group__image__L2__normrel_1image_L2_normrel) * [Image DotProd](image_statistics_functions.html#image-dotprod) * [DotProd](image_statistics_functions.html#group__image__dot__prod_1image_dot_prod) * [Image Count In Range](image_statistics_functions.html#image-count-in-range) * [CountInRange.](image_statistics_functions.html#group__image__count__in__range_1image_count_in_range) * [Image MaxEvery](image_statistics_functions.html#image-maxevery) * [MaxEvery](image_statistics_functions.html#group__image__maxevery_1image_maxevery) * [Image MinEvery](image_statistics_functions.html#image-minevery) * [MinEvery](image_statistics_functions.html#group__image__minevery_1image_minevery) * [Image Integral](image_statistics_functions.html#image-integral) * [Integral](image_statistics_functions.html#group__image__integral_1image_integral) * [Image Square Integral](image_statistics_functions.html#image-square-integral) * [SqrIntegral](image_statistics_functions.html#group__image__sqrintegral_1image_sqrintegral) * [Image RectStdDev](image_statistics_functions.html#image-rectstddev) * [RectStdDev](image_statistics_functions.html#group__image__rectstddev_1image_rectstddev) * [Image Histogram Even](image_statistics_functions.html#image-histogram-even) * [HistogramEven](image_statistics_functions.html#group__image__histogrameven_1image_histogrameven) * [Image Histogram Range](image_statistics_functions.html#image-histogram-range) * [HistogramRange](image_statistics_functions.html#group__image__histogramrange_1image_histogramrange) * [Image Proximity](image_statistics_functions.html#image-proximity) * [Image Proximity](image_statistics_functions.html#group__image__proximity_1image_proximity) * [General Introduction](image_statistics_functions.html#group__image__proximity_1general_introduction) * [Categorizations](image_statistics_functions.html#group__image__proximity_1category) * [Image Square Distance Full Norm](image_statistics_functions.html#image-square-distance-full-norm) * [SqrDistanceFull\_Norm](image_statistics_functions.html#group__sqrdistancefullnorm_1sqrdistancefullnorm) * [Image Square Distance Same Norm](image_statistics_functions.html#image-square-distance-same-norm) * [SqrDistanceSame\_Norm](image_statistics_functions.html#group__sqrdistancesamenorm_1sqrdistancesamenorm) * [Image Square Distance Valid Norm](image_statistics_functions.html#image-square-distance-valid-norm) * [SqrDistanceValid\_Norm](image_statistics_functions.html#group__sqrdistancevalidnorm_1sqrdistancevalidnorm) * [Image Cross Correlation Full Norm](image_statistics_functions.html#image-cross-correlation-full-norm) * [CrossCorrFull\_Norm](image_statistics_functions.html#group__crosscorrfullnorm_1crosscorrfullnorm) * [Image Cross Correlation Same Norm](image_statistics_functions.html#image-cross-correlation-same-norm) * [CrossCorrSame\_Norm](image_statistics_functions.html#group__crosscorrsamenorm_1crosscorrsamenorm) * [Image Cross Correlation Valid Norm](image_statistics_functions.html#image-cross-correlation-valid-norm) * [CrossCorrValid\_Norm](image_statistics_functions.html#group__crosscorrvalidnorm_1crosscorrvalidnorm) * [Image Cross Correlation Valid](image_statistics_functions.html#image-cross-correlation-valid) * [CrossCorrValid](image_statistics_functions.html#group__crosscorrvalid_1crosscorrvalid) * [Image Cross Correlation Full Norm Level](image_statistics_functions.html#image-cross-correlation-full-norm-level) * [CrossCorrFull\_NormLevel](image_statistics_functions.html#group__crosscorrfullnormlevel_1crosscorrfullnormlevel) * [Image Cross Correlation Same Norm Level](image_statistics_functions.html#image-cross-correlation-same-norm-level) * [CrossCorrSame\_NormLevel](image_statistics_functions.html#group__crosscorrsamenormlevel_1crosscorrsamenormlevel) * [Image Cross Correlation Valid Norm Level](image_statistics_functions.html#image-cross-correlation-valid-norm-level) * [CrossCorrValid\_NormLevel](image_statistics_functions.html#group__crosscorrvalidnormlevel_1crosscorrvalidnormlevel) * [Image Cross Correlation Full Norm Level Advanced](image_statistics_functions.html#image-cross-correlation-full-norm-level-advanced) * [CrossCorrFull\_NormLevelAdvanced](image_statistics_functions.html#group__crosscorrfullnormlevelAdvanced_1crosscorrfullnormlevelAdvanced) * [Image Cross Correlation Same Norm Level Advanced](image_statistics_functions.html#image-cross-correlation-same-norm-level-advanced) * [CrossCorrSame\_NormLevelAdvanced](image_statistics_functions.html#group__crosscorrsamenormlevelAdvanced_1crosscorrsamenormlevelAdvanced) * [Image Cross Correlation Valid Norm Level Advanced](image_statistics_functions.html#image-cross-correlation-valid-norm-level-advanced) * [CrossCorrValid\_NormLevelAdvanced](image_statistics_functions.html#group__crosscorrvalidnormlevelAdvanced_1crosscorrvalidnormlevelAdvanced) * [Image Quality Index](image_statistics_functions.html#image-quality-index) * [Image Quality Index](image_statistics_functions.html#group__image__quality__index_1image_quality_index) * [Image Maximum Error](image_statistics_functions.html#image-maximum-error) * [MaximumError](image_statistics_functions.html#group__image__maximum__error_1image_maximum_error) * [Common parameters for nppiMaximumError functions include:](image_statistics_functions.html#group__image__maximum__error_1CommonMaximumErrorParameters) * [Image Average Error](image_statistics_functions.html#image-average-error) * [AverageError](image_statistics_functions.html#group__image__average__error_1image_average_error) * [Image Maximum Relative Error](image_statistics_functions.html#image-maximum-relative-error) * [MaximumRelativeError](image_statistics_functions.html#group__image__maximum__relative__error_1image_maximum_relative_error) * [Image Average Relative Error](image_statistics_functions.html#image-average-relative-error) * [AverageRelativeError](image_statistics_functions.html#group__image__average__relative__error_1image_average_relative_error) * [Image Quality Assessment IQA](image_statistics_functions.html#image-quality-assessment-iqa) * [IQA](image_statistics_functions.html#group__image__quality__assessment_1image_quality_assessment) * [Image Batch Quality Assessment](image_statistics_functions.html#image-batch-quality-assessment) * [IQABatch](image_statistics_functions.html#group__batch__image__quality__assessment_1batch_image_quality_assessment) * [Image Advanced Batch Quality Assessment](image_statistics_functions.html#image-advanced-batch-quality-assessment) * [IQABatchAdvanced](image_statistics_functions.html#group__batch__image__quality__assessment__advanced_1batch_image_quality_assessment_advanced) * [Image Threshold And Compare Operations](image_threshold_and_compare_operations.html) * [Image Threshold Operations](image_threshold_and_compare_operations.html#image-threshold-operations) * [Threshold Operations](image_threshold_and_compare_operations.html#group__image__threshold__operations_1image_threshold_operations) * [Common parameters for nppiThreshold non-inplace and inplace functions:](image_threshold_and_compare_operations.html#group__image__threshold__operations_1CommonThresholdParameters) * [Image Threshold Greater Than Operations](image_threshold_and_compare_operations.html#image-threshold-greater-than-operations) * [Threshold Greater Than Operations](image_threshold_and_compare_operations.html#group__image__threshold__greater__than__operations_1image_threshold_greater_than_operations) * [Image Threshold Less Than Operations](image_threshold_and_compare_operations.html#image-threshold-less-than-operations) * [Threshold Less Than Operations](image_threshold_and_compare_operations.html#group__image__threshold__less__than__operations_1image_threshold_less_than_operations) * [Image Threshold Value Operations](image_threshold_and_compare_operations.html#image-threshold-value-operations) * [Threshold Value Operations](image_threshold_and_compare_operations.html#group__image__threshold__value__operations_1image_threshold_value_operations) * [Image Threshold Greater Than Value Operations](image_threshold_and_compare_operations.html#image-threshold-greater-than-value-operations) * [Threshold Greater Than Value Operations](image_threshold_and_compare_operations.html#group__image__threshold__greater__than__value__operations_1image_threshold_greater_than_value_operations) * [Image Threshold Less Than Value Operations](image_threshold_and_compare_operations.html#image-threshold-less-than-value-operations) * [Threshold Less Than Value Operations](image_threshold_and_compare_operations.html#group__image__threshold__less__than__value__operations_1image_threshold_less_than_value_operations) * [Image Fused AbsDiff Threshold Greater Than Value Operations](image_threshold_and_compare_operations.html#image-fused-absdiff-threshold-greater-than-value-operations) * [Fused AbsDiff Threshold Greater Than Value Operations](image_threshold_and_compare_operations.html#group__image__fused__absdiff__threshold__greater__than__value__operations_1image_fused_absdiff_threshold_greater_than_value_operations) * [Image Threshold Less Than Value Greater Than Value Operations](image_threshold_and_compare_operations.html#image-threshold-less-than-value-greater-than-value-operations) * [Threshold Less Than Value Or Greater Than Value Operations](image_threshold_and_compare_operations.html#group__image__threshold__less__than__value__greater__than__value__operations_1image_threshold_less_than_value_greater_than_value_operations) * [Image Comparison Operations](image_threshold_and_compare_operations.html#image-comparison-operations) * [Comparison Operations](image_threshold_and_compare_operations.html#group__image__comparison__operations_1image_comparison_operations) * [Compare Images Operations](image_threshold_and_compare_operations.html#compare-images-operations) * [Compare Images Operations](image_threshold_and_compare_operations.html#group__compare__images__operations_1compare_images_operations) * [Compare Image With Constant Operations](image_threshold_and_compare_operations.html#compare-image-with-constant-operations) * [Compare Image With Constant Operations](image_threshold_and_compare_operations.html#group__compare__image__with__constant__operations_1compare_image_with_constant_operations) * [Compare Image Differences With Epsilon Operations](image_threshold_and_compare_operations.html#compare-image-differences-with-epsilon-operations) * [Compare Image Differences With Epsilon Operations](image_threshold_and_compare_operations.html#group__compare__image__differences__with__epsilon__operations_1compare_image_differences_with_epsilon_operations) * [Compare Image Difference To Constant With Epsilon Operations](image_threshold_and_compare_operations.html#compare-image-difference-to-constant-with-epsilon-operations) * [Compare Image Difference With Constant Within Epsilon Operations](image_threshold_and_compare_operations.html#group__compare__image__difference__to__constant__with__epsilon__operations_1compare_image_difference_to_constant_with_epsilon_operations) * [Image Memory Management Functions](image_memory_management.html) * [Signal Arithmetic And Logical Operations](signal_arithmetic_and_logical_operations.html) * [Signal Arithmetic Functions](signal_arithmetic_and_logical_operations.html#signal-arithmetic-functions) * [Arithmetic Operations](signal_arithmetic_and_logical_operations.html#group__signal__arithmetic_1signal_arithmetic) * [Signal AddC](signal_arithmetic_and_logical_operations.html#signal-addc) * [AddC](signal_arithmetic_and_logical_operations.html#group__signal__addc_1signal_addc) * [Signal AddProductC](signal_arithmetic_and_logical_operations.html#signal-addproductc) * [AddProductC](signal_arithmetic_and_logical_operations.html#group__signal__addproductc_1signal_addproductc) * [Signal MulC](signal_arithmetic_and_logical_operations.html#signal-mulc) * [MulC](signal_arithmetic_and_logical_operations.html#group__signal__mulc_1signal_mulc) * [Signal SubC](signal_arithmetic_and_logical_operations.html#signal-subc) * [SubC](signal_arithmetic_and_logical_operations.html#group__signal__subc_1signal_subc) * [Signal SubCRev](signal_arithmetic_and_logical_operations.html#signal-subcrev) * [SubCRev](signal_arithmetic_and_logical_operations.html#group__signal__subcrev_1signal_subcrev) * [Signal DivC](signal_arithmetic_and_logical_operations.html#signal-divc) * [DivC](signal_arithmetic_and_logical_operations.html#group__signal__divc_1signal_divc) * [Signal DivCRev](signal_arithmetic_and_logical_operations.html#signal-divcrev) * [DivCRev](signal_arithmetic_and_logical_operations.html#group__signal__divcrev_1signal_divcrev) * [Signal Add](signal_arithmetic_and_logical_operations.html#signal-add) * [Add](signal_arithmetic_and_logical_operations.html#group__signal__add_1signal_add) * [Signal AddProduct](signal_arithmetic_and_logical_operations.html#signal-addproduct) * [AddProduct](signal_arithmetic_and_logical_operations.html#group__signal__addproduct_1signal_addproduct) * [Signal Mul](signal_arithmetic_and_logical_operations.html#signal-mul) * [Mul](signal_arithmetic_and_logical_operations.html#group__signal__mul_1signal_mul) * [Signal Sub](signal_arithmetic_and_logical_operations.html#signal-sub) * [Sub](signal_arithmetic_and_logical_operations.html#group__signal__sub_1signal_sub) * [Signal Div](signal_arithmetic_and_logical_operations.html#signal-div) * [Div](signal_arithmetic_and_logical_operations.html#group__signal__div_1signal_div) * [Signal Div Round](signal_arithmetic_and_logical_operations.html#signal-div-round) * [Div\_Round](signal_arithmetic_and_logical_operations.html#group__signal__divround_1signal_divround) * [Signal Abs](signal_arithmetic_and_logical_operations.html#signal-abs) * [Abs](signal_arithmetic_and_logical_operations.html#group__signal__abs_1signal_abs) * [Signal Square](signal_arithmetic_and_logical_operations.html#signal-square) * [Sqr](signal_arithmetic_and_logical_operations.html#group__signal__square_1signal_square) * [Signal Square Root](signal_arithmetic_and_logical_operations.html#signal-square-root) * [Sqrt](signal_arithmetic_and_logical_operations.html#group__signal__sqrt_1signal_sqrt) * [Signal Cube Root](signal_arithmetic_and_logical_operations.html#signal-cube-root) * [Cubrt](signal_arithmetic_and_logical_operations.html#group__signal__cuberoot_1signal_cuberoot) * [Signal Exp](signal_arithmetic_and_logical_operations.html#signal-exp) * [Exp](signal_arithmetic_and_logical_operations.html#group__signal__exp_1signal_exp) * [Signal Ln](signal_arithmetic_and_logical_operations.html#signal-ln) * [Ln](signal_arithmetic_and_logical_operations.html#group__signal__ln_1signal_ln) * [Signal 10Log10](signal_arithmetic_and_logical_operations.html#signal-10log10) * [10Log10](signal_arithmetic_and_logical_operations.html#group__signal__10log10_1signal_10log10) * [Signal SumLn](signal_arithmetic_and_logical_operations.html#signal-sumln) * [SumLn](signal_arithmetic_and_logical_operations.html#group__signal__sumln_1signal_sumln) * [Signal ArcTan](signal_arithmetic_and_logical_operations.html#signal-arctan) * [Arctan](signal_arithmetic_and_logical_operations.html#group__signal__inversetan_1signal_inversetan) * [Signal Normalize](signal_arithmetic_and_logical_operations.html#signal-normalize) * [Normalize](signal_arithmetic_and_logical_operations.html#group__signal__normalize_1signal_normalize) * [Signal Cauchy, CouchyD, And CouchyDD2](signal_arithmetic_and_logical_operations.html#signal-cauchy-couchyd-and-couchydd2) * [Cauchy, CauchyD, and CauchyDD2](signal_arithmetic_and_logical_operations.html#group__signal__cauchy_1signal_cauchy) * [Logical And Shift Operations](signal_arithmetic_and_logical_operations.html#logical-and-shift-operations) * [Logical And Shift Operations](signal_arithmetic_and_logical_operations.html#group__signal__logical__and__shift__operations_1signal_logical_and_shift_operations) * [Signal AndC](signal_arithmetic_and_logical_operations.html#signal-andc) * [AndC](signal_arithmetic_and_logical_operations.html#group__signal__andc_1signal_andc) * [Signal And](signal_arithmetic_and_logical_operations.html#signal-and) * [And](signal_arithmetic_and_logical_operations.html#group__signal__and_1signal_and) * [Signal OrC](signal_arithmetic_and_logical_operations.html#signal-orc) * [OrC](signal_arithmetic_and_logical_operations.html#group__signal__orc_1signal_orc) * [Signal Or](signal_arithmetic_and_logical_operations.html#signal-or) * [Or](signal_arithmetic_and_logical_operations.html#group__signal__or_1signal_or) * [Signal XorC](signal_arithmetic_and_logical_operations.html#signal-xorc) * [XorC](signal_arithmetic_and_logical_operations.html#group__signal__xorc_1signal_xorc) * [Signal Xor](signal_arithmetic_and_logical_operations.html#signal-xor) * [Xor](signal_arithmetic_and_logical_operations.html#group__signal__xor_1signal_xor) * [Signal Not](signal_arithmetic_and_logical_operations.html#signal-not) * [Not](signal_arithmetic_and_logical_operations.html#group__signal__not_1signal_not) * [Signal LShiftC](signal_arithmetic_and_logical_operations.html#signal-lshiftc) * [LShiftC](signal_arithmetic_and_logical_operations.html#group__signal__lshiftc_1signal_lshiftc) * [Signal RShiftC](signal_arithmetic_and_logical_operations.html#signal-rshiftc) * [RShiftC](signal_arithmetic_and_logical_operations.html#group__signal__rshiftc_1signal_rshiftc) * [Signal Conversion Functions](signal_conversion_functions.html) * [Signal Convert](signal_conversion_functions.html#signal-convert) * [Convert](signal_conversion_functions.html#group__signal__convert_1signal_convert) * [Signal Threshold](signal_conversion_functions.html#signal-threshold) * [Threshold](signal_conversion_functions.html#group__signal__threshold_1signal_threshold) * [Signal Filtering Functions](signal_filtering_functions.html) * [Integral](signal_filtering_functions.html#group__signal__integral_1signal_integral) * [Signal Initialization Functions](signal_initialization.html) * [Signal Set](signal_initialization.html#signal-set) * [Set](signal_initialization.html#group__signal__set_1signal_set) * [Signal Zero](signal_initialization.html#signal-zero) * [Zero](signal_initialization.html#group__signal__zero_1signal_zero) * [Signal Copy](signal_initialization.html#signal-copy) * [Copy](signal_initialization.html#group__signal__copy_1signal_copy) * [Signal Statistical Functions](signal_statistical_functions.html) * [Signal Min Every Or Max Every](signal_statistical_functions.html#signal-min-every-or-max-every) * [MinEvery And MaxEvery Functions](signal_statistical_functions.html#group__signal__min__every__or__max__every_1signal_min_every_or_max_every) * [Signal Sum](signal_statistical_functions.html#signal-sum) * [Sum](signal_statistical_functions.html#group__signal__sum_1signal_sum) * [Signal Maximum](signal_statistical_functions.html#signal-maximum) * [Maximum](signal_statistical_functions.html#group__signal__max_1signal_max) * [Signal Minimum](signal_statistical_functions.html#signal-minimum) * [Minimum](signal_statistical_functions.html#group__signal__min_1signal_min) * [Signal Mean](signal_statistical_functions.html#signal-mean) * [Mean](signal_statistical_functions.html#group__signal__mean_1signal_mean) * [Signal StdDev](signal_statistical_functions.html#signal-stddev) * [Standard Deviation](signal_statistical_functions.html#group__signal__standard__deviation_1signal_standard_deviation) * [Signal Mean And StdDev](signal_statistical_functions.html#signal-mean-and-stddev) * [Mean And Standard Deviation](signal_statistical_functions.html#group__signal__mean__and__standard__deviation_1signal_mean_and_standard_deviation) * [Signal MinMax](signal_statistical_functions.html#signal-minmax) * [Minimum Maximum](signal_statistical_functions.html#group__signal__min__max_1signal_min_max) * [Signal Norms](signal_statistical_functions.html#signal-norms) * [Signal Norm Inf](signal_statistical_functions.html#signal-norm-inf) * [Infinity Norm](signal_statistical_functions.html#group__signal__infinity__norm_1signal_infinity_norm) * [Signal Norm L1](signal_statistical_functions.html#signal-norm-l1) * [L1 Norm](signal_statistical_functions.html#group__signal__L1__norm_1signal_L1_norm) * [Signal Norm L2](signal_statistical_functions.html#signal-norm-l2) * [L2 Norm](signal_statistical_functions.html#group__signal__L2__norm_1signal_L2_norm) * [Signal Norm Inf NormDiff](signal_statistical_functions.html#signal-norm-inf-normdiff) * [Infinity Norm Diff](signal_statistical_functions.html#group__signal__infinity__norm__diff_1signal_infinity_norm_diff) * [Signal Norm L1 NormDiff](signal_statistical_functions.html#signal-norm-l1-normdiff) * [L1 Norm Diff](signal_statistical_functions.html#group__signal__L1__norm__diff_1signal_L1_norm_diff) * [Signal Norm L2 NormDiff](signal_statistical_functions.html#signal-norm-l2-normdiff) * [L2 Norm Diff](signal_statistical_functions.html#group__signal__L2__norm__diff_1signal_L2_norm_diff) * [Signal Dot Product](signal_statistical_functions.html#signal-dot-product) * [Dot Product](signal_statistical_functions.html#group__signal__dot__product_1signal_dot_product) * [Signal Count In Range](signal_statistical_functions.html#signal-count-in-range) * [Count In Range](signal_statistical_functions.html#group__signal__count__in__range_1signal_count_in_range) * [Signal Count Zero Crossings](signal_statistical_functions.html#signal-count-zero-crossings) * [Count Zero Crossings](signal_statistical_functions.html#group__signal__count__zero__crossings_1signal_count_zero_crossings) * [Signal Maximum Error](signal_statistical_functions.html#signal-maximum-error) * [MaximumError](signal_statistical_functions.html#group__signal__maximum__error_1signal_maximum_error) * [Signal Average Error](signal_statistical_functions.html#signal-average-error) * [AverageError](signal_statistical_functions.html#group__signal__average__error_1signal_average_error) * [Signal Maximum Relative Error](signal_statistical_functions.html#signal-maximum-relative-error) * [MaximumRelativeError](signal_statistical_functions.html#group__signal__maximum__relative__error_1signal_maximum_relative_error) * [Signal Average Relative Error](signal_statistical_functions.html#signal-average-relative-error) * [AverageRelativeError](signal_statistical_functions.html#group__signal__average__relative__error_1signal_average_relative_error) * [Signal Memory Management Functions](signal_memory_management.html) * [Malloc](signal_memory_management.html#group__signal__malloc_1signal_malloc) * [Free](signal_memory_management.html#group__signal__free_1signal_free) --- # 1. Introduction — nvFatbin 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * nvFatbin The User guide to nvFatbin library. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ The Fatbin Creator APIs are a set of APIs which can be used at runtime to combine multiple CUDA objects into one CUDA fat binary (fatbin). The APIs accept inputs in multiple formats, either device cubins, PTX, or LTO-IR. The output is a fatbin that can be loaded by `cuModuleLoadData` of the CUDA Driver API. The functionality in this library is similar to the `fatbinary` offline tool in the CUDA toolkit, with the following advantages: * Support for runtime fatbin creation. * The clients get fine grain control over the input process. * Supports direct input from memory, rather than requiring inputs be written to files. 2\. Getting Started[](#getting-started "Permalink to this headline") ====================================================================== 2.1. System Requirements[](#system-requirements "Permalink to this headline") ------------------------------------------------------------------------------- The Fatbin Creator library requires no special system configuration. It does not require a GPU. 2.2. Installation[](#installation "Permalink to this headline") ----------------------------------------------------------------- The Fatbin Creator library is part of the CUDA Toolkit release and the components are organized as follows in the CUDA toolkit installation directory: * On Windows: * `include\nvFatbin.h` * `lib\x64\nvFatbin.dll` * `lib\x64\nvFatbin_static.lib` * `doc\pdf\nvFatbin_User_Guide.pdf` * On Linux: * `include/nvFatbin.h` * `lib64/libnvfatbin.so` * `lib64/libnvfatbin_static.a` * `doc/pdf/nvFatbin_User_Guide.pdf` 3\. User Interface[](#user-interface "Permalink to this headline") ==================================================================== This chapter presents the Fatbin Creator APIs. Basic usage of the API is explained in [Basic Usage](index.html#basic-usage) . * [Error codes](index.html#error-codes) * [Creation](index.html#fatbinary-creation) * [Supported Options](index.html#supported-options) 3.1. Error codes[](#error-codes "Permalink to this headline") --------------------------------------------------------------- Enumerations [nvFatbinResult](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) The enumerated type nvFatbinResult defines API call result codes. Functions const char \* [nvFatbinGetErrorString](#group__error_1gaf7ab701548c5dcfaf9808a66770265ee) (nvFatbinResult result) nvFatbinGetErrorString returns an error description string for each error code. ### 3.1.1. Enumerations[](#enumerations "Permalink to this headline") enum nvFatbinResult[](#_CPPv414nvFatbinResult "Permalink to this definition") The enumerated type nvFatbinResult defines API call result codes. nvFatbin APIs return nvFatbinResult codes to indicate the result. _Values:_ enumerator NVFATBIN\_SUCCESS[](#_CPPv4N14nvFatbinResult16NVFATBIN_SUCCESSE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_INTERNAL[](#_CPPv4N14nvFatbinResult23NVFATBIN_ERROR_INTERNALE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_ELF\_ARCH\_MISMATCH[](#_CPPv4N14nvFatbinResult32NVFATBIN_ERROR_ELF_ARCH_MISMATCHE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_ELF\_SIZE\_MISMATCH[](#_CPPv4N14nvFatbinResult32NVFATBIN_ERROR_ELF_SIZE_MISMATCHE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_MISSING\_PTX\_VERSION[](#_CPPv4N14nvFatbinResult34NVFATBIN_ERROR_MISSING_PTX_VERSIONE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_NULL\_POINTER[](#_CPPv4N14nvFatbinResult27NVFATBIN_ERROR_NULL_POINTERE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_COMPRESSION\_FAILED[](#_CPPv4N14nvFatbinResult33NVFATBIN_ERROR_COMPRESSION_FAILEDE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED[](#_CPPv4N14nvFatbinResult39NVFATBIN_ERROR_COMPRESSED_SIZE_EXCEEDEDE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION[](#_CPPv4N14nvFatbinResult34NVFATBIN_ERROR_UNRECOGNIZED_OPTIONE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_INVALID\_ARCH[](#_CPPv4N14nvFatbinResult27NVFATBIN_ERROR_INVALID_ARCHE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_INVALID\_NVVM[](#_CPPv4N14nvFatbinResult27NVFATBIN_ERROR_INVALID_NVVME "Permalink to this definition") enumerator NVFATBIN\_ERROR\_EMPTY\_INPUT[](#_CPPv4N14nvFatbinResult26NVFATBIN_ERROR_EMPTY_INPUTE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_MISSING\_PTX\_ARCH[](#_CPPv4N14nvFatbinResult31NVFATBIN_ERROR_MISSING_PTX_ARCHE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_PTX\_ARCH\_MISMATCH[](#_CPPv4N14nvFatbinResult32NVFATBIN_ERROR_PTX_ARCH_MISMATCHE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_MISSING\_FATBIN[](#_CPPv4N14nvFatbinResult29NVFATBIN_ERROR_MISSING_FATBINE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_INVALID\_INDEX[](#_CPPv4N14nvFatbinResult28NVFATBIN_ERROR_INVALID_INDEXE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_IDENTIFIER\_REUSE[](#_CPPv4N14nvFatbinResult31NVFATBIN_ERROR_IDENTIFIER_REUSEE "Permalink to this definition") enumerator NVFATBIN\_ERROR\_INTERNAL\_PTX\_OPTION[](#_CPPv4N14nvFatbinResult34NVFATBIN_ERROR_INTERNAL_PTX_OPTIONE "Permalink to this definition") ### 3.1.2. Functions[](#functions "Permalink to this headline") const char \*nvFatbinGetErrorString([nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") result)[](#_CPPv422nvFatbinGetErrorString14nvFatbinResult "Permalink to this definition") nvFatbinGetErrorString returns an error description string for each error code. Parameters **result** – **\[in\]** error code Returns * nullptr, if result is NVFATBIN\_SUCCESS * a string, if result is not NVFATBIN\_SUCCESS 3.2. Fatbinary Creation[](#fatbinary-creation "Permalink to this headline") ----------------------------------------------------------------------------- Functions nvFatbinResult [nvFatbinAddCubin](#group__creation_1ga0b924dee60a8598e37a46b428eaaf656) (nvFatbinHandle handle, const void \*code, size\_t size, const char \*arch, const char \*identifier) nvFatbinAddCubin adds a CUDA binary to the fatbinary. nvFatbinResult [nvFatbinAddIndex](#group__creation_1ga3daf8c9e606b40e70d056c8936767411) (nvFatbinHandle handle, const void \*code, size\_t size, const char \*identifier) nvFatbinAddIndex adds an index file to the fatbinary. nvFatbinResult [nvFatbinAddLTOIR](#group__creation_1ga0fa2e39b30033a449eb67da4d3c74861) (nvFatbinHandle handle, const void \*code, size\_t size, const char \*arch, const char \*identifier, const char \*optionsCmdLine) nvFatbinAddLTOIR adds LTOIR to the fatbinary. nvFatbinResult [nvFatbinAddPTX](#group__creation_1gad8a824776549ba0a0b42b6de2ff93c40) (nvFatbinHandle handle, const char \*code, size\_t size, const char \*arch, const char \*identifier, const char \*optionsCmdLine) nvFatbinAddPTX adds PTX to the fatbinary. nvFatbinResult [nvFatbinAddReloc](#group__creation_1gaa325f2088eba9fce18763ffefa027817) (nvFatbinHandle handle, const void \*code, size\_t size) nvFatbinAddReloc adds relocatable PTX entries from a host object to the fatbinary. nvFatbinResult [nvFatbinCreate](#group__creation_1ga1778c540f98a6a0ac222dc58773d3577) (nvFatbinHandle \*handle\_indirect, const char \*\*options, size\_t optionsCount) nvFatbinCreate creates a new handle nvFatbinResult [nvFatbinDestroy](#group__creation_1gacea09629df06283cc15c7024508575bc) (nvFatbinHandle \*handle\_indirect) nvFatbinDestroy destroys the handle. nvFatbinResult [nvFatbinGet](#group__creation_1gad4c5cfbac5f644ffd504f09278767755) (nvFatbinHandle handle, void \*buffer) nvFatbinGet returns the completed fatbinary. nvFatbinResult [nvFatbinSize](#group__creation_1gabfcc71733f0776c9553db0c596d06dc5) (nvFatbinHandle handle, size\_t \*size) nvFatbinSize returns the fatbinary's size. nvFatbinResult [nvFatbinVersion](#group__creation_1gafda592094a7ddd66feae93848ad511af) (unsigned int \*major, unsigned int \*minor) nvFatbinVersion returns the current version of nvFatbin Typedefs [nvFatbinHandle](#group__creation_1ga8ff95a5c8cd1bac24c656fd34d657bb0) nvFatbinHandle is the unit of fatbin creation, and an opaque handle for a program. ### 3.2.1. Functions[](#id1 "Permalink to this headline") [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinAddCubin([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, const void \*code, size\_t size, const char \*arch, const char \*identifier)[](#_CPPv416nvFatbinAddCubin14nvFatbinHandlePKv6size_tPKcPKc "Permalink to this definition") nvFatbinAddCubin adds a CUDA binary to the fatbinary. User is responsible for making sure all strings are well-formed. Parameters * **handle** – **\[in\]** nvFatbin handle. * **code** – **\[in\]** The cubin. * **size** – **\[in\]** The size of the cubin. * **arch** – **\[in\]** The architecture that this cubin is for. * **identifier** – **\[in\]** Name of the cubin, useful when extracting the fatbin with tools like cuobjdump. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INVALID\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_ELF\_ARCH\_MISMATCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_ELF\_SIZE\_MISMATCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSION\_FAILED,](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_EMPTY\_INPUT](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinAddIndex([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, const void \*code, size\_t size, const char \*identifier)[](#_CPPv416nvFatbinAddIndex14nvFatbinHandlePKv6size_tPKc "Permalink to this definition") nvFatbinAddIndex adds an index file to the fatbinary. User is responsible for making sure all strings are well-formed. Parameters * **handle** – **\[in\]** nvFatbin handle. * **code** – **\[in\]** The index. * **size** – **\[in\]** The size of the index. * **identifier** – **\[in\]** Name of the index, useful when extracting the fatbin with tools like cuobjdump. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INVALID\_INDEX](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSION\_FAILED,](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_EMPTY\_INPUT](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinAddLTOIR([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, const void \*code, size\_t size, const char \*arch, const char \*identifier, const char \*optionsCmdLine)[](#_CPPv416nvFatbinAddLTOIR14nvFatbinHandlePKv6size_tPKcPKcPKc "Permalink to this definition") nvFatbinAddLTOIR adds LTOIR to the fatbinary. User is responsible for making sure all strings are well-formed. Parameters * **handle** – **\[in\]** nvFatbin handle. * **code** – **\[in\]** The LTOIR code. * **size** – **\[in\]** The size of the LTOIR code. * **arch** – **\[in\]** The architecture that this LTOIR is for. * **identifier** – **\[in\]** Name of the LTOIR, useful when extracting the fatbin with tools like cuobjdump. * **optionsCmdLine** – **\[in\]** Options used during JIT compilation. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INVALID\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSION\_FAILED,](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_EMPTY\_INPUT](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinAddPTX([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, const char \*code, size\_t size, const char \*arch, const char \*identifier, const char \*optionsCmdLine)[](#_CPPv414nvFatbinAddPTX14nvFatbinHandlePKc6size_tPKcPKcPKc "Permalink to this definition") nvFatbinAddPTX adds PTX to the fatbinary. User is responsible for making sure all string are well-formed. The size should be inclusive of the terminating null character (‘\\0’). If the final character is not ‘\\0’, one will be added automatically, but in doing so, the code will be copied if it hasn’t already been copied. Parameters * **handle** – **\[in\]** nvFatbin handle. * **code** – **\[in\]** The PTX code. * **size** – **\[in\]** The size of the PTX code. * **arch** – **\[in\]** The architecture that this PTX is for. * **identifier** – **\[in\]** Name of the PTX, useful when extracting the fatbin with tools like cuobjdump. * **optionsCmdLine** – **\[in\]** Options used during JIT compilation. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INVALID\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_PTX\_ARCH\_MISMATCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSION\_FAILED,](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_EMPTY\_INPUT](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_MISSING\_PTX\_VERSION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_MISSING\_PTX\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinAddReloc([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, const void \*code, size\_t size)[](#_CPPv416nvFatbinAddReloc14nvFatbinHandlePKv6size_t "Permalink to this definition") nvFatbinAddReloc adds relocatable PTX entries from a host object to the fatbinary. Note that each relocatable ptx source must have a unique identifier (the identifiers are taken from the object’s entries). This is enforced as only one entry per sm of each unique identifier. Note also that handle options are ignored for this operation. Instead, the host object’s options are copied over from each of its entries. Parameters * **handle** – **\[in\]** nvFatbin handle. * **code** – **\[in\]** The host object image. * **size** – **\[in\]** The size of the host object image code. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INVALID\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_PTX\_ARCH\_MISMATCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSION\_FAILED,](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_COMPRESSED\_SIZE\_EXCEEDED](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_EMPTY\_INPUT](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_MISSING\_PTX\_VERSION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_MISSING\_PTX\_ARCH](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_IDENTIFIER\_REUSE](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinCreate([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") \*handle\_indirect, const char \*\*options, size\_t optionsCount)[](#_CPPv414nvFatbinCreateP14nvFatbinHandlePPKc6size_t "Permalink to this definition") nvFatbinCreate creates a new handle Parameters * **handle\_indirect** – **\[out\]** Address of nvFatbin handle * **options** – **\[in\]** An array of strings, each containing a single option. * **optionsCount** – **\[in\]** Number of options. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinDestroy([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") \*handle\_indirect)[](#_CPPv415nvFatbinDestroyP14nvFatbinHandle "Permalink to this definition") nvFatbinDestroy destroys the handle. Use of any other pointers to the handle after calling this will result in undefined behavior. The passed in handle will be set to nullptr. Parameters **handle\_indirect** – **\[in\]** Pointer to the handle. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinGet([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, void \*buffer)[](#_CPPv411nvFatbinGet14nvFatbinHandlePv "Permalink to this definition") nvFatbinGet returns the completed fatbinary. User is responsible for making sure the buffer is appropriately sized for the `fatbinary`. You must call nvFatbinSize before using this, otherwise, it will return an error. See also [nvFatbinSize](#group__creation_1gabfcc71733f0776c9553db0c596d06dc5) Parameters * **handle** – **\[in\]** nvFatbin handle. * **buffer** – **\[out\]** memory to store fatbinary. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinSize([nvFatbinHandle](#_CPPv414nvFatbinHandle "nvFatbinHandle") handle, size\_t \*size)[](#_CPPv412nvFatbinSize14nvFatbinHandleP6size_t "Permalink to this definition") nvFatbinSize returns the fatbinary’s size. Parameters * **handle** – **\[in\]** nvFatbin handle. * **size** – **\[out\]** The fatbinary’s size Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) [nvFatbinResult](#_CPPv414nvFatbinResult "nvFatbinResult") nvFatbinVersion(unsigned int \*major, unsigned int \*minor)[](#_CPPv415nvFatbinVersionPjPj "Permalink to this definition") nvFatbinVersion returns the current version of nvFatbin Parameters * **major** – **\[out\]** The major version. * **minor** – **\[out\]** The minor version. Returns * [NVFATBIN\_SUCCESS](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_NULL\_POINTER](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) * [NVFATBIN\_ERROR\_INTERNAL](#group__error_1ga3684fe3591bb9b6ce168d8e48883fb34) ### 3.2.2. Typedefs[](#typedefs "Permalink to this headline") typedef struct \_nvFatbinHandle \*nvFatbinHandle[](#_CPPv414nvFatbinHandle "Permalink to this definition") nvFatbinHandle is the unit of fatbin creation, and an opaque handle for a program. To create a fatbin, an instance of nvFatbinHandle must be created first with [nvFatbinCreate()](#group__creation_1ga1778c540f98a6a0ac222dc58773d3577) . 3.3. Supported Options[](#supported-options "Permalink to this headline") --------------------------------------------------------------------------- nvFatbin supports the options below. Option names are prefixed with a single dash (`-`). Options that take a value have an assignment operator (`=`) followed by the option value, with no spaces, e.g. `"-host=windows"`. The supported options are: * `-32` Make entries 32 bit. * `-64` Make entries 64 bit. * `-c` Has no effect. (Deprecated, will be removed in the next major release. Didn’t do anything from the start.) * `-compress=` Enable (true) / disable (false) compression (default: true). * `-compress-all` Compress everything in the fatbin, even if it’s small. * `-compress-mode=` Choose the compression behavior. Valid options are “default”, “size” (smaller fatbin size), “speed” (faster decompression speed, currently same as default), “balance” (somewhere between size and speed), and “none” (no compression, similar to -compress=false but takes precedence over -compress=true). * `-cuda` Specify CUDA (rather than OpenCL). * `-g` Generate debug information. * `-host=` Specify host operating system. Valid options are “linux”, “windows”, and “mac” (deprecated). * `-opencl` Specify OpenCL (rather than CUDA). 4\. Basic Usage[](#basic-usage "Permalink to this headline") ============================================================== This section of the document uses a simple example to explain how to use the Fatbin Creator APIs to link a program. For brevity and readability, error checks on the API return values are not shown. This example assumes we want to create a fatbin with a CUBIN for sm\_52, PTX for sm\_61, and LTOIR for sm\_70. We can create an instance of the fatbin creator and obtain an api handle to it as shown in [Figure 1](index.html#basic-usage-fatbin-creation) . Figure 1. Fatbin Creator creation and initialization of a program nvFatbinHandle handle; nvFatbinCreate(&handle, nullptr, 0); Assume that we already have three inputs stored in `std::vector` ‘s (CUBIN, PTX, and LTOIR), which could be from code created with `nvrtc` and stored into vectors. (They do not have to be in vectors, this merely illustrates that both the data itself and its size are needed.) We can add the inputs as shown in [Figure 2](index.html#basic-usage-fatbin-inputs) . Figure 2. Inputs to the fatbin creator nvFatbinAddCubin(handle, cubin.data(), cubin.size(), "52", nullptr); nvFatbinAddPTX(handle, ptx.data(), ptx.size(), "61", nullptr, nullptr); nvFatbinAddLTOIR(handle, ltoir.data(), ltoir.size(), "70", nullptr, nullptr); The fatbin can now be obtained. To obtain this we first allocate memory for it. And to allocate memory, we need to query the size of the fatbin which is done as shown in [Figure 3](index.html#basic-usage-query-fatbin-size) . Figure 3. Query size of the created fatbin nvFatbinSize(linker, &fatbinSize); The fatbin can now be queried as shown in [Figure 4](index.html#basic-usage-query-fatbin) . This fatbin can then be executed on the GPU by passing this to the CUDA Driver APIs. Figure 4. Query the created fatbin void\* fatbin \= malloc(fatbinSize); nvFatbinGet(handle, fatbin); When the fatbin creator is not needed anymore, it can be destroyed as shown in [Figure 5](index.html#basic-usage-destroy-creator) . Figure 5. Destroy the fatbin creator nvFatbinDestroy(&handle); 5\. Compatibility[](#compatibility "Permalink to this headline") ================================================================== The nvFatbin library is compatible across releases. The library major version itself must be >= the maximum major version of the inputs. For example, you can create a fatbin from a cubin created with 11.8 and one with 12.4 if your nvFatbin library is at least version 12.x. 6\. Example: Runtime fatbin creation[](#example-runtime-fatbin-creation "Permalink to this headline") ======================================================================================================= This section demonstrates runtime fatbin creation. There are two cubins. The cubins are generated online using NVRTC. These two cubins are then passed to `nvFatbin*` API functions, which put the cubins into a fatbin. Note that this example requires a compatible GPU with drivers and NVRTC to work, even though the library doesn’t require either. 6.1. Code (online.cpp)[](#code-online-cpp "Permalink to this headline") ------------------------------------------------------------------------- #include #include #include #include #include #define NUM\_THREADS 128 #define NUM\_BLOCKS 32 #define NVRTC\_SAFE\_CALL(x) \\ do { \\ nvrtcResult result = x; \\ if (result != NVRTC\_SUCCESS) { \\ std::cerr << "\\nerror: " #x " failed with error " \\ << nvrtcGetErrorString(result) << '\\n'; \\ exit(1); \\ } \\ } while(0) #define CUDA\_SAFE\_CALL(x) \\ do { \\ CUresult result = x; \\ if (result != CUDA\_SUCCESS) { \\ const char \*msg; \\ cuGetErrorName(result, &msg); \\ std::cerr << "\\nerror: " #x " failed with error " \\ << msg << '\\n'; \\ exit(1); \\ } \\ } while(0) #define NVFATBIN\_SAFE\_CALL(x) \\ do \\ { \\ nvFatbinResult result = x; \\ if (result != NVFATBIN\_SUCCESS) \\ { \\ std::cerr << "\\nerror: " #x " failed with error " \\ << nvFatbinGetErrorString(result) << '\\n';\\ exit(1); \\ } \\ } while (0) const char \*fatbin\_saxpy \= " \\n\\ \_\_device\_\_ float compute(float a, float x, float y) { \\n\\ return a \* x + y; \\n\\ } \\n\\ \\n\\ extern \\"C\\" \_\_global\_\_ \\n\\ void saxpy(float a, float \*x, float \*y, float \*out, size\_t n) \\n\\ { \\n\\ size\_t tid = blockIdx.x \* blockDim.x + threadIdx.x; \\n\\ if (tid < n) { \\n\\ out\[tid\] = compute(a, x\[tid\], y\[tid\]); \\n\\ } \\n\\ } \\n"; size\_t process(const void\* input, const char\* input\_name, void\*\* output, const char\* arch) { // Create an instance of nvrtcProgram with the code string. nvrtcProgram prog; NVRTC\_SAFE\_CALL( nvrtcCreateProgram(&prog, // prog (const char\*) input, // buffer input\_name, // name 0, // numHeaders NULL, // headers NULL)); // includeNames const char \*opts\[1\]; opts\[0\] \= arch; nvrtcResult compileResult \= nvrtcCompileProgram(prog, // prog 1, // numOptions opts); // options // Obtain compilation log from the program. size\_t logSize; NVRTC\_SAFE\_CALL(nvrtcGetProgramLogSize(prog, &logSize)); char \*log \= new char\[logSize\]; NVRTC\_SAFE\_CALL(nvrtcGetProgramLog(prog, log)); std::cout << log << '\\n'; delete\[\] log; if (compileResult != NVRTC\_SUCCESS) { exit(1); } // Obtain generated CUBIN from the program. size\_t CUBINSize; NVRTC\_SAFE\_CALL(nvrtcGetCUBINSize(prog, &CUBINSize)); char \*CUBIN \= new char\[CUBINSize\]; NVRTC\_SAFE\_CALL(nvrtcGetCUBIN(prog, CUBIN)); // Destroy the program. NVRTC\_SAFE\_CALL(nvrtcDestroyProgram(&prog)); \*output \= (void\*) CUBIN; return CUBINSize; } int main(int argc, char \*argv\[\]) { void\* known \= NULL; size\_t known\_size \= process(fatbin\_saxpy, "fatbin\_saxpy.cu", &known, "-arch=sm\_52"); CUdevice cuDevice; CUcontext context; CUmodule module; CUfunction kernel; CUDA\_SAFE\_CALL(cuInit(0)); CUDA\_SAFE\_CALL(cuDeviceGet(&cuDevice, 0)); CUDA\_SAFE\_CALL(cuCtxCreate(&context, 0, cuDevice)); // Dynamically determine the arch to make one of the entries of the fatbin with int major \= 0; int minor \= 0; CUDA\_SAFE\_CALL(cuDeviceGetAttribute(&major, CU\_DEVICE\_ATTRIBUTE\_COMPUTE\_CAPABILITY\_MAJOR, cuDevice)); CUDA\_SAFE\_CALL(cuDeviceGetAttribute(&minor, CU\_DEVICE\_ATTRIBUTE\_COMPUTE\_CAPABILITY\_MINOR, cuDevice)); int arch \= major\*10 + minor; char smbuf\[16\]; sprintf(smbuf, "-arch=sm\_%d", arch); void\* dynamic \= NULL; size\_t dynamic\_size \= process(fatbin\_saxpy, "fatbin\_saxpy.cu", &dynamic, smbuf); sprintf(smbuf, "%d", arch); // Load the dynamic CUBIN and the statically known arch CUBIN // and put them in a fatbin together. nvFatbinHandle handle; const char\* fatbin\_options\[\] \= {"-cuda"}; NVFATBIN\_SAFE\_CALL(nvFatbinCreate(&handle, fatbin\_options, 1)); NVFATBIN\_SAFE\_CALL(nvFatbinAddCubin(handle, (void \*)dynamic, dynamic\_size, smbuf, "dynamic")); NVFATBIN\_SAFE\_CALL(nvFatbinAddCubin(handle, (void \*)known, known\_size, "52", "known")); size\_t fatbinSize; NVFATBIN\_SAFE\_CALL(nvFatbinSize(handle, &fatbinSize)); void \*fatbin \= malloc(fatbinSize); NVFATBIN\_SAFE\_CALL(nvFatbinGet(handle, fatbin)); NVFATBIN\_SAFE\_CALL(nvFatbinDestroy(&handle)); CUDA\_SAFE\_CALL(cuModuleLoadData(&module, fatbin)); CUDA\_SAFE\_CALL(cuModuleGetFunction(&kernel, module, "saxpy")); // Generate input for execution, and create output buffers. #define NUM\_THREADS 128 #define NUM\_BLOCKS 32 size\_t n \= NUM\_THREADS \* NUM\_BLOCKS; size\_t bufferSize \= n \* sizeof(float); float a \= 5.1f; float \*hX \= new float\[n\], \*hY \= new float\[n\], \*hOut \= new float\[n\]; for (size\_t i \= 0; i < n; ++i) { hX\[i\] \= static\_cast(i); hY\[i\] \= static\_cast(i \* 2); } CUdeviceptr dX, dY, dOut; CUDA\_SAFE\_CALL(cuMemAlloc(&dX, bufferSize)); CUDA\_SAFE\_CALL(cuMemAlloc(&dY, bufferSize)); CUDA\_SAFE\_CALL(cuMemAlloc(&dOut, bufferSize)); CUDA\_SAFE\_CALL(cuMemcpyHtoD(dX, hX, bufferSize)); CUDA\_SAFE\_CALL(cuMemcpyHtoD(dY, hY, bufferSize)); // Execute SAXPY. void \*args\[\] \= { &a, &dX, &dY, &dOut, &n }; CUDA\_SAFE\_CALL( cuLaunchKernel(kernel, NUM\_BLOCKS, 1, 1, // grid dim NUM\_THREADS, 1, 1, // block dim 0, NULL, // shared mem and stream args, 0)); // arguments CUDA\_SAFE\_CALL(cuCtxSynchronize()); // Retrieve and print output. CUDA\_SAFE\_CALL(cuMemcpyDtoH(hOut, dOut, bufferSize)); for (size\_t i \= 0; i < n; ++i) { std::cout << a << " \* " << hX\[i\] << " + " << hY\[i\] << " = " << hOut\[i\] << '\\n'; } // Release resources. CUDA\_SAFE\_CALL(cuMemFree(dX)); CUDA\_SAFE\_CALL(cuMemFree(dY)); CUDA\_SAFE\_CALL(cuMemFree(dOut)); CUDA\_SAFE\_CALL(cuModuleUnload(module)); CUDA\_SAFE\_CALL(cuCtxDestroy(context)); delete\[\] hX; delete\[\] hY; delete\[\] hOut; // Release resources. free(fatbin); delete\[\] ((char\*)known); delete\[\] ((char\*)dynamic); return 0; } 6.2. Build Instructions[](#build-instructions "Permalink to this headline") ----------------------------------------------------------------------------- Assuming the environment variable `CUDA_PATH` points to CUDA Toolkit installation directory, build this example as: * With nvFatbin shared library (note that if the test didn’t use nvrtc or run the code then it would not need to link with nvrtc or the CUDA driver API): * Windows: cl.exe online.cpp /Feonline ^ /I "%CUDA\_PATH%\\include" ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvfatbin.lib ^ "%CUDA\_PATH%"\\lib\\x64\\cuda.lib * Linux: g++ online.cpp -o online \\ -I $CUDA\_PATH/include \\ -L $CUDA\_PATH/lib64 \\ -lnvrtc -lnvfatbin -lcuda \\ -Wl,-rpath,$CUDA\_PATH/lib64 * With nvFatbin static library: * Windows: cl.exe online.cpp /Feonline ^ /I "%CUDA\_PATH%"\\include ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc-builtins\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvfatbin\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvptxcompiler\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\cuda.lib user32.lib Ws2\_32.lib * Linux: g++ online.cpp -o online \\ -I $CUDA\_PATH/include \\ -L $CUDA\_PATH/lib64 \\ -lnvrtc\_static -lnvrtc-builtins\_static -lnvfatbin\_static -lnvptxcompiler\_static -lcuda \\ -lpthread 6.3. Notices[](#notices "Permalink to this headline") ------------------------------------------------------- ### 6.3.1. Notice[](#notice "Permalink to this headline") This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. ### 6.3.2. OpenCL[](#opencl "Permalink to this headline") OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. ### 6.3.3. Trademarks[](#trademarks "Permalink to this headline") NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. © 2023 NVIDIA Corporation & affiliates. All rights reserved. --- # 1. Introduction — nvJitLink 12.8 documentation * [](../index.html) » * 1\. Introduction * v12.8 | [Archive](https://developer.nvidia.com/cuda-toolkit-archive)   * * * nvJitLink The User guide to nvJitLink library. 1\. Introduction[](#introduction "Permalink to this headline") ================================================================ The JIT Link APIs are a set of APIs which can be used at runtime to link together GPU devide code. The APIs accept inputs in multiple formats, either host objects, host libraries, fatbins (including with relocatable ptx), device cubins, PTX, index files or LTO-IR. The output is a linked cubin that can be loaded by `cuModuleLoadData` and `cuModuleLoadDataEx` of the CUDA Driver API. Link Time Optimization can also be performed when given LTO-IR or higher level formats that include LTO-IR. If an input does not contain GPU assembly code, it is first compiled and then linked. The functionality in this library is similar to the `cuLink*` APIs in the CUDA Driver, with the following advantages: * The `cuLink*` APIs have been deprecated for use with LTO-IR * Support for Link Time Optimization * Allow users to use runtime linking with the latest Toolkit version that is supported as part of CUDA Toolkit release. This support may not be available in the CUDA Driver APIs if the application is running with an older driver installed in the system. Refer to [CUDA Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) for more details. * The clients get fine grain control and can specify low-level compiler options during linking. 2\. Getting Started[](#getting-started "Permalink to this headline") ====================================================================== 2.1. System Requirements[](#system-requirements "Permalink to this headline") ------------------------------------------------------------------------------- The JIT Link library requires the following system configuration: * POSIX threads support for non-Windows platform. * GPU: Any GPU with CUDA Compute Capability 3.5 or higher. * CUDA Toolkit and Driver. 2.2. Installation[](#installation "Permalink to this headline") ----------------------------------------------------------------- The JIT Link library is part of the CUDA Toolkit release and the components are organized as follows in the CUDA toolkit installation directory: * On Windows: * `include\nvJitLink.h` * `lib\x64\nvJitLink.dll` * `lib\x64\nvJitLink_static.lib` * `doc\pdf\nvJitLink_User_Guide.pdf` * On Linux: * `include/nvJitLink.h` * `lib64/libnvJitLink.so` * `lib64/libnvJitLink_static.a` * `doc/pdf/nvJitLink_User_Guide.pdf` 3\. User Interface[](#user-interface "Permalink to this headline") ==================================================================== This chapter presents the JIT Link APIs. Basic usage of the API is explained in [Basic Usage](index.html#basic-usage) . * [Error codes](index.html#error-codes) * [Linking](index.html#linking) * [Supported Link Options](index.html#supported-link-options) 3.1. Error codes[](#error-codes "Permalink to this headline") --------------------------------------------------------------- Enumerations [nvJitLinkResult](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) The enumerated type nvJitLinkResult defines API call result codes. ### 3.1.1. Enumerations[](#enumerations "Permalink to this headline") enum nvJitLinkResult[](#_CPPv415nvJitLinkResult "Permalink to this definition") The enumerated type nvJitLinkResult defines API call result codes. nvJitLink APIs return nvJitLinkResult codes to indicate the result. _Values:_ enumerator NVJITLINK\_SUCCESS[](#_CPPv4N15nvJitLinkResult17NVJITLINK_SUCCESSE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_UNRECOGNIZED\_OPTION[](#_CPPv4N15nvJitLinkResult35NVJITLINK_ERROR_UNRECOGNIZED_OPTIONE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_MISSING\_ARCH[](#_CPPv4N15nvJitLinkResult28NVJITLINK_ERROR_MISSING_ARCHE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_INVALID\_INPUT[](#_CPPv4N15nvJitLinkResult29NVJITLINK_ERROR_INVALID_INPUTE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_PTX\_COMPILE[](#_CPPv4N15nvJitLinkResult27NVJITLINK_ERROR_PTX_COMPILEE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_NVVM\_COMPILE[](#_CPPv4N15nvJitLinkResult28NVJITLINK_ERROR_NVVM_COMPILEE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_INTERNAL[](#_CPPv4N15nvJitLinkResult24NVJITLINK_ERROR_INTERNALE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_THREADPOOL[](#_CPPv4N15nvJitLinkResult26NVJITLINK_ERROR_THREADPOOLE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_UNRECOGNIZED\_INPUT[](#_CPPv4N15nvJitLinkResult34NVJITLINK_ERROR_UNRECOGNIZED_INPUTE "Permalink to this definition") enumerator NVJITLINK\_ERROR\_FINALIZE[](#_CPPv4N15nvJitLinkResult24NVJITLINK_ERROR_FINALIZEE "Permalink to this definition") 3.2. Linking[](#linking "Permalink to this headline") ------------------------------------------------------- Enumerations [nvJitLinkInputType](#group__linking_1gaa66f143b76ecc08c215ba1a45dc0aeaf) The enumerated type nvJitLinkInputType defines the kind of inputs that can be passed to nvJitLinkAdd\* APIs. Functions nvJitLinkResult [nvJitLinkAddData](#group__linking_1ga9d70b5985c13679a9c64bdf621912f3b) (nvJitLinkHandle handle, nvJitLinkInputType inputType, const void \*data, size\_t size, const char \*name) nvJitLinkAddData adds data image to the link. nvJitLinkResult [nvJitLinkAddFile](#group__linking_1gaeade70d4845df4a329672bc796730f95) (nvJitLinkHandle handle, nvJitLinkInputType inputType, const char \*fileName) nvJitLinkAddFile reads data from file and links it in. nvJitLinkResult [nvJitLinkComplete](#group__linking_1ga9957fe993c155c0ef9426bf2466c6434) (nvJitLinkHandle handle) nvJitLinkComplete does the actual link. nvJitLinkResult [nvJitLinkCreate](#group__linking_1ga48ba234f5d1ae2dbabb037a319cd061b) (nvJitLinkHandle \*handle, uint32\_t numOptions, const char \*\*options) nvJitLinkCreate creates an instance of nvJitLinkHandle with the given input options, and sets the output parameter `handle` . nvJitLinkResult [nvJitLinkDestroy](#group__linking_1gafe20a6047c57c48cf18131f772336061) (nvJitLinkHandle \*handle) nvJitLinkDestroy frees the memory associated with the given handle and sets it to NULL. nvJitLinkResult [nvJitLinkGetErrorLog](#group__linking_1ga863a571d78822b20fb73049d249064f4) (nvJitLinkHandle handle, char \*log) nvJitLinkGetErrorLog puts any error messages in the log. nvJitLinkResult [nvJitLinkGetErrorLogSize](#group__linking_1ga3206e639d004f24e5ec9e3ceed35be75) (nvJitLinkHandle handle, size\_t \*size) nvJitLinkGetErrorLogSize gets the size of the error log. nvJitLinkResult [nvJitLinkGetInfoLog](#group__linking_1gad34de24b82e5e76d0f616061cef1f45f) (nvJitLinkHandle handle, char \*log) nvJitLinkGetInfoLog puts any info messages in the log. nvJitLinkResult [nvJitLinkGetInfoLogSize](#group__linking_1ga4515b56fcd42ec2131e7ab8f7a506aa6) (nvJitLinkHandle handle, size\_t \*size) nvJitLinkGetInfoLogSize gets the size of the info log. nvJitLinkResult [nvJitLinkGetLinkedCubin](#group__linking_1gaf47e10d4e9e32a1ae864584348a6c58c) (nvJitLinkHandle handle, void \*cubin) nvJitLinkGetLinkedCubin gets the linked cubin. nvJitLinkResult [nvJitLinkGetLinkedCubinSize](#group__linking_1ga4b802517c775509b6488599ee1276658) (nvJitLinkHandle handle, size\_t \*size) nvJitLinkGetLinkedCubinSize gets the size of the linked cubin. nvJitLinkResult [nvJitLinkGetLinkedPtx](#group__linking_1ga60547cee8fbc22a9c43edfa625b01642) (nvJitLinkHandle handle, char \*ptx) nvJitLinkGetLinkedPtx gets the linked ptx. nvJitLinkResult [nvJitLinkGetLinkedPtxSize](#group__linking_1ga9bdabb070beca8d99a72c0df459b68d8) (nvJitLinkHandle handle, size\_t \*size) nvJitLinkGetLinkedPtxSize gets the size of the linked ptx. nvJitLinkResult [nvJitLinkVersion](#group__linking_1ga04b722539d968953cb5cc70488e3f47a) (unsigned int \*major, unsigned int \*minor) nvJitLinkVersion returns the current version of nvJitLink. Typedefs [nvJitLinkHandle](#group__linking_1ga3e05e7d4a1826909af1701b1e01d8526) nvJitLinkHandle is the unit of linking, and an opaque handle for a program. ### 3.2.1. Enumerations[](#id1 "Permalink to this headline") enum nvJitLinkInputType[](#_CPPv418nvJitLinkInputType "Permalink to this definition") The enumerated type nvJitLinkInputType defines the kind of inputs that can be passed to nvJitLinkAdd\* APIs. _Values:_ enumerator NVJITLINK\_INPUT\_NONE[](#_CPPv4N18nvJitLinkInputType20NVJITLINK_INPUT_NONEE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_CUBIN[](#_CPPv4N18nvJitLinkInputType21NVJITLINK_INPUT_CUBINE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_PTX[](#_CPPv4N18nvJitLinkInputType19NVJITLINK_INPUT_PTXE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_LTOIR[](#_CPPv4N18nvJitLinkInputType21NVJITLINK_INPUT_LTOIRE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_FATBIN[](#_CPPv4N18nvJitLinkInputType22NVJITLINK_INPUT_FATBINE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_OBJECT[](#_CPPv4N18nvJitLinkInputType22NVJITLINK_INPUT_OBJECTE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_LIBRARY[](#_CPPv4N18nvJitLinkInputType23NVJITLINK_INPUT_LIBRARYE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_INDEX[](#_CPPv4N18nvJitLinkInputType21NVJITLINK_INPUT_INDEXE "Permalink to this definition") enumerator NVJITLINK\_INPUT\_ANY[](#_CPPv4N18nvJitLinkInputType19NVJITLINK_INPUT_ANYE "Permalink to this definition") ### 3.2.2. Functions[](#functions "Permalink to this headline") static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkAddData([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, [nvJitLinkInputType](#_CPPv418nvJitLinkInputType "nvJitLinkInputType") inputType, const void \*data, size\_t size, const char \*name)[](#_CPPv416nvJitLinkAddData15nvJitLinkHandle18nvJitLinkInputTypePKv6size_tPKc "Permalink to this definition") nvJitLinkAddData adds data image to the link. Parameters * **handle** – **\[in\]** nvJitLink handle. * **inputType** – **\[in\]** kind of input. * **data** – **\[in\]** pointer to data image in memory. * **size** – **\[in\]** size of the data. * **name** – **\[in\]** name of input object. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkAddFile([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, [nvJitLinkInputType](#_CPPv418nvJitLinkInputType "nvJitLinkInputType") inputType, const char \*fileName)[](#_CPPv416nvJitLinkAddFile15nvJitLinkHandle18nvJitLinkInputTypePKc "Permalink to this definition") nvJitLinkAddFile reads data from file and links it in. Parameters * **handle** – **\[in\]** nvJitLink handle. * **inputType** – **\[in\]** kind of input. * **fileName** – **\[in\]** name of file. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkComplete([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle)[](#_CPPv417nvJitLinkComplete15nvJitLinkHandle "Permalink to this definition") nvJitLinkComplete does the actual link. Parameters **handle** – **\[in\]** nvJitLink handle. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkCreate([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") \*handle, uint32\_t numOptions, const char \*\*options)[](#_CPPv415nvJitLinkCreateP15nvJitLinkHandle8uint32_tPPKc "Permalink to this definition") nvJitLinkCreate creates an instance of nvJitLinkHandle with the given input options, and sets the output parameter `handle`. It supports options listed in [Supported Link Options](#group__options) . See also nvJitLinkDestroy Parameters * **handle** – **\[out\]** Address of nvJitLink handle. * **numOptions** – **\[in\]** Number of options passed. * **options** – **\[in\]** Array of size `numOptions` of option strings. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_UNRECOGNIZED\_OPTION](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_MISSING\_ARCH](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkDestroy([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") \*handle)[](#_CPPv416nvJitLinkDestroyP15nvJitLinkHandle "Permalink to this definition") nvJitLinkDestroy frees the memory associated with the given handle and sets it to NULL. See also nvJitLinkCreate Parameters **handle** – **\[in\]** Address of nvJitLink handle. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetErrorLog([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, char \*log)[](#_CPPv420nvJitLinkGetErrorLog15nvJitLinkHandlePc "Permalink to this definition") nvJitLinkGetErrorLog puts any error messages in the log. User is responsible for allocating enough space to hold the `log`. See also nvJitLinkGetErrorLogSize Parameters * **handle** – **\[in\]** nvJitLink handle. * **log** – **\[out\]** The error log. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetErrorLogSize([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, size\_t \*size)[](#_CPPv424nvJitLinkGetErrorLogSize15nvJitLinkHandleP6size_t "Permalink to this definition") nvJitLinkGetErrorLogSize gets the size of the error log. See also nvJitLinkGetErrorLog Parameters * **handle** – **\[in\]** nvJitLink handle. * **size** – **\[out\]** Size of the error log. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetInfoLog([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, char \*log)[](#_CPPv419nvJitLinkGetInfoLog15nvJitLinkHandlePc "Permalink to this definition") nvJitLinkGetInfoLog puts any info messages in the log. User is responsible for allocating enough space to hold the `log`. See also nvJitLinkGetInfoLogSize Parameters * **handle** – **\[in\]** nvJitLink handle. * **log** – **\[out\]** The info log. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetInfoLogSize([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, size\_t \*size)[](#_CPPv423nvJitLinkGetInfoLogSize15nvJitLinkHandleP6size_t "Permalink to this definition") nvJitLinkGetInfoLogSize gets the size of the info log. See also nvJitLinkGetInfoLog Parameters * **handle** – **\[in\]** nvJitLink handle. * **size** – **\[out\]** Size of the info log. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetLinkedCubin([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, void \*cubin)[](#_CPPv423nvJitLinkGetLinkedCubin15nvJitLinkHandlePv "Permalink to this definition") nvJitLinkGetLinkedCubin gets the linked cubin. User is responsible for allocating enough space to hold the `cubin`. See also nvJitLinkGetLinkedCubinSize Parameters * **handle** – **\[in\]** nvJitLink handle. * **cubin** – **\[out\]** The linked cubin. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetLinkedCubinSize([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, size\_t \*size)[](#_CPPv427nvJitLinkGetLinkedCubinSize15nvJitLinkHandleP6size_t "Permalink to this definition") nvJitLinkGetLinkedCubinSize gets the size of the linked cubin. See also nvJitLinkGetLinkedCubin Parameters * **handle** – **\[in\]** nvJitLink handle. * **size** – **\[out\]** Size of the linked cubin. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetLinkedPtx([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, char \*ptx)[](#_CPPv421nvJitLinkGetLinkedPtx15nvJitLinkHandlePc "Permalink to this definition") nvJitLinkGetLinkedPtx gets the linked ptx. Linked PTX is only available when using the `-lto` option. User is responsible for allocating enough space to hold the `ptx`. See also nvJitLinkGetLinkedPtxSize Parameters * **handle** – **\[in\]** nvJitLink handle. * **ptx** – **\[out\]** The linked PTX. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) static inline [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkGetLinkedPtxSize([nvJitLinkHandle](#_CPPv415nvJitLinkHandle "nvJitLinkHandle") handle, size\_t \*size)[](#_CPPv425nvJitLinkGetLinkedPtxSize15nvJitLinkHandleP6size_t "Permalink to this definition") nvJitLinkGetLinkedPtxSize gets the size of the linked ptx. Linked PTX is only available when using the `-lto` option. See also nvJitLinkGetLinkedPtx Parameters * **handle** – **\[in\]** nvJitLink handle. * **size** – **\[out\]** Size of the linked PTX. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) [nvJitLinkResult](#_CPPv415nvJitLinkResult "nvJitLinkResult") nvJitLinkVersion(unsigned int \*major, unsigned int \*minor)[](#_CPPv416nvJitLinkVersionPjPj "Permalink to this definition") nvJitLinkVersion returns the current version of nvJitLink. Parameters * **major** – **\[out\]** The major version. * **minor** – **\[out\]** The minor version. Returns * [NVJITLINK\_SUCCESS](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INVALID\_INPUT](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) * [NVJITLINK\_ERROR\_INTERNAL](#group__error_1ga02b99d1f1327b90cb1ff1747a38eb1a3) ### 3.2.3. Typedefs[](#typedefs "Permalink to this headline") typedef struct nvJitLink \*nvJitLinkHandle[](#_CPPv415nvJitLinkHandle "Permalink to this definition") nvJitLinkHandle is the unit of linking, and an opaque handle for a program. To link inputs, an instance of nvJitLinkHandle must be created first with nvJitLinkCreate(). 3.3. Supported Link Options[](#supported-link-options "Permalink to this headline") ------------------------------------------------------------------------------------- nvJitLink supports the link options below. Option names are prefixed with a single dash (`-`). Options that take a value have an assignment operator (`=`) followed by the option value, with no spaces, e.g. `"-arch=sm_90"`. The supported options are: * `-arch=sm_` Pass SM architecture value. See nvcc for valid values of . Can use compute\_ value instead if only generating PTX. This is a required option. * `-maxrregcount=` Maximum register count. * `-time` Print timing information to InfoLog. * `-verbose` Print verbose messages to InfoLog. * `-lto` Do link time optimization. * `-ptx` Emit ptx after linking instead of cubin; only supported with `-lto` * `-O` Optimization level. Only 0 and 3 are accepted. * `-g` Generate debug information. * `-lineinfo` Generate line information. * `-ftz=` Flush to zero. * `-prec-div=` Precise divide. * `-prec-sqrt=` Precise square root. * `-fma=` Fast multiply add. * `-kernels-used=` Pass list of kernels that are used; any not in the list can be removed. This option can be specified multiple times. * `-variables-used=` Pass list of variables that are used; any not in the list can be removed. This option can be specified multiple times. * `-optimize-unused-variables` Normally device code optimization is limited by not knowing what the host code references. With this option it can assume that if a variable is not referenced in device code then it can be removed. * `-Xptxas=` Pass to ptxas. This option can be called multiple times. * `-split-compile=` Split compilation maximum thread count. Use 0 to use all available processors. Value of 1 disables split compilation (default). * `-split-compile-extended=` A more aggressive form of split compilation available in LTO mode only. Accepts a maximum thread count value. Use 0 to use all available processors. Value of 1 disables extended split compilation (default). Note: This option can potentially impact performance of the compiled binary. * `-jump-table-density=` When doing LTO, specify the case density percentage in switch statements, and use it as a minimal threshold to determine whether jump table(brx.idx instruction) will be used to implement a switch statement. Default value is 101. The percentage ranges from 0 to 101 inclusively. * `-no-cache` Don’t cache the intermediate steps of nvJitLink. * `-device-stack-protector` Enable stack canaries in device code. Stack canaries make it more difficult to exploit certain types of memory safety bugs involving stack-local variables. The compiler uses heuristics to assess the risk of such a bug in each function. Only those functions which are deemed high-risk make use of a stack canary. 4\. Basic Usage[](#basic-usage "Permalink to this headline") ============================================================== This section of the document uses a simple example to explain how to use the JIT Link APIs to link a program. For brevity and readability, error checks on the API return values are not shown. This example assumes we want to link for sm\_80, but whatever arch is installed on the system should be used. We can create the linker and obtain a handle to it as shown in [Figure 1](index.html#basic-usage-linker-creation) . Figure 1. Linker creation and initialization of a program nvJitLink\_t linker; const char\* link\_options\[\] \= { "-arch=sm\_80" }; nvJitLinkCreate(&linker, 1, link\_options); Assume that we already have two relocatable input files (a.o and b.o), which could be created with the `nvcc -dc` command. We can add the input files as show in [Figure 2](index.html#basic-usage-link-inputs) . Figure 2. Inputs to linker nvJitLinkAddFile(linker, NVJITLINK\_INPUT\_OBJECT, "a.o"); nvJitLinkAddFile(linker, NVJITLINK\_INPUT\_OBJECT, "b.o"); Now the actual link can be done as shown in [Figure 3](index.html#basic-usage-linking-of-program) . > Figure 3. Linking of the PTX program nvJitLinkComplete(linker); The linked GPU assembly code can now be obtained. To obtain this we first allocate memory for it. And to allocate memory, we need to query the size of the image of the linked GPU assembly code which is done as shown in [Figure 4](index.html#basic-usage-query-image-size) . Figure 4. Query size of the linked assembly image nvJitLinkGetLinkedCubinSize(linker, &cubinSize); The image of the linked GPU assembly code can now be queried as shown in [Figure 5](index.html#basic-usage-query-image) . This image can then be executed on the GPU by passing this image to the CUDA Driver APIs. Figure 5. Query the linked assembly image elf \= (char\*) malloc(cubinSize); nvJitLinkGetLinkedCubin(linker, (void\*)elf); When the linker is not needed anymore, it can be destroyed as shown in [Figure 6](index.html#basic-usage-destroy-linker) . Figure 6. Destroy the linker nvJitLinkDestroy(&linker); 5\. Compatibility[](#compatibility "Permalink to this headline") ================================================================== The nvJitLink library is compatible across minor versions in a release, but may not be compatible across major versions. The library version itself must be >= the maximum version of the inputs, and the shared library version must be >= the version that was linked with. For example, you can link an object created with 12.0 and one with 12.1 if your nvJitLink library is version 12.x where x >= 1. If it was linked with 12.1, then you can replace and use the nvJitLink shared library with any version 12.x where x >= 1. On the flip side, you cannot use 12.0 to link 12.1 objects, nor use 12.0 nvJitLink library to run 12.1 code. Linking across major versions (like 11.x with 12.x) works for ELF and PTX inputs, but does not work with LTOIR inputs. If using LTO, then compatibility is only guaranteed within a major release. Linking extended ISA sources (like sm\_90a) against any other sm version will always fail. Linking with PTX sources from different architectures (such as compute\_89 and compute\_90) will work as long as the final link is the newest of all of the architectures being linked. That is, for any compute\_X and compute\_Y, the link is valid if the target is sm\_N where N >= max(X,Y). Linking with LTO sources from different architectures (such as lto\_89 and lto\_90) will work as long as the final link is the newest of all of the architectures being linked. That is, for any lto\_X and lto\_Y, the link is valid if the target is sm\_N where N >= max(X,Y). Linking with non-PTX, non-LTO sources is limited to link-compatible architectures, such as how sm\_70 and sm\_75 can link with each other but not sm\_80. 6\. Example: Device LTO (link time optimization)[](#example-device-lto-link-time-optimization "Permalink to this headline") ============================================================================================================================= This section demonstrates device link time optimization (LTO). There are two units of LTO IR. The first unit is generated offline using `nvcc`, by specifying the architecture as ‘`-arch lto_XX`’ (see offline.cu). The generated LTO IR is packaged in a fatbinary. The second unit is generated online using NVRTC, by specifying the flag ‘`-dlto`’ (see online.cpp). These two units are then passed to `libnvJitLink*` API functions, which link together the LTO IR, run the optimizer on the linked IR, and generate a cubin (see online.cpp). The cubin is then loaded on the GPU and executed. 6.1. Code (offline.cu)[](#code-offline-cu "Permalink to this headline") ------------------------------------------------------------------------- \_\_device\_\_ float compute(float a, float x, float y) { return a \* x + y; } 6.2. Code (online.cpp)[](#code-online-cpp "Permalink to this headline") ------------------------------------------------------------------------- #include #include #include #include #include #define NUM\_THREADS 128 #define NUM\_BLOCKS 32 #define NVRTC\_SAFE\_CALL(x) \\ do { \\ nvrtcResult result = x; \\ if (result != NVRTC\_SUCCESS) { \\ std::cerr << "\\nerror: " #x " failed with error " \\ << nvrtcGetErrorString(result) << '\\n'; \\ exit(1); \\ } \\ } while(0) #define CUDA\_SAFE\_CALL(x) \\ do { \\ CUresult result = x; \\ if (result != CUDA\_SUCCESS) { \\ const char \*msg; \\ cuGetErrorName(result, &msg); \\ std::cerr << "\\nerror: " #x " failed with error " \\ << msg << '\\n'; \\ exit(1); \\ } \\ } while(0) #define NVJITLINK\_SAFE\_CALL(h,x) \\ do { \\ nvJitLinkResult result = x; \\ if (result != NVJITLINK\_SUCCESS) { \\ std::cerr << "\\nerror: " #x " failed with error " \\ << result << '\\n'; \\ size\_t lsize; \\ result = nvJitLinkGetErrorLogSize(h, &lsize); \\ if (result == NVJITLINK\_SUCCESS && lsize > 0) { \\ char \*log = (char\*)malloc(lsize); \\ result = nvJitLinkGetErrorLog(h, log); \\ if (result == NVJITLINK\_SUCCESS) { \\ std::cerr << "error: " << log << '\\n'; \\ free(log); \\ } \\ } \\ exit(1); \\ } \\ } while(0) const char \*lto\_saxpy \= " \\n\\ extern \_\_device\_\_ float compute(float a, float x, float y); \\n\\ \\n\\ extern \\"C\\" \_\_global\_\_ \\n\\ void saxpy(float a, float \*x, float \*y, float \*out, size\_t n) \\n\\ { \\n\\ size\_t tid = blockIdx.x \* blockDim.x + threadIdx.x; \\n\\ if (tid < n) { \\n\\ out\[tid\] = compute(a, x\[tid\], y\[tid\]); \\n\\ } \\n\\ } \\n"; int main(int argc, char \*argv\[\]) { size\_t numBlocks \= 32; size\_t numThreads \= 128; // Create an instance of nvrtcProgram with the code string. nvrtcProgram prog; NVRTC\_SAFE\_CALL( nvrtcCreateProgram(&prog, // prog lto\_saxpy, // buffer "lto\_saxpy.cu", // name 0, // numHeaders NULL, // headers NULL)); // includeNames // specify that LTO IR should be generated for LTO operation const char \*opts\[\] \= {"-dlto", "--relocatable-device-code=true"}; nvrtcResult compileResult \= nvrtcCompileProgram(prog, // prog 2, // numOptions opts); // options // Obtain compilation log from the program. size\_t logSize; NVRTC\_SAFE\_CALL(nvrtcGetProgramLogSize(prog, &logSize)); char \*log \= new char\[logSize\]; NVRTC\_SAFE\_CALL(nvrtcGetProgramLog(prog, log)); std::cout << log << '\\n'; delete\[\] log; if (compileResult != NVRTC\_SUCCESS) { exit(1); } // Obtain generated LTO IR from the program. size\_t LTOIRSize; NVRTC\_SAFE\_CALL(nvrtcGetLTOIRSize(prog, <OIRSize)); char \*LTOIR \= new char\[LTOIRSize\]; NVRTC\_SAFE\_CALL(nvrtcGetLTOIR(prog, LTOIR)); // Destroy the program. NVRTC\_SAFE\_CALL(nvrtcDestroyProgram(&prog)); CUdevice cuDevice; CUcontext context; CUmodule module; CUfunction kernel; CUDA\_SAFE\_CALL(cuInit(0)); CUDA\_SAFE\_CALL(cuDeviceGet(&cuDevice, 0)); CUDA\_SAFE\_CALL(cuCtxCreate(&context, 0, cuDevice)); // Load the generated LTO IR and the LTO IR generated offline // and link them together. nvJitLinkHandle handle; // Dynamically determine the arch to link for int major \= 0; int minor \= 0; CUDA\_SAFE\_CALL(cuDeviceGetAttribute(&major, CU\_DEVICE\_ATTRIBUTE\_COMPUTE\_CAPABILITY\_MAJOR, cuDevice)); CUDA\_SAFE\_CALL(cuDeviceGetAttribute(&minor, CU\_DEVICE\_ATTRIBUTE\_COMPUTE\_CAPABILITY\_MINOR, cuDevice)); int arch \= major\*10 + minor; char smbuf\[16\]; sprintf(smbuf, "-arch=sm\_%d", arch); const char \*lopts\[\] \= {"-lto", smbuf}; NVJITLINK\_SAFE\_CALL(handle, nvJitLinkCreate(&handle, 2, lopts)); // NOTE: assumes "offline.fatbin" is in the current directory // The fatbinary contains LTO IR generated offline using nvcc NVJITLINK\_SAFE\_CALL(handle, nvJitLinkAddFile(handle, NVJITLINK\_INPUT\_FATBIN, "offline.fatbin")); NVJITLINK\_SAFE\_CALL(handle, nvJitLinkAddData(handle, NVJITLINK\_INPUT\_LTOIR, (void \*)LTOIR, LTOIRSize, "lto\_online")); // The call to nvJitLinkComplete causes linker to link together the two // LTO IR modules (offline and online), do optimization on the linked LTO IR, // and generate cubin from it. NVJITLINK\_SAFE\_CALL(handle, nvJitLinkComplete(handle)); size\_t cubinSize; NVJITLINK\_SAFE\_CALL(handle, nvJitLinkGetLinkedCubinSize(handle, &cubinSize)); void \*cubin \= malloc(cubinSize); NVJITLINK\_SAFE\_CALL(handle, nvJitLinkGetLinkedCubin(handle, cubin)); NVJITLINK\_SAFE\_CALL(handle, nvJitLinkDestroy(&handle)); CUDA\_SAFE\_CALL(cuModuleLoadData(&module, cubin)); CUDA\_SAFE\_CALL(cuModuleGetFunction(&kernel, module, "saxpy")); // Generate input for execution, and create output buffers. size\_t n \= NUM\_THREADS \* NUM\_BLOCKS; size\_t bufferSize \= n \* sizeof(float); float a \= 5.1f; float \*hX \= new float\[n\], \*hY \= new float\[n\], \*hOut \= new float\[n\]; for (size\_t i \= 0; i < n; ++i) { hX\[i\] \= static\_cast(i); hY\[i\] \= static\_cast(i \* 2); } CUdeviceptr dX, dY, dOut; CUDA\_SAFE\_CALL(cuMemAlloc(&dX, bufferSize)); CUDA\_SAFE\_CALL(cuMemAlloc(&dY, bufferSize)); CUDA\_SAFE\_CALL(cuMemAlloc(&dOut, bufferSize)); CUDA\_SAFE\_CALL(cuMemcpyHtoD(dX, hX, bufferSize)); CUDA\_SAFE\_CALL(cuMemcpyHtoD(dY, hY, bufferSize)); // Execute SAXPY. void \*args\[\] \= { &a, &dX, &dY, &dOut, &n }; CUDA\_SAFE\_CALL( cuLaunchKernel(kernel, NUM\_BLOCKS, 1, 1, // grid dim NUM\_THREADS, 1, 1, // block dim 0, NULL, // shared mem and stream args, 0)); // arguments CUDA\_SAFE\_CALL(cuCtxSynchronize()); // Retrieve and print output. CUDA\_SAFE\_CALL(cuMemcpyDtoH(hOut, dOut, bufferSize)); for (size\_t i \= 0; i < n; ++i) { std::cout << a << " \* " << hX\[i\] << " + " << hY\[i\] << " = " << hOut\[i\] << '\\n'; } // Release resources. CUDA\_SAFE\_CALL(cuMemFree(dX)); CUDA\_SAFE\_CALL(cuMemFree(dY)); CUDA\_SAFE\_CALL(cuMemFree(dOut)); CUDA\_SAFE\_CALL(cuModuleUnload(module)); CUDA\_SAFE\_CALL(cuCtxDestroy(context)); free(cubin); delete\[\] hX; delete\[\] hY; delete\[\] hOut; delete\[\] LTOIR; return 0; } 6.3. Build Instructions[](#build-instructions "Permalink to this headline") ----------------------------------------------------------------------------- Assuming the environment variable `CUDA_PATH` points to CUDA Toolkit installation directory, build this example as: * Compile offline.cu to fatbinary containing LTO IR (change `lto_52` to a different `lto_XX` architecture as appropriate). nvcc -arch lto\_52 -rdc=true -fatbin offline.cu * With nvJitLink shared library (note that if test didn’t use nvrtc then it would not need to link with nvrtc): * Windows: cl.exe online.cpp /Feonline ^ /I "%CUDA\_PATH%\\include" ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvJitLink.lib ^ "%CUDA\_PATH%"\\lib\\x64\\cuda.lib * Linux: g++ online.cpp -o online \\ -I $CUDA\_PATH/include \\ -L $CUDA\_PATH/lib64 \\ -lnvrtc -lnvJitLink -lcuda \\ -Wl,-rpath,$CUDA\_PATH/lib64 * With nvJitLink static library (when linking with static library then need to also link with nvptxcompiler\_static, but with is implicitly included): * Windows: cl.exe online.cpp /Feonline ^ /I "%CUDA\_PATH%"\\include ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvrtc-builtins\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvJitLink\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\nvptxcompiler\_static.lib ^ "%CUDA\_PATH%"\\lib\\x64\\cuda.lib user32.lib Ws2\_32.lib * Linux: g++ online.cpp -o online \\ -I $CUDA\_PATH/include \\ -L $CUDA\_PATH/lib64 \\ -lnvrtc\_static -lnvrtc-builtins\_static -lnvJitLink\_static -lnvptxcompiler\_static -lcuda \\ -lpthread 6.4. Notices[](#notices "Permalink to this headline") ------------------------------------------------------- ### 6.4.1. Notice[](#notice "Permalink to this headline") This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. ### 6.4.2. OpenCL[](#opencl "Permalink to this headline") OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc. ### 6.4.3. Trademarks[](#trademarks "Permalink to this headline") NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. © 2022-2022 NVIDIA Corporation & affiliates. All rights reserved. ---