# Table of Contents - [404 | Firedancer](#404-firedancer) - [Firedancer](#firedancer) - [Firedancer](#firedancer) - [fdctl Command Line Interface | Firedancer](#fdctl-command-line-interface-firedancer) - [Getting Started | Firedancer](#getting-started-firedancer) - [Initializing | Firedancer](#initializing-firedancer) - [Performance Tuning | Firedancer](#performance-tuning-firedancer) - [Monitoring | Firedancer](#monitoring-firedancer) - [Configuring | Firedancer](#configuring-firedancer) - [Frequently Asked Questions | Firedancer](#frequently-asked-questions-firedancer) - [Troubleshooting | Firedancer](#troubleshooting-firedancer) - [Netlink Integration | Firedancer](#netlink-integration-firedancer) - [Net Tile | Firedancer](#net-tile-firedancer) - [firedancer Command Line Interface | Firedancer](#firedancer-command-line-interface-firedancer) - [Metrics | Firedancer](#metrics-firedancer) - [Firedancer WebSocket API | Firedancer](#firedancer-websocket-api-firedancer) --- # 404 | Firedancer [Skip to content](https://docs.firedancer.io/guide/#VPContent) 404 PAGE NOT FOUND ============== > But if you don't change your direction, and if you keep looking, you may end up where you are heading. [Take me home](https://docs.firedancer.io/) --- # Firedancer [Skip to content](https://docs.firedancer.io/#VPContent) Firedancer 🔥💃Solana validator =============================== A new Solana validator client, built from the ground up for performance [Get Started](https://docs.firedancer.io/guide/getting-started.html) [View on GitHub](https://github.com/firedancer-io/firedancer) Fast ---- Designed from the ground up to be fast. The concurrency model draws from experience in the low latency trading space, and the code contains many novel high performance reimplementations of core Solana primitives. Secure ------ The architecture of the validator allows it to run with a highly restrictive sandbox and almost no system calls. Independent ----------- Firedancer is written from scratch. This brings client diversity to the Solana network and helps it stay resilient to supply chain attacks in build tooling or dependencies. --- # Firedancer [Skip to content](https://docs.firedancer.io/guide/firedancer.html#VPContent) On this page Firedancer [​](https://docs.firedancer.io/guide/firedancer.html#firedancer) ============================================================================ Why is Jump Trading Group creating Firedancer? [​](https://docs.firedancer.io/guide/firedancer.html#why-is-jump-trading-group-creating-firedancer) --------------------------------------------------------------------------------------------------------------------------------------------------- There are many blockchains in existence today, each with different advantages and disadvantages. What differentiates Solana is its design which enables the dissemination of frequent, inexpensive information. It is cheap to use so it does not impose an onerous cost on the end user. Solana can handle the high throughput requirements that come with the bursts of activity. Over the past twenty years, Jump has built a high-capacity, high-reliability, low-latency global network to support our trading and address many of the scaling problems that Solana is currently facing. In doing so, we have had to think hard about how to take a flood of market data and process it more efficiently than our competitors, which means pushing the limits of our hardware. Leveraging our experience over the last two decades in building this global network, we believe we are well-positioned to significantly improve the power, reliability, and security of the Solana ecosystem. The Firedancer project requires a significant commitment of Jump developer resources, and we are appreciative of the support from the Solana Foundation to make this project possible. Firedancer’s goal of improving Solana: [​](https://docs.firedancer.io/guide/firedancer.html#firedancer-s-goal-of-improving-solana) ----------------------------------------------------------------------------------------------------------------------------------- Firedancer is intended to replicate Solana’s functionality but with higher performance. Consistent with Solana’s vision, the goal is to eliminate software inefficiencies and drive the performance to the limits of the hardware, while attempting to not create any new vulnerabilities. Firedancer makes minimal use of third party libraries for cryptographic and network functionality where there are currently bottlenecks. As a second validator codebase, Firedancer removes the risk of a single point of failure. As nodes are distributed more evenly among several independent codebases, the overall network will become more robust against any one bug. Additionally, Firedancer is written in C, in contrast to the Rust codebase of the original Solana validator, which further reduces the likelihood of a vulnerability affecting the entire network. Firedancer’s security model is inspired by methods used in the browser world which is under perpetual attack. Diversification of security approaches is another way the second validator intends to make the network more robust. Improved performance also means current performance needs can be achieved with lower cost hardware. This should encourage additional validator operators to join the network who were otherwise prohibited by current hardware requirements. What does the Firedancer team aim to deliver? [​](https://docs.firedancer.io/guide/firedancer.html#what-does-the-firedancer-team-aim-to-deliver) ------------------------------------------------------------------------------------------------------------------------------------------------- At a high level, the three functional blocks of Solana are network, runtime, and the consensus mechanism. The network layer is essentially code complete, and is being tuned and undergoing reviews by auditors. The first Firedancer validator to go into testnet \[is version 0.1 and\] is nicknamed Frankendancer. It is Firedancer’s networking layer grafted onto the Agave runtime and consensus code. Although the networking layer will be much faster, it is likely that the performance of the Frankendancer validator will be bottlenecked by the legacy runtime and consensus code. Frankendancer still offers an advantage with DoS mitigation. This document explains how to bring up Frankendancer in testnet. We hope to get feedback from validators while we go through security audits and continue to refine the codebase. The goal is to get Frankendancer to mainnet in Q2 2024. In parallel the team has been working on replicating the functionality of the runtime and consensus code, with the goal of delivering a fully independent validator by the end of 2024. Frankendancer [​](https://docs.firedancer.io/guide/firedancer.html#frankendancer) ---------------------------------------------------------------------------------- Firedancer is a new Solana validator with a new codebase, but it is being developed incrementally. To enable testing and deployment before the entire Solana protocol has been implemented we rely on the existing Agave validator code to provide functionality that is missing. This side-by-side configuration is referred to as "frankendancer". Currently, Firedancer has implemented the following parts: * The QUIC and UDP ingress networking pieces, using high performance kernel bypass networking. * The block distribution engine and egress networking, also using kernel bypass. The engine contains a full reimplementation of erasure coding and the Solana turbine protocol for packet routing. * Signature verification with a custom AVX512 ED25519 implementation. * The block packing logic. All other functionality is retained by Agave, including the runtime itself which tracks account state and executes transactions. This means building and running a Firedancer validator also builds a Agave validator and runs it as a child process. The Solana process is fully built into and managed by Firedancer and you do not need to start it up or otherwise configure it. Because of this side by side operation, existing knowledge and processes you have from running an Agave validator will likely apply to Firedancer too. For example, the monitoring and metrics are left the same, and you will run the same `solana` CLI command. This Book [​](https://docs.firedancer.io/guide/firedancer.html#this-book) -------------------------------------------------------------------------- This book is intended for operators and administrators already familiar with running an Agave validator, as Firedancer heavily leans on the same concepts and components. To get up to speed there, read the Solana Labs [validators guide](https://docs.solana.com/running-validator) . The guide describes how to start and run a healthy Firedancer validator node. The validator is under heavy development and should be considered beta software, thank you for being on the bleeding edge. Help [​](https://docs.firedancer.io/guide/firedancer.html#help) ---------------------------------------------------------------- Firedancer is beta software and you should expect hiccups when getting started with, and when operating the validator. The Firedancer development team can help diagnose any issues. The main place to discuss Firedancer operational issues is the #firedancer channel in the [Solana Tech discord](https://discord.com/invite/solana) . Contributing [​](https://docs.firedancer.io/guide/firedancer.html#contributing) -------------------------------------------------------------------------------- Firedancer is open source and being developed on GitHub. All contributions are welcome. Jump is currently the primary contributor as the project is in the initial development phase, but Firedancer is a decentralized, community validator and project stewardship will be transitioned to the community as the project matures. License [​](https://docs.firedancer.io/guide/firedancer.html#license) ---------------------------------------------------------------------- Firedancer is available under the [Apache-2.0 license](https://www.apache.org/licenses/LICENSE-2.0) . Firedancer also includes external libraries that are available under a variety of licenses. See [LICENSE](https://github.com/firedancer-io/firedancer/blob/main/LICENSE) for the full license text and [NOTICES](https://github.com/firedancer-io/firedancer/blob/main/NOTICE) for additional details on external libraries. --- # fdctl Command Line Interface | Firedancer [Skip to content](https://docs.firedancer.io/api/cli.html#VPContent) On this page `fdctl` Command Line Interface [​](https://docs.firedancer.io/api/cli.html#fdctl-command-line-interface) ========================================================================================================= The Firedancer binary `fdctl` contains many subcommands which can be run from the command line. `run` [​](https://docs.firedancer.io/api/cli.html#run) ------------------------------------------------------- Runs the validator. This command does not exit until the validator does, an error in running the validator will be propagated to the exit code of the process. The command can be run with the capabilities listed below but it is suggested to run it as `sudo`. The command writes an abbreviated log output to `stderr` and nothing will be written to `stdout`. | Arguments | Description | | --- | --- | | `--config ` | Path to a configuration TOML file to run the validator with | Capabilities | Capability | Reason | | --- | --- | | `CAP_NET_RAW` | call `socket(2)` to bind to a raw socket for use by XDP | | `CAP_SYS_ADMIN` | call `bpf(2)` with the `BPF_OBJ_GET` command to initialize XDP | | `CAP_SYS_ADMIN` | call `unshare(2)` with `CLONE_NEWUSER` to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces | | `CAP_SETUID` | call `setresuid(2)` to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID | | `CAP_SETGID` | call `setresgid(2)` to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID | | `CAP_SYS_RESOURCE` | call `rlimit(2)` to increase `RLIMIT_MEMLOCK` so all memory can be locked with `mlock(2)`. Not required if the process already has a high enough limit | | `CAP_SYS_RESOURCE` | call `setpriority(2)` to increase thread priorities. Not required if the process already has a nice value of -19 | | `CAP_SYS_RESOURCE` | call `rlimit(2) to increase` RLIMIT\_NOFILE\` to allow more open files for Agave. Not required if the resource limit is already high enough | | `CAP_NET_BIND_SERVICE` | call `bind(2)` to bind to a privileged port for serving metrics. Only required if the bind port is below 1024 | $ fdctl run --config config.toml Log at "/tmp/fd-0.0.0_755725_fd_fd_2024_07_01_03_24_27_470038344_GMT+00" NOTICE 07-01 03:24:27.471949 755725 f0 main src/disco/topo/fd_topo.c(445): SUMMARY Total Tiles: 16 Total Memory Locked: 28145897472 bytes (26 GiB + 218 MiB + 20 KiB) Required Gigantic Pages: 26 Required Huge Pages: 109 Required Normal Pages: 36 Required Gigantic Pages (NUMA node 0): 26 Required Huge Pages (NUMA node 0): 109 [...] NOTICE 07-01 03:24:28.772209 755745 16 metric:0 src/disco/topo/fd_topo_run.c(30): booting tile metric:0 pid:755742 tid:755745 NOTICE 07-01 03:24:28.773617 755747 13 shred:0 src/disco/topo/fd_topo_run.c(30): booting tile shred:0 pid:755740 tid:755747 NOTICE 07-01 03:24:28.773798 755748 9 pack:0 src/disco/topo/fd_topo_run.c(30): booting tile pack:0 pid:755739 tid:755748 [...] `monitor` [​](https://docs.firedancer.io/api/cli.html#monitor) --------------------------------------------------------------- Monitors a validator that is running locally on this machine. This is a low level performance monitor mostly useful for diagnosing throughput issues. The monitor takes over the controlling terminal and refreshes it many times a second with up to date information. You can exit the monitor by sending Ctrl+C or `SIGINT`. | Arguments | Description | | --- | --- | | `--config ` | Path to a configuration TOML file to run the monitor with. This must be the same configuration file the validator was started with | Capabilities | Capability | Reason | | --- | --- | | `CAP_SYS_ADMIN` | call `unshare(2)` with `CLONE_NEWUSER` to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces | | `CAP_SETUID` | call `setresuid(2)` to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID | | `CAP_SETGID` | call `setresgid(2)` to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID | | `CAP_SYS_RESOURCE` | call `rlimit(2)` to increase `RLIMIT_MEMLOCK` so all memory can be locked with `mlock(2)`. Not required if the process already has a high enough limit | $ fdctl monitor --config config.toml snapshot for 2024-07-01 04:31:34.669413032 GMT+00 tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish ---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+---------- net | 763643 | - | - | run( run) | -( -) | 0( +0) | 42.300 | 0.000 | 57.700 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 quic | 763638 | - | - | run( run) | -( -) | 0( +0) | 0.331 | 0.000 | 99.669 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 verify | 763641 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.509 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 verify | 763639 | - | - | run( run) | -( -) | 0( +0) | 0.488 | 0.000 | 99.417 | 0.000 | 0.000 | 0.000 | 0.000 | 0.095 verify | 763640 | - | - | run( run) | -( -) | 0( +0) | 0.483 | 0.000 | 99.516 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 verify | 763644 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.510 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 verify | 763642 | - | - | run( run) | -( -) | 0( +0) | 0.489 | 0.000 | 99.511 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 dedup | 763634 | - | - | run( run) | -( -) | 0( +0) | 0.049 | 0.000 | 99.951 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 pack | 763635 | - | - | run( run) | -( -) | 0( +0) | 0.060 | 0.000 | 99.937 | 0.000 | 0.000 | 0.000 | 0.000 | 0.003 bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.287 | 0.000 | 99.584 | 0.000 | 0.000 | 0.000 | 0.000 | 0.129 bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.286 | 0.000 | 99.714 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 poh | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.463 | 0.000 | 99.517 | 0.000 | 0.000 | 0.000 | 0.018 | 0.002 shred | 763637 | - | - | run( run) | -( -) | 0( +0) | 0.404 | 0.000 | 99.519 | 0.000 | 0.000 | 0.000 | 0.000 | 0.077 store | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.293 | 0.000 | 99.447 | 0.000 | 0.000 | 0.000 | 0.000 | 0.260 sign | 763636 | - | - | run( run) | -( -) | 0( +0) | 0.239 | 0.000 | 99.729 | 0.000 | 0.000 | 0.000 | 0.000 | 0.032 metric | 763630 | - | - | run( run) | -( -) | 0( +0) | 100.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 link | tot TPS | tot bps | uniq TPS | uniq bps | ha tr% | uniq bw% | filt tr% | filt bw% | ovrnp cnt | ovrnr cnt | slow cnt | tx seq ------------------+----------+----------+----------+----------+----------+----------+----------+----------+---------------------+---------------------+---------------------+------------------- quic->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) shred->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) net->quic | 15.4 | 44.4K | 15.4 | 44.4K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) sign->quic | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) quic->verify | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 16( +0) verify->dedup | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 16( +1) verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0) verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0) verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0) dedup->pack | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1) poh->pack | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) poh->pack | 15.4 | 5907.3 | 15.4 | 5907.3 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) pack->bank | 30.8 | 519.8K | 30.8 | 519.8K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3) pack->bank | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 306( +3) bank->poh | 30.8 | 529.7K | 30.8 | 529.7K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 229( +2) bank->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 0( +0) poh->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) pack->poh | 46.2 | 519.8K | 46.2 | 519.8K | 100.000 | 100.000 | 100.000 | 100.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3) net->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) poh->shred | 969.2 | 597.5K | 969.2 | 597.5K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 4930( +63) poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) sign->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) shred->store | 30.8 | 10.3M | 30.8 | 10.3M | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 154( +2) quic->sign | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) shred->sign | 15.4 | 3938.2 | 15.4 | 3938.2 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0) `configure` [​](https://docs.firedancer.io/api/cli.html#configure) ------------------------------------------------------------------- Configures the operating system so that it can run Firedancer. See [the guide](https://docs.firedancer.io/guide/initializing.html) for more information. There are the following stages to each configure command: * `hugetlbfs` Reserves huge and gigantic pages for use by Firedancer and mounts huge page filesystems for then under a path in the configuration TOML file. * `sysctl` Set required kernel parameters. * `hyperthreads` Disables hyperthreaded pair for critical CPU cores. * `bonding` Prepares bonded network devices for XDP networking. * `ethtool-channels` Configures the number of channels on the network device. * `ethtool-offloads` Modify offload feature flags on the network device. * `ethtool-loopback` Disables UDP segmentation on the loopback device. | Arguments | Description | | --- | --- | | `--config ` | Path to a configuration TOML file to configure the validator with. This must be the same configuration file the validator will be started with | config.toml toml [hugetlbfs] mount_path = "/mnt/.fd" [layout] net_tile_count = 2 [tiles] [net] interface = "ens3f0" ### `configure init ...` [​](https://docs.firedancer.io/api/cli.html#configure-init-stage) Prepare the operating system environment to run Firedancer. This will reserve and mount the huge page filesystems, set the kernel parameters, and configure the number of combined channels on the network device. Capabilities | Capability | Reason | | --- | --- | | `root` | increase `/proc/sys/vm/nr_hugepages` and mount hugetlbfs filesystems. Only applies for the `hugetlbfs` stage | | `root` | increase network device channels with `ethtool --set-channels`. Only applies for the `ethtool-channels` stage | | `root` | disable network device offloads with `ethtool --offload IFACE FEATURE off`. Only applies for the `ethtool-offloads` stage | | `root` | disable network device tx-udp-segmentation with `ethtool --offload lo tx-udp-segmentation off`. Only applies for the `ethtool-loopback` stage | | `CAP_SYS_ADMIN` | set kernel parameters in `/proc/sys`. Only applies for the `sysctl` stage | $ fdctl configure init all NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist NOTICE hugetlbfs ... configuring NOTICE RUN: `mkdir -p /mnt/.fd/.huge` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=241172480` NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424` NOTICE sysctl ... unconfigured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000) NOTICE RUN: `echo "1000000" > /proc/sys/vm/max_map_count` NOTICE RUN: `echo "1024000" > /proc/sys/fs/file-max` NOTICE RUN: `echo "1024000" > /proc/sys/fs/nr_open` NOTICE RUN: `echo "2" > /proc/sys/net/ipv4/conf/lo/rp_filter` NOTICE RUN: `echo "1" > /proc/sys/net/ipv4/conf/lo/accept_local` WARNING kernel parameter `/proc/sys/net/core/bpf_jit_enable` is too low (got 0 but expected at least 1). Proceeding but performance may be reduced. WARNING kernel parameter `/proc/sys/kernel/numa_balancing` is set to 1, not the expected value of 0. Proceeding but performance may be reduced. NOTICE hyperthreads ... unconfigured ... pack cpu 5 has hyperthread pair 37 which should be offline NOTICE hyperthreads ... configuring NOTICE RUN: `echo "0" > /sys/devices/system/cpu/cpu37/online` NOTICE RUN: `echo "0" > /sys/devices/system/cpu/cpu40/online` NOTICE ethtool-channels ... unconfigured ... device `ens3f0` does not have right number of channels (got 1 but expected 2) NOTICE ethtool-channels ... configuring NOTICE ethtool-channels ... RUN: `ethtool --set-channels ens3f0 combined 2` NOTICE ethtool-offloads ... unconfigured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled NOTICE ethtool-offloads ... configuring NOTICE ethtool-offloads ... RUN: `ethtool --offload ens3f0 generic-receive-offload off` NOTICE ethtool-loopback ... unconfigured ... device `lo` has tx-udp-segmentation enabled. Should be disabled NOTICE ethtool-loopback ... configuring NOTICE ethtool-loopback ... RUN: `ethtool --offload lo tx-udp-segmentation off` ### `configure check ...` [​](https://docs.firedancer.io/api/cli.html#configure-check-stage) Check if the operating system environment is properly configured. Exits with a non-zero exit code if it is not, after printing relevant diagnostics to `stderr`. $ fdctl configure check all WARNING hugetlbfs ... not configured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist WARNING sysctl ... not configured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000) WARNING ethtool-channels ... not configured ... device `ens3f0` does not have right number of channels (got 1 but expected 2) WARNING ethtool-offloads ... not configured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled WARNING ethtool-loopback ... not configured ... device `lo` has tx-udp-segmentation enabled. Should be disabled ERR failed to configure some stages ### `configure fini ...` [​](https://docs.firedancer.io/api/cli.html#configure-fini-stage) Remove any Firedancer specific operating system configuration still lingering. This only unmounts the `hugetlbfs` stages and returns the reserved huge and gigantic pages to the kernel pool. It will not reduce sysctls that were earlier increased, or change the network channel count back as we no longer know what the original value was. Capabilities | Capability | Reason | | --- | --- | | `root` | remove directories from `/mnt`, unmount hugetlbfs. Only applies for the `hugetlbfs` stage | $ fdctl configure fini all NOTICE ethtool ... finishing NOTICE sysctl ... finishing NOTICE hugetlbfs ... finishing NOTICE RUN: `umount /mnt/.fd/.huge` NOTICE RUN: `rmdir /mnt/.fd/.huge` NOTICE RUN: `umount /mnt/.fd/.gigantic` NOTICE RUN: `rmdir /mnt/.fd/.gigantic` NOTICE RUN: `rmdir /mnt/.fd/.normal` NOTICE RUN: `rmdir /mnt/.fd` `version` [​](https://docs.firedancer.io/api/cli.html#version) --------------------------------------------------------------- Prints the current version of the validator to the standard output and exits. The command writes diagnostic messages from logs to `stderr`. sh $ fdctl version 0.101.11814 `shred-version` [​](https://docs.firedancer.io/api/cli.html#shred-version) --------------------------------------------------------------------------- Prints the current shred version of the cluster being joined, according to the entrypoints, to standard output and exits. The command writes diagnostic messages from logs to `stderr`. sh $ fdctl shred-version 9065 `metrics` [​](https://docs.firedancer.io/api/cli.html#metrics) --------------------------------------------------------------- Prints the current validator metrics to stdout. Metrics can typically be accessed via HTTP when the `metric` tile is enabled, but the command can be used even if the metrics server is not enabled, or the validator has crashed. sh $ fdctl metrics --config ~/config.toml # HELP tile_pid The process ID of the tile. # TYPE tile_pid gauge tile_pid{kind="netlnk",kind_id="0"} 627750 tile_pid{kind="net",kind_id="0"} 627759 `set-identity` [​](https://docs.firedancer.io/api/cli.html#set-identity) ------------------------------------------------------------------------- Changes the identity key of a running validator. The `` argument is required and must be the path to an Agave style `identity.json` keypair file. If the path is specified as `-` the key will instead be read from `stdin`. WARNING `set-identity` must be called with the configuration file you started the validator with, like `fdctl set-identity --config `, if the `config` argument is not provided, the command may not update the key on all tiles and your validator may start skipping slots. It is not generally safe to call `set-identity`, as another validator might be running with the same identity, and if they both produce a block or vote concurrently, the validator may violate consensus and be subject to (future) slashing. Best practice requires copying the `tower.bin` file from the prior to the new validator, to ensure that vote lockouts are respected. The validator will not change identity in the middle of a leader slot, and will wait until any in-progress leader slot completes before switching to the new identity. It is safe to call during or near a leader slot because of this wait. The command exits successfully (with an exit code of 0) if the identity key was changed, otherwise it will fail and print diagnostic messages to `stderr`. Reasons for failure include the validator being unable to open or load the tower, when `--require-tower` is specified, or being unable to load or verify the provided identity key. Currently due to implementation limitations, the key can be partially changed if the `set-identity` command is cancelled (for example with Ctrl+C) while running. The next call to `set-identity` might need to provide the `--force` argument to succeed if this occurs, to reset this partial state and proceed with setting a new key. | Arguments | Description | | --- | --- | | `` | Path to a `identity.json` keypair file, or `-` to read the JSON formatted key from `stdin` | | `--config ` | Path to a configuration TOML file of the validator to change identity for. This must be the same configuration file the validator was started with | | `--require-tower` | If specified, refuse to set the validator identity if saved tower state is not found | | `--force` | If a `set-identity` operation is abandoned part way through, you will need to specify `--force` to reset the validator key state when trying again | $ sudo fdctl set-identity --config ~/config.toml ~/keys/validator-keypair.json NOTICE Validator identity key switched to `4UCZB7zfquCVN7GafWETFVLTceNH3nm2mndyHeDuSggC` `keys` [​](https://docs.firedancer.io/api/cli.html#keys) --------------------------------------------------------- ### `keys pubkey ` [​](https://docs.firedancer.io/api/cli.html#keys-pubkey-path) Prints the base58 encoding of the public key in the file at `` to the standard output and exits. The file at `` should be an Agave style `identity.json` key file. The command writes diagnostic messages from logs to `stderr`. sh $ fdctl keys pubkey ~/.firedancer/fd1/identity.json Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF ### `keys new ` [​](https://docs.firedancer.io/api/cli.html#keys-new-path) Creates a new keypair from the kernel random number generator and writes it to the file specified at ``. The default user for the operation is the user running the command and should have write access to ``. The user can be changed by specifying it in the TOML configuration file. | Arguments | Description | | --- | --- | | `--config` | Path to a configuration TOML file which determines the user creating the file. | config.toml toml user = "firedancer" $ fdctl keys new ~/identity.json --config config.toml NOTICE successfully created keypair in `/home/firedancer/identity.json` `mem` [​](https://docs.firedancer.io/api/cli.html#mem) ------------------------------------------------------- Prints information about the memory requirements and the tile configuration and layout of the validator to `stdout` before exiting. The command writes diagnostic messages from logs to `stderr`. Firedancer preallocates and locks all memory it needs from huge and gigantic page mounts before booting, and the `hugetlbfs` stage of `fdctl configure` will reserve the memory described here for exclusive use by Firedancer. | Arguments | Description | | --- | --- | | `--config` | Path to a configuration TOML file to print memory usage information with | | `--sort` | List all memory allocations sorted by size in decreasing order, including a percentage of total and exact byte count | sh $ fdctl mem --config config.toml SUMMARY Total Tiles: 17 Total Memory Locked: 27088932864 bytes (25 GiB + 234 MiB + 20 KiB) Required Gigantic Pages: 25 Required Huge Pages: 117 Required Normal Pages: 27 Required Gigantic Pages (NUMA node 0): 25 Required Huge Pages (NUMA node 0): 117 WORKSPACES 0 ( 1 GiB): net_quic page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904 1 ( 1 GiB): net_shred page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904 [...] LINKS 0 ( 32 MiB): net_quic kind_id=0 wksp_id=0 depth=16384 mtu=2048 burst=1 1 ( 32 MiB): net_shred kind_id=0 wksp_id=1 depth=16384 mtu=2048 burst=1 [...] TILES 0 ( 3 GiB): net kind_id=0 wksp_id=18 cpu_idx=1 out_link=-1 in=[-2, -3] out=[ 0, 1] 1 ( 3 GiB): quic kind_id=0 wksp_id=19 cpu_idx=2 out_link=4 in=[ 0, -21] out=[ 2, 20] [...] --- # Getting Started | Firedancer [Skip to content](https://docs.firedancer.io/guide/getting-started.html#VPContent) On this page Getting Started [​](https://docs.firedancer.io/guide/getting-started.html#getting-started) =========================================================================================== Frankendancer [​](https://docs.firedancer.io/guide/getting-started.html#frankendancer) --------------------------------------------------------------------------------------- This guide details building and running the Frankendancer validator which is a hybrid of Firedancer and Agave code running side by side. Frankendancer replaces the Agave networking stack and block production components to perform better while leader. It is not yet possible to run a full Firedancer validator, which is in heavy development. Hardware Requirements [​](https://docs.firedancer.io/guide/getting-started.html#hardware-requirements) ------------------------------------------------------------------------------------------------------- Because Firedancer currently depends on the Agave validator, the hardware requirements are at least [what's recommended](https://docs.anza.xyz/operations/requirements) for that validator. Firedancer hopes to reduce these over time. **Minimum** * 24-Core AMD or Intel CPU @ >2.8GHz * 256GB RAM * 2TB PCI Gen3 NVME SSD (High TBW) **Recommended** * 32-Core CPU @ >3GHz with AVX512 support * 512GB RAM with ECC memory * Same capacity with separate disks for Accounts and Ledger * 1 Gigabit/s Network Bandwidth Validator operators also refer to [https://solanahcl.org/](https://solanahcl.org/) which has a lot of useful information about hardware. Installing [​](https://docs.firedancer.io/guide/getting-started.html#installing) --------------------------------------------------------------------------------- ### Prerequisites [​](https://docs.firedancer.io/guide/getting-started.html#prerequisites) Firedancer must be built from source and currently only supports building and running on Linux. Firedancer requires a recent Linux kernel, at least v4.18. This corresponds to Ubuntu 20.04, Fedora 29, Debian 11, or RHEL 8. * GCC version 8.5 or higher. Only GCC version 11, 12, and 13 are supported and tested by the Firedancer developers. * [rustup](https://rustup.rs/) * clang, git, and make NOTE Firedancer currently builds the [Agave](https://docs.solana.com/running-validator/validator-reqs) validator as a dependency, which requires a full Rust toolchain. Once Firedancer is able to stand alone, this will no longer be required. Other dependencies of the Firedancer build can be installed with a convenience script. First, clone the source code with: sh $ git clone --recurse-submodules https://github.com/firedancer-io/firedancer.git $ cd firedancer $ git checkout v0.819.30111 # Or the latest Frankendancer release Then you can run the `deps.sh` script to install system packages and compile library dependencies. System packages will be installed via the package manager on your system, while library dependencies will be compiled and output placed under `./opt`. sh $ ./deps.sh Releases [​](https://docs.firedancer.io/guide/getting-started.html#releases) ----------------------------------------------------------------------------- Firedancer does not produce pre-built binaries and you must build from source, but Firedancer releases are made available as tags. The following naming convention is used, * `main` This should not be used. The main branch is bleeding edge and includes all Firedancer development and changes that could break Frankendancer. * `v0.xxx.yyyyy` Official Frankendancer releases. The Frankendancer versioning has three components, * Major version is always `0`. The first full Firedancer release will be `1.x` * Minor version increments by 100 for each new Frankendancer release. The minor version will then increment by 1 for new minor versions within this release. * The patch number encodes the Agave validator version. An Agave version of `v1.17.14` is represented as `11714`. ================= main branch ================= \ \ \ v0.100.11814 \ v0.200.11901 \ \ \ v0.100.11815 \ v0.201.11902 \ \ v0.101.11815 Building [​](https://docs.firedancer.io/guide/getting-started.html#building) ----------------------------------------------------------------------------- Once dependencies are installed, you can build Firedancer. Because Firedancer depends on the Agave validator, this will also build some Agave components. sh $ make -j fdctl solana You will need around 32GiB of available memory to build Firedancer. If you run out of memory compiling, make can return a variety of errors. TIP The Firedancer production validator is built as a single binary `fdctl` short for Firedancer control. You can start, stop, and monitor the Firedancer instance from this one program. The `solana` CLI binary can be built with make as well for convenience so you can run RPC commands like `solana transfer`. Firedancer automatically detects the hardware it is being built on and enables architecture specific instructions for maximum performance if possible. This means binaries built on one machine may not be able to run on another. If you wish to target a lower machine architecture you can compile for a specific target by setting the `MACHINE` environment variable to one of the targets under `config/`. sh $ MACHINE=linux_gcc_x86_64 make -j fdctl solana The default target is `native`, and compiled binaries will be placed in `./build/native/gcc/bin`. Updating [​](https://docs.firedancer.io/guide/getting-started.html#updating) ----------------------------------------------------------------------------- If you checked out Firedancer using Git, run through these steps to check out a newer version, update dependencies, and rebuild binaries. sh git fetch --tags git checkout v0.819.30111 git submodule update make -j fdctl solana Running [​](https://docs.firedancer.io/guide/getting-started.html#running) --------------------------------------------------------------------------- ### Configuration [​](https://docs.firedancer.io/guide/getting-started.html#configuration) Firedancer has many configuration options which are [discussed later](https://docs.firedancer.io/guide/configuring.html) . For now, we override only the essential options needed to start the validator on Testnet. config.toml toml user = "firedancer" [gossip] entrypoints = [\ "entrypoint.testnet.solana.com:8001",\ "entrypoint2.testnet.solana.com:8001",\ "entrypoint3.testnet.solana.com:8001",\ ] [consensus] identity_path = "/home/firedancer/validator-keypair.json" vote_account_path = "/home/firedancer/vote-keypair.json" known_validators = [\ "5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on",\ "dDzy5SR3AXdYWVqbDEkVFdvSPCtS9ihF5kJkHCtXoFs",\ "Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN",\ "eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ",\ "9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv",\ ] [rpc] port = 8899 full_api = true private = true [reporting] solana_metrics_config = "host=https://metrics.solana.com:8086,db=tds,u=testnet_write,p=c4fa841aa918bf8274e3e2a44d77568d9861b3ea" This configuration will cause Firedancer to run as the user `firedancer` on the local machine. The `identity_path` and `vote_account_path` should be Agave style keys, which can be generated using the [`fdctl keys`. subcommand](https://docs.firedancer.io/api/cli.html#keys-new-path) . The `vote_account_path` can also be the public key of an existing vote account. This will put the ledger in `/home/firedancer/.firedancer/fd1/ledger`. To customize this path, refer to the [configuration guide](https://docs.firedancer.io/guide/configuring.html#ledger) . LEDGER The Firedancer blockstore in the ledger directory is compatible with the one for the Agave validator, and it is possible to switch between validator clients while keeping the `ledger` directory in place. Additionally, this configuration enables the full RPC API at port 8899. Although the port will not be published to other validators in gossip, use a firewall to restrict access to this port for maximum security. The Firedancer client can report diagnostic metrics similar to an Agave client. It is recommended to set the `[reporting.solana_metrics_config]` in the config file to the appropriate value for the cluster. The options for the different clusters are listed in the `default.toml` file in the [`reporting`](https://github.com/firedancer-io/firedancer/blob/main/src/app/fdctl/config/default.toml#L144) section. ### Permissions [​](https://docs.firedancer.io/guide/getting-started.html#permissions) There are two users involved in running Firedancer. The user that you launch `fdctl` with, and the user Firedancer switches to after it has started. The requirements for these users are very different: * The user Firedancer starts as is not specified in configuration, but is simply the user that launches the process. For most commands, including `fdctl run` and `configure` it needs to be `root` or have various capabilities described below to setup kernel bypass networking. It is recommended to simply use the `root` user when launching. * The user Firedancer switches to after it has booted up and performed privileged initialization. This is given by the `user` option in your configuration TOML file. Firedancer requires nothing from this user and it should be as minimally permissioned as possible. It should never be `root` or another superuser, and the user should not be present in the sudoers file or have any other privileges. Only the `fdctl run` and `monitor` commands will switch to the non-privileged user, and other commands will run as the startup user until they complete. Most commands can be started with capabilities rather than as the `root` user, although this isn't recommended. If you are an advanced operator, you can see which capabilities are required for a command by running it unprivileged: $ ./build/native/gcc/bin/fdctl run WARNING run ... process requires capability `CAP_NET_RAW` to call `socket(2)` to bind to a raw socket for use by XDP WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `bpf(2)` with the `BPF_OBJ_GET` command to initialize XDP WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `unshare(2)` with `CLONE_NEWUSER` to sandbox the process in a user namespace WARNING run ... process requires capability `CAP_SETUID` to call `setresuid(2)` to switch uid to the sandbox user WARNING run ... process requires capability `CAP_SETGID` to call `setresgid(2)` to switch gid to the sandbox user ERR insufficient permissions to execute command `run`. It is recommended to start Firedancer as the root user, but you can also start it with the missing capabilities listed above. The program only needs to start with elevated permissions to do privileged operations at boot, and will immediately drop permissions and switch to the user specified in your configuration file once they are complete. Firedancer will not execute outside of the boot process as root, and will refuse to start if it cannot drop privileges. Firedancer needs to be started privileged to configure high performance networking with XDP. For additional layers of defense against local privilege escalation, it is not suggested to `setcap(8)` the `fdctl` binary as this can create a larger attack surface. ### Initialization [​](https://docs.firedancer.io/guide/getting-started.html#initialization) The validator uses some Linux features that must be enabled and configured before it can be started correctly. It is possible for advanced operators to do this configuration manually, but `fdctl` provides a command to check and automate this step. WARNING Running any `fdctl configure` command may make permanent changes to your system. You should be careful before running these commands on a production host. The initialization steps are described [in detail](https://docs.firedancer.io/guide/initializing.html) later. But plowing ahead at the moment: sh $ sudo ./build/native/gcc/bin/fdctl configure init all --config ~/config.toml You will be told what steps are performed: NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist NOTICE hugetlbfs ... configuring NOTICE RUN: `mkdir -p /mnt/.fd/.huge` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=241172480` NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424` NOTICE sysctl ... already valid NOTICE bonding ... already valid NOTICE ethtool-channels ... already valid NOTICE ethtool-offloads ... already valid NOTICE ethtool-loopback ... already valid It is strongly suggested to run the `configure` command when the system boots, and it needs to be run each time the system is rebooted. ### Running [​](https://docs.firedancer.io/guide/getting-started.html#running-1) Finally, we can run Firedancer: sh $ sudo ./build/native/gcc/bin/fdctl run --config ~/config.toml Firedancer logs selected output to `stderr` and a more detailed log to a local file. Every tile in Firedancer runs in a separate process for security isolation, so you will see a complete process tree get launched. sh $ pstree 1741904 -as systemd --switched-root --system --deserialize 17 └─sudo ./build/native/gcc/bin/fdctl run --config ~/config.toml └─fdctl run --config ~/config.toml └─fdctl run --config ~/config.toml ├─fdctl run-agave --config-fd 0 │ └─35*[{fdctl}] ├─diag:0 run1 diag 0 --pipe-fd 20 --config-fd 0 ├─dedup:0 run1 dedup 0 --pipe-fd 15 --config-fd 0 ├─gui:0 run1 gui 0 --pipe-fd 22 --config-fd 0 ├─metric:0 run1 metric 0 --pipe-fd 19 --config-fd 0 ├─net:0 run1 net 0 --pipe-fd 7 --config-fd 0 ├─pack:0 run1 pack 0 --pipe-fd 16 --config-fd 0 ├─plugin:0 run1 plugin 0 --pipe-fd 21 --config-fd 0 ├─quic:0 run1 quic 0 --pipe-fd 8 --config-fd 0 ├─shred:0 run1 shred 0 --pipe-fd 17 --config-fd 0 ├─sign:0 run1 sign 0 --pipe-fd 18 --config-fd 0 ├─verify:0 run1 verify 0 --pipe-fd 9 --config-fd 0 ├─verify:1 run1 verify 1 --pipe-fd 10 --config-fd 0 ├─verify:2 run1 verify 2 --pipe-fd 11 --config-fd 0 ├─verify:3 run1 verify 3 --pipe-fd 12 --config-fd 0 ├─verify:4 run1 verify 4 --pipe-fd 13 --config-fd 0 └─verify:5 run1 verify 5 --pipe-fd 14 --config-fd 0 If any of the processes dies or is killed it will bring all of the others down with it. ### Networking [​](https://docs.firedancer.io/guide/getting-started.html#networking) Firedancer uses `AF_XDP`, a Linux API for high performance networking. For more background see the [kernel documentation](https://www.kernel.org/doc/html/next/networking/af_xdp.html) . Although `AF_XDP` works with any ethernet network interface, results may vary across drivers. Popular well tested drivers include: * `ixgbe` — Intel X540 * `i40e` — Intel X710 series * `ice` — Intel E800 series Firedancer installs an XDP program on the network interface `[net.interface]` and `lo` while it is running. This program redirects traffic on ports that Firedancer is listening on via `AF_XDP`. Traffic targeting any other applications (e.g. an SSH or HTTP server running on the system) passes through as usual. The XDP program is unloaded when the Firedancer process exits. `AF_XDP` requires `CAP_SYS_ADMIN` and `CAP_NET_RAW` privileges. This is one of the reasons why Firedancer requires root permissions on Linux. WARNING Packets received and sent via `AF_XDP` will not appear under standard network monitoring tools like `tcpdump`. --- # Initializing | Firedancer [Skip to content](https://docs.firedancer.io/guide/initializing.html#VPContent) On this page Initializing [​](https://docs.firedancer.io/guide/initializing.html#initializing) ================================================================================== Overview [​](https://docs.firedancer.io/guide/initializing.html#overview) -------------------------------------------------------------------------- The `fdctl configure` command is used to setup the host operator system so Firedancer can run correctly. It does the following: * **hugetlbfs** Reserves huge and gigantic pages for use by Firedancer. * **sysctl** Sets required kernel parameters. * **hyperthreads** Checks hyperthreaded pair for critical CPU cores. * **bonding** Prepares bonded network devices for XDP networking. * **ethtool-channels** Configures the number of channels on the network device. * **ethtool-offloads** Modify offload feature flags on the network device. * **ethtool-loopback** Disable tx-udp-segmentation on the loopback device. The `hugetlbfs` configuration must be performed every time the system is rebooted, to remount the `hugetlbfs` filesystems, as do `sysctl`, `ethtool-channels` and `ethtool-offloads` to reconfigure the networking device. The configure command is run like `fdctl configure ...` where `mode` is one of: * `init` Configures the provided stages if they are not already configured. * `check` Check if each stage is already configured. The command will exit with an error code if they are not. `check` never requires privileges and will not make any changes to the system. * `fini` Unconfigure (reverse) the stage if it is reversible. `stage` can be one or more of `hugetlbfs`, `sysctl`, `hyperthreads`, `bonding`, `ethtool-channels`, `ethtool-offloads`, `ethtool-loopback`, and `snapshots` and these stages are described below. You can also use the stage `all` which will configure everything. Stages have different privilege requirements, which you can see by trying to run the stage without privileges. The `check` mode never requires privileges, and the `init` mode will only require privileges if it needs to actually change something. hugetlbfs [​](https://docs.firedancer.io/guide/initializing.html#hugetlbfs) ---------------------------------------------------------------------------- The `hugetlbfs` stage is used to reserve `huge` (2MiB) and `gigantic` (1GiB) memory pages from the Linux kernel for use by Firedancer. See also the [kernel documentation](https://docs.kernel.org/admin-guide/mm/hugetlbpage.html) of these pages. Almost all memory in Firedancer is allocated out of these pages for performance reasons. This is a two step process. First, the number of `huge` and `gigantic` pages available on the entire system is increased in the kernel by increasing `/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages` until the `free_hugepages` value is high enough for all the memory needs of the validator. Once the pages have been reserved globally in the kernel pool, they are assigned specifically to Firedancer by creating a `hugetlbfs` mount at each of `/mnt/.fd/.gigantic/` and `/mnt/.fd/.huge` for gigantic and huge pages respectively. These paths can be configured in the TOML file under the `[hugetlbfs]` section. Lets run it: $ sudo fdctl configure init hugetlbfs NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist NOTICE hugetlbfs ... configuring NOTICE RUN: `mkdir -p /mnt/.fd/.huge` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=228589568` NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic` NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424` $ cat /proc/mounts | grep \\.fd none /mnt/.fd/.gigantic hugetlbfs rw,seclabel,relatime,pagesize=1024M,min_size=540092137472 0 0 none /mnt/.fd/.huge hugetlbfs rw,seclabel,relatime,pagesize=2M,min_size=95124124 0 0 This stage requires root privileges, and cannot be performed with capabilities. If the required hugetlbfs mounts are already present, with at least the amount of memory reserved that we required then the `init` mode does nothing and the `check` mode will return successfully without requiring privileges. The `fini` mode will unmount the two filesystems, and remove them from `/mnt/.fd/`, although it will leave the `/mnt/.fd/` directory in place. The `fini` mode will not succeed if memory from the mounts is mapped into a running process. If fini succeeds, the huge and gigantic pages that Firedancer had reserved will be returned to the kernel global pool so they can be used by other programs, but the global pool size will not be decreased, even if it was earlier increased during `init`. TIP The `hugetlbfs` step should be run immediately when the system is booted. If run later, it may fail because the operating system memory is fragmented and a large contiguous block cannot be reserved. sysctl [​](https://docs.firedancer.io/guide/initializing.html#sysctl) ---------------------------------------------------------------------- It is suggested to run Firedancer with certain kernel parameters tuned for best performance. The `sysctl` stage will check and configure these parameters. The stage will only increase values to meet the minimum, and will not decrease them if the minimum is already met. | Sysctl | Minimum | Required | Description | | --- | --- | --- | --- | | /proc/sys/vm/max\_map\_count | 1000000 | Yes | Agave accounts database requires mapping many files. | | /proc/sys/fs/file-max | 1024000 | Yes | Agave accounts database requires opening many files. | | /proc/sys/fs/nr\_open | 1024000 | Yes | Agave accounts database requires opening many files. | | /proc/sys/net/ipv4/conf/lo/rp\_filter | 2 | Yes | If sending QUIC transactions to Firedancer over loopback, this must be enabled to receive a response. Otherwise Linux will drop response packets due to limitations in the kernel eBPF networking stack. The sendTransaction RPC call will send over loopback. | | /proc/sys/net/ipv4/conf/lo/accept\_local | 1 | Yes | If sending QUIC transactions to Firedancer over loopback, this must be enabled to receive a response. Otherwise Linux will drop response packets due to limitations in the kernel eBPF networking stack. The sendTransaction RPC call will send over loopback. | | /proc/sys/net/core/bpf\_jit\_enable | 1 | No | Firedancer uses BPF for kernel bypass networking. BPF JIT makes this faster. | | /proc/sys/kernel/numa\_balancing | 0 | No | Firedancer assigns all memory to the right NUMA node, and rebalancing will make the system slower. | Sysctls that are not required will produce a warning if they are not set correctly, but configuration will proceed and exit normally. The `init` mode requires either `root` privileges, or to be run with `CAP_SYS_ADMIN`. The `fini` mode does nothing and kernel parameters will never be reduced or changed back as a result of running `configure`. hyperthreads [​](https://docs.firedancer.io/guide/initializing.html#hyperthreads) ---------------------------------------------------------------------------------- Most work in Firedancer can be scaled with the number of CPU cores, but there are two jobs (tiles) which must run serially on a single core: * **pack** Responsible for scheduling transactions for execution when we are leader. * **poh** Performs repeated `sha256` hashes, and periodically stamps these hashes into in-progress blocks when we are leader. Because any interruption, context switch, or sharing of the CPU core that these jobs run on could cause skipped leader slots or unfull blocks, Firedancer expects them to get a dedicated core. This means on machines with a hyperthreaded CPU, the hyperthreaded pair of these tiles should be switched to offline. This stage looks to see if the CPU is hyperthreaded, and will print a warning for the operator if the pair of these tiles are used or online. A typical warning on a hyperthreaded system with `auto` layout looks like this: $ sudo fdctl configure init hyperthreads WARNING pack cpu 5 has hyperthread pair cpu 29 which should be offline. Proceeding but performance may be reduced. WARNING poh cpu 9 has hyperthread pair cpu 33 which should be offline. Proceeding but performance may be reduced. When using the `auto` layout, Firedancer will ensure no other tiles are assigned to run on the hyperthread pairs, but if using a manual layout, it is possible to assign another tile to the pair. This stage has no dependencies on any other stage, but it is dependent on the topology specified in your configuration. It is recommended that you turn off the CPUs specified in the warning for optimal performance. bonding [​](https://docs.firedancer.io/guide/initializing.html#bonding) ------------------------------------------------------------------------ Validator hosts commonly use bonded network devices to increase bandwidth and fault tolerance. Common Linux distributions pick defaults for bonding driver configuration that are incompatible with the XDP mode of some network drivers. When attaching XDP to a network device part of such an incorrectly configured bond, the bonding driver may erroneously consider that device as failed, leading to connectivity loss. Firedancer reconfigures bonding driver parameters to tolerate brief downtime when configuring XDP networking. This is done by increasing the `miimode`, `downdelay`, and `peer_notif_delay` timeout parameters in `/sys/class/net/bond0/bonding/` to five seconds. $ sudo fdctl configure init bonding NOTICE bonding ... unconfigured ... /sys/class/net/bond0/bonding/miimon is 200ms, want at least 5000ms NOTICE bonding ... configuring NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/miimon` NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/downdelay` NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/peer_notif_delay` NOTICE bonding ... done $ cat /sys/class/net/bond0/bonding/miimon 5000 Changing device settings with `bonding` requires root privileges, and cannot be performed with capabilities. ethtool-channels [​](https://docs.firedancer.io/guide/initializing.html#ethtool-channels) ------------------------------------------------------------------------------------------ In addition to XDP, Firedancer uses receive side scaling (RSS) to improve network performance. This uses functionality of modern NICs to steer packets to different queues to distribute processing among CPUs. See the [kernel documentation](https://docs.kernel.org/networking/scaling.html) for more information. In Firedancer, each `net` tile serves just one network queue, so the `ethtool-channels` stage will modify the network device `[net.interface]` configuration such that all packets needed by Firedancer are steered to the proper queue(s). There are three modes, selectable in your configuration, that govern this behavior: * **simple** mode modifies the combined channel count of the configured network device to be the same as the number of `net` tiles, `[layout.net_tile_count]`. If your NIC does not support the required number of queues, you will need to reduce the number of `net` tiles, potentially down to one for NICs which don't support queues at all. This is the default mode and should work for all network devices. Because the queue count is reduced system-wide, not solely for Firedancer, this can have a negative performance impact on non-Firedancer network traffic. * **dedicated** mode reserves a dedicated hardware queue for each `net` tile. This is the more advanced mode and may not work with all network devices. By modifying the RXFH indirection table and installing ntuple rules, Firedancer traffic is directed onto the dedicated queues and all other traffic is sharded amongst the rest. This has a performance benefit for both Firedancer and non-Firedancer traffic. * **auto** mode attempts to initialize the device in dedicated mode and automatically falls back to simple mode if any failure occurs. The command run by the stage in simple mode is similar to running `ethtool --set-channels combined ` but it also supports bonded devices. We can check that it worked: $ sudo fdctl configure init ethtool-channels NOTICE ethtool-channels ... unconfigured ... device `ens3f0` does not have right number of channels (got 1 but expected 2) NOTICE ethtool-channels ... configuring NOTICE ethtool-channels ... RUN: `ethtool --set-channels ens3f0 combined 2` $ ethtool --show-channels ens3f0 Channel parameters for ens3f0: Pre-set maximums: RX: 64 TX: 64 Other: 1 Combined: 64 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 2 The stage only needs to be run once after boot but before running Firedancer. It has no dependencies on any other stage, although it is dependent on the number of `net` tiles in your configuration. Changing device settings with `ethtool-channels` requires root privileges, and cannot be performed with capabilities. ethtool-offloads [​](https://docs.firedancer.io/guide/initializing.html#ethtool-offloads) ------------------------------------------------------------------------------------------ XDP is incompatible with a feature of network devices called `rx-udp-gro-forwarding`. This feature must be disabled for Firedancer to work. GRE segmentation offload is also disabled. The command run by the stage is similar to running `ethtool --offload off` but it also supports bonded devices. We can check that it worked: $ sudo fdctl configure init ethtool-offloads NOTICE ethtool-offloads ... unconfigured ... device `ens3f0np0` has generic-receive-offload enabled. Should be disabled NOTICE ethtool-offloads ... configuring NOTICE RUN: `ethtool --offload ens3f0np0 generic-receive-offload off` NOTICE RUN: `ethtool --features ens3f0np0 tx-gre-segmentation off` NOTICE RUN: `ethtool --offload lo generic-receive-offload off` $ ethtool --show-offload ens3f0np0 | grep generic-receive-offload generic-receive-offload: off $ ethtool --show-offload ens3f0np0 | grep tx-gre-segmentation tx-gre-segmentation: off The stage only needs to be run once after boot but before running Firedancer. It has no dependencies on any other stage. Changing device settings with `ethtool-offloads` requires root privileges, and cannot be performed with capabilities. ethtool-loopback [​](https://docs.firedancer.io/guide/initializing.html#ethtool-loopback) ------------------------------------------------------------------------------------------ XDP is incompatible with localhost UDP traffic using a feature called `tx-udp-segmentation`. This feature must be disabled when connecting Agave clients to Firedancer over loopback, or when using Frankendancer. The command run by the stage is `ethtool --offload lo tx-udp-segmentation off`. We can check that it worked: $ sudo fdctl configure init ethtool-loopback NOTICE ethtool-loopback ... unconfigured ... device `lo` has tx-udp-segmentation enabled. Should be disabled NOTICE ethtool-loopback ... configuring NOTICE ethtool-loopback ... RUN: `ethtool --offload lo tx-udp-segmentation off` $ ethtool --show-offload lo | grep tx-udp-segmentation tx-udp-segmentation: off The stage only needs to be run once after boot but before running Firedancer. It has no dependencies on any other stage. Changing device settings with `ethtool-loopback` requires root privileges, and cannot be performed with capabilities. snapshots [​](https://docs.firedancer.io/guide/initializing.html#snapshots) ---------------------------------------------------------------------------- When starting up, validators must load a snapshot to catch up to the current state of the blockchain. Snapshots are downloaded from other validator peers in the cluster and are stored to a snapshots directory. In init, the snapshots configure phase will create the snapshots directory if it does not exist. In fini, the snapshots configure phase will remove the snapshots directory recursively. NOTE The snapshots configure phase is only enabled in the Firedancer binary. --- # Performance Tuning | Firedancer [Skip to content](https://docs.firedancer.io/guide/tuning.html#VPContent) On this page Performance Tuning [​](https://docs.firedancer.io/guide/tuning.html#performance-tuning) ======================================================================================== Overview [​](https://docs.firedancer.io/guide/tuning.html#overview) -------------------------------------------------------------------- The Firedancer validator is composed of a handful of threads, each performing one of fifteen distinct jobs. Some jobs only need one thread to do them, but certain jobs require many threads performing the same work in parallel. Each thread is given a CPU core to run on, and threads take ownership of the core: never sleeping or letting the operating system use it for another purpose. The combination of a job, and the thread it runs on, and the CPU core it is assigned to is called a tile. The fifteen kinds of tile are, | Tile | Description | | --- | --- | | `net` | Sends and receives network packets from the network device | | `quic` | Receives transactions from clients, performing all connection management and packet processing to manage and implement the QUIC protocol | | `verify` | Verifies the cryptographic signature of incoming transactions, filtering invalid ones | | `dedup` | Checks for and filters out duplicated incoming transactions | | `pack` | Collects incoming transactions and smartly schedules them for execution when we are leader | | `bank` | Executes transactions that have been scheduled when we are leader | | `poh` | Continuously hashes in the background, and mixes the hash in with executed transactions to prove passage of time | | `shred` | Distributes block data to the network when leader, and receives and retransmits block data when not leader | | `store` | Receives block data when we are leader, or from other nodes when they are leader, and stores it locally in a database on disk | | `metric` | Collects monitoring information about other tiles and serves it on an HTTP endpoint | | `sign` | Holds the validator private key, and receives and responds to signing requests from other tiles | | `resolv` | Resolves address lookup tables before transactions are scheduled | | `diag` | Counts context switches and diagnostic information of other tiles | | `plugin` | Provides data to the `gui` tile | | `gui` | Receives data from the validator and serves an HTTP endpoint to clients to view it | These tiles communicate with each other via shared memory queues. The work each tile performs and how they communicate with each other is fixed, but the count of each tile kind and which CPU cores they are assigned to is set by your configuration, and this is the primary way to tune the performance of Firedancer. Configuration [​](https://docs.firedancer.io/guide/tuning.html#configuration) ------------------------------------------------------------------------------ The default configuration provided if no options are specified is given in the [`default.toml`](https://github.com/firedancer-io/firedancer/blob/main/src/app/fdctl/config/default.toml) file: default.toml toml [layout] affinity = "1-16" agave_affinity = "17-31" net_tile_count = 1 quic_tile_count = 1 verify_tile_count = 4 bank_tile_count = 2 shred_tile_count = 1 Only `net`, `quic`, `verify`, `bank`, and `shred` tile counts are configurable. There may be 0 or 1 `plugin` and `gui` tiles if the GUI is disabled or enabled. The rest are fixed at one thread each. The assignment of tiles to CPU cores is determined by the `affinity` string, which is documented fully in the [`default.toml`](https://github.com/firedancer-io/firedancer/blob/main/src/app/fdctl/config/default.toml) file itself. The Frankendancer validator currently starts an Agave process to perform functionality like replay, gossip, and repair that is not yet implemented in Firedancer. The `agave_affinity` string determines the CPU cores that are given to the threads of this Agave process. The following table shows the performance of the adjustable tiles on an Intel Icelake core, along with some performance notes and recommendations for `mainnet-beta`, | Tile | Default | Notes | | --- | --- | --- | | `net` | 1 | Handles >1M TPS per tile. Designed to scale out for future network conditions, but there is no need to run more than 1 net tile at the moment on `mainnet-beta` | | `quic` | 1 | Handles >1M TPS per tile. Designed to scale out for future network conditions, but there is no need to run more than 1 QUIC tile at the moment on `mainnet-beta` | | `verify` | 4 | Handles 20-40k TPS per tile. Recommend running many verify tiles, as signature verification is the primary bottleneck of the application | | `bank` | 4 | Handles 20-40k TPS per tile, with diminishing returns from adding more tiles. Designed to scale out for future network conditions, but 4 tiles is enough to handle current `mainnet-beta` conditions. Can be increased further when benchmarking to test future network performance | | `shred` | 1 | Throughput is mainly dependent on cluster size, 1 tile is enough to handle current `mainnet-beta` conditions. In benchmarking, if the cluster size is small, 1 tile can handle >1M TPS | Testing [​](https://docs.firedancer.io/guide/tuning.html#testing) ------------------------------------------------------------------ Firedancer includes a simple benchmarking tool for measuring the transaction throughput of the validator when it is leader, in transactions per second (TPS). In practice, the Solana network performance is limited by two factors that are unrelated to what this tool measures: * The replay performance of the slowest nodes in the network, and if they can keep up * The consensus limits on block size and data size In particular, consensus limits on the Solana protocol limit the network strictly to around 81,000 TPS. But the tool can be useful for testing local affinity and layout configurations. The benchmark runs on a single machine and performs the following: 1. A new genesis is created, and set of accounts are pre-funded 2. A set of CPU cores is assigned to generating and signing simple transactions using these accounts as fast as possible 3. Another set of CPU cores is assigned to sending these transfers via QUIC over loopback to the locally running validator 4. Around once a second, an RPC call is made to get the total count of transactions that have executed on the chain, and this information is printed to the console The benchmark is currently quite synthetic, as it only measures single node performance, in an idealized case where all transactions are non-conflicting. Running [​](https://docs.firedancer.io/guide/tuning.html#running) ------------------------------------------------------------------ The benchmark command is part of the `fddev` development binary, which can be built with `make -j fddev`. With the binary in hand, we can run our benchmark, here it will be on a 32 physical core AMD EPYC 7513: sh $ lscpu Architecture: x86_64 CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD Model name: AMD EPYC 7513 32-Core Processor $ ./build/native/gcc/bin/fddev bench NOTICE main configure.c(106): kill ... configuring NOTICE main configure.c(102): hugetlbfs ... already valid NOTICE main configure.c(102): sysctl ... already valid [ ... snip ... ] NOTICE bencho:0 fd_bencho.c(137): 63373 txn/s NOTICE bencho:0 fd_bencho.c(137): 63340 txn/s We have not provided a configuration file to the bench command, so it is using the stock configuration from `default.toml` and reaching around 63,000 TPS. Let's take a look at the performance with the `monitor` command and see if we can figure out what's going on. $ ./build/native/gcc/bin/fddev monitor snapshot for 2024-07-29 16:44:59.066296419 GMT+00 tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish ---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+---------- net | 2555236 | - | - | run( run) | -( -) | 0( +0) | 35.511 | 0.000 | 54.233 | 0.000 | 0.000 | 0.000 | 10.255 | 0.000 quic | 2555236 | - | - | run( run) | -( -) | 0( +0) | 0.405 | 0.000 | 95.501 | 0.000 | 0.000 | 0.000 | 0.000 | 4.094 verify | 2555236 | - | - | run( run) | -( -) | 0( +0) | 0.081 | 0.000 | 0.047 | 0.000 | 0.000 | 0.084 | 0.000 | 99.788 verify | 2555236 | - | - | run( run) | -( -) | 0( +0) | 0.082 | 0.000 | 0.036 | 0.000 | 0.000 | 0.078 | 0.000 | 99.804 verify | 2555236 | - | - | run( run) | -( -) | 0( +0) | 0.123 | 0.000 | 0.000 | 0.000 | 0.000 | 0.111 | 0.000 | 99.765 [ ... snip ... ] If we narrow in on just the verify tiles we can see the problem: all of the verify tiles are completely busy processing incoming transactions, and so additional transactions are being dropped. Here `% finish` indicates the percentage of time the tile is occupied doing work, while `overnp cnt` indicates that the tile is being overrun by the quic tile and dropping transactions.  tile | sig | in backp | backp cnt | % backp | % finish ---------+------------+----------+---------------+----------+---------- verify | run( run) | -( -) | 0( +0) | 0.000 | 99.788 verify | run( run) | -( -) | 0( +0) | 0.000 | 99.804 verify | run( run) | -( -) | 0( +0) | 0.000 | 99.765 verify | run( run) | -( -) | 0( +0) | 0.000 | 99.745 link | tot TPS | ovrnp cnt | ovrnr cnt | slow cnt | tx seq ------------------+----------+---------------------+---------------------+---------------------+------------------- quic->verify | 17.2K | 9( +1) | 0( +0) | 0( +0) | 507134( +7149) quic->verify | 17.2K | 9( +1) | 0( +0) | 0( +0) | 507134( +7149) quic->verify | 11.8K | 15( +0) | 0( +0) | 0( +0) | 507134( +7149) quic->verify | 17.0K | 9( +1) | 0( +0) | 0( +0) | 507134( +7149) This configuration is not ideal. With some tuning to increase the number of verify tiles, and a few other changes we can try to achieve a higher TPS rate, bench-zen3-32core.toml toml [ledger] # Place the ledger in memory rather than on disk so that writing the # ledger is not a performance bottleneck path = "/data/shm/{name}/ledger" [layout] # We will need a lot of verify tiles, and a few more bank tiles to be # able to execute at higher TPS rates. Increase their core counts, and # assign the tiles to cores. We only need 1 shred tile, since there is # only 1 node in the cluster it can handle a high TPS rate by itself affinity = "14-57,f1" agave_affinity = "58-63" verify_tile_count = 30 bank_tile_count = 6 shred_tile_count = 1 [development.genesis] # The default amount of accounts to use for the benchmark is 1024, but # to reach higher transaction throughput we need more accounts so that # more transfers can be handled in parallel fund_initial_accounts = 32768 [development.bench] # benchg tiles are used to generate and sign transactions in the # benchmarking tool, we are going to need more of them to test higher # TPS rate benchg_tile_count = 12 # benchs tiles are for sending the transactions to Firedancer over # loopback, and we will need an extra one of these as well benchs_tile_count = 2 # Assign these benchg, benchs (and the bencho tile which orchestrates # the benchmarking) to some CPU cores. The bencho assignment is # floating as it is not performance sensitive affinity = "f1,0-13" # The Solana protocol consensus limits restrict the benchmark to # around 81,000 TPS. We have special options to increase these limits # for testing and benchmarking larger_max_cost_per_block = true larger_shred_limits_per_block = true [rpc] # Tracking certain transaction history and metadata to serve RPC # requests is expensive and can slow down our validator, turn this # functionality off transaction_history = false extended_tx_metadata_storage = false Now try running again, $ ./build/native/gcc/bin/fddev bench --config src/app/fdctl/config/bench-zen3-32core.toml [ ... snip ... ] NOTICE bencho:0 fd_bencho.c(137): 191180 txn/s NOTICE bencho:0 fd_bencho.c(137): 308027 txn/s NOTICE bencho:0 fd_bencho.c(137): 12389 txn/s NOTICE bencho:0 fd_bencho.c(137): 89564 txn/s NOTICE bencho:0 fd_bencho.c(137): 92380 txn/s We start out with a higher TPS rate but it quickly falls back to around 90k TPS. We can try to figure out why by running the `monitor` command. $ ./build/native/gcc/bin/fddev monitor --config src/app/fdctl/config/bench-zen3-32core.toml snapshot for 2024-07-29 16:44:59.066296419 GMT+00 tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish ---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+---------- net | 2239972 | - | - | run( run) | -( -) | 0( +0) | 48.576 | 0.000 | 51.424 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 quic | 2239972 | - | - | run( run) | -( -) | 0( +0) | 0.553 | 0.000 | 88.238 | 0.000 | 0.000 | 0.000 | 0.000 | 11.208 verify | 2239972 | - | - | run( run) | -( -) | 0( +0) | 0.199 | 0.000 | 0.000 | 0.000 | 0.000 | 0.804 | 0.000 | 98.997 verify | 2239972 | - | - | run( run) | -( -) | 0( +0) | 0.177 | 0.000 | 0.000 | 0.000 | 0.000 | 0.796 | 0.000 | 99.027 verify | 2239972 | - | - | run( run) | -( -) | 0( +0) | 0.177 | 0.000 | 0.000 | 0.000 | 0.000 | 0.796 | 0.000 | 99.027 [ ... snip ... ] The culprit is visible in the output, which will be clearer if we filter it down to the relevant information,  tile | sig | in backp | backp cnt | % backp | % finish ---------+------------+----------+---------------+----------+---------- poh | run( run) | err(err) | 1002( +22) | 76.918 | 7.892 shred | run( run) | err(err) | 1462( +22) | 95.857 | 3.386 store | run( run) | -( -) | 0( +0) | 0.000 | 99.973 Here we see what is happening. The blockstore is completely busy spending 99.973% of its time storing data, while the PoH and shred tiles are in back-pressure waiting for the blockstore to catch up. The blockstore is an Agave component built on RocksDB that is not rewritten as part of Frankendancer. bench-zen3-32core.toml toml [development.bench] disable_blockstore_from_slot = 1 We can disable the blockstore specifically for benchmarking, to show the performance of just the Firedancer components in the leader pipeline. Now we can run one more time and see a reasonably good value for the TPS throughput of Firedancer on this machine. $ ./build/native/gcc/bin/fddev bench --config src/app/fdctl/config/bench-zen3-32core.toml [ ... snip ... ] NOTICE bencho:0 fd_bencho.c(137): 272840 txn/s NOTICE bencho:0 fd_bencho.c(137): 278380 txn/s NOTICE bencho:0 fd_bencho.c(137): 268246 txn/s --- # Monitoring | Firedancer [Skip to content](https://docs.firedancer.io/guide/monitoring.html#VPContent) On this page Monitoring [​](https://docs.firedancer.io/guide/monitoring.html#monitoring) ============================================================================ Agave CLI [​](https://docs.firedancer.io/guide/monitoring.html#agave-cli) -------------------------------------------------------------------------- Because Frankendancer runs the Agave validator, it can be monitored with the standard Agave command line tools. First make sure you have built the `solana` CLI binary: sh $ make solana Finished release-with-debug [optimized + debuginfo] target(s) in 0.44s Similar to `fdctl`, the compiled binary will be placed in `./build/native/gcc/bin` by default. NOTE Many commands require RPC to be enabled on the validator, see the [configuring](https://docs.firedancer.io/guide/configuring.html) guide for more information. **gossip:** check if the validator has joined gossip sh $ solana -ut gossip IP Address | Identity | Gossip | TPU | RPC Address | Version | Feature Set ----------------+----------------------------------------------+--------+-------+-----------------------+------------+---------------- 74.118.136.198 | 2CY8VXH2jummjSmwcusSj2jGMiaHE4eo7WQ9LScxykvt | 8001 | 9001 | 74.118.136.198:8899 | 0.106.11814| 4215500110 **catchup:** check if the validator is caught up sh $ solana -ut catchup --our-localhost ⠤ 1 slot(s) behind (us:123 them:124) **validators:** ensure the validator is voting sh $ solana -ut validators Identity Vote Account Commission Last Vote Root Slot Skip Rate Credits Version Active Stake 2CY8VXH2jummjSmwcusSj2jGMiaHE4eo7WQ9LScxykvt uhiGpdNqcqPGzYuRfVxjiHQWKKJPwKRSaPiXXxwSy9K 100% 279227304 ( -1) 279227273 ( -1) 0.00% 54287 0.106.11814 70100.022292880 SOL (0.03%) **block-production:** ensure the validator is producing blocks sh $ solana -ut block-production Identity Leader Slots Blocks Produced Skipped Slots Skip Rate 2CY8VXH2jummjSmwcusSj2jGMiaHE4eo7WQ9LScxykvt 16 16 0 0.00% NOTE You can also use the `agave-validator --ledger monitor` command with Frankendancer. For that, you need to build the `agave-validator` binary from the `agave` repository. Metrics [​](https://docs.firedancer.io/guide/monitoring.html#metrics) ---------------------------------------------------------------------- Firedancer exposes a large set of prometheus compatible metrics at a HTTP endpoint, by default on port `7999` but this is configurable in the TOML file. sh $ curl http://localhost:7999/metrics # HELP tile_pid The process ID of the tile. # TYPE tile_pid gauge tile_pid{kind="net",kind_id="0"} 1108973 tile_pid{kind="quic",kind_id="0"} 1108975 tile_pid{kind="verify",kind_id="0"} 1108978 See the [metrics API documentation](https://docs.firedancer.io/api/metrics.html) for more information on the available data. Live monitoring [​](https://docs.firedancer.io/guide/monitoring.html#live-monitoring) -------------------------------------------------------------------------------------- Firedancer ships with a monitoring tool included in `fdctl`, which you can run on the same host as the running validator to view tile and other performance information. sh $ fdctl monitor --config ~/config.toml snapshot for 2024-06-25 17:32:25.795577630 GMT+00 tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish ---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+---------- net | 1108973 | - | - | run( run) | -( -) | 0( +0) | 40.118 | 0.000 | 59.882 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 quic | 1108975 | - | - | run( run) | -( -) | 0( +0) | 0.325 | 0.000 | 99.675 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 verify | 1108978 | - | - | run( run) | -( -) | 0( +0) | 0.496 | 0.000 | 99.504 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 [...] You can also view the Firedancer GUI in a browser. Refer to the [configuration section](https://docs.firedancer.io/guide/configuring.html#gui) to enable it. --- # Configuring | Firedancer [Skip to content](https://docs.firedancer.io/guide/configuring.html#VPContent) On this page Configuring [​](https://docs.firedancer.io/guide/configuring.html#configuring) =============================================================================== Overview [​](https://docs.firedancer.io/guide/configuring.html#overview) ------------------------------------------------------------------------- Firedancer is configured via a [TOML](https://toml.io/en/) file. Almost all options have a recommended default value that is set automatically by Firedancer, and an operator needs only to specify values for options they wish to change. The full list of options is as specified in the [`default.toml`](https://github.com/firedancer-io/firedancer/blob/main/src/app/fdctl/config/default.toml) file is documented below. MIGRATING The Agave validator is configured with command line options like `--identity identity.json --rpc-port 8899`. When migrating your scripts, these command line options will need to move to the corresponding configuration option in the TOML file. The full list of available options and their defaults are documented below. An example TOML file overriding select options needed for a new validator on testnet might look like: testnet.toml toml user = "firedancer" [gossip] entrypoints = [\ "entrypoint.testnet.solana.com:8001",\ "entrypoint2.testnet.solana.com:8001",\ "entrypoint3.testnet.solana.com:8001",\ ] [consensus] expected_genesis_hash = "4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY" known_validators = [\ "5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on",\ "dDzy5SR3AXdYWVqbDEkVFdvSPCtS9ihF5kJkHCtXoFs",\ "Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN",\ "eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ",\ "9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv",\ ] [rpc] port = 9099 full_api = true private = true [consensus] identity_path = "/home/firedancer/validator-keypair.json" vote_account_path = "/home/firedancer/vote-keypair.json" Once your configuration file is created you can use it by either setting the `FIREDANCER_CONFIG_TOML` environment variable, or by passing it to your command with the `--config` option. NOTE The same configuration file must be supplied to all commands, especially when configuring and later running the validator. Using a different file for different commands may cause them to fail. Logging [​](https://docs.firedancer.io/guide/configuring.html#logging) ----------------------------------------------------------------------- By default Firedancer will maintain two logs. One permanent log which is written to a file, and an ephemeral log for fast visual inspection which is written to stderr. The Agave runtime and consensus components also output logs which are a part of the Firedancer's logs. You can increase the ephemeral log output in the configuration TOML. toml [log] level_stderr = "INFO" Layout [​](https://docs.firedancer.io/guide/configuring.html#layout) --------------------------------------------------------------------- One way that Firedancer is fast is that it pins a dedicated thread to each CPU core on the system. Each thread can do one specific kind of work, for example, a `verify` tile can verify the signatures of incoming transactions. Tiles are connected together in a graph to form an efficient pipeline for processing transactions. WARNING Each tile needs a dedicated CPU core and it will be saturated at 100% utilization. The Agave process will run on the cores under the `agave_affinity` and this should not overlap with tile cores. The configuration file has options for how many of each kind of tile should be started. toml [layout] affinity = "1-18" quic_tile_count = 2 verify_tile_count = 4 bank_tile_count = 4 agave_affinity = "19-31" It is suggested to run as many tiles as possible and tune the tile counts for maximum system throughput so that the Solana network can run faster. There are some example tuned configurations in the `src/app/fdctl/config/` folder to work from. TIP You can set the `affinity` and `agave_affinity` to `"auto"`. This will let Firedancer detect the topology of the system and automatically configure the assignment of the tiles to cpu cores. This setting is the default option. GUI [​](https://docs.firedancer.io/guide/configuring.html#gui) --------------------------------------------------------------- Firedancer has a GUI that can provide a lot of useful information about the validator. You can enable it by adding the following in the config TOML: toml [tiles.gui] enabled = true By default it will listen on `127.0.0.1:80`. You can configure this using the `[tile.gui.gui_listen_address]` and `[tiles.gui.gui_listen_port]` options. Options [​](https://docs.firedancer.io/guide/configuring.html#options) ----------------------------------------------------------------------- The list of all available configuration options and their default values is provided below. You only need to override options which you wish to change. default.toml toml # Name of this Firedancer instance. This name serves as a unique token # so that multiple Firedancer instances can run side by side without # conflicting when they need to share a system or kernel namespace. # When starting a Firedancer instance, it will potentially load, reset, # or overwrite any state created by a prior or currently running # instance with the same name. name = "fd1" # The operating system user to permission data and run Firedancer as. # Firedancer needs to start privileged, either with various capabilities # or as root, so that it can configure kernel bypass networking. Once # this configuration has been performed, the process will enter a highly # restrictive sandbox, drop all privileges, and switch to the user given # here. When running the configuration steps of `fdctl configure` data # will be permissioned so that is it writable for this user and not the # user which is performing the configuration. # # Firedancer requires nothing from this user, and it should be as # minimally permissioned as is possible. It is suggested to run # Firedancer as a separate user from other processes on the machine so # that they cannot attempt to send signals, ptrace, or otherwise # interfere with the process. You should under no circumstances use a # superuser or privileged user here, and Firedancer will not allow you # to use the root user. It is also not a good idea to use a user that # has access to `sudo` or has other entries in the sudoers file. # # Firedancer will automatically determine a user to run as if none is # provided. By default, the user is determined by the following # sequence: # # 1. The `SUDO_USER`, `LOGNAME`, `USER`, `LNAME`, or `USERNAME` # environment variables are checked in this order, and if one of # them is set that user is used # 2. The `/proc/self/loginuid` file is used to determine the UID, and # the username is looked up in nss (the name service switch). # # This means if running as sudo, the user will be the terminal user # which invoked sudo, not the root user. user = "" # Absolute directory path to place scratch files used during setup and # operation. The ledger and accounts databases will also be placed in # here by default, although that can be overridden by other options. # # Two substitutions will be performed on this string. If "{user}" is # present it will be replaced with the user running Firedancer, as # above, and "{name}" will be replaced with the name of the Firedancer # instance. scratch_directory = "/home/{user}/.firedancer/{name}" # Port range used for various incoming network listeners, in the form # `-` inclusive. The range used includes min, but # not max [min, max). Ports are used for receiving transactions and\ # votes from clients and other validators.\ #\ # For Firedancer, ports are assigned statically in later parts of this\ # configuration file, and this option is passed to the Agave\ # client with the `--dynamic-port-range` argument. Agave will use\ # this to determine port locations for services not yet rewritten as\ # part of Firedancer, including gossip and RPC. This port range should\ # NOT overlap with any of the static ports used by Firedancer below.\ dynamic_port_range = "8900-9000"\ \ # Firedancer logs to two places by default: stderr and a logfile.\ # stdout is not used for logging, and will only be used to print command\ # output or boot errors. Messages to "stderr" are abbreviated and not\ # as fully detailed as those to the log file. The log file is intended\ # for long term archival purposes. The log levels mirror the Linux\ # syslog levels, which are, from lowest to highest priority:\ #\ # - DEBUG Development and diagnostic messages.\ # - INFO Less important informational notice.\ # - NOTICE More important informational notice.\ # - WARNING Unexpected condition, shouldn't happen. Should be\ # investigated.\ # - ERR Kills Firedancer. Routine error, likely programmer error.\ # - CRIT Kills Firedancer. Critical errors.\ # - ALERT Kills Firedancer. Alert requiring immediate attention.\ # - EMERG Kills Firedancer. Emergency requiring immediate\ # attention, security or risk issue.\ #\ # Default behaviors are:\ #\ # - DEBUG messages are not written to either stream.\ # - INFO messages are written in detailed form to the log file.\ # - NOTICE is INFO + messages are written in summary form to\ # stderr.\ # - WARNING is NOTICE + the log file is flushed to disk.\ # - ERR and above are WARNING + the program will be exited with an\ # error code of 1.\ #\ # All processes in Firedancer share one log file, and they all inherit\ # STDERR and STDOUT from the launcher. An example log message would\ # look something like:\ #\ # NOTICE 01-23 04:56:07.890123 45678 f0 0 src/file.c(901): 1 is the loneliest number\ #\ # to the ephemeral log (stderr) and log something like:\ #\ # NOTICE 2023-01-23 04:56:07.890123456 GMT-06 45678:45678 user:host:f0 app:thread:0 src/file.c(901)[func]: 1 is the loneliest number\ #\ # to the permanent log (log file).\ #\ # Firedancer does not support rotating the log file in response to\ # SIGUSR1 or SIGUSR2. Instead, it will silently swallow and ignore\ # these signals. To perform log rotation, it is suggested to use a\ # mechanism like the `copytruncate` directive of `logrotate`.\ [log]\ # Absolute file path of where to place the log file. It will be\ # appended to, or created if it does not already exist. The\ # shortened ephemeral log will always be written to stderr.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # If no path is provided, the default is to place the log file in\ # /tmp with a name that will be unique. If specified as "-", the\ # permanent log will be written to stdout.\ path = ""\ \ # Firedancer can colorize the stderr ephemeral log using ANSI escape\ # codes so that it looks pretty in a terminal. This option must be\ # one of "auto", "true", or "false". If set to "auto" stderr output\ # will be colorized if we can detect the terminal supports it. The\ # log file output will never be colorized.\ colorize = "auto"\ \ # The minimum log level which will be written to the log file. Log\ # levels lower than this will be skipped. Must be one of the levels\ # described above.\ level_logfile = "INFO"\ \ # The minimum log level which will be written to stderr. Log levels\ # lower than this will be skipped. Must be one of the levels\ # described above. This should be at least the same as the level\ # for the log file.\ level_stderr = "NOTICE"\ \ # The minimum log level which will immediately flush the log file to\ # disk. Must be one of the levels described above.\ level_flush = "WARNING"\ \ # The client supports sending health reports, and performance and\ # diagnostic information to a remote server for collection and analysis.\ # This reporting powers the Solana Validator Dashboard and is often used\ # by developers to monitor network health.\ [reporting]\ # A metrics environment string describing where to report the\ # diagnostic event data to. The options for public clusters are\ # described at https://docs.solanalabs.com/clusters/available,\ # and are:\ #\ # mainnet-beta:\ # "host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password"\ #\ # devnet:\ # "host=https://metrics.solana.com:8086,db=devnet,u=scratch_writer,p=topsecret"\ #\ # testnet:\ # "host=https://metrics.solana.com:8086,db=tds,u=testnet_write,p=c4fa841aa918bf8274e3e2a44d77568d9861b3ea"\ #\ # If no option is provided here, event reporting is disabled.\ #\ # This string is passed to the Agave client with the\ # `SOLANA_METRICS_CONFIG` environment variable.\ solana_metrics_config = ""\ \ # The ledger is the set of information that can be replayed to get back\ # to the current state of the chain. In Solana, it is considered a\ # combination of the genesis, and the recent unconfirmed blocks. The\ # accounts database (the current balance of all the accounts) is\ # information that is derived from the ledger.\ [ledger]\ # Absolute directory path to place the ledger. Firedancer currently\ # spawns a Agave validator to execute transactions that it\ # receives. If no ledger path is provided, it is defaulted to the\ # `ledger` subdirectory of the scratch directory above.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # The ledger path constructed here is passed to the Agave\ # client with the `--ledger` argument.\ path = ""\ \ # Absolute directory path to place the accounts database in. If\ # this is empty, it will default to the `accounts` subdirectory of\ # the ledger `path` above.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # This option is passed to the Agave client with the `--accounts` argument.\ accounts_path = ""\ \ # Maximum number of shreds to keep in root slots in the ledger\ # before discarding.\ #\ # The default is chosen to allow enough time for a validator to\ # download a snapshot from a peer and boot from it, and to make sure\ # that if a validator needs to reboot from its own snapshot, it has\ # enough slots locally to catch back up to where it was when it\ # stopped. It works out to around 400GB of space.\ #\ # This option is passed to the Agave client with the\ # `--limit-ledger-size` argument.\ limit_size = 200_000_000\ \ # If nonempty, enable an accounts index indexed by the specified\ # field. The account field must be one of "program-id",\ # "spl-token-owner", or "spl-token-mint". These options are passed\ # to the Agave client with the `--account-index` argument.\ account_indexes = []\ \ # If account indexes are enabled, exclude these keys from the index.\ # These options are passed to the Agave client with the\ # `--account-index-exclude-key` argument.\ account_index_exclude_keys = []\ \ # If account indexes are enabled, only include these keys in the\ # index. This overrides `account_index_exclude_keys` if specified\ # and that value will be ignored. These options are passed to the\ # Agave client with the `--account-index-include-key` argument.\ account_index_include_keys = []\ \ # If account indexes are enabled, they are flushed to disk\ # periodically. Otherwise, the entire index is kept in memory.\ # Setting this to false may reduce disk I/O strain. This option is\ # passed to the Agave client with the `--enable-accounts-disk-index`\ # argument.\ enable_accounts_disk_index = false\ \ # If account indexes are enabled, this is the absolute directory\ # path to place the accounts index in. If this is empty, it\ # will default to the `accounts_index` subdirectory of the ledger\ # `path` above. This option is passed to the Agave client with\ # the `--accounts-index-path` argument. Setting this will\ # automatically set the [ledger.enable_accounts_disk_index] to true.\ accounts_index_path = ""\ \ # Every epoch, the client calculates a hash of all the accounts\ # on-chain called the Epoch Accounts Hash. As a part of this\ # process, the client iterates over the entire accounts database\ # to calculate serialized hashes for every account which are then\ # stored in several files in a directory on disk.\ #\ # This option specifies the absolute directory path to place the\ # accounts hash cache data in. If this is empty, it will default\ # to the `accounts_hash_cache` subdirectory of the ledger `path`\ # above. This option is passed to the Agave client with the\ # `--accounts-hash-cache-path` argument.\ accounts_hash_cache_path = ""\ \ # Whether to use compression when storing snapshots. This option is\ # passed to the Agave client with the\ # `--snapshot-archive-format` argument.\ snapshot_archive_format = "zstd"\ \ # Refuse to start if saved tower state is not found. This option is\ # passed to the Agave client with the `--require-tower`\ # argument.\ require_tower = false\ \ [gossip]\ # Routable DNS name or IP address and port number to use to\ # rendezvous with the gossip cluster. These entrypoints are passed\ # to the Agave client with the `--entrypoint` argument.\ entrypoints = []\ \ # If true, checks at startup that at least one of the provided\ # entrypoints can connect to this validator on all necessary ports.\ # If it can't, then the validator will exit.\ #\ # This option is passed to the Agave client inverted with the\ # `--no-port-check` argument.\ port_check = true\ \ # The port number to use for receiving gossip messages on this\ # validator. This option is passed to the Agave client with\ # the `--gossip-port` argument. This argument is required. Agave\ # treats this as an optional argument, but the code that tries\ # to find a default will always fail.\ port = 8001\ \ [rpc]\ # If nonzero, enable JSON RPC on this port, and use the next port\ # for the RPC websocket. If zero, disable JSON RPC. This option is\ # passed to the Agave client with the `--rpc-port` argument.\ port = 0\ \ # If true, all RPC operations are enabled on this validator,\ # including non-default RPC methods for querying chain state and\ # transaction history. This option is passed to the Agave\ # client with the `--full-rpc-api` argument.\ full_api = false\ \ # If the RPC is private, the validator's open RPC port is not\ # published in the `solana gossip` command for use by others. This\ # option is passed to the Agave client with the\ # `--private-rpc` argument.\ private = false\ \ # The address to bind the RPC port. If no address is specified,\ # the default RPC bind address is 127.0.0.1 if [rpc.private] = true,\ # otherwise it's 0.0.0.0. This option is passed to the Agave client\ # with the `--rpc-bind-address` argument.\ bind_address = ""\ \ # RPC address for the validator to advertise publicly in gossip.\ # Useful for validators running behind a load balancer or proxy.\ # This option is passed to the Agave client\ # with the `--public-rpc-address` argument.\ public_address = ""\ \ # Enable historical transaction info over JSON RPC, including the\ # `getConfirmedBlock` API. This will cause an increase in disk\ # usage and IOPS. This option is passed to the Agave client\ # with the `--enable-rpc-transaction-history` argument.\ transaction_history = false\ \ # If enabled, include CPI inner instructions, logs, and return data\ # in the historical transaction info stored. This option is passed\ # to the Agave client with the\ # `--enable-extended-tx-metadata-storage` argument.\ extended_tx_metadata_storage = false\ \ # If true, use the RPC service of known validators only. This\ # option is passed to the Agave client with the\ # `--only-known-rpc` argument.\ only_known = true\ \ # If true, enable the unstable RPC PubSub `blockSubscribe`\ # subscription. This option is passed to the Agave client\ # with the `--rpc-pubsub-enable-block-subscription` argument.\ pubsub_enable_block_subscription = false\ \ # If true, enable the unstable RPC PubSub `voteSubscribe`\ # subscription. This option is passed to the Agave client\ # with the `--rpc-pubsub-enable-vote-subscription` argument.\ pubsub_enable_vote_subscription = false\ \ # If enabled, fetch historical transaction info from a BigTable\ # instance as a fallback to local ledger data when serving RPC\ # requests. The `GOOGLE_APPLICATION_CREDENTIALS` environment\ # variable must be set to access BigTable.\ #\ # This option is passed to the Agave client with the\ # `--enable-rpc-bigtable-ledger-storage` argument.\ bigtable_ledger_storage = false\ \ # The Agave client periodically takes and stores snapshots of the\ # chain's state. Other clients, especially as they bootstrap or catch\ # up to the head of the chain, may request a snapshot.\ [snapshots]\ # If false, all snapshots (both full and incremental) will not be\ # produced. This option is passed to the Agave client (inverted)\ # with the `--no-snapshots` argument.\ enabled = true\ \ # Enable incremental snapshots by setting this flag. This option is\ # passed to the Agave client (inverted) with the\ # `--no-incremental-snapshots` argument.\ incremental_snapshots = true\ \ # Set how frequently full snapshots are taken, measured in slots,\ # where one slot is about 400ms on production chains. It's\ # recommended to leave this to the default or to set it to the same\ # value that the known validators are using. Must be a multiple of\ # the incremental snapshot interval. This options is passed to the\ # Agave client with the `--full-snapshot-interval-slots` argument.\ full_snapshot_interval_slots = 25000\ \ # Set how frequently incremental snapshots are taken, measured in\ # slots. Must be a multiple of the accounts hash interval (which is\ # 100 by default). This options is passed to the Agave client with\ # the `--snapshot-interval-slots` argument.\ incremental_snapshot_interval_slots = 100\ \ # Set the maximum number of full snapshot archives to keep when\ # purging older snapshots. This option is passed to the Agave\ # client with the `--maximum-full-snapshots-to-retain` argument.\ maximum_full_snapshots_to_retain = 2\ \ # Set the maximum number of incremental snapshot archives to keep\ # when purging older snapshots. This option is passed to the Agave\ # client with the `--maximum-incremental-snapshots-to-retain`\ # argument.\ maximum_incremental_snapshots_to_retain = 4\ \ # Set the minimum snapshot download speed in bytes per second. If\ # the initial download speed falls below this threshold, the\ # validator will retry the download against a different RPC node.\ #\ # The default value is 10MB/s. This option is passed to the Agave\ # client with the `--minimum-snapshot-download-speed` argument.\ minimum_snapshot_download_speed = 10485760\ \ # The maximum number of times to abort and retry when encountering a\ # slow snapshot download.\ #\ # The default value is 5 retries. This option is passed to the\ # Agave client with the `--maximum-snapshot-download-abort`\ # argument.\ maximum_snapshot_download_abort = 5\ \ # Absolute directory path for storing snapshots. If no path is\ # provided, it defaults to the [ledger.path] option from above.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # The snapshot path constructed here is passed to the Agave\ # client with the `--snapshots` argument.\ path = ""\ \ # Absolute directory path for storing incremental snapshots. If no\ # path is provided, defaults to the [snapshots.path] option above,\ # or if that is not provided, the [ledger.path] option above.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # The snapshot path constructed here is passed to the Agave\ # client with the `--incremental-snapshot-archive-path` argument.\ incremental_path = ""\ \ [consensus]\ # Absolute path to a `keypair.json` file containing the identity of\ # the validator. When connecting to dev, test, or mainnet it is\ # required to provide an identity file.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # When running a local cluster, Firedancer will generate a keypair\ # if one is not provided (or has not already been generated) and\ # place it in this path. If no path is provided, it is defaulted\ # to inside the scratch directory, under path `identity.json`.\ #\ # This option is passed to the Agave client with the\ # `--identity` argument.\ identity_path = ""\ \ # Absolute path to a `keypair.json` containing the voting account\ # of the validator. You can alternatively provide the base58\ # encoded pubkey of the vote account to use with this validator. If\ # no voting account is provided, voting will be disabled and the\ # validator will cast no votes.\ #\ # Two substitutions will be performed on this string. If "{user}"\ # is present it will be replaced with the user running Firedancer,\ # as above, and "{name}" will be replaced with the name of the\ # Firedancer instance.\ #\ # The authorized voter for the vote account must be either the\ # identity keypair or one of the authorized-voter keypairs.\ #\ # When running a local cluster, Firedancer will generate a keypair\ # if one is not provided (or has not already been generated) and\ # place it in this path. If no path is provided, it is defaulted\ # to inside the scratch directory, under path `vote-account.json`.\ #\ # This option is passed to the Agave client with the\ # `--vote-account` argument.\ vote_account_path = ""\ \ # List of absolute paths to authorized-voter keypairs for the vote\ # account. This is not needed if no vote account is specified.\ # If a vote account is specified and this is empty, the identity\ # account will be used as the authorized-voter account.\ #\ # Two substitutions will be performed on each string in this list.\ # If "{user}" is present it will be replaced with the user running\ # Firedancer, as above, and "{name}" will be replaced with the\ # name of the Firedancer instance.\ #\ # These options are passed to the Agave client with the\ # `--authorized-voter` argument.\ authorized_voter_paths = []\ \ # If false, do not attempt to fetch a snapshot from the cluster,\ # instead start from a local snapshot if one is present. A snapshot\ # is required to run the validator, so either one must be present,\ # or you need to fetch it. The snapshot will be fetched from a\ # validator in the list of entrypoints. If no validators are listed\ # there, starting the validator will fail. This option is passed\ # (inverted) to the Agave client with the `--no-snapshot-fetch`\ # argument.\ snapshot_fetch = true\ \ # If false, do not attempt to fetch the genesis from the cluster.\ # This option is passed (inverted) to the Agave client with\ # the `--no-genesis-fetch` argument.\ genesis_fetch = true\ \ # On startup, do some simulations to see how fast the validator can\ # generate proof of history. If it is too slow to keep up with the\ # network, exit out during boot. It is recommended to leave this on\ # to ensure you can keep up with the network. This option is passed\ # to the Agave client (inverted) with the\ # `--no-poh-speed-test` argument.\ poh_speed_test = true\ \ # If set, require the genesis block to have the given hash. If it\ # does not the validator will abort with an error. This option is\ # passed to the Agave client with the\ # `--expected-genesis-hash` argument.\ expected_genesis_hash = ""\ \ # If nonzero, after processing the ledger, and the next slot is the\ # provided value, wait until a supermajority of stake is visible on\ # gossip before starting proof of history. This option is passed to\ # the Agave client with the `--wait-for-supermajority`\ # argument.\ wait_for_supermajority_at_slot = 0\ \ # If there is a hard fork, it might be required to provide an\ # expected bank hash to ensure the correct fork is being selected.\ # If this is not provided, or we are not waiting for a\ # supermajority, the bank hash is not checked. Otherwise, we require\ # the bank at the supermajority slot to have this hash. This option\ # is passed to the Agave client with the\ # `--expected-bank-hash` argument.\ expected_bank_hash = ""\ \ # The shred version is a small hash of the genesis block and any\ # subsequent hard forks. The Agave client uses it to filter\ # out any shred traffic from validators that disagree with this\ # validator on the genesis hash or the set of hard forks. If\ # nonzero, ignore any shreds that have a different shred version\ # than this value. If zero, the expected shred version is\ # automatically determined by copying the shred version that the\ # entrypoint validator is using. This option is passed to the\ # Agave client with the `--expected-shred-version` argument.\ expected_shred_version = 0\ \ # If the validator starts up with no ledger, it will wait to start\ # block production until it sees a vote land in a rooted slot. This\ # prevents double signing. Turn off to risk double signing a block.\ # This option is passed to the Agave client (inverted) with\ # the `--no-wait-for-vote-to-start-leader` argument.\ wait_for_vote_to_start_leader = true\ \ # Perform a network speed test on starting up the validator. If\ # this is not disabled, and the speed test fails, the validator will\ # refuse to start. This option is passed to the Agave client\ # (inverted) with the `--no-os-network-limits-test` argument.\ os_network_limits_test = true\ \ # If nonempty, add a hard fork at each of the provided slots. These\ # options are passed to the Agave client with the\ # `--hard-fork` argument.\ hard_fork_at_slots = []\ \ # A set of validators we trust to publish snapshots. If a snapshot\ # is not published by a validator with one of these keys, it is\ # ignored. If no known validators are specified, any hash will be\ # accepted. These options are passed to the Agave client with\ # the `--known-validator` argument.\ known_validators = []\ \ # CPU cores in Firedancer are carefully managed. Where a typical\ # program lets the operating system scheduler determine which threads to\ # run on which cores and for how long, Firedancer overrides most of this\ # behavior by pinning threads to CPU cores.\ #\ # The validator splits all work into eleven distinct jobs, with each\ # thread running one of the jobs:\ #\ # - net Sends and receives network packets from the network\ # device\ #\ # - quic Receives transactions from clients, performing all\ # connection management and packet processing to manage\ # and implement the QUIC protocol\ #\ # - verify Verifies the cryptographic signature of incoming\ # transactions, filtering invalid ones\ #\ # - dedup Checks for and filters out duplicated incoming\ # transactions\ #\ # - pack Collects incoming transactions and smartly schedules\ # them for execution when we are leader\ #\ # - bank Executes transactions that have been scheduled when we\ # are leader\ #\ # - poh Continuously hashes in the background, and mixes the\ # hash in with executed transactions to prove passage of\ # time\ #\ # - shred Distributes block data to the network when leader, and\ # receives and retransmits block data when not leader\ #\ # - store Receives block data when we are leader, or from other\ # nodes when they are leader, and stores it locally in a\ # database on disk\ #\ # - metric Collects monitoring information about other tiles and\ # serves it on an HTTP endpoint\ #\ # - sign Holds the validator private key, and receives and\ # responds to signing requests from other tiles\ #\ # The jobs involved in producing blocks when we are leader are organized\ # in a pipeline, where transactions flow through the system in a linear\ # sequence.\ #\ # net -> quic -> verify -> dedup -> pack -> bank -> poh -> shred -> store\ #\ # Some of these jobs (net, quic, verify, bank, and shred) can be\ # parallelized, and run on multiple CPU cores at once. For example, we\ # could structure the pipeline like this for performance:\ #\ # net -> quic +-> verify -+> dedup -> pack +-> bank -+> poh -> shred -> store\ # +-> verify -+ +-> bank -+\ # +-> verify -+\ # +-> verify -+\ #\ # Each instance of a job running on a CPU core is called a tile. In\ # this configuration we are running 4 verify tiles and 2 bank tiles.\ #\ # The problem of deciding which cores to use, and what job to run on\ # each core we call layout. Layout is system dependent and the highest\ # throughput layout will vary depending on the specific hardware\ # available.\ #\ # Tiles communicate with each other using message queues. If a queue\ # between two tiles fills up, the producer will either block, waiting\ # until there is free space to continue which is referred to as\ # backpressure, or it will drop transactions or data and continue.\ #\ # A slow tile can cause backpressure through the rest of the system\ # causing it to halt, and the goal of adding more tiles is to increase\ # throughput of a job, preventing dropped transactions. For example,\ # if the QUIC server was producing 100,000 transactions a second, but\ # each verify tile could only handle 20,000 transactions a second, five\ # of the verify tiles would be needed to keep up without dropping\ # transactions.\ #\ # A full Firedancer layout spins up these eleven tasks onto a variety of\ # CPU cores and connects them together with queues so that data can flow\ # in and out of the system with maximum throughput and minimal drops.\ [layout]\ # Logical CPU cores to run Firedancer tiles on. Can be specified as\ # a single core like "0", a range like "0-10", or a range with\ # stride like "0-10/2". Stride is useful when CPU cores should be\ # skipped due to hyperthreading. You can also have a number\ # preceded by a 'f' like 'f5' which means the next five tiles are\ # not pinned and will float on the original core set that Firedancer\ # was started with.\ #\ # For example, if Firedancer has six tiles numbered 0..5, and the\ # affinity is specified as\ #\ # f1,0-1,2-4/2,f1\ #\ # Then the tile to core mapping looks like,\ #\ # tile | core\ # -----+-----\ # 0 | floating\ # 1 | 0\ # 2 | 1\ # 3 | 2\ # 4 | 4\ # 5 | floating\ #\ # If the value is specified as auto, Firedancer will attempt to\ # determine the best layout for the system. This is the default\ # value although for best performance it is recommended to specify\ # the layout manually. If the layout is specified as auto, the\ # agave_affinity below must also be set to auto.\ affinity = "auto"\ \ # In addition to the Firedancer tiles which use a core each, the\ # current version of Firedancer hosts an Agave validator as\ # a subprocess.\ #\ # This affinity controls which logical CPU cores the Agave\ # subprocess and all of its threads are allowed to run on. This is\ # specified in the same format as the above Firedancer affinity.\ #\ # It is strongly suggested that you do not overlap the Firedancer\ # affinity with the Agave affinity, as Firedancer tiles expect\ # to have exclusive use of their core. Unexpected latency spikes\ # due to context switching may decrease performance overall.\ #\ # If the value is specified as "auto", the [layout.affinity] field\ # must also be set to "auto", and the Agave affinity will be\ # determined automatically as well.\ agave_affinity = "auto"\ \ # Logical CPU cores to blocklist from being used by Firedancer, when\ # using auto affinity. If not using auto affinity, this option is\ # ignored.\ #\ # Blocklisting cores is useful if there are specific cores that are\ # used by other processes on the system, and you do not want\ # Firedancer running there and competing for the core.\ #\ # Cores are specified as a comma separated list of single cores like\ # "0", ranges like "0-10", or ranges with stride like "0-10/2".\ # Cores can also be specified with an 'h' suffix to indicate both\ # the core and its hyperthread sibling should be blocklisted. For\ # example, "2h" blocklists core 2 and its hyperthread sibling (if\ # any, if there is no hyperthread sibling the program will proceed).\ #\ # By default, both core 0 and its hyperthread sibling are\ # blocklisted to prevent interference with OS kernel threads that\ # often run on these cores.\ blocklist_cores = "0h"\ \ # The number of threads to spawn per-fork for the unified scheduler.\ # The replay stage, which is a part of the Agave subprocess, uses\ # these threads for transaction execution. The threads stay within\ # the cores dedicated to the Agave subprocess.\ #\ # If set to 0, the default depends on the number of cores available\ # to the agave subprocess.\ #\ # agave_cores >= 8 => agave_cores - 4\ # 4 <= agave_cores < 8 => 4\ # agave_cores < 4 => agave_cores\ #\ # Increasing the value for this parameter might help during the\ # start-up phase when the validator is trying to catch up to the\ # cluster. It may also help the node stay caught up if it keeps\ # falling behind.\ agave_unified_scheduler_handler_threads = 0\ \ # How many net tiles to run. Should be set to 1. This is\ # configurable and designed to scale out for future network\ # conditions, but there is no need to run more than 1 net tile given\ # current `mainnet-beta` conditions.\ #\ # Net tiles are responsible for sending and receiving packets from\ # the network device configured in the [net] section below.\ # Each net tile will service exactly one queue from the device, and\ # Firedancer will error on boot if the number of queues on the\ # device is not configured correctly.\ #\ # The net tile is designed to scale linearly when adding more tiles.\ #\ # See the comments for the [net] section below for more information.\ net_tile_count = 1\ \ # How many QUIC tiles to run. Should be set to 1. This is\ # configurable and designed to scale out for future network\ # conditions. There is no need to run more than 1 QUIC tile given\ # current `mainnet-beta` conditions, unless the validator is the\ # subject of an attack.\ #\ # QUIC tiles are responsible for parsing incoming QUIC protocol\ # messages, managing connections and responding to clients.\ # Connections from the net tiles will be evenly distributed\ # between the available QUIC tiles round-robin style.\ #\ # QUIC tiles are designed to scale linearly when adding more tiles,\ quic_tile_count = 1\ \ # How many resolver tiles to run. Should be set to 1. This is\ # configurable and designed to scale out for future network\ # conditions. There is no need to run more than 1 resolver tile\ # given current `mainnet-beta` conditions, unless the validator is\ # under a DoS or spam attack.\ #\ # Resolve tiles are responsible for resolving address lookup tables\ # before transactions are scheduled.\ resolh_tile_count = 1\ \ # How many verify tiles to run. Verify tiles perform signature\ # verification on incoming transactions, an expensive operation that\ # is often the bottleneck of the validator.\ #\ # Verify tiles are designed to scale linearly when adding more\ # tiles, and the verify tile count should be increased until the\ # validator is not dropping incoming QUIC transactions from clients.\ #\ # On modern hardware, each verify tile can handle around 20-40K\ # transactions per second. Six tiles seems to be enough to handle\ # current `mainnet-beta` traffic, unless the validator is under a\ # denial of service or spam attack.\ verify_tile_count = 6\ \ # How many bank tiles to run. Should be set to 4 for perf and\ # balanced scheduling modes. Bank tiles execute transactions, so\ # the validator can include the results of the transaction into a\ # block when we are leader. Because of current consensus limits\ # restricting blocks to around 32,000 transactions per block, there\ # is typically no need to use more than 4 bank tiles on\ # mainnet-beta, except when using unique scheduling strategies. For\ # development and benchmarking, it can be useful to increase this\ # number further.\ #\ # Bank tiles do not scale linearly. The current implementation uses\ # the agave runtime for execution, which takes various locks and\ # uses concurrent data structures which slow down with multiple\ # parallel users.\ bank_tile_count = 4\ \ # How many shred tiles to run. Should be set to 1. This is\ # configurable and designed to scale out for future network\ # conditions. There is no need to run more than 1 shred tile given\ # current `mainnet-beta` conditions. There is however a need to run\ # 2 shred tiles under current `testnet` conditions.\ #\ # Shred tiles distribute block data to the network when we are\ # leader, and receive and retransmit it to other nodes when we are\ # not leader.\ #\ # Shred tile performance heavily dependent on the number of peer\ # nodes in the cluster, as computing where data should go is an\ # expensive function with this list of peers as the input. In\ # development and benchmarking, 1 tile is also sufficient to hit\ # very high TPS rates because the cluster size will be very small.\ shred_tile_count = 1\ \ # All memory that will be used in Firedancer is pre-allocated in two\ # kinds of pages: huge and gigantic. Huge pages are 2 MiB and gigantic\ # pages are 1 GiB. This is done to prevent TLB misses which can have a\ # high performance cost. There are three important steps in this\ # configuration,\ #\ # 1. At boot time or soon after, the kernel is told to allocate a\ # certain number of both huge and gigantic pages to a special pool\ # so that they are reserved for later use by privileged programs.\ #\ # 2. At configuration time, one (pseudo) file system of type hugetlbfs\ # for each of huge and gigantic pages is mounted on a local\ # directory. Any file created within these file systems will be\ # backed by in-memory pages of the desired size.\ #\ # 3. At Firedancer initialization time, Firedancer creates a\ # "workspace" file in one of these mounts. The workspace is a\ # single mapped memory region within which the program lays out and\ # initializes all the data structures it will need in advance.\ # Most Firedancer allocations occur at initialization time, and this\ # memory is fully managed by special purpose allocators.\ #\ # A typical layout of the mounts looks as follows,\ #\ # /mnt/.fd [Mount parent directory specified below]\ # +-- .gigantic [Files created in this mount use 1 GiB\ # pages]\ # +-- firedancer1.wksp\ # +-- .huge [Files created in this mount use 2 MiB\ # pages]\ # +-- scratch1.wksp\ # +-- scratch2.wksp\ [hugetlbfs]\ # The absolute path to a directory in the file system. Firedancer\ # will mount the hugetlbfs file system for gigantic pages at a\ # subdirectory named .gigantic under this path, or if the entire\ # path already exists, will use it as-is. Firedancer will also\ # mount the hugetlbfs file system for huge pages at a subdirectory\ # named .huge under this path, or if the entire path already exists,\ # will use it as-is. If the mount already exists it should be\ # writable by the Firedancer user.\ mount_path = "/mnt/.fd"\ \ # The largest supported page size on the system. Possible values\ # are either "huge" (2 MiB) or "gigantic" (1 GiB). Larger sizes\ # yield better performance. Smaller values may be required on\ # virtualized environments like cloud providers.\ max_page_size = "gigantic"\ \ # Workspaces are either fully backed by "huge" or "gigantic" pages.\ # Workspaces with footprint equal to or greater than the given value\ # in MiB are backed by "gigantic" pages unless disabled via\ # max_page_size.\ gigantic_page_threshold_mib = 128\ \ # The network stack is provided by net tiles, which are responsible for\ # sending and receiving packets on the network.\ #\ # Net tiles will multiplex in both directions, fanning out packets\ # to multiple parts of Firedancer that can receive and handle them,\ # like QUIC tiles and the Turbine retransmission engine. Then also\ # fanning in from these various network senders to transmit on the\ # queues we have available.\ [net]\ # The provider determines which Linux networking stack is used by\ # Firedancer. Possible values for the option are:\ #\ # "xdp" (default)\ # Use Linux express data path APIs for high performance networking.\ # These APIs map memory from the network device directly into\ # Firedancer to bypass parts of the kernel when reading and writing\ # data.\ #\ # "socket"\ # Use UDP sockets and system calls like `send(2)` for networking.\ # Sockets are slower and socket support is provided on a best-effort\ # basis. The socket option is provided as a fallback for hosts which\ # do not support XDP, and to support experimental features that are\ # not yet implemented in the XDP stack.\ #\ # Using the XDP networking stack is strongly preferred where possible,\ # as it is faster and better tested by the development team.\ provider = "xdp"\ \ # Which interface to bind to for network traffic. Currently,\ # only one interface is supported for networking. If this is\ # empty, the default is the interface used to route to 8.8.8.8,\ # you can check what this is with `ip route get 8.8.8.8`\ interface = ""\ \ # Optional IPv4 bind address for peer-to-peer traffic. For\ # incoming packets, specifies the destination address to listen\ # on. For outgoing packets, sets the default source address.\ # When left unspecified (default), listens on all addresses, and\ # derives the source address for outgoing packets based on\ # address and route tables. Should be an IPv4 address, e.g.\ # "198.51.100.2". This option is passed to the Agave client\ # with the `--bind-address` argument.\ bind_address = ""\ \ # The maximum number of packets in-flight between a net tile and\ # downstream consumers, after which additional packets begin to\ # replace older ones, which will be dropped. Smaller values use\ # less memory. Larger values decrease link overruns caused by\ # excessive latency.\ ingress_buffer_size = 16384\ \ # This section contains XDP-specific network configuration.\ # Only active if [net.provider] is set to "xdp".\ [net.xdp]\ # XDP is redundantly provided by different parts in the kernel.\ # This option selects the XDP provider. Possible values for\ # this option are:\ #\ # "skb" (default)\ # XDP provided by the kernel's generic network code. This is\ # the slowest mode, but it is well tested and compatible with\ # all Linux network devices and drivers.\ #\ # "drv"\ # XDP provided by the device driver. This mode is much faster\ # than "skb" but only works for certain hardware. May require\ # very recent Linux versions. Can be less stable than "skb"\ # mode. Tested extensively with ConnectX NICs (mlx5), Intel\ # X710 (i40e), and Intel E810 (ice).\ # Full list of network drivers supporting XDP:\ # https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#xdp\ #\ # "default"\ # Selects "skb" or "drv" automatically.\ #\ # If the kernel/hardware does not support driver mode, then\ # `fdctl run` will fail to start up with "operation not\ # supported".\ xdp_mode = "skb"\ \ # This option helps reduce CPU usage when receiving packets.\ # If enabled, the net tile instructs the network device to copy\ # packets directly (DMA) into the working memory of downstream\ # tiles (such as the quic tile). This allows scaling ingress up\ # to 100 Gbps per net tile.\ #\ # Only works when XDP is provided by the network driver (see\ # the xdp_mode option above). If the kernel/hardware does not\ # support zero copy, `fdctl run` will fail to start up with\ # "operation not supported".\ xdp_zero_copy = false\ \ # XDP uses metadata queues shared across the kernel and\ # userspace to relay events about incoming and outgoing packets.\ # This setting defines the number of entries in these metadata\ # queues, which has to be a power of two.\ #\ # Smaller values use less memory. Larger values reduce packet\ # loss caused by excessive scheduling or inter-CPU latency.\ xdp_rx_queue_size = 32768\ xdp_tx_queue_size = 32768\ \ # When the oldest packet in a batch has been queued for more than\ # the specified duration, flushes the batch out to the network\ # device. Smaller values improve latency, larger values may\ # improve throughput.\ flush_timeout_micros = 20\ \ # RSS (RX hardware flow steering) queue mode\ #\ # Each XDP net tile services exactly one hardware queue. Thus we\ # must ensure the system routes all incoming Firedancer packets to\ # the correct queue(s). The queue setup and initialization has the\ # following options:\ #\ # "simple" (default)\ # Reduces the total queue count to equal the number of net tiles.\ # All packets (both interesting to Firedancer and not) are sharded\ # amongst these queues. Simple, reliable, and works everywhere\ # but suboptimal performance impact, especially to socket-based\ # traffic.\ #\ # "dedicated"\ # Reserves a dedicated hardware queue for each net tile. Uses\ # ethtool 'ntuple' rules to route incoming packets based on\ # UDP dst port. Packets are load balanced across the dedicated\ # queues based on lowest bits of source address. This mode\ # may not work with some network device setups.\ #\ # "auto"\ # Attempts to run in "dedicated" mode but automatically falls\ # back to "simple" mode if necessary.\ rss_queue_mode = "simple"\ \ # Enable native network bonding support.\ #\ # If native_bond is set to 'true' and [net.interface] is a 'bond'\ # type network device, install XDP config at bond slave devices\ # instead of the bond device itself.\ #\ # Requires net_tile_count to be divisble by the number of slave\ # devices. This causes outgoing traffic to be load balanced with\ # an arbitrary/opaque hash policy. Supports up to 16 slave devices.\ #\ # Assumes that bond memberships don't change, i.e. running\ # `ip link set dev eth1 master bond0` will crash Firedancer.\ # However, it is fine to bring individual bonded devices down,\ # or unplug cables for maintenance. In other words, executing\ # `ip link set dev eth1 down` while Firedancer is running is\ # just fine.\ native_bond = false\ \ [net.socket]\ # Sets the socket receive buffer size via SO_RCVBUF.\ # Raises net.core.rmem_max accordingly\ receive_buffer_size = 134217728\ \ # Sets the socket receive buffer size via SO_SNDBUF.\ # Raises net.core.wmem_max accordingly\ send_buffer_size = 134217728\ \ # Tiles are described in detail in the layout section above. While the\ # layout configuration determines how many of each tile to place on\ # which CPU core to create a functioning system, below is the individual\ # settings that can change behavior of the tiles.\ [tiles]\ # The netlink tile forwards Linux network configuration to net tiles.\ # This config section contains advanced options that typically do not\ # need to be changed.\ # For further info, see https://docs.firedancer.io/guide/netlink.html\ [tiles.netlink]\ # The maximum number of routes per route table.\ #\ # The netlink tile imports two route tables from Linux, namely\ # `local` and `main`. You can view them by running\ # `ip route show table main`. Decreasing this option can result\ # in connectivity issues. Increasing this option can drastically\ # decrease performance.\ #\ # For virtually all cloud and bare-metal server providers, the\ # number of routes per table does not exceed 16, excluding the\ # /32 routes, see below.\ max_routes = 128\ \ # This setting configures the maximum number of IPv4 routes with a\ # /32 netmask per route table. This setting is relevant for Double Zero\ # accelerated networking, which uses one /32 route per peer on the\ # network. The number of /32 routes should not exceed 100,000.\ max_peer_routes = 8192\ \ # The maximum number of Ethernet neighbors.\ #\ # This should be roughly as large as the size your Ethernet subnet.\ # E.g. if your IP address is 198.51.100.3/24, then your subnet has\ # up to 256 neighbors (2^(32-24)).\ max_neighbors = 4096\ \ # QUIC tiles are responsible for serving network traffic, including\ # parsing and responding to packets and managing connection timeouts\ # and state machines. These tiles implement the QUIC protocol,\ # along with receiving regular (non-QUIC) UDP transactions, and\ # forward well-formed (but not necessarily valid) ones to verify\ # tiles.\ [tiles.quic]\ # Which port to listen on for incoming, regular UDP transactions\ # that are not over QUIC. These could be votes, user\ # transactions, or transactions forwarded from another\ # validator.\ regular_transaction_listen_port = 9001\ \ # Which port to listen on for incoming QUIC transactions.\ # Currently this must be exactly 6 more than the\ # transaction_listen_port.\ quic_transaction_listen_port = 9007\ \ # Maximum number of simultaneous QUIC connections which can be\ # open. New connections which would exceed this limit will not\ # be accepted.\ #\ # This must be >= 2 and also a power of 2.\ max_concurrent_connections = 131072\ \ # Controls how many transactions coming in via TPU can be\ # reassembled at the same time. Reassembly is required for user\ # transactions larger than ca ~1200 bytes, as these arrive\ # fragmented. This parameter should scale linearly with line\ # rate. Usually, clients send all fragments at once, such that\ # each reassembly only takes a few microseconds.\ #\ # Higher values reduce TPU packet loss over unreliable networks.\ # If this parameter is set too low, packet loss can cause some\ # large transactions to get dropped. Must be 2 or larger.\ txn_reassembly_count = 131072\ \ # QUIC has a handshake process which establishes a secure\ # connection between two endpoints. The handshake process is\ # very expensive. So we allow only a limited number of\ # handshakes to occur concurrently.\ #\ max_concurrent_handshakes = 4096\ \ # QUIC has a concept of an idle connection, one where neither\ # the client nor the server has sent any packet to the other for\ # a period of time. Once this timeout is reached the connection\ # will be terminated.\ #\ # An idle connection will be terminated if it remains idle\ # longer than this threshold.\ idle_timeout_millis = 10000\ \ # Max delay for outgoing ACKs.\ ack_delay_millis = 50\ \ # QUIC retry is a feature to combat new connection request\ # spamming. See rfc9000 8.1.2 for more details. This flag\ # determines whether the feature is enabled in the validator.\ retry = true\ \ # Log TLS encryption keys to decrypt QUIC traffic to this file\ # path in NSS SSLKEYLOGFILE format. An empty string (the\ # default) disables key logging.\ ssl_key_log_file = ""\ \ # Verify tiles perform signature verification of incoming\ # transactions, making sure that the data is well-formed, and that\ # it is signed by the appropriate private key.\ [tiles.verify]\ # The verify tiles have a cache of signatures they have seen\ # recently, used to discard duplicate transactions before they\ # get verified to save signature verification resources. See\ # [tiles.dedup.signature_cache_size] below for more information.\ signature_cache_size = 4194302\ \ # The maximum number of messages in-flight between a QUIC tile\ # and associated verify tile, after which earlier messages might\ # start being overwritten, and get dropped so that the system\ # can keep up.\ receive_buffer_size = 16384\ \ # After being verified, all transactions are sent to a dedup tile to\ # ensure the same transaction is not repeated multiple times. The\ # dedup tile keeps a rolling history of signatures it has seen and\ # drops any that are duplicated, before forwarding unique ones on.\ [tiles.dedup]\ # The size of the cache that stores unique signatures we have\ # seen to deduplicate. This is the maximum number of signatures\ # that can be remembered before we will let a duplicate through.\ #\ # If a duplicated transaction is let through, it will waste more\ # resources downstream before we are able to determine that it\ # is invalid and has already been executed. If a lot of memory\ # is available, it can make sense to increase this cache size to\ # protect against denial of service from high volumes of\ # transaction spam.\ #\ # The default value here, 2^26 - 2 is a good balance, as it fits\ # in 1 GiB of memory using a single gigantic page.\ signature_cache_size = 33554430\ \ # The bundle tile receives bundles from a block producer remote\ # endpoint and forwards them for execution to pack.\ [tiles.bundle]\ # True if a bundle tile should be enabled, which receives\ # bundles from a remote block engine and forwards them for\ # atomic execution to the pack tile.\ enabled = false\ \ # The URL of the block engine to receive blocks from.\ url = ""\ \ # The domain name of the TLS certificate to use when verifying\ # the peer. If none is provided, the domain from the URL above\ # will be used by default.\ #\ # This is only needed in special cases where the block engine\ # peer presents a certificate for a different domain that where\ # it is being accessed.\ tls_domain_name = ""\ \ # The tip distribution program is used to control the\ # distribution of tips that transactions from bundles may pay.\ # This should be set to the public key of the tip distribution\ # program. Consult the block engine provider's documentation\ # for the appropriate value of this field.\ tip_distribution_program_addr = ""\ \ # The tip payment program handles the regular collection of tips\ # into the account configured by the tip distribution program.\ # This should be set to the public key of the tip payment\ # program. Consult the block engine provider's documentation\ # for the appropriate value of this field.\ tip_payment_program_addr = ""\ \ # A validator may want to distribute tips from transactions to\ # their stakers or to any other relevant parties. Authority is\ # delegated to the public key provided in the\ # tip_distribution_authority to determine the distribution by\ # uploading a hash tree. This field should be set to the\ # base58 public key.\ tip_distribution_authority = ""\ \ # The standard tip distribution mechanism distributes tips\ # pro rata to stakers with the validator optionally taking a\ # portion as a fee. The commission set below, measured in basis\ # points (100ths of a percent), determines the validator's\ # portion.\ commission_bps = 0\ \ # The validator keeps the gRPC connection to the bundle server\ # alive indefinitely, even outside of leader slots. This is\ # done using HTTP/2 PING frames. This option roughly configures\ # the time between keepalives. The actual timing is randomized\ # to reduce burst loads on the network. Must be within range\ # [3e3,3600e3]\ keepalive_interval_millis = 5000\ \ # By default, the TLS certificate of the bundle server is\ # verified against the CA certs in /etc/ssl/certs, if the URL\ # specifies https. To disable certificate verification, set\ # this option to 'false'.\ tls_cert_verify = true\ \ # The pack tile takes incoming transactions that have been verified\ # by the verify tile and then deduplicated, and attempts to order\ # them in an optimal way to generate the most fees per compute\ # resource used to execute them.\ [tiles.pack]\ # The pack tile receives transactions while it is waiting to\ # become leader and stores them for future execution. This\ # option determines the maximum number of transactions that\ # will be stored before those with the lowest estimated\ # profitability get dropped. The maximum allowed, and default\ # value is 65524. It is not recommended to change this.\ max_pending_transactions = 65524\ \ # When a transaction consumes fewer CUs than it requests, the\ # bank and pack tiles work together to adjust the block limits\ # so that a different transaction can consume the unspent CUs.\ # This is normally desirable, as it typically leads to\ # producing blocks with more transactions.\ #\ # In situations where transactions typically do not\ # significantly over-request CUs, or when the CU limit is high\ # enough so that over-requesting CUs does not impact how many\ # transactions fit in a block, this can be disabled to improve\ # performance. It's not recommended (but allowed) to disable\ # this option in a production cluster.\ use_consumed_cus = true\ \ # The pack tile can schedule using different strategies with\ # different tradeoffs. The primary tradeoff is that executing\ # transactions quickly is best for the network as a whole, but\ # can reduce revenue opportunities for this validator.\ #\ # Over time, as the network gets faster and compute unit limits\ # are raised, this trade-off will become less pronounced.\ #\ # The Firedancer team supports two different strategies which\ # are described below:\ #\ # "perf"\ # Try to fill the block as fast as possible using the highest-\ # paying transactions available. This strategy results in\ # consistently 100% full blocks.\ #\ # When using the perf scheduling mode, there is no reason to\ # increase the bank tile count beyond the default of 4.\ #\ # "balanced" (default)\ # Try to fill the block at a rate that is just fast enough to\ # fill it by the end. This strategy optimizes for revenue\ # from priority fees, but can result in blocks that are not\ # always 100% full, and can be poor at capturing MEV if using\ # an external block builder.\ #\ # When using the balanced mode, there is no reason to increase\ # bank tile count beyond the default of 4.\ #\ # In cases of network congestion, high volume, or if seeing poor\ # block packing performance for any reason, it is recommended to\ # revert to the "balanced" strategy.\ schedule_strategy = "balanced"\ \ # The bank tile is what executes transactions and updates the\ # accounting state as a result of any operations performed by the\ # transactions. Currently, the bank tile is implemented by the\ # Agave execution engine and is not configurable.\ [tiles.bank]\ \ # The proof of history tile receives transactions from bank tiles\ # and mixes their hashes into a continuous stream of hashes that\ # proves to other validators time is passing.\ [tiles.pohh]\ # A validator will always be leader for at least four\ # consecutive slots, which looks like ....\ #\ # +-------------+----+----+----+----+-------------+\ # | ... | s0 | s1 | s2 | s3 | ... |\ # +-------------+----+----+----+----+-------------+\ # t0 t1\ #\ # Leader slots are 400 milliseconds, so in a well-behaved\ # validator the start time of slot s1 (t1) will be 400\ # milliseconds after s0 (t0).\ #\ # Agave validators start late, at some time t0 + 400 + x, where\ # x is the time it takes the validator to "freeze" slot s0.\ # This is typically around 100 milliseconds, but can be up to\ # 300 or so in degenerate cases. This is not good for the\ # network because it increases confirmation times and skip\ # rates, but it is beneficial to the individual validator (a\ # tragedy of the commons) because the validator is leader\ # longer, can receive higher paying transactions for longer,\ # and ultimately generate more fees for the operator.\ #\ # Firedancer defaults to matching the Agave behavior here and\ # starting late, but you can disable the option to use a more\ # healthy behavior, where taking a long time to "freeze" slots\ # cuts into your own next slot time, and does not increase it.\ lagged_consecutive_leader_start = false\ \ # The shred tile distributes processed transactions that have been\ # executed to the rest of the cluster in the form of shred packets.\ [tiles.shred]\ # When this validator is not the leader, it receives the most\ # recent processed transactions from the leader and other\ # validators in the form of shred packets. Shreds are grouped\ # in sets for error correction purposes, and the full validation\ # of a shred packet requires receiving at least half of the set.\ # Since shreds frequently arrive out of order, the shred tile\ # needs a relatively large buffer to hold sets of shreds until\ # they can be fully validated. This option specifies the size\ # of this buffer.\ #\ # To compute an appropriate value, multiply the expected Turbine\ # worst-case latency (tenths of seconds) by the expected\ # transaction rate, and divide by approx 25.\ max_pending_shred_sets = 512\ \ # The shred tile listens on a specific port for shreds to\ # forward. This argument controls which port that is. The port\ # is broadcast over gossip so other validators know how to reach\ # this one.\ shred_listen_port = 8003\ \ # Shreds can also be forwarded to specific addresses, for\ # example to run an unstaked RPC or for archiving. Each new, valid\ # shred that the validator receives will be forwarded to the\ # addresses specified in additional_shred_destinations_retransmit.\ # Each shred that the validor produced when it is leader will be\ # sent to the addresses specified in\ # additional_shred_destinations_leader. Destinations must be in the\ # form "ip:port", for example "1.2.3.4:5566". Shreds will\ # be sent to these destinations first, prior to sending to other\ # validators.\ additional_shred_destinations_retransmit = []\ additional_shred_destinations_leader = []\ \ # The metric tile receives metrics updates published from the rest\ # of the tiles and serves them via. a Prometheus compatible HTTP\ # endpoint.\ [tiles.metric]\ # The address to listen on. By default, metrics are only\ # accessible from the local machine. If you wish to expose them\ # to the network, you can change the listen address.\ #\ # The Firedancer team makes a best effort to secure the metrics\ # endpoint but exposing it to the internet from a production\ # validator is not recommended as it increases the attack\ # surface of the validator.\ prometheus_listen_address = "127.0.0.1"\ \ # The port to listen on for HTTP request for Prometheus metrics.\ # Firedancer serves metrics at a URI like 127.0.0.1:7999/metrics\ prometheus_listen_port = 7999\ \ # The gui tile receives data from the validator and serves an HTTP\ # endpoint to clients to view it.\ [tiles.gui]\ # If the GUI is enabled.\ enabled = true\ \ # The address to listen on. By default, if enabled, the GUI\ # will only be accessible from the local machine. If you wish\ # to expose it to the network, you can change the listen\ # address.\ #\ # The Firedancer team makes a best effort to secure the GUI\ # endpoint but exposing it to the internet from a production\ # validator is not recommended as it increases the attack\ # surface of the validator.\ gui_listen_address = "127.0.0.1"\ \ # The port to listen on.\ gui_listen_port = 80\ \ # Maximum number of simultaneous HTTP connections which can be\ # open. The server closes an HTTP connection after each\ # request. When no more connections are free, evicts the oldest\ # first.\ max_http_connections = 1024\ \ # Maximum number of simultaneous WebSocket connections which can\ # be open. Every browser tab currently showing the GUI has one\ # persistent WebSocket connection. When a new WebSocket is\ # created but no connections are free, the oldest one is closed.\ max_websocket_connections = 1024\ \ # Maximum length of an HTTP request including headers.\ max_http_request_length = 8192\ \ # All GUI connections share buffer space for outgoing GUI\ # WebSocket updates and HTTP response headers. If this buffer\ # runs out, the connections with the slowest recipients are\ # dropped. Larger values make the GUI server more tolerant to\ # slow clients but also increase memory usage.\ send_buffer_size_mb = 5120\ \ # These options can be useful for development, but should not be used\ # when connecting to a live cluster, as they may cause the validator to\ # be unstable or have degraded performance or security. The program\ # will check that these options are set correctly in production and\ # refuse to start otherwise.\ [development]\ # For enhanced security, Firedancer runs itself in a restrictive\ # sandbox in production. The sandbox prevents most system calls and\ # restricts the capabilities of the process after initialization to\ # make the attack surface smaller. This is required in production,\ # but might be too restrictive during development.\ #\ # In development, you can disable the sandbox for testing and\ # debugging with the `--no-sandbox` argument to `fddev`.\ sandbox = true\ \ # As part of the security sandboxing, Firedancer will run every tile\ # in a separate process. This can be annoying for debugging where\ # you want control of all the tiles under one inferior, so we also\ # support a development mode where tiles are run as threads instead\ # and the system operates inside a single process. This does not\ # impact performance and threads still get pinned.\ #\ # This option cannot be enabled in production. In development, you\ # can also launch Firedancer as a single process for with the\ # `--no-clone` argument to `fddev`.\ no_clone = false\ \ # As part of security sandboxing, the dumpable bit of each process\ # is cleared with prctl( PR_SET_DUMPABLE, 0 ). This prevents other\ # processes from attaching via. `ptrace(2)` which could be a\ # security issue, and also prevents core dumps from being generated\ # which might leak sensitive information. If the sandbox is\ # disabled, core dumps are always produced and this option has no\ # effect.\ #\ # This option controls whether core dumps are produced when the\ # sandbox is enabled, and what is in them. The option can be\ # enabled in production although it is not always recommended as it\ # removes some minor security protections from the sandbox. The\ # possible options are,\ #\ # "disabled" (default)\ # No core dumps are produced.\ #\ # "minimal"\ # Core dumps are produced, but they contain only stack memory,\ # registers, and other core process state. No heap memory or\ # workspaces are included.\ #\ # "regular"\ # Core dumps are produced including most useful process state,\ # including heap memory and important workspaces mapped into the\ # process. Certain large workspaces that are typically not useful\ # will not be included, such as the accounts database and\ # blockstore.\ #\ # "full"\ # Core dumps are with almost all process memory, including things\ # like the accounts database. The only memory which will not be\ # in the dump are sensitive pages containing private key material\ # from the running validator. These dumps can be extremely large\ # and it is not recommended to use this option.\ core_dump = "disabled"\ \ # Firedancer currently hosts a Agave client as a child process\ # when it starts up, to provide functionality that has not yet been\ # implemented. For development sometimes it is desirable to not\ # launch this subprocess, although it will prevent the validator\ # from operating correctly.\ #\ # In development, you can disable agave for testing and debugging\ # with the `--no-agave` argument to `fddev`.\ no_agave = false\ \ # Sometimes, it may be useful to run a bootstrap Firedancer\ # validator, either for development or for testing purposes. The\ # `fddev` tool is provided for this purpose, which creates the\ # bootstrap keys and does the cluster genesis using some parameters\ # that are typically useful for development.\ #\ # Enabling this allows decoupling the genesis and key creation from\ # the validator startup. The bootstrap validator can then be\ # started up with `fdctl`. It will expect the genesis to already\ # exist at [ledger.path]. The keys used during genesis should be\ # the same as the ones supplied in the [consensus.identity_path] and\ # [consensus.vote_account_path]. This option will not be effective\ # if [gossip.entrypoints] is non-empty.\ bootstrap = false\ \ [development.gossip]\ # Under normal operating conditions, a validator should always\ # reach out to a host located on the public internet. If this\ # value is true, it allows the validator to gossip with nodes\ # configuration item allows Firedancer to gossip with nodes\ # located on a private internet (rfc1918).\ #\ # This option is passed to the Agave client with the\ # `--allow-private-addr` argument.\ allow_private_address = false\ \ [development.genesis]\ # When creating a new chain from genesis during development,\ # this option can be used to specify the number of hashes in\ # each tick of the proof history component.\ #\ # A value of one is the default, and will be used to mean that\ # the proof of history component will run in low power mode,\ # and use one hash per tick. This is equivalent to a value of\ # "sleep" when providing hashes-per-tick to the Agave\ # genesis.\ #\ # A value of zero means the genesis will automatically determine\ # the number of hashes in each tick based on how many hashes\ # the generating computer can do in the target tick duration\ # specified below.\ #\ # This value specifies the initial value for the chain in the\ # genesis, but it might be overridden at runtime if the related\ # features which increase this value are enabled. The features\ # are named like `update_hashes_per_tick2`.\ #\ # A value of 62,500 is the same as mainnet-beta, devnet, and\ # testnet, following activation of the `update_hashes_per_tick6`\ # feature.\ hashes_per_tick = 62_500\ \ # How long each tick of the proof of history component should\ # take, in microseconds. This value specifies the initial value\ # of the chain and it will not change at runtime. The default\ # value used here is the same as mainnet, devnet, and testnet.\ target_tick_duration_micros = 6250\ \ # The number of ticks in each slot. This value specifies the\ # initial value of the chain and it will not change at runtime.\ # The default value used here is the same as mainnet, devnet,\ # and testnet.\ ticks_per_slot = 64\ \ # The count of accounts to pre-fund in the genesis block with\ # SOL. Useful for benchmarking and development. The specific\ # accounts that will be funded are those with private-keys of\ # 0, 1, 2, .... N.\ fund_initial_accounts = 1024\ \ # The amount of SOL to pre-fund each account with.\ fund_initial_amount_lamports = 50000000000000\ \ # The number of lamports to stake on the voting account of the\ # validator that starts up from the genesis. Genesis creation\ # will fund the staking account passed to it with this amount\ # and then stake it on the voting account. Note that the voting\ # account key used in the genesis needs to be the same as the\ # one that is used by the bootstrap validator.\ vote_account_stake_lamports = 500000000\ \ # Setting warmup epochs to true will allow shorter epochs towards\ # the beginning of the cluster. This allows for faster stake\ # activation. The first epoch will be 32 slots long and the\ # duration of each subsequent epoch will be double that of the\ # one before it until it reaches the desired epoch duration of the\ # cluster.\ warmup_epochs = false\ \ [development.bench]\ # How many benchg tiles to run when benchmarking. benchg tiles\ # are responsible for generating and signing outgoing\ # transactions to the validator which is expensive.\ benchg_tile_count = 4\ \ # How many benchs tiles to run when benchmarking. benchs tiles\ # are responsible for sending transactions to the validator, for\ # example by calling send() on a socket. On loopback, a single\ # benchs tile can send around 320k packets a second.\ benchs_tile_count = 2\ \ # Which cores to run the benchmarking tiles on. By default, the\ # cores will be floating but to get good performance\ # measurements on your machine, you should create a topology\ # where these generators get their own cores.\ #\ # The tiles included in this affinity are,\ #\ # bencho, benchg1, .... benchgN, benchs1, ... benchsN\ #\ # If the [layout.affinity] above is set to "auto" then this\ # value must also be set to "auto" and it will be determined\ # automatically.\ affinity = "auto"\ \ # Solana has a hard-coded maximum CUs per block limit of\ # 50,000,000+ which works out to around 83,000 transfers a\ # second, since each consumes about 1500 CUs. When\ # benchmarking, this can be the limiting bottleneck of the\ # system, so this option is provided to raise the limit. If set\ # to true, the limit will be lifted to 624,000,000 for a little\ # over 1 million transfers per second.\ #\ # This option should not be used in production, and would cause\ # consensus violations and a consensus difference between the\ # validator and the rest of the network.\ larger_max_cost_per_block = false\ \ # Solana has a consensus-agreed-upon limit of 32,768 data and\ # parity shreds per block. This limit prevents a malicious (or\ # mistaken) validator from slowing down the network by producing\ # a huge block. When benchmarking, this limit can be the\ # bottleneck of the whole system, so this option is provided to\ # raise the limit. If set to true, the limit will be raised to\ # 131,072 data and parity shreds per block, for about 1,200,000\ # small transactions per second.\ #\ # This option should not be used in a production network, since\ # it would cause consensus violations between the validator and\ # the rest of the network.\ larger_shred_limits_per_block = false\ \ # Frankendancer currently depends on the Agave blockstore\ # component for storing block data to disk. This component can\ # frequently be a bottleneck when testing the throughput of the\ # leader pipeline, so this option is provided to disable it\ # starting from a certain slot.\ #\ # This option should not be used in a production network; it\ # causes the validator to not be able to serve repair requests,\ # snapshots, or participate in other consensus critical\ # operations. It is only useful for benchmarking the leader\ # TPU performance in a single node cluster case.\ #\ # A value of 0 means this option will not be used. A value of 1\ # disables the blockstore entirely. A common use case for any\ # other positive value is to create a snapshot before disabling\ # the blockstore. This is useful in cases when benchmarking an\ # entire cluster. The leader needs to create a first snapshot\ # that the followers need to fetch in order to join the cluster.\ # In such a case, it is useful to set this value to the same\ # number as [snapshots.full_snapshot_interval_slots].\ disable_blockstore_from_slot = 0\ \ # Frankendancer currently depends on the Agave status cache to\ # prevent double-spend attacks. The data structure that backs\ # the Agave status cache frequently causes lots of page faults\ # and contention during benchmarking. It severely limits\ # banking stage scalability, so this option is provided to\ # disable it.\ #\ # This option should not be used in a production network. If\ # set to true, Frankendancer will not be able to identify a\ # transaction it has received as a duplicate if it occurred in\ # another leader's block, causing it to produce invalid blocks.\ # It is only useful for benchmarking when it is known that\ # duplicate transactions will not be submitted and the validator\ # with this option enabled will always be leader.\ disable_status_cache = false\ \ # Experimental options for the bundle tile. These should not be\ # changed on a production validator.\ [development.bundle]\ # Specify the maximum Protobuf message size in KiB.\ #\ # This affects the size of various gRPC-related buffers,\ # including socket buffer sizes, the TLS buffer, the HTTP/2\ # frame buffer, and a Protobuf encode buffer.\ buffer_size_kib = 16384\ \ # SSL library heap size in MiB.\ #\ # When connecting to an HTTPS bundle endpoint, the SSL library\ # dynamic allocations in a fixed size heap region. This option\ # controls the size of that region.\ ssl_heap_size_mib = 64\ \ # Log TLS encryption keys for block engine connection. An empty\ # string (the default) disables key logging.\ #\ # Keys are appended in NSS SSLKEYLOGFILE format to the given\ # file path. A file is created if none exists. The resulting\ # file can be used to decrypt packet captures of gRPC conns\ # made by the bundle tile.\ ssl_key_log_file = ""\ \ # The following options relate to the 'fddev pktgen' tool only.\ # pktgen tests net tile transmit throughput by generating a flood of\ # non-routable Ethernet frames. Only use 'fddev pktgen' if you know\ # what you are doing! It can cause damage to your system and network\ # infrastructure.\ [development.pktgen]\ # Which cores to run the 'net' and 'pktgen' tiles on.\ # By default, two fixed cores will be used. To get reliable\ # measurements, avoid the use of floating tiles.\ # This setting overrides [layout.affinity] above.\ affinity = "auto"\ \ # fake_dst_ip should be set to the IP address of a neighbor on the\ # Ethernet network. (For example the router the system is directly\ # connected to.)\ #\ # The net tile queries the route and neighbor tables as part of the\ # benchmark. In short, these resolve the dst MAC address of the\ # outgoing Ethernet packets. Since pktgen does not produce\ # valid IPv4 packets, no traffic actually gets sent to this IP.\ #\ # The IP address should fulfill these requirements:\ # - 'ip route get ' resolves to the XDP interface\ # (see [net.interface])\ # - 'ip neigh get dev ' returns REACHABLE\ # If these requirements are not met, no packets will be sent out.\ fake_dst_ip = "" # e.g. "192.0.2.64"\ \ # The following options relate to the 'fddev udpecho' tool only.\ # This testing feature echos back incoming UDP packets using the\ # production networking stack.\ [development.udpecho]\ affinity = "auto"\ \ # Development only options for the GUI tile. These should not be\ # change on a production validator.\ [development.gui]\ # Enables support for WebSocket compression. Compression may\ # improve startup speed when loading the GUI and decrease\ # overall bandwidth use. GUI WebSocket messages are by default\ # always a single WebSocket text frame with plaintext JSON\ # contents. If this option is enabled, a client may indicate\ # support for receiving compressed messages with the\ # Sec-WebSocket-Protocol header.\ #\ # If compression is successfully negotiated, the server will\ # optionally compress large frames with ZSTD, indicating this\ # by marking the frame as binary instead of text.\ #\ # This option has not been tested or security audited and should\ # not currently be used in production.\ websocket_compression = false --- # Frequently Asked Questions | Firedancer [Skip to content](https://docs.firedancer.io/guide/faq.html#VPContent) Return to top Frequently Asked Questions [​](https://docs.firedancer.io/guide/faq.html#frequently-asked-questions) ===================================================================================================== What hardware do I need to run Frankendancer? The current Frankendancer hardware requirements are the same as that of an Agave validator. Refer to the [Hardware](https://docs.firedancer.io/guide/getting-started.html#hardware-requirements) section in the [Getting Started](https://docs.firedancer.io/guide/getting-started.html) guide for more details. How can I obtain the Frankendancer binaries? Frankendancer does not currently provide pre-built binaries. It is recommended to build the binaries on the same host where you are planning to run the validator. Frankendancer detects system properties and tries to build a binary tuned for the particular host. Take a look at the [getting started](https://docs.firedancer.io/guide/getting-started.html) guide for requirements and instructions. What branch or tag should I build from? You should checkout and build from a tagged release. The latest version is `v0.819.30111`. For more information, refer to the [releases](https://docs.firedancer.io/guide/getting-started.html#releases) section. How do I resolve errors encountered while starting up Frankendancer? The Frankendancer binary `fdctl` tries to provide helpful error messages to identify the problem and sometimes even suggests solutions. Take a look at the [troubleshooting](https://docs.firedancer.io/guide/troubleshooting.html) guide for some easy steps that can mitigate some common issues. Can Agave and Frankendancer use the same ledger and snapshots? Yes, Frankendancer is fully compatible with both the snapshot and the ledger formats of the Agave validator. How can I monitor the status of my Frankendancer node? You can use most of the regular monitoring tools and commands that you typically would use with an Agave validator to monitor Frankendancer as well. Refer to the [monitoring](https://docs.firedancer.io/guide/monitoring.html) guide for some helpful commands. Why is my node still delinquent? There could be several reasons, some of which include the validator being unable to catchup and the validator not voting properly among others. Take a look at the [tuning](https://docs.firedancer.io/guide/tuning.html) guide for some tips on how to configure Frankendancer to increase the performance of the replay stage so the validator catches up faster. Also make sure that your node is staked and the stake is active. --- # Troubleshooting | Firedancer [Skip to content](https://docs.firedancer.io/guide/troubleshooting.html#VPContent) On this page Troubleshooting [​](https://docs.firedancer.io/guide/troubleshooting.html#troubleshooting) =========================================================================================== This page has a collection of common troubleshooting steps when operators encounter errors while building and running Frankendancer. If these do not address the problem, send a message in the `#firedancer-operators` channel on the Solana Tech Discord or file an issue on GitHub. Building [​](https://docs.firedancer.io/guide/troubleshooting.html#building) ----------------------------------------------------------------------------- ### General Recommendations [​](https://docs.firedancer.io/guide/troubleshooting.html#general-recommendations) * It is always a good idea to retry building everything again from scratch. Do a fresh clone of the repository, following the instructions in the [Getting Started](https://docs.firedancer.io/guide/getting-started.html#prerequisites) guide. Remember to check if you're using a supported compiler and to run `./deps.sh`! * If you're updating an existing repository clone, be sure to update the solana submodule _after_ pulling the latest changes. For example: sh ~/firedancer $ git fetch ~/firedancer $ git checkout v0.819.30111 ~/firedancer $ git submodule update ### Specific Errors [​](https://docs.firedancer.io/guide/troubleshooting.html#specific-errors) * Missing `cargo` binary from rust toolchain sh error: the 'cargo' binary, normally provided by the 'cargo' component, is not applicable to the '1.75.0-x86_64-unknown-linux-gnu' toolchain + exec cargo +1.75.0 build --profile=release-with-debug --lib -p agave-validator error: the 'cargo' binary, normally provided by the 'cargo' component, is not applicable to the '1.75.0-x86_64-unknown-linux-gnu' toolchain make: *** [src/app/fdctl/Local.mk:107: cargo-validator] Error 1 This typically happens due to a race condition between trying to install the correct version of the rust toolchain and using it. Separately re-installing the toolchain fixes it (replace `1.75.0` with the appropriate version): sh rustup toolchain uninstall 1.75.0-x86_64-unknown-linux-gnu rustup toolchain install 1.75.0-x86_64-unknown-linux-gnu Configuring [​](https://docs.firedancer.io/guide/troubleshooting.html#configuring) ----------------------------------------------------------------------------------- ### General Recommendations [​](https://docs.firedancer.io/guide/troubleshooting.html#general-recommendations-1) * If there are errors during `fdctl configure init all --config ~/config.toml`, consider running `fdctl configure fini all --config ~/config.toml` to remove all existing configuration and try the `init` command again. You can also re-run a specific configure stage, for example, `fdctl configure init workspace --config ~/config.toml`. * Make sure the `config.toml` specified during this command is the same as the one specified with the `run` command. Also make sure that the content is valid TOML. * Read the output of the command carefully, `fdctl` often prints out a helpful message that contains suggestions on how to resolve some errors. Be sure to try them out! Running [​](https://docs.firedancer.io/guide/troubleshooting.html#running) --------------------------------------------------------------------------- ### General Recommendations [​](https://docs.firedancer.io/guide/troubleshooting.html#general-recommendations-2) * Make sure the `~/config.toml` being used is the same in the `configure` and `run` commands. --- # Netlink Integration | Firedancer [Skip to content](https://docs.firedancer.io/guide/internals/netlink.html#VPContent) On this page Netlink Integration [​](https://docs.firedancer.io/guide/internals/netlink.html#netlink-integration) ===================================================================================================== Summary [​](https://docs.firedancer.io/guide/internals/netlink.html#summary) ----------------------------------------------------------------------------- Firedancer's userland networking stack sources configuration from netlink to allow mostly zero config interoperability with Linux. This contrasts with other fast networking stacks which typically require complex network configuration or a dedicated IP address. The following describes the netlink integration in detail. Tile Overview [​](https://docs.firedancer.io/guide/internals/netlink.html#tile-overview) ----------------------------------------------------------------------------------------- Firedancer uses XDP for fast networking. This means that some packet processing steps traditionally done in the kernel (with UDP sockets) now have to be done in the Firedancer software. Specifically routing and resolving link-level neighbors. The required information in these steps is requested from the kernel via the [rtnetlink API](https://man7.org/linux/man-pages/man7/rtnetlink.7.html) . Doing all netlink requests in the data path (i.e. in the net tile) bears security risk and is slow. The reasons netlink requests are done in a separate tile are: * **Improved security architecture.** Firedancer's sandbox isolates the netlink interface from untrusted user traffic * **Better performance.** The netlink tile provides shared memory caches that greatly reduce the amount of netlink requests. ### "Netbase" shared memory region [​](https://docs.firedancer.io/guide/internals/netlink.html#netbase-shared-memory-region) The netlink tile keeps a read-only cache of the following information: * Interface table * IPv4 route tables `local` and `main` * Neighbor tables (only for XDP-enabled Ethernet interfaces) The objects containing the above information are stored in the "netbase" workspace. (A workspace is a shared memory region) ### Security [​](https://docs.firedancer.io/guide/internals/netlink.html#security) A netlink tile requires an rtnetlink socket. On startup, it subscribes to route and neighbor table changes. It will also issue RTM\_GETROUTE and RTM\_GETNEIGH requests. On RHEL 8 with a Linux 4.18 kernel, all netlink interactions (including creation of the socket) can be done from a regular unprivileged user without capabilities. The kernel's netlink interface exposes a large attack surface. Therefore, this tile attempts to isolate itself from direct untrusted inputs. ### Data flows [​](https://docs.firedancer.io/guide/internals/netlink.html#data-flows) * `[net tiles] <-- [netbase]` Net tiles have read only access to the shared memory region backing the netbase object. A malicious netlink tile can compromise net tiles by corrupting the netbase object, but not vice versa. * `[changes by sysadmin] --> [netlink] --> [netlink tile]` Route table updates are forwarded to the netlink tile. This occurs rarely (typically if the sysadmin performs manual changes or if due to a system daemon). * `[netlink tile] --> [netbase]` The netlink tile writes neighbor and route table updates to a shared memory region. * `[neighbor discovery] --> [netlink] --> [netlink tile]` Neighbor table updates are forwarded ot the netlink tile. This path has limited throughput (few ~100K updates per second). * `[untrusted traffic] --> [net tile] --> [app tile]` `--> [net tile] --> [netlink tile] --> [neighbor discovery]` App tiles will blindly respond to the source IP found in untrusted packets. This source IP can be spoofed. Neighbor solicitation might be required in order to find out the MAC address of that IP. On IPv4, these are ARP requests broadcasted to the local network. Net tiles cannot solicit neighbors directly, so they notify the netlink tile that neighbor solicitation is needed. (Potentially at line rate if network configuration is part of a huge subnet) The netlink tile will deduplicate these requests and forward them to the kernel. This path is the only direct 'untrusted traffic' -> 'netlink tile' data flow, so the internal neighbor solicit message format is kept as simple as possible for security. ### Neighbor discovery (ARP) [​](https://docs.firedancer.io/guide/internals/netlink.html#neighbor-discovery-arp) A concurrent open addressed hash table is used to store ARP entries (henceforth called "neighbor table"). This table attempts to continuously stay in sync with the kernel. The netlink tile requests neighbor solicitations via the netlink equivalent of `ip neigh add dev DEVICE IP use`. ### Routing [​](https://docs.firedancer.io/guide/internals/netlink.html#routing) The Firedancer network stack supports very simple routing tables as typically seen on cloud instances, servers directly connected to an Ethernet switch, or a router. Only the "local" and "main" routing tables are synchronized. Policy based routing and additional routing tables are NOT supported. Outgoing traffic matching the "local" table is sent to the loopback device. --- # Net Tile | Firedancer [Skip to content](https://docs.firedancer.io/guide/internals/net_tile.html#VPContent) On this page Net Tile [​](https://docs.firedancer.io/guide/internals/net_tile.html#net-tile) ================================================================================ Net tiles provide fast networking to a Firedancer system. At a high level, a net tile acts as a translation layer between the Internet (IPv4) and the Firedancer messaging subsystem ("tango"). Currently, all versions of Firedancer use the Linux AF\_XDP APIs, which bypass most of the Linux network stack. AF\_XDP bypasses large parts of the kernel's routing code, avoids expensive context switches, and most notably offloads (most) data copies to network hardware (often called "zero copy" I/O). A net tile never sleeps (busy polling). It consists of a simple event loop that passes incoming packets down to app tiles, and routes outgoing packets to the right network interface. Internally, it wakes up the kernel ~20k times a second to do batches of RX and TX. ### Terminology [​](https://docs.firedancer.io/guide/internals/net_tile.html#terminology) * NIC: "Network interface" * XDP: "eXpress Data Path", a family of Linux APIs * AF\_XDP: XDP sockets family * XSK: "XDP socket" (singular) * rtnetlink: A Linux API to query various networking information * netlink: In this context, refers to rtnetlink * ethtool: A Linux API to configure Ethernet devices * RX: "Receive" (Packets from outside entering a Firedancer host) * TX: "Transmit" (Packets sent by a Firedancer app tile) * UMEM: Buffer space for packets * UMEM frame: A single buffer that can fit one packet XDP modes [​](https://docs.firedancer.io/guide/internals/net_tile.html#xdp-modes) ---------------------------------------------------------------------------------- The terms `skb` and `drv` are used throughout the codebase and documentation. Simply put, `drv` is the fast mode that is not available everywhere. `skb` is the slow fallback that is always supported. The term `skb` comes from [`struct sk_buff`](https://docs.kernel.org/networking/skbuff.html) , a major part of the Linux network stack. The term `drv` refers to XDP support being implemented in a network device driver before `struct sk_buff` would be allocated. `drv` mode is considered less stable than `skb`. This is because each device driver contains a fair amount of XDP implementation boilerplate that changes rather frequently (due to new XDP features being added to the kernel and support for new devices being added to drivers). HISTORY The performance target for XDP RX is ~20 million packets per second. A proof-of-concept net tile rewrite achieved this number on an Ivy Bridge CPU and an Intel XL710 that was flooded with externally generated packets. The kernel configuration was a recent Linux version with i40e running in XDP drv mode, preferred busy polling, and zero copy I/O. The net tile achieves far less currently but is being optimized incrementally. XDP Lifecycle [​](https://docs.firedancer.io/guide/internals/net_tile.html#xdp-lifecycle) ------------------------------------------------------------------------------------------ REFERENCE The XDP API is explained in detail in [`src/waltz/xdp/fd_xsk.h`](https://raw.githubusercontent.com/firedancer-io/firedancer/refs/heads/main/src/waltz/xdp/fd_xsk.h) : [https://github.com/firedancer-io/firedancer/blob/main/src/waltz/xdp/fd\_xsk.h](https://github.com/firedancer-io/firedancer/blob/main/src/waltz/xdp/fd_xsk.h) ### Persistent config [​](https://docs.firedancer.io/guide/internals/net_tile.html#persistent-config) Running `fdctl configure init all` does various NIC configuration. This configuration persists across Firedancer restarts but gets wiped after a reboot. Linux has some undocumented problems with ethtool offloads like GRO and UDP segmentation. Those speed up regular sockets I/O, so are often enabled by default, but cause packet corruption in XDP sockets. The NIC is configured to steer all Firedancer traffic to a single queue per net tile. This is because one XDP socket is required per NIC channel. Multiple XDP sockets per net tile does not scale well. In the future, the net tile may also reconfigure /sys/class/net params such as `gro_flush_timeout` and `napi_defer_hard_irqs`. Those change the scheduling of kernel networking code. ### Ephemeral config [​](https://docs.firedancer.io/guide/internals/net_tile.html#ephemeral-config) Running `fdctl run` sets up the following ephemeral kernel config. This part of the setup is scoped to the lifetime of the Firedancer net tile processes: * Installs an XDP program * Deploys various supporting eBPF maps * Creates AF\_XDP sockets and XDP metadata rings * Binds UMEM regions to AF\_XDP sockets * Creates an rtnetlink socket to monitor for config changes When a Firedancer instance exits gracefully or gets killed, the kernel will undo this config: * Removes the XDP program * Releases the UMEM region (instead using kernel-allocated packet buffers) * Destroys the rtnetlink socket * Destroys the XDP socket * Destroys the eBPF maps Topology [​](https://docs.firedancer.io/guide/internals/net_tile.html#topology) -------------------------------------------------------------------------------- ### TX Links [​](https://docs.firedancer.io/guide/internals/net_tile.html#tx-links) The net tile is mostly agnostic to the TX topology. It only expects tango standard practices: * The metadata ring is an mcache * "mcache publish marks data freed" pattern (assumes TSO memory model) * The data region is a dcache * Chunk addressing with base in The net tile exports flow control information (fseq, fctl) allowing app tiles to be configured for backpressuring TX, although this is not recommended. ### RX Links [​](https://docs.firedancer.io/guide/internals/net_tile.html#rx-links) RX links are inherently unreliable (don't backpressure). Each app tile _should_ be able to sift through packets at line rate. The RX topology is created by the `fd_net_tile_topo` API. For each combination of (net tile, app tile kind) there is one RX mcache. Multiple app tiles of the same kind will each read from the same RX mcache but "take turns" picking out packets based on a load balancing policy. So, if there are 2 net tiles, 3 quic tiles, and 1 shred tile, the following RX links exist. * `net:0` → `quic` * `net:0` → `shred` * `net:1` → `quic` * `net:1` → `shred` There is one UMEM dcache object per net tile. All RX mcaches of each net tile share that same dcache. * `net:0` → `UMEM:0` * `net:1` → `UMEM:1` ### UMEM region [​](https://docs.firedancer.io/guide/internals/net_tile.html#umem-region) A "UMEM" region is the term for XDP packet buffers that appears in the XDP documentation. In Firedancer, a UMEM is a 4K aligned memory region subdivided into 2048 byte frames. Each frame can carry one Ethernet packet. In recent Linux versions UMEMs have become more flexible, but Firedancer does not use those advanced features. Each net tile manages a UMEM region allocated via a dcache object. This UMEM region is bound to one or more XDP sockets. The UMEM region is used for both RX and TX. Each frame is statically assigned for either purpose on startup. What makes the UMEM region special is that it is shared throughout the system. It is accessible to: * Firedancer app tiles (read-only) * Firedancer net tiles (read-write) * The Linux kernel (read-write) * PCIe network devices (read-write via IOMMU) With the right XDP flags (`XDP_FLAGS_DRV_MODE` and `XDP_ZEROCOPY`), this allows for "zero copy I/O". Wherein PCIe network devices can copy incoming Ethernet packets all the way to application tiles without software copies along the way and without compromising on security. Under the hood, the PCIe device initiates writes to DRAM (DMA) and signals to the device driver that a new packet arrived. The driver then passes a pointer to the packet through to the kernel's XDP code. Which in turn passes it to the net tile. The net tile identifies the recipient app tile for that packet, and passes along that pointer to the corresponding mcache. NOTE BY RIPATEL Firedancer did not initially have a zero copy RX path. Yet, changing to one required less than 500 lines of code to be changed at the net tile. No changes to app tile code were required. I've seen the same task take major refactors in other code bases. This is the subtle beauty of KB's Tango message queue design, shaped by over a decade of high-performance networking. ### Security Protections [​](https://docs.firedancer.io/guide/internals/net_tile.html#security-protections) As any other tile, the net tile and network-facing app tiles are heavily sandboxed by using seccomp, user namespaces, and dropping capabilities. UMEM regions and RX mcaches are mapped read-only to app tiles. This prevents a malicious app tile from corrupting unrelated traffic. It does, however, allow any app tile to eavesdrop on any other incoming packets (including packets that are not sent to Firedancer). TIP Completely isolating control plane traffic from Firedancer requires the use of separate physical network interfaces. The net tile read-only maps TX mcaches and dcaches containing outgoing packets from app tiles. Each app tile can only access its own TX link but cannot see any other TX links. The net tile speculatively copies TX packets but checks for overruns. RX Lifecycle [​](https://docs.firedancer.io/guide/internals/net_tile.html#rx-lifecycle) ---------------------------------------------------------------------------------------- ┌──────┐ ┌──────┐ │ FILL ┼─────► RX │ └─▲────┘ └────┬─┘ │ │ │ ┌────────┐ │ │ │ MCACHE │ │ ├──┼ QUIC ◄───┤ │ └────────┘ │ │ │ │ ┌────────┐ │ │ │ MCACHE │ │ └──┼ SHRED ◄───┘ └────────┘ Figure visualizing how ownership of RX buffers transitions across ring buffers / FIFO queues. The RX lifecycle of each incoming packet involves three steps. The net tile keeps track of the state of each buffer by storing a pointer to each buffer in three corresponding rings. 1. FILL ring: Give free packet buffers to the kernel for the NIC to fill 2. RX ring: Reap filled buffers containing newly arrived packets 3. MCACHE: Make packets visible to app tiles while simultaneously freeing old buffers Each RX buffer (in UMEM) endlessly cycles through these three states. Each step is explained in more detail below. ### FILL ring [​](https://docs.firedancer.io/guide/internals/net_tile.html#fill-ring) There is one FILL ring per XDP socket. The XDP FILL ring consists of bare 64-bit aligned pointers addressed relative to the start of the UMEM region. The FILL ring is allocated by the kernel and read-write mapped to userspace by the net tile. The pointers in the fill ring indicate to the kernel / NIC: "You can write incoming packet data to the buffers behind these pointers". If the FILL ring is empty, incoming packets will get dropped as there is no space to write them to. All RX buffers start out in the FILL ring. Whenever another buffer frees up (see below for free conditions), it gets added back to the XDP FILL ring. ### RX ring [​](https://docs.firedancer.io/guide/internals/net_tile.html#rx-ring) There is one RX ring per XDP socket. The kernel publishes XDP descriptors of newly arrived packets to the XDP RX ring: [`struct xdp_desc`](https://elixir.bootlin.com/linux/v6.13.2/source/include/uapi/linux/if_xdp.h#L155) The net tile consumes XDP descriptors from the XDP RX ring. After briefly deciding what to do with the packet, the net tile either frees the buffer immediately (returning it to the FILL ring) or forwards it to the mcache ring, see below. LINUX NAPI The process how the kernel gets notified of a newly arrived packet is fairly involved and deserves a multi-page writeup itself. The NAPI kernel docs are a good start: [https://docs.kernel.org/networking/napi.html](https://docs.kernel.org/networking/napi.html) ### mcache ring (RX) [​](https://docs.firedancer.io/guide/internals/net_tile.html#mcache-ring-rx) There is one mcache ring per downstream tile interested in traffic. The number of UMEM buffers in 'MCACHE' state is equal to the depths of all mcache RX rings. The `chunk` field of each mcache slot/line points to a different UMEM buffer. Whenever the net tile produces a new message onto the mcache, it changes the `chunk` field to the new buffer. The old buffer (based on the previous value of the `chunk` field) is immediately sent to the FILL ring. The RX mcache ring does not backpressure. Too slow app tiles will get overridden. The net tile produces tango descriptors of arrived packets to the mcache ring: [`fd_frag_meta_t`](https://github.com/firedancer-io/firedancer/blob/v0.305.20111/src/tango/fd_tango_base.h#L146) For incoming packets, a frag\_meta is interpreted as follows. (Outgoing packets use a slightly different schema) | Name | Type | Description | | --- | --- | --- | | `seq` | `u64` | Local sequence number | | `sig` | `u64` | Compressed `netmux_sig` fields | | `chunk` | `u32` | wksp offset right shifted by 6 bits | | `sz` | `u16` | Packet size including Ethernet header | | `ctl` | `u16` | Low 6 bits of wksp offset | | `tsorig` | `u32` | Unused | | `tspub` | `u32` | Unused | Given the workspace base address (see `fd_wksp.h`) of the UMEM, the absolute address of a packet is reconstructed as follows: c static inline void const * rx_pkt_laddr( void * base, ulong chunk, ulong ctl ) { return (void *)( (ulong)base + (chunk<<6) + ctl ); } NOTE The "chunk index" field is a 64 byte aligned offset. This is fine for most other tiles because they only produce 64 byte aligned messages. Most other tiles also don't use the `ctl` field. The XDP API does not have alignment guarantees for received packets, so the net tile just reuses the `ctl` field to encode the low bits of the packet offset (a value in range `[0,63]`). The sig field allows for fast payload filtering without having to read packet contents. It consists of the following bit fields: | Bits | Type | Description | | --- | --- | --- | | `[0,32)` | `u32` | IPv4 source address (big endian) |\ | `[32,40)` | `u8` | Protocol index |\ | `[40,44)` | `u4` | Compressed offset of UDP payload |\ | `[44,64)` | `u20` | Load balancing hash |\ \ The load balancing hash is derived from the packet's IP source address and source port. In case there are multiple tiles of the same kind (e.g. quic), packets with the same load balancing hash will be handled by the same tile.\ \ TX Lifecycle [​](https://docs.firedancer.io/guide/internals/net_tile.html#tx-lifecycle)\ \ ----------------------------------------------------------------------------------------\ \ The TX lifecycle of each incoming packet involves three steps:\ \ ┌──────┐ ┌──────┐\ │ FREE ┼──────► TX │\ └─▲────┘ └────┬─┘\ │ │\ │ ┌────────────┐ │\ └─┼ COMPLETION ◄─┘\ └────────────┘\ \ Figure visualizing how ownership of TX buffers\ transitions across ring buffers / FIFO queues.\ \ ### mcache rings (TX) [​](https://docs.firedancer.io/guide/internals/net_tile.html#mcache-rings-tx)\ \ App tiles instruct net tiles to send packets via mcache TX rings. All net tiles listen for new outgoing packets on all TX mcaches, taking turns according to the packet's load balancing hash.\ \ The frag\_meta of these rings is interpreted as follows:\ \ | Name | Type | Description |\ | --- | --- | --- |\ | `seq` | `u64` | Local sequence number |\ | `sig` | `u64` | Compressed `netmux_sig` fields |\ | `chunk` | `u32` | wksp offset right shifted by 6 bits |\ | `sz` | `u16` | Packet size including Ethernet header |\ | `ctl` | `u16` | Ignored |\ | `tsorig` | `u32` | Ignored |\ | `tspub` | `u32` | Ignored |\ \ Packet data is 64 byte aligned.\ \ The sig field consists of the following subfields.\ \ | Bits | Type | Description |\ | --- | --- | --- |\ | `[0,32)` | `u32` | IPv4 dest address (big endian) |\ | `[40,44)` | `u4` | Compressed offset of UDP payload |\ | `[44,64)` | `u20` | Load balancing hash |\ \ ### FREE, TX rings [​](https://docs.firedancer.io/guide/internals/net_tile.html#free-tx-rings)\ \ Once a net tile found a packet to send in a mcache TX ring, it attempts to allocate a UMEM TX frame from the tx\_free ring. The packet payload is copied into the frame, and the frame is entered into the TX ring.\ \ ### Completion ring [​](https://docs.firedancer.io/guide/internals/net_tile.html#completion-ring)\ \ Once the kernel finished processing the TX frame, it moves it to the completion ring.\ \ The net tile moves completed frames back to the free ring.\ \ Loopback [​](https://docs.firedancer.io/guide/internals/net_tile.html#loopback)\ \ --------------------------------------------------------------------------------\ \ The first net tile (`net:0`) sets up XDP on the loopback device, for two main reasons:\ \ * For testing and development.\ * The Agave code sends local traffic to itself to as part of routine operation (e.g., when it's the leader it sends votes to its own TPU socket).\ \ The Linux kernel routes outgoing packets addressed to IP addresses owned by the system via loopback. (See `ip route show table local`) The net tile partially matches this behavior. For better performance and simplicity, a second XDP socket is used.\ \ Alternatively, the net tile could have sent such traffic out to the public gateway, in hopes that the traffic gets mirrored back.\ \ But for now, Firedancer also binds XDP to loopback. This is a small performance hit for other traffic, but otherwise won't interfere.\ \ The loopback device only supports XDP in SKB mode.\ \ Development [​](https://docs.firedancer.io/guide/internals/net_tile.html#development)\ \ --------------------------------------------------------------------------------------\ \ ### `fddev pktgen` [​](https://docs.firedancer.io/guide/internals/net_tile.html#fddev-pktgen)\ \ The `fddev pktgen` subcommand floods an Ethernet neighbor with small packets. The purpose of this tool is to benchmark the TX path of the net tile.\ \ `fddev pktgen` runs a single net tile and a special pktgen tile generating about 10 million 64 byte sized Ethernet frames per second.\ \ While this application will cause significant load on the Ethernet network, it cannot harm any other IP networks, because the packets generated are not routable.\ \ Running pktgen may result in no packets getting sent out. This happens typically because the `netlnk` tile couldn't resolve the Ethernet neighbor behind the IP address `[development.pktgen.fake_dst_ip]`.\ \ To add a fake neighbor, run:\ \ shell\ \ ip neigh add FAKE_DST_IP dev DEVICE lladdr 00:00:00:00:00:00\ \ Where `FAKE_DST_IP` is an IP like `10.0.0.1` and `DEVICE` is the Linux network device name like `eth0`.\ \ Considerations [​](https://docs.firedancer.io/guide/internals/net_tile.html#considerations)\ \ --------------------------------------------------------------------------------------------\ \ These are the main known considerations and limitations of the Firedancer v0.4 net tile. This list is likely to become out of date.\ \ * Firedancer does not support IPv6\ * It would be a trivial exercise to add IPv6 support\ * As of February 2025, practically all Solana traffic uses IPv4, to the point of command-line apps and public validator dashboards not even displaying IPv6 addresses\ * IPv6 is considerably more expensive than IPv4 due to far lower MTUs (1280 on IPv6 vs 1500 on IPv4), mandatory UDP checksums, and longer addresses requiring a more complex route table lookup routine\ * The net tile does not (yet) use `SO_PREFERRED_BUSY_POLL`\ * The net tile supports only one external network interface (in addition to loopback)\ * fdctl does not yet configure IRQ affinity, nor disable NIC interrupts, which can lead hard IRQs and ksoftirqd/NAPI stealing CPU time from random other pinned tiles\ * Firedancer cannot share a network interface with other AF\_XDP apps\ * The net tile only supports simple route tables, see [Netlink](https://docs.firedancer.io/guide/internals/netlink.html)\ .\ * Running Firedancer is believed to cause a performance detriment to apps using Linux networking on the shared interfaces\ * The number of RX mcaches is `O(n*m)` where n is the number of net tiles and m is the number of app tiles. It could be `O(max(n,m))`.\ * The tx\_free ring is probably obsolete. Buffers could be moved from COMPLETION to TX directly. --- # firedancer Command Line Interface | Firedancer [Skip to content](https://docs.firedancer.io/api/firedancer-cli.html#VPContent) On this page `firedancer` Command Line Interface [​](https://docs.firedancer.io/api/firedancer-cli.html#firedancer-command-line-interface) ============================================================================================================================== The Firedancer binary `firedancer` contains many subcommands which can be run from the command line. `firedancer` also supports subcommands from the `fdctl` binary. WARNING There is no stable release for the `firedancer` binary. Run the client and use these commands at your own risk. `add-authorized-voter` [​](https://docs.firedancer.io/api/firedancer-cli.html#add-authorized-voter) ---------------------------------------------------------------------------------------------------- Adds an authorized voter to the running validator. The `` argument is required and must be the path to an Agave style `voter.json` keypair file. If the path is specified as `-` the key will instead be read from `stdin`. WARNING `add-authorized-voter` is only supported with `firedancer` and not `fdctl`. In other words, the command is only supported while running the full client validator and not Frankendancer. WARNING `add-authorized-voter` must be called with the configuration file you started the validator with, like `firedancer add-authorized-voter --config `, if the `config` argument is not provided, the command may not add the authorized voter key on all tiles and your validator may produce invalid votes. It is not generally safe to call `add-authorized-voter`, as another validator might be running with the same authorized voter and vote account. If they both vote concurrently, the validator may violate consensus and be subject to (future) slashing. It is safe to call the command while the validator is running and voting as the client guarantees that votes will not be produced with the new authorized voter key until the key has been gracefully added to the running validator. The command exits successfully (with an exit code of 0) if the authorized voter was added, otherwise it will fail and print diagnostic messages to `stderr`. Reasons for failure include the validator being unable to load or verify the provided authorized voter key, if the provided key is a duplicate that the validator is already using, or if there are too many authorized voters for the running validator (more than 16). | Arguments | Description | | --- | --- | | `` | Path to a `voter.json` keypair file, or `-` to read the JSON formatted key from `stdin` | | `--config ` | Path to a configuration TOML file of the validator to add an authorized voter for. This must be the same configuration file the validator was started with | $ sudo firedancer add-authorized-voter --config ~/config.toml ~/keys/validator-keypair.json NOTICE Authorized voter key added `4UCZB7zfquCVN7GafWETFVLTceNH3nm2mndyHeDuSggC` --- # Metrics | Firedancer [Skip to content](https://docs.firedancer.io/api/metrics.html#VPContent) On this page Metrics [​](https://docs.firedancer.io/api/metrics.html#metrics) ================================================================= Overview [​](https://docs.firedancer.io/api/metrics.html#overview) ------------------------------------------------------------------- Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a [Prometheus](https://prometheus.io/docs/introduction/overview/) HTTP endpoint: config.toml toml [tiles.metric] prometheus_listen_port = 7999 sh $ curl http://localhost:7999/metrics # HELP tile_pid The process ID of the tile. # TYPE tile_pid gauge tile_pid{kind="net",kind_id="0"} 1527373 tile_pid{kind="quic",kind_id="0"} 1527370 tile_pid{kind="quic",kind_id="1"} 1527371 tile_pid{kind="verify",kind_id="0"} 1527369 tile_pid{kind="verify",kind_id="1"} 1527374 tile_pid{kind="dedup",kind_id="0"} 1527365 ... WARNING Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time. There are three metric types reported by Firedancer, following the [Prometheus data model](https://prometheus.io/docs/concepts/metric_types/) : * `counter` — A cumulative metric representing a monotonically increasing counter. * `gauge` — A single numerical value that can go arbitrarily up or down. * `histogram` — Samples observations like packet sizes and counts them in buckets. All Links [​](https://docs.firedancer.io/api/metrics.html#all-links) --------------------------------------------------------------------- There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a `link_kind` and `link_kind_id`) will potentially have multiple consumers reporting the metric, one per consumer (identified by a `kind` and `kind_id`). sh # HELP link_published_size_bytes The total number of bytes read by the link consumer. # TYPE link_published_size_bytes counter link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0 link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0 These link related metrics have four labels, which are to identify the link the metric is for: * `kind` — The name of the tile consuming from the link. * `kind_id` — The tile index of the tile which is consuming from the link. * `link_kind` — The name of the link being consumed. * `link_kind_id` — The link index of the link which is being consumed. | Metric | Type | Description | | --- | --- | --- | | link\_​consumed\_​count | counter | The number of times the link reader has consumed a fragment | | link\_​consumed\_​size\_​bytes | counter | The total number of bytes read by the link consumer | | link\_​filtered\_​count | counter | The number of fragments that were filtered and not consumed | | link\_​filtered\_​size\_​bytes | counter | The total number of bytes read by the link consumer that were filtered | | link\_​overrun\_​polling\_​count | counter | The number of times the link has been overrun while polling | | link\_​overrun\_​polling\_​frag\_​count | counter | The number of fragments the link has not processed because it was overrun while polling | | link\_​overrun\_​reading\_​count | counter | The number of input overruns detected while reading metadata by the consumer | | link\_​overrun\_​reading\_​frag\_​count | counter | The number of fragments the link has not processed because it was overrun while reading | | link\_​slow\_​count | counter | The number of times the consumer was detected as rate limiting consumer by the producer | All Tiles [​](https://docs.firedancer.io/api/metrics.html#all-tiles) --------------------------------------------------------------------- Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice: sh # HELP quic_connections_created The total number of connections that have been created. # TYPE quic_connections_created counter quic_connections_created{kind="quic",kind_id="0"} 42145 quic_connections_created{kind="quic",kind_id="1"} 38268 # HELP quic_connections_aborted Number of connections aborted. # TYPE quic_connections_aborted counter quic_connections_aborted{kind="quic",kind_id="0"} 14 quic_connections_aborted{kind="quic",kind_id="1"} 23 All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for: * `kind` — The tile name the metric is being reported for. * `kind_id` — The tile index of the tile which is reporting the metric. | Metric | Type | Description | | --- | --- | --- | | tile\_​pid | gauge | The process ID of the tile | | tile\_​tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development | | tile\_​last\_​cpu | gauge | Index of the CPU last executed on | | tile\_​context\_​switch\_​involuntary\_​count | counter | The number of involuntary context switches | | tile\_​context\_​switch\_​voluntary\_​count | counter | The number of voluntary context switches | | tile\_​page\_​fault\_​major\_​count | counter | The number of major page faults | | tile\_​page\_​fault\_​minor\_​count | counter | The number of minor page faults | | tile\_​status | gauge | The current status of the tile. 0 is booting, 1 is running. 2 is shutdown | | tile\_​heartbeat | gauge | The last UNIX timestamp in nanoseconds that the tile heartbeated | | tile\_​in\_​backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0 | | tile\_​backpressure\_​count | counter | Number of times the tile has had to wait for one of more consumers to catch up to resume publishing | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="caught\_​up\_​housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Housekeeping) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="processing\_​housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Housekeeping) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="backpressure\_​housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Backpressure + Housekeeping) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="caught\_​up\_​prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Prefrag) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="processing\_​prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Prefrag) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="backpressure\_​prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Backpressure + Prefrag) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="caught\_​up\_​postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Postfrag) | | tile\_​regime\_​duration\_​nanos
{tile\_​regime="processing\_​postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Postfrag) | | tile\_​cpu\_​duration\_​nanos
{cpu\_​regime="wait"} | counter | CPU time spent in each CPU regime (Wait (task was runnable but not scheduled)) | | tile\_​cpu\_​duration\_​nanos
{cpu\_​regime="idle"} | counter | CPU time spent in each CPU regime (Idle (task was not runnable)) | | tile\_​cpu\_​duration\_​nanos
{cpu\_​regime="user"} | counter | CPU time spent in each CPU regime (User (task was scheduled and executing in user mode)) | | tile\_​cpu\_​duration\_​nanos
{cpu\_​regime="system"} | counter | CPU time spent in each CPU regime (System (task was scheduled and executing in kernel mode)) | Genesi Tile [​](https://docs.firedancer.io/api/metrics.html#genesi-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | Ipecho Tile [​](https://docs.firedancer.io/api/metrics.html#ipecho-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | ipecho\_​current\_​shred\_​version | gauge | The current shred version used by the validator | | ipecho\_​connection\_​count | gauge | The number of active connections to the ipecho service | | ipecho\_​connections\_​closed\_​ok | counter | The number of connections to the ipecho service that have been made and closed normally | | ipecho\_​connections\_​closed\_​error | counter | The number of connections to the ipecho service that have been made and closed abnormally | | ipecho\_​bytes\_​read | counter | The total number of bytes read from all connections to the ipecho service | | ipecho\_​bytes\_​written | counter | The total number of bytes written to all connections to the ipecho service | Snapct Tile [​](https://docs.firedancer.io/api/metrics.html#snapct-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapct\_​state | gauge | State of the tile | | snapct\_​full\_​num\_​retries | counter | Number of times we aborted and retried full snapshot download because the peer was too slow | | snapct\_​incremental\_​num\_​retries | counter | Number of times we aborted and retried incremental snapshot download because the peer was too slow | | snapct\_​full\_​bytes\_​read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted | | snapct\_​full\_​bytes\_​written | gauge | Number of bytes written so far from the full snapshot. Might decrease if snapshot load is aborted and restarted | | snapct\_​full\_​bytes\_​total | gauge | Total size of the full snapshot file. Might change if snapshot load is aborted and restarted | | snapct\_​full\_​download\_​retries | gauge | Number of times we retried the full snapshot download because the peer was too slow | | snapct\_​incremental\_​bytes\_​read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted | | snapct\_​incremental\_​bytes\_​written | gauge | Number of bytes written so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted | | snapct\_​incremental\_​bytes\_​total | gauge | Total size of the incremental snapshot file. Might change if snapshot load is aborted and restarted | | snapct\_​incremental\_​download\_​retries | gauge | Number of times we retried the incremental snapshot download because the peer was too slow | | snapct\_​predicted\_​slot | gauge | The predicted slot from which replay starts after snapshot loading finishes. Might change if snapshot load is aborted and restarted | | snapct\_​ssl\_​alloc\_​errors | counter | Number of SSL allocation errors encountered. | Snapld Tile [​](https://docs.firedancer.io/api/metrics.html#snapld-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapld\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapld\_​ssl\_​alloc\_​errors | counter | Number of SSL allocation errors encountered. | Snapdc Tile [​](https://docs.firedancer.io/api/metrics.html#snapdc-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapdc\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapdc\_​full\_​compressed\_​bytes\_​read | gauge | Number of bytes read so far from the compressed full snapshot file. Might decrease if snapshot load is aborted and restarted | | snapdc\_​full\_​decompressed\_​bytes\_​written | gauge | Number of bytes decompressed so far from the full snapshot. Might decrease if snapshot load is aborted and restarted | | snapdc\_​incremental\_​compressed\_​bytes\_​read | gauge | Number of bytes read so far from the compressed incremental snapshot file. Might decrease if snapshot load is aborted and restarted | | snapdc\_​incremental\_​decompressed\_​bytes\_​written | gauge | Number of bytes decompressed so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted | Snapin Tile [​](https://docs.firedancer.io/api/metrics.html#snapin-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapin\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapin\_​full\_​bytes\_​read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted | | snapin\_​incremental\_​bytes\_​read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted | | snapin\_​accounts\_​loaded | gauge | Number of accounts seen during snapshot loading. Includes duplicates. Resets if snapshot load restarts | | snapin\_​accounts\_​replaced | gauge | Number of previously inserted accounts replaced by a later duplicate. Resets if snapshot load restarts | | snapin\_​accounts\_​ignored | gauge | Number of stale duplicate accounts dropped because a previously inserted account was newer. Resets if snapshot load restarts | Snapwr Tile [​](https://docs.firedancer.io/api/metrics.html#snapwr-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapwr\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 4=SHUTDOWN | | snapwr\_​file\_​capacity\_​bytes | gauge | Account database file capacity in bytes | | snapwr\_​file\_​used\_​bytes | gauge | Current number of bytes used in the account database file | Snapwh Tile [​](https://docs.firedancer.io/api/metrics.html#snapwh-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapwh\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 4=SHUTDOWN | Snapla Tile [​](https://docs.firedancer.io/api/metrics.html#snapla-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapla\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapla\_​full\_​accounts\_​hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | | snapla\_​incremental\_​accounts\_​hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | Snapls Tile [​](https://docs.firedancer.io/api/metrics.html#snapls-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapls\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapls\_​full\_​accounts\_​hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | | snapls\_​incremental\_​accounts\_​hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | Snapwm Tile [​](https://docs.firedancer.io/api/metrics.html#snapwm-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snapwm\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snapwm\_​accounts\_​loaded | gauge | Number of accounts seen during snapshot loading. Includes duplicates. Resets if snapshot load restarts | | snapwm\_​accounts\_​replaced | gauge | Number of previously inserted accounts replaced by a later duplicate. Resets if snapshot load restarts | | snapwm\_​accounts\_​ignored | gauge | Number of stale duplicate accounts dropped because a previously inserted account was newer. Resets if snapshot load restarts | | snapwm\_​accounts\_​active | gauge | Current number of accounts in index. Resets if snapshot load restarts | Snaplh Tile [​](https://docs.firedancer.io/api/metrics.html#snaplh-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snaplh\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snaplh\_​full\_​accounts\_​hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | | snaplh\_​incremental\_​accounts\_​hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | Snaplv Tile [​](https://docs.firedancer.io/api/metrics.html#snaplv-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | snaplv\_​state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN | | snaplv\_​full\_​duplicate\_​accounts\_​hashed | gauge | Number of duplicate accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | | snaplv\_​incremental\_​duplicate\_​accounts\_​hashed | gauge | Number of duplicate accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted | Netlnk Tile [​](https://docs.firedancer.io/api/metrics.html#netlnk-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | netlnk\_​drop\_​events | counter | Number of netlink drop events caught | | netlnk\_​link\_​full\_​syncs | counter | Number of full link table syncs done | | netlnk\_​route\_​full\_​syncs | counter | Number of full route table syncs done | | netlnk\_​updates
{netlink\_​msg="link"} | counter | Number of netlink live updates processed (Link) | | netlnk\_​updates
{netlink\_​msg="neigh"} | counter | Number of netlink live updates processed (Neighbor Table Entry) | | netlnk\_​updates
{netlink\_​msg="ipv4\_​route"} | counter | Number of netlink live updates processed (IPv4 Route Table Entry) | | netlnk\_​interface\_​count | gauge | Number of network interfaces | | netlnk\_​route\_​count
{route\_​table="local"} | gauge | Number of IPv4 routes (Local) | | netlnk\_​route\_​count
{route\_​table="main"} | gauge | Number of IPv4 routes (Main) | | netlnk\_​neigh\_​probe\_​sent | counter | Number of neighbor solicit requests sent to kernel | | netlnk\_​neigh\_​probe\_​fails | counter | Number of neighbor solicit requests that failed to send (kernel too slow) | | netlnk\_​neigh\_​probe\_​rate\_​limit\_​host | counter | Number of neighbor solicit that exceeded the per-host rate limit | | netlnk\_​neigh\_​probe\_​rate\_​limit\_​global | counter | Number of neighbor solicit that exceeded the global rate limit | Net Tile [​](https://docs.firedancer.io/api/metrics.html#net-tile) ------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | net\_​rx\_​pkt\_​cnt | counter | Packet receive count | | net\_​rx\_​bytes\_​total | counter | Total number of bytes received (including Ethernet header) | | net\_​rx\_​undersz\_​cnt | counter | Number of incoming packets dropped due to being too small | | net\_​rx\_​fill\_​blocked\_​cnt | counter | Number of incoming packets dropped due to fill ring being full | | net\_​rx\_​backpressure\_​cnt | counter | Number of incoming packets dropped due to backpressure | | net\_​rx\_​busy\_​cnt | gauge | Number of receive buffers currently busy | | net\_​rx\_​idle\_​cnt | gauge | Number of receive buffers currently idle | | net\_​tx\_​submit\_​cnt | counter | Number of packet transmit jobs submitted | | net\_​tx\_​complete\_​cnt | counter | Number of packet transmit jobs marked as completed by the kernel | | net\_​tx\_​bytes\_​total | counter | Total number of bytes transmitted (including Ethernet header) | | net\_​tx\_​route\_​fail\_​cnt | counter | Number of packet transmit jobs dropped due to route failure | | net\_​tx\_​neighbor\_​fail\_​cnt | counter | Number of packet transmit jobs dropped due to unresolved neighbor | | net\_​tx\_​full\_​fail\_​cnt | counter | Number of packet transmit jobs dropped due to XDP TX ring full or missing completions | | net\_​tx\_​busy\_​cnt | gauge | Number of transmit buffers currently busy | | net\_​tx\_​idle\_​cnt | gauge | Number of transmit buffers currently idle | | net\_​xsk\_​tx\_​wakeup\_​cnt | counter | Number of XSK sendto syscalls dispatched | | net\_​xsk\_​rx\_​wakeup\_​cnt | counter | Number of XSK recvmsg syscalls dispatched | | net\_​xdp\_​rx\_​dropped\_​other | counter | Dropped for other reasons (xdp\_statistics\_v0.rx\_dropped) | | net\_​xdp\_​rx\_​invalid\_​descs | counter | Dropped due to invalid descriptor (xdp\_statistics\_v0.rx\_invalid\_descs) | | net\_​xdp\_​tx\_​invalid\_​descs | counter | Dropped due to invalid descriptor (xdp\_statistics\_v0.tx\_invalid\_descs) | | net\_​xdp\_​rx\_​ring\_​full | counter | Dropped due to rx ring being full (xdp\_statistics\_v1.rx\_ring\_full) | | net\_​xdp\_​rx\_​fill\_​ring\_​empty\_​descs | counter | Failed to retrieve item from fill ring (xdp\_statistics\_v1.rx\_fill\_ring\_empty\_descs) | | net\_​xdp\_​tx\_​ring\_​empty\_​descs | counter | Failed to retrieve item from tx ring (xdp\_statistics\_v1.tx\_ring\_empty\_descs) | | net\_​rx\_​gre\_​cnt | counter | Number of valid GRE packets received | | net\_​rx\_​gre\_​invalid\_​cnt | counter | Number of invalid GRE packets received | | net\_​rx\_​gre\_​ignored\_​cnt | counter | Number of received but ignored GRE packets | | net\_​tx\_​gre\_​cnt | counter | Number of GRE packet transmit jobs submitted | | net\_​tx\_​gre\_​route\_​fail\_​cnt | counter | Number of GRE packets transmit jobs dropped due to route failure | | net\_​rx\_​src\_​addr\_​invalid\_​cnt | counter | Number of incoming packets dropped due to invalid source IP address | Sock Tile [​](https://docs.firedancer.io/api/metrics.html#sock-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | sock\_​syscalls\_​sendmmsg
{sock\_​err="no\_​error"} | counter | Number of sendmmsg syscalls dispatched (No error) | | sock\_​syscalls\_​sendmmsg
{sock\_​err="slow"} | counter | Number of sendmmsg syscalls dispatched (ENOBUFS, EAGAIN error) | | sock\_​syscalls\_​sendmmsg
{sock\_​err="perm"} | counter | Number of sendmmsg syscalls dispatched (EPERM error (blocked by netfilter)) | | sock\_​syscalls\_​sendmmsg
{sock\_​err="unreach"} | counter | Number of sendmmsg syscalls dispatched (ENETUNREACH, EHOSTUNREACH error) | | sock\_​syscalls\_​sendmmsg
{sock\_​err="down"} | counter | Number of sendmmsg syscalls dispatched (ENONET, ENETDOWN, EHOSTDOWN error) | | sock\_​syscalls\_​sendmmsg
{sock\_​err="other"} | counter | Number of sendmmsg syscalls dispatched (Unrecognized error code) | | sock\_​syscalls\_​recvmmsg | counter | Number of recvmsg syscalls dispatched | | sock\_​rx\_​pkt\_​cnt | counter | Number of packets received | | sock\_​tx\_​pkt\_​cnt | counter | Number of packets sent | | sock\_​tx\_​drop\_​cnt | counter | Number of packets failed to send | | sock\_​tx\_​bytes\_​total | counter | Total number of bytes transmitted (including Ethernet header) | | sock\_​rx\_​bytes\_​total | counter | Total number of bytes received (including Ethernet header) | Quic Tile [​](https://docs.firedancer.io/api/metrics.html#quic-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | quic\_​txns\_​overrun | counter | Count of txns overrun before reassembled (too small txn\_reassembly\_count) | | quic\_​txn\_​reasms\_​started | counter | Count of fragmented txn receive ops started | | quic\_​txn\_​reasms\_​active | gauge | Number of fragmented txn receive ops currently active | | quic\_​frags\_​ok | counter | Count of txn frags received | | quic\_​frags\_​gap | counter | Count of txn frags dropped due to data gap | | quic\_​frags\_​dup | counter | Count of txn frags dropped due to dup (stream already completed) | | quic\_​txns\_​received
{tpu\_​recv\_​type="udp"} | counter | Count of txns received via TPU (TPU/UDP) | | quic\_​txns\_​received
{tpu\_​recv\_​type="quic\_​fast"} | counter | Count of txns received via TPU (TPU/QUIC unfragmented) | | quic\_​txns\_​received
{tpu\_​recv\_​type="quic\_​frag"} | counter | Count of txns received via TPU (TPU/QUIC fragmented) | | quic\_​txns\_​abandoned | counter | Count of txns abandoned because a conn was lost | | quic\_​txn\_​undersz | counter | Count of txns received via QUIC dropped because they were too small | | quic\_​txn\_​oversz | counter | Count of txns received via QUIC dropped because they were too large | | quic\_​legacy\_​txn\_​undersz | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet | | quic\_​legacy\_​txn\_​oversz | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction | | quic\_​received\_​packets | counter | Number of IP packets received | | quic\_​received\_​bytes | counter | Total bytes received (including IP, UDP, QUIC headers) | | quic\_​sent\_​packets | counter | Number of IP packets sent | | quic\_​sent\_​bytes | counter | Total bytes sent (including IP, UDP, QUIC headers) | | quic\_​connections\_​alloc | gauge | The number of currently allocated QUIC connections | | quic\_​connections\_​state
{quic\_​conn\_​state="invalid"} | gauge | The number of QUIC connections in each state (Freed) | | quic\_​connections\_​state
{quic\_​conn\_​state="handshake"} | gauge | The number of QUIC connections in each state (Handshaking peer) | | quic\_​connections\_​state
{quic\_​conn\_​state="handshake\_​complete"} | gauge | The number of QUIC connections in each state (Handshake complete, confirming with peer) | | quic\_​connections\_​state
{quic\_​conn\_​state="active"} | gauge | The number of QUIC connections in each state (Active connection) | | quic\_​connections\_​state
{quic\_​conn\_​state="peer\_​close"} | gauge | The number of QUIC connections in each state (Peer requested close) | | quic\_​connections\_​state
{quic\_​conn\_​state="abort"} | gauge | The number of QUIC connections in each state (Connection terminating due to error) | | quic\_​connections\_​state
{quic\_​conn\_​state="close\_​pending"} | gauge | The number of QUIC connections in each state (Connection is closing) | | quic\_​connections\_​state
{quic\_​conn\_​state="dead"} | gauge | The number of QUIC connections in each state (Connection about to be freed) | | quic\_​connections\_​created | counter | The total number of connections that have been created | | quic\_​connections\_​closed | counter | Number of connections gracefully closed | | quic\_​connections\_​aborted | counter | Number of connections aborted | | quic\_​connections\_​timed\_​out | counter | Number of connections timed out | | quic\_​connections\_​retried | counter | Number of connections established with retry | | quic\_​connection\_​error\_​no\_​slots | counter | Number of connections that failed to create due to lack of slots | | quic\_​connection\_​error\_​retry\_​fail | counter | Number of connections that failed during retry (e.g. invalid token) | | quic\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="initial"} | counter | Number of packets with an unknown connection ID (Initial) | | quic\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="retry"} | counter | Number of packets with an unknown connection ID (Retry) | | quic\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="handshake"} | counter | Number of packets with an unknown connection ID (Handshake) | | quic\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="one\_​rtt"} | counter | Number of packets with an unknown connection ID (1-RTT) | | quic\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="success"} | counter | Results of attempts to acquire QUIC frame metadata (Success) | | quic\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="fail\_​empty\_​pool"} | counter | Results of attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) | | quic\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="fail\_​conn\_​max"} | counter | Results of attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) | | quic\_​initial\_​token\_​len
{quic\_​initial\_​token\_​len="zero"} | counter | Number of Initial packets grouped by token length (No token) | | quic\_​initial\_​token\_​len
{quic\_​initial\_​token\_​len="fd\_​quic\_​len"} | counter | Number of Initial packets grouped by token length (fd\_quic retry token length) | | quic\_​initial\_​token\_​len
{quic\_​initial\_​token\_​len="invalid\_​len"} | counter | Number of Initial packets grouped by token length (Invalid token length) | | quic\_​handshakes\_​created | counter | Number of handshake flows created | | quic\_​handshake\_​error\_​alloc\_​fail | counter | Number of handshakes dropped due to alloc fail | | quic\_​handshake\_​evicted | counter | Number of handshakes dropped due to eviction | | quic\_​stream\_​received\_​events | counter | Number of stream RX events | | quic\_​stream\_​received\_​bytes | counter | Total stream payload bytes received | | quic\_​received\_​frames
{quic\_​frame\_​type="unknown"} | counter | Number of QUIC frames received (Unknown frame type) | | quic\_​received\_​frames
{quic\_​frame\_​type="ack"} | counter | Number of QUIC frames received (ACK frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="reset\_​stream"} | counter | Number of QUIC frames received (RESET\_STREAM frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="stop\_​sending"} | counter | Number of QUIC frames received (STOP\_SENDING frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="crypto"} | counter | Number of QUIC frames received (CRYPTO frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="new\_​token"} | counter | Number of QUIC frames received (NEW\_TOKEN frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="stream"} | counter | Number of QUIC frames received (STREAM frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="max\_​data"} | counter | Number of QUIC frames received (MAX\_DATA frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="max\_​stream\_​data"} | counter | Number of QUIC frames received (MAX\_STREAM\_DATA frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="max\_​streams"} | counter | Number of QUIC frames received (MAX\_STREAMS frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="data\_​blocked"} | counter | Number of QUIC frames received (DATA\_BLOCKED frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="stream\_​data\_​blocked"} | counter | Number of QUIC frames received (STREAM\_DATA\_BLOCKED frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="streams\_​blocked"} | counter | Number of QUIC frames received (STREAMS\_BLOCKED(bidi) frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="new\_​conn\_​id"} | counter | Number of QUIC frames received (NEW\_CONN\_ID frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="retire\_​conn\_​id"} | counter | Number of QUIC frames received (RETIRE\_CONN\_ID frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="path\_​challenge"} | counter | Number of QUIC frames received (PATH\_CHALLENGE frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="path\_​response"} | counter | Number of QUIC frames received (PATH\_RESPONSE frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="conn\_​close\_​quic"} | counter | Number of QUIC frames received (CONN\_CLOSE(transport) frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="conn\_​close\_​app"} | counter | Number of QUIC frames received (CONN\_CLOSE(app) frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="handshake\_​done"} | counter | Number of QUIC frames received (HANDSHAKE\_DONE frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="ping"} | counter | Number of QUIC frames received (PING frame) | | quic\_​received\_​frames
{quic\_​frame\_​type="padding"} | counter | Number of QUIC frames received (PADDING frame) | | quic\_​ack\_​tx
{quic\_​ack\_​tx="noop"} | counter | ACK events (Non-ACK-eliciting packet) | | quic\_​ack\_​tx
{quic\_​ack\_​tx="new"} | counter | ACK events (New ACK range) | | quic\_​ack\_​tx
{quic\_​ack\_​tx="merged"} | counter | ACK events (Merged into existing ACK range) | | quic\_​ack\_​tx
{quic\_​ack\_​tx="drop"} | counter | ACK events (Out of buffers) | | quic\_​ack\_​tx
{quic\_​ack\_​tx="cancel"} | counter | ACK events (ACK suppressed by handler) | | quic\_​service\_​duration\_​seconds | histogram | Duration spent in service | | quic\_​receive\_​duration\_​seconds | histogram | Duration spent processing packets | | quic\_​frame\_​fail\_​parse | counter | Number of QUIC frames failed to parse | | quic\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="initial"} | counter | Number of packets that failed decryption (Initial) | | quic\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="early"} | counter | Number of packets that failed decryption (Early data) | | quic\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="handshake"} | counter | Number of packets that failed decryption (Handshake) | | quic\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="app"} | counter | Number of packets that failed decryption (App data) | | quic\_​pkt\_​no\_​key
{quic\_​enc\_​level="initial"} | counter | Number of packets that failed decryption due to missing key (Initial) | | quic\_​pkt\_​no\_​key
{quic\_​enc\_​level="early"} | counter | Number of packets that failed decryption due to missing key (Early data) | | quic\_​pkt\_​no\_​key
{quic\_​enc\_​level="handshake"} | counter | Number of packets that failed decryption due to missing key (Handshake) | | quic\_​pkt\_​no\_​key
{quic\_​enc\_​level="app"} | counter | Number of packets that failed decryption due to missing key (App data) | | quic\_​pkt\_​net\_​header\_​invalid | counter | Number of packets dropped due to weird IP or UDP header | | quic\_​pkt\_​quic\_​header\_​invalid | counter | Number of packets dropped due to weird QUIC header | | quic\_​pkt\_​undersz | counter | Number of QUIC packets dropped due to being too small | | quic\_​pkt\_​oversz | counter | Number of QUIC packets dropped due to being too large | | quic\_​pkt\_​verneg | counter | Number of QUIC version negotiation packets received | | quic\_​retry\_​sent | counter | Number of QUIC Retry packets sent | | quic\_​pkt\_​retransmissions
{quic\_​enc\_​level="initial"} | counter | Number of QUIC packets that retransmitted (Initial) | | quic\_​pkt\_​retransmissions
{quic\_​enc\_​level="early"} | counter | Number of QUIC packets that retransmitted (Early data) | | quic\_​pkt\_​retransmissions
{quic\_​enc\_​level="handshake"} | counter | Number of QUIC packets that retransmitted (Handshake) | | quic\_​pkt\_​retransmissions
{quic\_​enc\_​level="app"} | counter | Number of QUIC packets that retransmitted (App data) | Bundle Tile [​](https://docs.firedancer.io/api/metrics.html#bundle-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | bundle\_​transaction\_​received | counter | Total count of transactions received, including transactions within bundles | | bundle\_​packet\_​received | counter | Total count of packets received | | bundle\_​proto\_​received\_​bytes | counter | Total count of bytes from received grpc protobuf payloads | | bundle\_​bundle\_​received | counter | Total count of bundles received | | bundle\_​errors
{bundle\_​error="protobuf"} | counter | Number of gRPC errors encountered (Protobuf decode/encode error) | | bundle\_​errors
{bundle\_​error="transport"} | counter | Number of gRPC errors encountered (Transport error) | | bundle\_​errors
{bundle\_​error="timeout"} | counter | Number of gRPC errors encountered (I/O timeout) | | bundle\_​errors
{bundle\_​error="no\_​fee\_​info"} | counter | Number of gRPC errors encountered (Bundle dropped due to missing fee info) | | bundle\_​errors
{bundle\_​error="ssl\_​alloc"} | counter | Number of gRPC errors encountered (OpenSSL alloc fail) | | bundle\_​heap\_​size | gauge | Workspace heap size | | bundle\_​heap\_​free\_​bytes | gauge | Approx free space in workspace | | bundle\_​shredstream\_​heartbeats | counter | Number of ShredStream heartbeats successfully sent | | bundle\_​keepalives | counter | Number of HTTP/2 PINGs acknowledged by server | | bundle\_​connected | gauge | 1 if connected to the bundle server, 0 if not | | bundle\_​rtt\_​sample | gauge | Latest RTT sample at scrape time (nanoseconds) | | bundle\_​rtt\_​smoothed | gauge | RTT moving average (nanoseconds) | | bundle\_​rtt\_​var | gauge | RTT variance (nanoseconds) | | bundle\_​message\_​rx\_​delay\_​nanos | histogram | Message receive delay in nanoseconds from bundle server to bundle client | | bundle\_​pending\_​transactions | gauge | Number of transactions buffered and waiting to be published | | bundle\_​transaction\_​dropped\_​backpressure | counter | Transactions dropped because the pending transaction buffer was full (should never happen) | Verify Tile [​](https://docs.firedancer.io/api/metrics.html#verify-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | verify\_​transaction\_​result
{verify\_​tile\_​result="success"} | counter | Count of transaction results through verify tile (Transaction verified successfully) | | verify\_​transaction\_​result
{verify\_​tile\_​result="bundle\_​peer\_​failure"} | counter | Count of transaction results through verify tile (Peer transaction in the bundle failed) | | verify\_​transaction\_​result
{verify\_​tile\_​result="parse\_​failure"} | counter | Count of transaction results through verify tile (Transaction failed to parse) | | verify\_​transaction\_​result
{verify\_​tile\_​result="dedup\_​failure"} | counter | Count of transaction results through verify tile (Transaction failed deduplication) | | verify\_​transaction\_​result
{verify\_​tile\_​result="verify\_​failure"} | counter | Count of transaction results through verify tile (Transaction failed signature verification) | | verify\_​gossiped\_​votes\_​received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path | Dedup Tile [​](https://docs.firedancer.io/api/metrics.html#dedup-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | dedup\_​transaction\_​result
{dedup\_​tile\_​result="success"} | counter | Count of transaction results through dedup tile (Transaction deduplicated successfully) | | dedup\_​transaction\_​result
{dedup\_​tile\_​result="bundle\_​peer\_​failure"} | counter | Count of transaction results through dedup tile (Peer transaction in the bundle failed) | | dedup\_​transaction\_​result
{dedup\_​tile\_​result="dedup\_​failure"} | counter | Count of transaction results through dedup tile (Transaction failed deduplication) | | dedup\_​gossiped\_​votes\_​received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path | Resolv Tile [​](https://docs.firedancer.io/api/metrics.html#resolv-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | resolv\_​no\_​bank\_​drop | counter | Count of transactions dropped because the bank was not available | | resolv\_​stash\_​operation
{resolve\_​stash\_​operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) | | resolv\_​stash\_​operation
{resolve\_​stash\_​operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) | | resolv\_​stash\_​operation
{resolve\_​stash\_​operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) | | resolv\_​stash\_​operation
{resolve\_​stash\_​operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​lookup\_​index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="account\_​uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​account\_​data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​account\_​owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="account\_​not\_​found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) | | resolv\_​lut\_​resolved
{lut\_​resolve\_​result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) | | resolv\_​blockhash\_​expired | counter | Count of transactions that failed to resolve because the blockhash was expired | | resolv\_​transaction\_​bundle\_​peer\_​failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed | Pack Tile [​](https://docs.firedancer.io/api/metrics.html#pack-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | pack\_​schedule\_​microblock\_​duration\_​seconds | histogram | Duration of scheduling one microblock | | pack\_​no\_​sched\_​microblock\_​duration\_​seconds | histogram | Duration of discovering that there are no schedulable transactions | | pack\_​insert\_​transaction\_​duration\_​seconds | histogram | Duration of inserting one transaction into the pool of available transactions | | pack\_​complete\_​microblock\_​duration\_​seconds | histogram | Duration of the computation associated with marking one microblock as complete | | pack\_​total\_​transactions\_​per\_​microblock\_​count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes | | pack\_​votes\_​per\_​microblock\_​count | histogram | Count of simple vote transactions in a scheduled microblock | | pack\_​normal\_​transaction\_​received | counter | Count of transactions received via the normal TPU path | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="instr\_​acct\_​cnt"} | counter | Result of inserting a transaction into the pack object (Transaction has an instruction that references too many accounts) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonce\_​conflict"} | counter | Result of inserting a transaction into the pack object (Bundle with two conflicting durable nonce transactions) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="bundle\_​blacklist"} | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the bundle blacklist) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="invalid\_​nonce"} | counter | Result of inserting a transaction into the pack object (Transaction is an invalid durable nonce transaction) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="write\_​sysvar"} | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="estimation\_​fail"} | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="duplicate\_​account"} | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="too\_​many\_​accounts"} | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="too\_​large"} | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="expired"} | counter | Result of inserting a transaction into the pack object (Transaction already expired) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="addr\_​lut"} | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="unaffordable"} | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="duplicate"} | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonce\_​priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonvote\_​add"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="vote\_​add"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonvote\_​replace"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="vote\_​replace"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonce\_​nonvote\_​add"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction added to pending transactions) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="unused"} | counter | Result of inserting a transaction into the pack object (Unused because durable nonce transactions can't be simple votes) | | pack\_​transaction\_​inserted
{pack\_​txn\_​insert\_​return="nonce\_​nonvote\_​replace"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​no\_​execle\_​no\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​no\_​execle\_​no\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​execle\_​no\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, had execle but wasn't leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​execle\_​no\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, had execle but wasn't leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​no\_​execle\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available execle) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​no\_​execle\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available execle) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​execle\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had available execle but no transactions) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​execle\_​leader\_​no\_​microblock"} | counter | Time in nanos spent in each state (Pack had execle and transactions available but couldn't schedule anything non-conflicting) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​no\_​execle\_​no\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​no\_​execle\_​no\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​execle\_​no\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​execle\_​no\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​no\_​execle\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all execle were busy) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​no\_​execle\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all execle were busy) | | pack\_​metric\_​timing
{pack\_​timing\_​state="no\_​txn\_​execle\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) | | pack\_​metric\_​timing
{pack\_​timing\_​state="txn\_​execle\_​leader\_​microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) | | pack\_​transaction\_​dropped\_​from\_​extra | counter | Transactions dropped from the extra transaction storage because it was full | | pack\_​transaction\_​inserted\_​to\_​extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full | | pack\_​transaction\_​inserted\_​from\_​extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage | | pack\_​transaction\_​expired | counter | Transactions deleted from pack because their TTL expired | | pack\_​transaction\_​deleted | counter | Transactions dropped from pack because they were requested to be deleted | | pack\_​transaction\_​already\_​executed | counter | Transactions dropped from pack because they were already executed (in either the replay or leader pipeline) | | pack\_​transaction\_​dropped\_​partial\_​bundle | counter | Transactions dropped from pack because they were part of a partial bundle | | pack\_​available\_​transactions
{avail\_​txn\_​type="all"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (All transactions in any treap) | | pack\_​available\_​transactions
{avail\_​txn\_​type="regular"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes in the main treap) | | pack\_​available\_​transactions
{avail\_​txn\_​type="votes"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Simple votes) | | pack\_​available\_​transactions
{avail\_​txn\_​type="conflicting"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes that write to a hotly-contended account) | | pack\_​available\_​transactions
{avail\_​txn\_​type="bundles"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Transactions that are part of a bundle) | | pack\_​pending\_​transactions\_​heap\_​size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions | | pack\_​smallest\_​pending\_​transaction | gauge | A lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling | | pack\_​microblock\_​per\_​block\_​limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached | | pack\_​data\_​per\_​block\_​limit | counter | The number of times pack did not pack a microblock because it reached the data per block limit at the start of trying to schedule a microblock | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="taken"} | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="cu\_​limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="fast\_​path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="byte\_​limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="alloc\_​limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the data allocation limit) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="write\_​cost"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="slow\_​path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) | | pack\_​transaction\_​schedule
{pack\_​txn\_​schedule="defer\_​skip"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times) | | pack\_​bundle\_​crank\_​status
{bundle\_​crank\_​result="not\_​needed"} | counter | Result of considering whether bundle cranks are needed (On-chain state in the correct state) | | pack\_​bundle\_​crank\_​status
{bundle\_​crank\_​result="inserted"} | counter | Result of considering whether bundle cranks are needed (Inserted an initializer bundle to update the on-chain state) | | pack\_​bundle\_​crank\_​status
{bundle\_​crank\_​result="creation\_​failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but creation failed) | | pack\_​bundle\_​crank\_​status
{bundle\_​crank\_​result="insertion\_​failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) | | pack\_​cus\_​consumed\_​in\_​block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block | | pack\_​cus\_​scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. | | pack\_​cus\_​rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. | | pack\_​cus\_​net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. | | pack\_​cus\_​pct | histogram | The percent of the total block cost limit used for each block pack produced. | | pack\_​delete\_​missed | counter | Count of attempts to delete a transaction that wasn't found | | pack\_​delete\_​hit | counter | Count of attempts to delete a transaction that was found and deleted | Execle Tile [​](https://docs.firedancer.io/api/metrics.html#execle-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | execle\_​transaction\_​result
{transaction\_​result="success"} | counter | Result of loading and executing a transaction (Transaction executed successfully) | | execle\_​transaction\_​result
{transaction\_​result="instructon\_​error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) | | execle\_​transaction\_​result
{transaction\_​result="account\_​not\_​found"} | counter | Result of loading and executing a transaction (The transaction fee payer address was not found) | | execle\_​transaction\_​result
{transaction\_​result="program\_​account\_​not\_​found"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was not found) | | execle\_​transaction\_​result
{transaction\_​result="insufficient\_​funds\_​for\_​fee"} | counter | Result of loading and executing a transaction (The transaction fee payer did not have balance to pay the fee) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​account\_​for\_​fee"} | counter | Result of loading and executing a transaction (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) | | execle\_​transaction\_​result
{transaction\_​result="already\_​processed"} | counter | Result of loading and executing a transaction (The transaction has already been processed in a recent block) | | execle\_​transaction\_​result
{transaction\_​result="blockhash\_​not\_​found"} | counter | Result of loading and executing a transaction (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​program\_​for\_​execution"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was no executable. TODO: No longer needed with SIMD-0162) | | execle\_​transaction\_​result
{transaction\_​result="address\_​lookup\_​table\_​not\_​found"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that does not exist or is inactive) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​address\_​lookup\_​table\_​owner"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that is not owned by the ALUT program account) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​address\_​lookup\_​table\_​data"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that contains data which is not a valid ALUT) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​address\_​lookup\_​table\_​index"} | counter | Result of loading and executing a transaction (The transaction references an account offset from the ALUT which does not exist) | | execle\_​transaction\_​result
{transaction\_​result="max\_​loaded\_​accounts\_​data\_​size\_​exceeded"} | counter | Result of loading and executing a transaction (The total account data size of the loaded accounts exceeds the consensus limit) | | execle\_​transaction\_​result
{transaction\_​result="duplicate\_​instruction"} | counter | Result of loading and executing a transaction (A compute budget program instruction was invoked more than once) | | execle\_​transaction\_​result
{transaction\_​result="invalid\_​loaded\_​accounts\_​data\_​size\_​limit"} | counter | Result of loading and executing a transaction (The compute budget program was invoked and set the loaded accounts data size to zero) | | execle\_​transaction\_​result
{transaction\_​result="nonce\_​already\_​advanced"} | counter | Result of loading and executing a transaction (The transaction references a nonce account that is already advanced) | | execle\_​transaction\_​result
{transaction\_​result="nonce\_​advance\_​failed"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the advance instruction was not valid or failed) | | execle\_​transaction\_​result
{transaction\_​result="nonce\_​wrong\_​blockhash"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the blockhash is not the correct one) | | execle\_​transaction\_​result
{transaction\_​result="account\_​in\_​use"} | counter | Result of loading and executing a transaction (The transaction conflicts with another transaction in the microblock. TODO: No longer possible with smart dispatcher) | | execle\_​transaction\_​result
{transaction\_​result="account\_​loaded\_​twice"} | counter | Result of loading and executing a transaction (The transaction references the same account twice) | | execle\_​transaction\_​result
{transaction\_​result="signature\_​failure"} | counter | Result of loading and executing a transaction (The transaction had an invalid signature) | | execle\_​transaction\_​result
{transaction\_​result="too\_​many\_​account\_​locks"} | counter | Result of loading and executing a transaction (The transaction references too many accounts. TODO: No longer possible with smart dispatcher) | | execle\_​transaction\_​result
{transaction\_​result="insufficient\_​funds\_​for\_​rent"} | counter | Result of loading and executing a transaction (The transaction would leave an account with a lower balance than the rent-exempt minimum) | | execle\_​transaction\_​result
{transaction\_​result="unbalanced\_​transaction"} | counter | Result of loading and executing a transaction (The total referenced account lamports before and after the transaction was unbalanced) | | execle\_​transaction\_​result
{transaction\_​result="bundle\_​peer"} | counter | Result of loading and executing a transaction (The transaction was part of a bundle and an earlier transaction in the bundle failed) | | execle\_​transaction\_​landed
{transaction\_​landed="landed\_​success"} | counter | Whether a transaction landed in the block or not (Transaction landed) | | execle\_​transaction\_​landed
{transaction\_​landed="landed\_​fees\_​only"} | counter | Whether a transaction landed in the block or not (Transaction landed, but was fees only and did not execute) | | execle\_​transaction\_​landed
{transaction\_​landed="landed\_​failed"} | counter | Whether a transaction landed in the block or not (Transaction landed, but failed to execute) | | execle\_​transaction\_​landed
{transaction\_​landed="unlanded"} | counter | Whether a transaction landed in the block or not (Transaction did not land) | | execle\_​compute\_​units\_​total | counter | Estimated number of compute units executed since tile start | Poh Tile [​](https://docs.firedancer.io/api/metrics.html#poh-tile) ------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | Sign Tile [​](https://docs.firedancer.io/api/metrics.html#sign-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | sign\_​sign\_​duration\_​seconds | histogram | Duration of signing a message | Shred Tile [​](https://docs.firedancer.io/api/metrics.html#shred-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | shred\_​cluster\_​contact\_​info\_​cnt | histogram | Number of contact infos in the cluster contact info message | | shred\_​microblocks\_​abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot | | shred\_​invalid\_​block\_​id | counter | The number of times a block was created with unknown parent block\_id | | shred\_​batch\_​sz | histogram | The size (in bytes) of each microblock batch that is shredded | | shred\_​batch\_​microblock\_​cnt | histogram | The number of microblocks in each microblock batch that is shredded | | shred\_​shredding\_​duration\_​seconds | histogram | Duration of producing one FEC set from the shredder | | shred\_​add\_​shred\_​duration\_​seconds | histogram | Duration of verifying and processing one shred received from the network | | shred\_​shred\_​processed
{shred\_​processing\_​result="bad\_​slot"} | counter | The result of processing a shred from the network (Shred was for a slot for which we don't know the leader) | | shred\_​shred\_​processed
{shred\_​processing\_​result="parse\_​failed"} | counter | The result of processing a shred from the network (Shred parsing failed) | | shred\_​shred\_​processed
{shred\_​processing\_​result="equivoc"} | counter | The result of processing a shred from the network (Shred was equivocated with another shred) | | shred\_​shred\_​processed
{shred\_​processing\_​result="rejected"} | counter | The result of processing a shred from the network (Shred was invalid for one of many reasons) | | shred\_​shred\_​processed
{shred\_​processing\_​result="ignored"} | counter | The result of processing a shred from the network (Shred was ignored because we had already reconstructed the FEC set, or was for an old slot) | | shred\_​shred\_​processed
{shred\_​processing\_​result="duplicate"} | counter | The result of processing a shred from the network (Shred was a duplicate of a previously received valid shred of an incomplete FEC set) | | shred\_​shred\_​processed
{shred\_​processing\_​result="okay"} | counter | The result of processing a shred from the network (Shred accepted to an incomplete FEC set) | | shred\_​shred\_​processed
{shred\_​processing\_​result="completes"} | counter | The result of processing a shred from the network (Shred accepted and resulted in a valid, complete FEC set) | | shred\_​fec\_​set\_​spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space | | shred\_​shred\_​rejected\_​initial | counter | The number of shreds that were rejected before any resources were allocated for the FEC set | | shred\_​shred\_​rejected\_​unchained | counter | The number of shreds that were rejected because they're not chained merkle shreds | | shred\_​fec\_​rejected\_​fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid | | shred\_​bad\_​nonce | counter | The number of shreds we received that seemed to be repair responses but had an incorrect nonce | | shred\_​shred\_​repair\_​rcv | counter | The number of times we received a repair shred | | shred\_​shred\_​repair\_​rcv\_​bytes | counter | The number bytes received from network packets with repair shreds. Bytes include network headers | | shred\_​shred\_​turbine\_​rcv | counter | The number of times we received a turbine shred | | shred\_​shred\_​turbine\_​rcv\_​bytes | counter | The number bytes received from network packets with turbine shreds. Bytes include network headers | Gossvf Tile [​](https://docs.firedancer.io/api/metrics.html#gossvf-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​pull\_​request"} | counter | Number of gossip messages processed (Pull Request (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​pull\_​response"} | counter | Number of gossip messages processed (Pull Response (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​push"} | counter | Number of gossip messages processed (Push (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​prune"} | counter | Number of gossip messages processed (Prune (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​ping"} | counter | Number of gossip messages processed (Ping (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="success\_​pong"} | counter | Number of gossip messages processed (Pong (success)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​unparseable"} | counter | Number of gossip messages processed (Unparseable) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​not\_​contact\_​info"} | counter | Number of gossip messages processed (Pull Request (not contact info)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​loopback"} | counter | Number of gossip messages processed (Pull Request (loopback)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​inactive"} | counter | Number of gossip messages processed (Pull Request (inactive)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​wallclock"} | counter | Number of gossip messages processed (Pull Request (wallclock)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​signature"} | counter | Number of gossip messages processed (Pull Request (signature)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​shred\_​version"} | counter | Number of gossip messages processed (Pull Request (shred version)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​mask\_​bits"} | counter | Number of gossip messages processed (Pull Request (mask bits)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​prune\_​destination"} | counter | Number of gossip messages processed (Prune (destination)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​prune\_​wallclock"} | counter | Number of gossip messages processed (Prune (wallclock)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​prune\_​signature"} | counter | Number of gossip messages processed (Prune (signature)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​push\_​no\_​valid\_​crds"} | counter | Number of gossip messages processed (Push (no valid crds)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pull\_​response\_​no\_​valid\_​crds"} | counter | Number of gossip messages processed (Pull Response (no valid crds)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​ping\_​signature"} | counter | Number of gossip messages processed (Ping (signature)) | | gossvf\_​message\_​rx\_​count
{gossvf\_​message\_​outcome="dropped\_​pong\_​signature"} | counter | Number of gossip messages processed (Pong (signature)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​pull\_​request"} | counter | Total wire bytes of gossip messages processed (Pull Request (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​pull\_​response"} | counter | Total wire bytes of gossip messages processed (Pull Response (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​push"} | counter | Total wire bytes of gossip messages processed (Push (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​prune"} | counter | Total wire bytes of gossip messages processed (Prune (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​ping"} | counter | Total wire bytes of gossip messages processed (Ping (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="success\_​pong"} | counter | Total wire bytes of gossip messages processed (Pong (success)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​unparseable"} | counter | Total wire bytes of gossip messages processed (Unparseable) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​not\_​contact\_​info"} | counter | Total wire bytes of gossip messages processed (Pull Request (not contact info)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​loopback"} | counter | Total wire bytes of gossip messages processed (Pull Request (loopback)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​inactive"} | counter | Total wire bytes of gossip messages processed (Pull Request (inactive)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​wallclock"} | counter | Total wire bytes of gossip messages processed (Pull Request (wallclock)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​signature"} | counter | Total wire bytes of gossip messages processed (Pull Request (signature)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​shred\_​version"} | counter | Total wire bytes of gossip messages processed (Pull Request (shred version)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​request\_​mask\_​bits"} | counter | Total wire bytes of gossip messages processed (Pull Request (mask bits)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​prune\_​destination"} | counter | Total wire bytes of gossip messages processed (Prune (destination)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​prune\_​wallclock"} | counter | Total wire bytes of gossip messages processed (Prune (wallclock)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​prune\_​signature"} | counter | Total wire bytes of gossip messages processed (Prune (signature)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​push\_​no\_​valid\_​crds"} | counter | Total wire bytes of gossip messages processed (Push (no valid crds)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pull\_​response\_​no\_​valid\_​crds"} | counter | Total wire bytes of gossip messages processed (Pull Response (no valid crds)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​ping\_​signature"} | counter | Total wire bytes of gossip messages processed (Ping (signature)) | | gossvf\_​message\_​rx\_​bytes
{gossvf\_​message\_​outcome="dropped\_​pong\_​signature"} | counter | Total wire bytes of gossip messages processed (Pong (signature)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="success\_​pull\_​response"} | counter | Number of CRDS values processed (Pull Response (success)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="success\_​push"} | counter | Number of CRDS values processed (Push (success)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​duplicate"} | counter | Number of CRDS values processed (Pull Response (duplicate)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​signature"} | counter | Number of CRDS values processed (Pull Response (signature)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​origin\_​no\_​contact\_​info"} | counter | Number of CRDS values processed (Pull Response (origin no contact info)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​origin\_​shred\_​version"} | counter | Number of CRDS values processed (Pull Response (origin shred version)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​inactive"} | counter | Number of CRDS values processed (Pull Response (inactive)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​wallclock"} | counter | Number of CRDS values processed (Pull Response (wallclock)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​push\_​signature"} | counter | Number of CRDS values processed (Push (signature)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​push\_​origin\_​no\_​contact\_​info"} | counter | Number of CRDS values processed (Push (origin no contact info)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​push\_​origin\_​shred\_​version"} | counter | Number of CRDS values processed (Push (origin shred version)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​push\_​inactive"} | counter | Number of CRDS values processed (Push (inactive)) | | gossvf\_​crds\_​rx\_​count
{gossvf\_​crds\_​outcome="dropped\_​push\_​wallclock"} | counter | Number of CRDS values processed (Push (wallclock)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="success\_​pull\_​response"} | counter | Total wire bytes of CRDS values processed (Pull Response (success)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="success\_​push"} | counter | Total wire bytes of CRDS values processed (Push (success)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​duplicate"} | counter | Total wire bytes of CRDS values processed (Pull Response (duplicate)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​signature"} | counter | Total wire bytes of CRDS values processed (Pull Response (signature)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​origin\_​no\_​contact\_​info"} | counter | Total wire bytes of CRDS values processed (Pull Response (origin no contact info)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​origin\_​shred\_​version"} | counter | Total wire bytes of CRDS values processed (Pull Response (origin shred version)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​inactive"} | counter | Total wire bytes of CRDS values processed (Pull Response (inactive)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​pull\_​response\_​wallclock"} | counter | Total wire bytes of CRDS values processed (Pull Response (wallclock)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​push\_​signature"} | counter | Total wire bytes of CRDS values processed (Push (signature)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​push\_​origin\_​no\_​contact\_​info"} | counter | Total wire bytes of CRDS values processed (Push (origin no contact info)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​push\_​origin\_​shred\_​version"} | counter | Total wire bytes of CRDS values processed (Push (origin shred version)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​push\_​inactive"} | counter | Total wire bytes of CRDS values processed (Push (inactive)) | | gossvf\_​crds\_​rx\_​bytes
{gossvf\_​crds\_​outcome="dropped\_​push\_​wallclock"} | counter | Total wire bytes of CRDS values processed (Push (wallclock)) | Gossip Tile [​](https://docs.firedancer.io/api/metrics.html#gossip-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | gossip\_​ping\_​tracker\_​capacity | counter | The peer tracking capacity of the ping tracker | | gossip\_​ping\_​tracker\_​count
{ping\_​tracked\_​type="unpinged"} | gauge | The number of peers being tracked for ping/pong (Peer is known but has not been pinged yet) | | gossip\_​ping\_​tracker\_​count
{ping\_​tracked\_​type="invalid"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged but has not yet responded, or responded with an invalid pong) | | gossip\_​ping\_​tracker\_​count
{ping\_​tracked\_​type="valid"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged and is currently valid) | | gossip\_​ping\_​tracker\_​count
{ping\_​tracked\_​type="valid\_​refreshing"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged and is currently valid, and is being refreshed with a new ping) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="staked"} | counter | The result of processing a pong response to a ping (Pong ignored as node is staked) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="entrypoint"} | counter | The result of processing a pong response to a ping (Pong ignored as node is an entrypoint) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="untracked"} | counter | The result of processing a pong response to a ping (Pong ignored as node has no outstanding ping) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="address"} | counter | The result of processing a pong response to a ping (Pong ignored as responding node doesn't match pinged address) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="token"} | counter | The result of processing a pong response to a ping (Pong ignored as token was not valid) | | gossip\_​ping\_​tracker\_​pong\_​result
{pong\_​response="success"} | counter | The result of processing a pong response to a ping (Pong accepted) | | gossip\_​ping\_​tracker\_​evicted\_​count | counter | The number of peers evicted from the ping tracker to make space for new peers | | gossip\_​ping\_​tracked\_​count | counter | The total number of peers ever tracked for ping/pong | | gossip\_​ping\_​tracker\_​stake\_​changed\_​count | counter | The number of times a tracked peer was removed from tracking as it became staked | | gossip\_​ping\_​tracker\_​address\_​changed\_​count | counter | The number of times a tracked peer was removed from tracking as it's gossip address changed | | gossip\_​crds\_​capacity | gauge | The capacity of the data store | | gossip\_​crds\_​count
{crds\_​value="contact\_​info\_​v1"} | gauge | The number of entries in the data store (Contact Info V1) | | gossip\_​crds\_​count
{crds\_​value="vote"} | gauge | The number of entries in the data store (Vote) | | gossip\_​crds\_​count
{crds\_​value="lowest\_​slot"} | gauge | The number of entries in the data store (Lowest Slot) | | gossip\_​crds\_​count
{crds\_​value="snapshot\_​hashes"} | gauge | The number of entries in the data store (Snapshot Hashes) | | gossip\_​crds\_​count
{crds\_​value="accounts\_​hashes"} | gauge | The number of entries in the data store (Accounts Hashes) | | gossip\_​crds\_​count
{crds\_​value="epoch\_​slots"} | gauge | The number of entries in the data store (Epoch Slots) | | gossip\_​crds\_​count
{crds\_​value="version\_​v1"} | gauge | The number of entries in the data store (Version V1) | | gossip\_​crds\_​count
{crds\_​value="version\_​v2"} | gauge | The number of entries in the data store (Version V2) | | gossip\_​crds\_​count
{crds\_​value="node\_​instance"} | gauge | The number of entries in the data store (Node Instance) | | gossip\_​crds\_​count
{crds\_​value="duplicate\_​shred"} | gauge | The number of entries in the data store (Duplicate Shred) | | gossip\_​crds\_​count
{crds\_​value="incremental\_​snapshot\_​hashes"} | gauge | The number of entries in the data store (Incremental Snapshot Hashes) | | gossip\_​crds\_​count
{crds\_​value="contact\_​info\_​v2"} | gauge | The number of entries in the data store (Contact Info V2) | | gossip\_​crds\_​count
{crds\_​value="restart\_​last\_​voted\_​fork\_​slots"} | gauge | The number of entries in the data store (Restart Last Voted Fork Slots) | | gossip\_​crds\_​count
{crds\_​value="restart\_​heaviest\_​fork"} | gauge | The number of entries in the data store (Restart Heaviest Fork) | | gossip\_​crds\_​expired\_​count | counter | The number of entries expired from the data store due to age | | gossip\_​crds\_​evicted\_​count | counter | The number of entries evicted from the data store to make space for new entries | | gossip\_​crds\_​peer\_​capacity | gauge | The capacity for storing peers in the data store | | gossip\_​crds\_​peer\_​staked\_​count | gauge | The number of staked peers in the data store | | gossip\_​crds\_​peer\_​unstaked\_​count | gauge | The number of unstaked peers in the data store | | gossip\_​crds\_​peer\_​total\_​stake | gauge | The total visible stake in the data store, in lamports | | gossip\_​crds\_​peer\_​evicted\_​count | counter | The number of peers evicted from the data store to make space for new peers | | gossip\_​crds\_​purged\_​capacity | gauge | The capacity of the list of purged data store entries | | gossip\_​crds\_​purged\_​count | gauge | The number of purged data store entries | | gossip\_​crds\_​purged\_​expired\_​count | counter | The number of purged entries expired from the purged list due to age | | gossip\_​crds\_​purged\_​evicted\_​count | counter | The number of purged entries evicted from the data store to make space for new entries | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="upserted\_​pull\_​response"} | counter | Outcome of incoming CRDS messages (Pull Response (upserted)) | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="upserted\_​push"} | counter | Outcome of incoming CRDS messages (Push (upserted)) | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="dropped\_​pull\_​response\_​stale"} | counter | Outcome of incoming CRDS messages (Pull Response (newer entry already present in table)) | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="dropped\_​pull\_​response\_​duplicate"} | counter | Outcome of incoming CRDS messages (Pull Response (duplicate)) | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="dropped\_​push\_​stale"} | counter | Outcome of incoming CRDS messages (Push (newer entry already present in table)) | | gossip\_​crds\_​rx\_​count
{gossip\_​crds\_​outcome="dropped\_​push\_​duplicate"} | counter | Outcome of incoming CRDS messages (Push (duplicate)) | | gossip\_​message\_​tx\_​count
{gossip\_​message="pull\_​request"} | counter | Number of gossip messages sent (Pull Request) | | gossip\_​message\_​tx\_​count
{gossip\_​message="pull\_​response"} | counter | Number of gossip messages sent (Pull Response) | | gossip\_​message\_​tx\_​count
{gossip\_​message="push"} | counter | Number of gossip messages sent (Push) | | gossip\_​message\_​tx\_​count
{gossip\_​message="prune"} | counter | Number of gossip messages sent (Prune) | | gossip\_​message\_​tx\_​count
{gossip\_​message="ping"} | counter | Number of gossip messages sent (Ping) | | gossip\_​message\_​tx\_​count
{gossip\_​message="pong"} | counter | Number of gossip messages sent (Pong) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="pull\_​request"} | counter | Total wire bytes sent in gossip messages (Pull Request) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="pull\_​response"} | counter | Total wire bytes sent in gossip messages (Pull Response) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="push"} | counter | Total wire bytes sent in gossip messages (Push) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="prune"} | counter | Total wire bytes sent in gossip messages (Prune) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="ping"} | counter | Total wire bytes sent in gossip messages (Ping) | | gossip\_​message\_​tx\_​bytes
{gossip\_​message="pong"} | counter | Total wire bytes sent in gossip messages (Pong) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="contact\_​info\_​v1"} | counter | Number of CRDS values sent in push messages (Contact Info V1) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="vote"} | counter | Number of CRDS values sent in push messages (Vote) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="lowest\_​slot"} | counter | Number of CRDS values sent in push messages (Lowest Slot) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="snapshot\_​hashes"} | counter | Number of CRDS values sent in push messages (Snapshot Hashes) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="accounts\_​hashes"} | counter | Number of CRDS values sent in push messages (Accounts Hashes) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="epoch\_​slots"} | counter | Number of CRDS values sent in push messages (Epoch Slots) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="version\_​v1"} | counter | Number of CRDS values sent in push messages (Version V1) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="version\_​v2"} | counter | Number of CRDS values sent in push messages (Version V2) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="node\_​instance"} | counter | Number of CRDS values sent in push messages (Node Instance) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="duplicate\_​shred"} | counter | Number of CRDS values sent in push messages (Duplicate Shred) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="incremental\_​snapshot\_​hashes"} | counter | Number of CRDS values sent in push messages (Incremental Snapshot Hashes) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="contact\_​info\_​v2"} | counter | Number of CRDS values sent in push messages (Contact Info V2) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="restart\_​last\_​voted\_​fork\_​slots"} | counter | Number of CRDS values sent in push messages (Restart Last Voted Fork Slots) | | gossip\_​crds\_​tx\_​push\_​count
{crds\_​value="restart\_​heaviest\_​fork"} | counter | Number of CRDS values sent in push messages (Restart Heaviest Fork) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="contact\_​info\_​v1"} | counter | Total wire bytes of CRDS sent out in push messages (Contact Info V1) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="vote"} | counter | Total wire bytes of CRDS sent out in push messages (Vote) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="lowest\_​slot"} | counter | Total wire bytes of CRDS sent out in push messages (Lowest Slot) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="snapshot\_​hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Snapshot Hashes) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="accounts\_​hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Accounts Hashes) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="epoch\_​slots"} | counter | Total wire bytes of CRDS sent out in push messages (Epoch Slots) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="version\_​v1"} | counter | Total wire bytes of CRDS sent out in push messages (Version V1) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="version\_​v2"} | counter | Total wire bytes of CRDS sent out in push messages (Version V2) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="node\_​instance"} | counter | Total wire bytes of CRDS sent out in push messages (Node Instance) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="duplicate\_​shred"} | counter | Total wire bytes of CRDS sent out in push messages (Duplicate Shred) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="incremental\_​snapshot\_​hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Incremental Snapshot Hashes) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="contact\_​info\_​v2"} | counter | Total wire bytes of CRDS sent out in push messages (Contact Info V2) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="restart\_​last\_​voted\_​fork\_​slots"} | counter | Total wire bytes of CRDS sent out in push messages (Restart Last Voted Fork Slots) | | gossip\_​crds\_​tx\_​push\_​bytes
{crds\_​value="restart\_​heaviest\_​fork"} | counter | Total wire bytes of CRDS sent out in push messages (Restart Heaviest Fork) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="contact\_​info\_​v1"} | counter | Number of CRDS values sent in pull response messages (Contact Info V1) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="vote"} | counter | Number of CRDS values sent in pull response messages (Vote) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="lowest\_​slot"} | counter | Number of CRDS values sent in pull response messages (Lowest Slot) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="snapshot\_​hashes"} | counter | Number of CRDS values sent in pull response messages (Snapshot Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="accounts\_​hashes"} | counter | Number of CRDS values sent in pull response messages (Accounts Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="epoch\_​slots"} | counter | Number of CRDS values sent in pull response messages (Epoch Slots) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="version\_​v1"} | counter | Number of CRDS values sent in pull response messages (Version V1) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="version\_​v2"} | counter | Number of CRDS values sent in pull response messages (Version V2) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="node\_​instance"} | counter | Number of CRDS values sent in pull response messages (Node Instance) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="duplicate\_​shred"} | counter | Number of CRDS values sent in pull response messages (Duplicate Shred) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="incremental\_​snapshot\_​hashes"} | counter | Number of CRDS values sent in pull response messages (Incremental Snapshot Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="contact\_​info\_​v2"} | counter | Number of CRDS values sent in pull response messages (Contact Info V2) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="restart\_​last\_​voted\_​fork\_​slots"} | counter | Number of CRDS values sent in pull response messages (Restart Last Voted Fork Slots) | | gossip\_​crds\_​tx\_​pull\_​response\_​count
{crds\_​value="restart\_​heaviest\_​fork"} | counter | Number of CRDS values sent in pull response messages (Restart Heaviest Fork) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="contact\_​info\_​v1"} | counter | Total wire bytes of CRDS sent out in pull response messages (Contact Info V1) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="vote"} | counter | Total wire bytes of CRDS sent out in pull response messages (Vote) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="lowest\_​slot"} | counter | Total wire bytes of CRDS sent out in pull response messages (Lowest Slot) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="snapshot\_​hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Snapshot Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="accounts\_​hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Accounts Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="epoch\_​slots"} | counter | Total wire bytes of CRDS sent out in pull response messages (Epoch Slots) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="version\_​v1"} | counter | Total wire bytes of CRDS sent out in pull response messages (Version V1) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="version\_​v2"} | counter | Total wire bytes of CRDS sent out in pull response messages (Version V2) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="node\_​instance"} | counter | Total wire bytes of CRDS sent out in pull response messages (Node Instance) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="duplicate\_​shred"} | counter | Total wire bytes of CRDS sent out in pull response messages (Duplicate Shred) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="incremental\_​snapshot\_​hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Incremental Snapshot Hashes) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="contact\_​info\_​v2"} | counter | Total wire bytes of CRDS sent out in pull response messages (Contact Info V2) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="restart\_​last\_​voted\_​fork\_​slots"} | counter | Total wire bytes of CRDS sent out in pull response messages (Restart Last Voted Fork Slots) | | gossip\_​crds\_​tx\_​pull\_​response\_​bytes
{crds\_​value="restart\_​heaviest\_​fork"} | counter | Total wire bytes of CRDS sent out in pull response messages (Restart Heaviest Fork) | | gossip\_​wfs\_​staked\_​peers\_​online | gauge | Number of staked peers with recent gossip activity | | gossip\_​wfs\_​staked\_​peers\_​total | gauge | Total number of staked peers being tracked | | gossip\_​wfs\_​stake\_​online | gauge | Lamports of stake from peers with recent gossip activity | | gossip\_​wfs\_​stake\_​total | gauge | Total lamports of stake being tracked | | gossip\_​wfs\_​state | gauge | Wait for supermajority state. 1=INIT, 2=WAIT, 3=PUBLISH, 4=DONE | Repair Tile [​](https://docs.firedancer.io/api/metrics.html#repair-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | repair\_​total\_​pkt\_​count | counter | How many network packets we have sent, including reqs, pings, pongs, etc | | repair\_​sent\_​pkt\_​types
{repair\_​sent\_​request\_​types="needed\_​window"} | counter | What types of client messages are we sending (Need Window) | | repair\_​sent\_​pkt\_​types
{repair\_​sent\_​request\_​types="needed\_​highest\_​window"} | counter | What types of client messages are we sending (Need Highest Window) | | repair\_​sent\_​pkt\_​types
{repair\_​sent\_​request\_​types="needed\_​orphan"} | counter | What types of client messages are we sending (Need Orphans) | | repair\_​sent\_​pkt\_​types
{repair\_​sent\_​request\_​types="pong"} | counter | What types of client messages are we sending (Pong) | | repair\_​repaired\_​slots | counter | Until which slots have we fully repaired | | repair\_​current\_​slot | counter | Our view of the current cluster slot, max slot received | | repair\_​request\_​peers | counter | How many peers have we requested | | repair\_​sign\_​tile\_​unavail | counter | How many times no sign tiles were available to send request | | repair\_​eager\_​repair\_​aggresses | counter | How many times we pass eager repair threshold | | repair\_​rerequest\_​queue | counter | How many times we re-request a shred from the inflights queue | | repair\_​malformed\_​ping | counter | How many times we received a malformed ping | | repair\_​unknown\_​peer\_​ping | counter | How many times we received a ping from an unknown peer | | repair\_​failed\_​sigverify\_​ping | counter | How many times we failed to verify the signature of a ping | | repair\_​slot\_​complete\_​time | histogram | Time in seconds it took to complete a slot | | repair\_​response\_​latency | histogram | Time in nanoseconds it took to receive a repair request response | | repair\_​sign\_​duration\_​seconds | histogram | Duration of signing a message | | repair\_​blk\_​evicted | counter | How many times we evicted a block from the forest | | repair\_​blk\_​failed\_​insert | counter | How many times we failed to insert a block into the forest due to failed eviction | Replay Tile [​](https://docs.firedancer.io/api/metrics.html#replay-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | replay\_​store\_​query\_​acquire | counter | Count of store slock acquires for query | | replay\_​store\_​query\_​release | counter | Count of store slock releases for query | | replay\_​store\_​query\_​wait | histogram | Time in seconds spent waiting to acquire the slock for query | | replay\_​store\_​query\_​work | histogram | Time in seconds spent working before releasing the slock for query | | replay\_​store\_​query\_​cnt | counter | Count of queries | | replay\_​store\_​query\_​missing\_​cnt | counter | Count of queries where merkle root was missing | | replay\_​store\_​query\_​mr | gauge | First 8 bytes of most recently queried merkle root | | replay\_​store\_​query\_​missing\_​mr | gauge | First 8 bytes of most recently queried missing merkle root | | replay\_​root\_​slot | gauge | The slot at which our node has most recently rooted | | replay\_​root\_​distance | gauge | The distance in slots between our current root and the current reset slot | | replay\_​leader\_​slot | gauge | The slot at which we are currently leader, or 0 if none | | replay\_​next\_​leader\_​slot | gauge | The slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot | | replay\_​reset\_​slot | gauge | The slot at which we last reset the replay stage, or 0 if unknown | | replay\_​max\_​live\_​banks | gauge | The maximum number of banks we can have alive | | replay\_​live\_​banks | gauge | The number of banks we currently have alive | | replay\_​reasm\_​free | gauge | The number of free FEC sets in the reassembly queue | | replay\_​reasm\_​latest\_​slot | gauge | Slot of the latest FEC set in the reassembly queue that can be replayed | | replay\_​reasm\_​latest\_​fec\_​idx | gauge | FEC set index of the latest FEC set in the reassembly queue that can be replayed | | replay\_​sched\_​active\_​bank\_​idx | gauge | The bank that the replay scheduler is currently dispatching work for | | replay\_​sched\_​last\_​dispatch\_​bank\_​idx | gauge | The bank that the replay scheduler last dispatched work for | | replay\_​sched\_​last\_​dispatch\_​time\_​nanos | gauge | Timestamp when the replay scheduler last dispatched work | | replay\_​sched\_​staging\_​lane\_​popcnt | gauge | The total number of staging lanes that are currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once | | replay\_​sched\_​staging\_​lane\_​popcnt\_​wmk | gauge | The high watermark number of staging lanes that were at one point occupied in the replay scheduler | | replay\_​sched\_​staging\_​lane\_​head\_​bank\_​idx0 | gauge | The bank at the head of staging lane 0 of the replay dispatcher | | replay\_​sched\_​staging\_​lane\_​head\_​bank\_​idx1 | gauge | The bank at the head of staging lane 1 of the replay dispatcher | | replay\_​sched\_​staging\_​lane\_​head\_​bank\_​idx2 | gauge | The bank at the head of staging lane 2 of the replay dispatcher | | replay\_​sched\_​staging\_​lane\_​head\_​bank\_​idx3 | gauge | The bank at the head of staging lane 3 of the replay dispatcher | | replay\_​sched\_​txn\_​pool\_​popcnt | gauge | The total number of transactions that are available to be scheduled in the replay scheduler | | replay\_​sched\_​txn\_​pool\_​size | gauge | The maximum number of transactions that the replay scheduler can consider. This value is fixed at Firedancer startup but is a useful reference for TxnPoolPopcnt | | replay\_​sched\_​mblk\_​pool\_​popcnt | gauge | The total number of microblocks that are available to be scheduled in the replay scheduler | | replay\_​sched\_​mblk\_​pool\_​size | gauge | The maximum number of microblocks that the replay scheduler can consider. This value is fixed at Firedancer startup but is a useful reference for MblkPoolPopcnt | | replay\_​sched\_​block\_​pool\_​popcnt | gauge | The total number of blocks that are alive in the replay scheduler | | replay\_​sched\_​block\_​pool\_​size | gauge | The maximum number of blocks that the replay scheduler can keep track of. This value is fixed at Firedancer startup but is a useful reference for BlockPoolPopcnt | | replay\_​sched\_​block\_​added\_​staged | counter | Blocks added to the replay scheduler as staged | | replay\_​sched\_​block\_​added\_​unstaged | counter | Blocks added to the replay scheduler as unstaged | | replay\_​sched\_​block\_​replayed | counter | Blocks that the replay scheduler considers fully replayed | | replay\_​sched\_​block\_​abandoned | counter | Blocks that the replay scheduler abandoned for any reason | | replay\_​sched\_​block\_​bad | counter | Blocks that the replay scheduler considers bad | | replay\_​sched\_​block\_​promoted | counter | Blocks that the replay scheduler promoted from unstaged to a staging lane | | replay\_​sched\_​block\_​demoted | counter | Blocks that the replay scheduler demoted from a staging lane to unstaged | | replay\_​sched\_​deactivate\_​no\_​child | counter | Number of times that the replay scheduler stopped dispatching because a block fully replayed and there was no child block available yet | | replay\_​sched\_​deactivate\_​no\_​work | counter | Number of times that the replay scheduler stopped dispatching because a partially replayed block ran out of work to dispatch | | replay\_​sched\_​deactivate\_​abandoned | counter | Number of times that the replay scheduler stopped dispatching because a block got abandoned for any reason | | replay\_​sched\_​deactivate\_​minority | counter | Number of times that the replay scheduler stopped dispatching because a block got abandoned for being on a minority fork | | replay\_​sched\_​lane\_​switch | counter | Number of times that the replay scheduler switched the lane that it is dispatching from | | replay\_​sched\_​lane\_​promote | counter | Number of times that the replay scheduler promoted one or more blocks onto a staging lane | | replay\_​sched\_​lane\_​demote | counter | Number of times that the replay scheduler demoted one or more blocks from a staging lane | | replay\_​sched\_​fork\_​observed | counter | Number of forks that the replay scheduler has observed | | replay\_​sched\_​alut\_​success | counter | Number of transactions with address lookup tables that the replay scheduler successfully expanded | | replay\_​sched\_​alut\_​failure | counter | Number of transactions with address lookup tables that the replay scheduler failed to expand | | replay\_​sched\_​txn\_​abandoned\_​parsed | counter | Number of transactions that were parsed but ended up being in abandoned blocks | | replay\_​sched\_​txn\_​abandoned\_​exec | counter | Number of transactions that were executed but ended up being in abandoned blocks | | replay\_​sched\_​txn\_​abandoned\_​done | counter | Number of transactions that were fully replayed but ended up being in abandoned blocks | | replay\_​sched\_​weighted\_​in\_​flight | counter | Cumulative number of transactions that have been in-flight from the replay scheduler's point of view, weighted by the duration of each transaction. This value is useful as the dividend for TxnWeightedInFlightDuration | | replay\_​sched\_​weighted\_​in\_​flight\_​duration | counter | Duration over which TxnWeightedInFlight was accumulated. Use this as divisor to get the average number of in-flight transactions | | replay\_​sched\_​none\_​in\_​flight\_​duration | counter | Duration over which the replay scheduler had no transaction in-flight. This value is useful in a ratio against WeightedInFlightDuration to see how often the replay transaction execution pipeline was idle | | replay\_​sched\_​txn\_​parsed | counter | Number of transactions that the replay pipeline has parsed | | replay\_​sched\_​txn\_​exec | counter | Number of transactions that the replay pipeline has executed | | replay\_​sched\_​txn\_​sigverify | counter | Number of transactions that the replay pipeline has sigverified | | replay\_​sched\_​txn\_​mixin | counter | Number of transactions that the replay pipeline has performed PoH mixin | | replay\_​sched\_​txn\_​done | counter | Number of transactions that were fully replayed | | replay\_​sched\_​mblk\_​parsed | counter | Number of microblocks that the replay pipeline has parsed | | replay\_​sched\_​mblk\_​hashed | counter | Number of microblocks that the replay pipeline has fully hashed | | replay\_​sched\_​mblk\_​done | counter | Number of microblocks that were fully replayed | | replay\_​sched\_​bytes\_​ingested | counter | Number of bytes that the replay scheduler has ingested | | replay\_​sched\_​bytes\_​ingested\_​padding | counter | Number of bytes that the replay scheduler ingested but did not parse for being padding | | replay\_​sched\_​bytes\_​dropped | counter | Number of bytes that the replay scheduler refused to ingest because the block is considered abandoned | | replay\_​sched\_​fec | counter | Number of FEC sets that the replay scheduler has been given | | replay\_​slots\_​total | counter | Count of slots replayed successfully or leader slots packed and shredded successfully | | replay\_​transactions\_​total | counter | Count of transactions processed overall on the current fork | | replay\_​sched\_​full | counter | Times where sched is full and a FEC set can't be processed | | replay\_​reasm\_​empty | counter | Times where reasm is empty and a FEC set can't be processed | | replay\_​leader\_​bid\_​wait | counter | Times where replay is blocked by the PoH tile not sending an end of leader message | | replay\_​banks\_​full | counter | Times where banks are full and a FEC set can't be processed | | replay\_​storage\_​root\_​behind | counter | Times where the storage root is behind the consensus root and can't be advanced | | replay\_​accdb\_​created | counter | Number of account database records created | | replay\_​accdb\_​reverted | counter | Number of account database records reverted | | replay\_​accdb\_​rooted | counter | Number of account database entries rooted | | replay\_​accdb\_​rooted\_​bytes | counter | Number of bytes in account database entries rooted (including overhead) | | replay\_​accdb\_​gc\_​root | counter | Number of account database entries garbage collected | | replay\_​accdb\_​reclaimed | counter | Number of account database entries reclaimed (deletion rooted) | | replay\_​root\_​slot\_​duration\_​seconds | histogram | Time in seconds spent updating the rooted account store (one sample per block) | | replay\_​root\_​account\_​duration\_​seconds | histogram | Time in seconds spent updating the rooted account store (one sample per block, normalized by account count) | | replay\_​root\_​elapsed\_​seconds
{root\_​phase="db"} | counter | Total time in seconds spent rooting accounts (Waiting on database server) | | replay\_​root\_​elapsed\_​seconds
{root\_​phase="copy"} | counter | Total time in seconds spent rooting accounts (Copying account data) | | replay\_​root\_​elapsed\_​seconds
{root\_​phase="gc"} | counter | Total time in seconds spent rooting accounts (Garbage collecting old account data) | | replay\_​progcache\_​rooted | counter | Number of program cache entries rooted | | replay\_​progcache\_​gc\_​root | counter | Number of program cache entries garbage collected while rooting | | replay\_​progcache\_​free\_​parts | gauge | Number of program cache heap partitions free (indicates fragmentation) | | replay\_​progcache\_​free\_​bytes | gauge | Program cache heap utilization (free bytes) | | replay\_​progcache\_​size\_​bytes | gauge | Program cache heap utilization (total size) | | replay\_​progcache\_​free\_​part\_​max\_​bytes | gauge | Largest free heap partition in program cache | | replay\_​progcache\_​used\_​part\_​median\_​bytes | gauge | Median used heap partition size in program cache | | replay\_​progcache\_​used\_​part\_​mean\_​bytes | gauge | Mean used heap partition size in program cache | | replay\_​progcache\_​time\_​seconds | counter | Total time in seconds spent doing program cache tasks | | replay\_​accdb\_​cache\_​free\_​parts | gauge | Number of account database cache heap partitions free (indicates fragmentation) | | replay\_​accdb\_​cache\_​free\_​bytes | gauge | Account database cache heap utilization (free bytes) | | replay\_​accdb\_​cache\_​size\_​bytes | gauge | Account database cache heap utilization (total size) | | replay\_​accdb\_​cache\_​free\_​part\_​max\_​bytes | gauge | Largest free heap partition in account database cache | | replay\_​accdb\_​cache\_​used\_​part\_​median\_​bytes | gauge | Median used heap partition size in account database cache | | replay\_​accdb\_​cache\_​used\_​part\_​mean\_​bytes | gauge | Mean used heap partition size in account database cache | Execrp Tile [​](https://docs.firedancer.io/api/metrics.html#execrp-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | execrp\_​accdb\_​created | counter | Number of account database records created | | execrp\_​txn\_​regime
{txn\_​regime="setup"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction setup) | | execrp\_​txn\_​regime
{txn\_​regime="exec"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction execution (includes VM setup/execution)) | | execrp\_​txn\_​regime
{txn\_​regime="commit"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction result commit) | | execrp\_​vm\_​regime
{vm\_​regime="setup"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM setup) | | execrp\_​vm\_​regime
{vm\_​regime="commit"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM commit) | | execrp\_​vm\_​regime
{vm\_​regime="setup\_​cpi"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM setup (CPI)) | | execrp\_​vm\_​regime
{vm\_​regime="commit\_​cpi"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM commit (CPI)) | | execrp\_​vm\_​regime
{vm\_​regime="interpreter"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM interpreter execution) | | execrp\_​txn\_​account\_​changes
{account\_​change="unchanged\_​nonexist"} | counter | Transaction account change event counters (Account did not exist before and still does not) | | execrp\_​txn\_​account\_​changes
{account\_​change="created"} | counter | Transaction account change event counters (Account created) | | execrp\_​txn\_​account\_​changes
{account\_​change="delete"} | counter | Transaction account change event counters (Account deleted) | | execrp\_​txn\_​account\_​changes
{account\_​change="modify"} | counter | Transaction account change event counters (Account modified) | | execrp\_​txn\_​account\_​changes
{account\_​change="unchanged"} | counter | Transaction account change event counters (Account unchanged) | | execrp\_​compute\_​units\_​total | counter | Estimated number of compute units executed since tile start | | execrp\_​progcache\_​lookups | counter | Program cache lookup counter | | execrp\_​progcache\_​hits | counter | Program cache hit counter | | execrp\_​progcache\_​misses | counter | Program cache miss counter | | execrp\_​progcache\_​oom\_​heap | counter | Program cache out-of-memory event counter (heap) | | execrp\_​progcache\_​oom\_​desc | counter | Program cache out-of-memory event counter (descriptor table) | | execrp\_​progcache\_​fills | counter | Number of program cache insertions | | execrp\_​progcache\_​fill\_​bytes | counter | Number of bytes inserted into program cache | | execrp\_​progcache\_​spills | counter | Program cache spill counter (OOM fallback mechanism) | | execrp\_​progcache\_​spill\_​bytes | counter | Number of bytes spilled from program cache (OOM fallback mechanism) | | execrp\_​progcache\_​evictions | counter | Program cache eviction counter | | execrp\_​progcache\_​eviction\_​bytes | counter | Number of bytes evicted from program cache | | execrp\_​progcache\_​duration\_​total\_​seconds | counter | Total time in seconds spent on program cache operations | | execrp\_​progcache\_​duration\_​load\_​seconds | counter | Total time in seconds spent loading programs | Accdb Tile [​](https://docs.firedancer.io/api/metrics.html#accdb-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | accdb\_​accounts | gauge | Current number of accounts | | accdb\_​read\_​ops
{storage\_​type="shared\_​cache"} | counter | Total number of read operations (Record cache) | | accdb\_​read\_​ops
{storage\_​type="io\_​cache"} | counter | Total number of read operations (I/O layer cache) | | accdb\_​read\_​ops
{storage\_​type="file"} | counter | Total number of read operations (File access) | | accdb\_​read\_​bytes
{storage\_​type="shared\_​cache"} | counter | Total number of bytes read (Record cache) | | accdb\_​read\_​bytes
{storage\_​type="io\_​cache"} | counter | Total number of bytes read (I/O layer cache) | | accdb\_​read\_​bytes
{storage\_​type="file"} | counter | Total number of bytes read (File access) | | accdb\_​write\_​ops
{storage\_​type="shared\_​cache"} | counter | Total number of write operations (Record cache) | | accdb\_​write\_​ops
{storage\_​type="io\_​cache"} | counter | Total number of write operations (I/O layer cache) | | accdb\_​write\_​ops
{storage\_​type="file"} | counter | Total number of write operations (File access) | | accdb\_​write\_​bytes
{storage\_​type="shared\_​cache"} | counter | Total number of bytes written (Record cache) | | accdb\_​write\_​bytes
{storage\_​type="io\_​cache"} | counter | Total number of bytes written (I/O layer cache) | | accdb\_​write\_​bytes
{storage\_​type="file"} | counter | Total number of bytes written (File access) | | accdb\_​file\_​capacity\_​bytes | gauge | Account database file capacity in bytes | | accdb\_​file\_​used\_​bytes | gauge | Current number of bytes used in account database file | | accdb\_​bstream\_​seq
{bstream\_​seq="ancient"} | gauge | Current bstream sequence number (Blocks between ancient and past have been written and forgotten (no read, no write)) | | accdb\_​bstream\_​seq
{bstream\_​seq="past"} | gauge | Current bstream sequence number (Blocks between past and present have been written (read only)) | | accdb\_​bstream\_​seq
{bstream\_​seq="present"} | gauge | Current bstream sequence number (Blocks between present and future are being written (write only)) | | accdb\_​bstream\_​seq
{bstream\_​seq="future"} | gauge | Current bstream sequence number (Blocks between future and ancient have not been written (no read, no write)) | | accdb\_​request\_​batches | counter | Number of request batches processed | | accdb\_​requests
{vinyl\_​request="acquire"} | counter | Number of requests processed (Acquire record) | | accdb\_​requests
{vinyl\_​request="release"} | counter | Number of requests processed (Release record) | | accdb\_​requests
{vinyl\_​request="erase"} | counter | Number of requests processed (Erase record) | | accdb\_​blocks
{vinyl\_​blocks="pair"} | counter | Number of blocks written to bstream (Record) | | accdb\_​blocks
{vinyl\_​blocks="dead"} | counter | Number of blocks written to bstream (Record deletion) | | accdb\_​blocks
{vinyl\_​blocks="part"} | counter | Number of blocks written to bstream (Partition/divider) | | accdb\_​garbage\_​bytes | gauge | | | accdb\_​cum\_​gc\_​bytes | counter | Total number of record bytes that were garbage collected | | accdb\_​account\_​index\_​remaining\_​free | gauge | Remaining free slots in the account database index (validator crashes when this number reaches zero) | Tower Tile [​](https://docs.firedancer.io/api/metrics.html#tower-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | tower\_​ignored\_​cnt | counter | Number of replay\_slot\_completed frags we ignored | | tower\_​ignored\_​slot | gauge | Most recent ignored replay\_slot\_completed frag | | tower\_​eqvoc\_​cnt | counter | Number of replay\_slot\_completed frags we detect as equivocations | | tower\_​eqvoc\_​slot | gauge | Most recent equivocating replay\_slot\_completed frag | | tower\_​replay\_​slot | gauge | Replay slot | | tower\_​vote\_​slot | gauge | Most recent vote slot, ULONG\_MAX if no vote cast | | tower\_​reset\_​slot | gauge | Reset slot | | tower\_​root\_​slot | gauge | Root slot | | tower\_​init\_​slot | gauge | Init slot | | tower\_​not\_​ready | counter | Frag arrived before tower was initialized | | tower\_​txn\_​bad\_​deser | counter | Vote txn failed to deserialize | | tower\_​txn\_​bad\_​tower | counter | Vote txn deserialized but tower was invalid | | tower\_​txn\_​not\_​tower\_​sync | counter | Vote txn was not a TowerSync instruction | | tower\_​txn\_​empty\_​tower | counter | Vote txn had an empty tower (validator hasn't voted) | | tower\_​eqvoc\_​success\_​merkle | counter | Merkle root conflict | | tower\_​eqvoc\_​success\_​meta | counter | Coding metadata conflict | | tower\_​eqvoc\_​success\_​last | counter | Last shred index conflict | | tower\_​eqvoc\_​success\_​overlap | counter | Overlapping FEC set conflict | | tower\_​eqvoc\_​success\_​chained | counter | Chained merkle root conflict | | tower\_​eqvoc\_​err\_​serde | counter | Invalid serialization | | tower\_​eqvoc\_​err\_​slot | counter | Shreds were for different slots | | tower\_​eqvoc\_​err\_​version | counter | Either shred had wrong shred version | | tower\_​eqvoc\_​err\_​type | counter | Wrong shred type (must be chained merkle) | | tower\_​eqvoc\_​err\_​merkle | counter | Failed to derive merkle root | | tower\_​eqvoc\_​err\_​signature | counter | Failed to sig verify | | tower\_​eqvoc\_​err\_​chunk\_​cnt | counter | num\_chunks != FD\_EQVOC\_CHUNK\_CNT | | tower\_​eqvoc\_​err\_​chunk\_​idx | counter | chunk\_index >= FD\_EQVOC\_CHUNK\_CNT | | tower\_​eqvoc\_​err\_​chunk\_​len | counter | chunk\_len does not match expected length for chunk\_index | | tower\_​eqvoc\_​err\_​ignored\_​from | counter | Unrecognized from address | | tower\_​eqvoc\_​err\_​ignored\_​slot | counter | Slot older than root or unable to derive leader schedule | | tower\_​eqvoc\_​proof\_​constructed | counter | Number of duplicate proofs we constructed from shreds | | tower\_​eqvoc\_​proof\_​verified | counter | Number of duplicate proofs we verified from gossip | | tower\_​ghost\_​not\_​voted | counter | Ghost vote skipped because voter hasn't voted | | tower\_​ghost\_​too\_​old | counter | Ghost vote slot was behind the root | | tower\_​ghost\_​already\_​voted | counter | Ghost vote slot was not newer than previous vote | | tower\_​hfork\_​unknown\_​vtr | counter | Hfork voter not in voter set | | tower\_​hfork\_​already\_​voted | counter | Hfork voter already voted for this block\_id | | tower\_​hfork\_​too\_​old | counter | Hfork vote slot not newer than previous | | tower\_​hfork\_​matched\_​slot | gauge | Highest slot where 52%+ of stake agreed on our bank hash | | tower\_​hfork\_​mismatched\_​slot | gauge | Highest slot where 52%+ of stake agreed on a different bank hash than ours (we hard forked) | | tower\_​ancestor\_​rollback | counter | Rollback to an ancestor of our prev vote (can't vote) | | tower\_​sibling\_​confirmed | counter | Duplicate sibling got confirmed (can't vote) | | tower\_​same\_​fork | counter | Same fork as prev vote (can vote) | | tower\_​switch\_​pass | counter | Prev vote was on a different fork, but we are allowed to switch (can vote) | | tower\_​switch\_​fail | counter | Prev vote was on a different fork, and we are not allowed to switch (can't vote) | | tower\_​lockout\_​fail | counter | Locked out (can't vote) | | tower\_​threshold\_​fail | counter | Did not pass threshold check (can't vote) | | tower\_​propagated\_​fail | counter | Prev leader block did not propagate (can't vote) | | tower\_​votes\_​too\_​old | counter | Vote slot was behind the votes root | | tower\_​votes\_​too\_​new | counter | Vote slot was too far ahead of the votes root | | tower\_​votes\_​unknown\_​vtr | counter | Vote account was not in the voter set | | tower\_​votes\_​already\_​voted | counter | Voter already voted for this slot | | tower\_​votes\_​unknown\_​slot | counter | Vote txn was for a slot we haven't replayed | | tower\_​votes\_​unknown\_​block\_​id | counter | Vote txn was for a block id we don't recognize | Txsend Tile [​](https://docs.firedancer.io/api/metrics.html#txsend-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | txsend\_​received\_​packets | counter | Total count of QUIC packets received | | txsend\_​received\_​bytes | counter | Total bytes received via QUIC | | txsend\_​sent\_​packets | counter | Total count of QUIC packets sent | | txsend\_​sent\_​bytes | counter | Total bytes sent via QUIC | | txsend\_​retry\_​sent | counter | Total count of QUIC retry packets sent | | txsend\_​connections\_​alloc | gauge | Number of currently allocated QUIC connections | | txsend\_​connections\_​state
{quic\_​conn\_​state="invalid"} | gauge | Number of QUIC connections in each state (Freed) | | txsend\_​connections\_​state
{quic\_​conn\_​state="handshake"} | gauge | Number of QUIC connections in each state (Handshaking peer) | | txsend\_​connections\_​state
{quic\_​conn\_​state="handshake\_​complete"} | gauge | Number of QUIC connections in each state (Handshake complete, confirming with peer) | | txsend\_​connections\_​state
{quic\_​conn\_​state="active"} | gauge | Number of QUIC connections in each state (Active connection) | | txsend\_​connections\_​state
{quic\_​conn\_​state="peer\_​close"} | gauge | Number of QUIC connections in each state (Peer requested close) | | txsend\_​connections\_​state
{quic\_​conn\_​state="abort"} | gauge | Number of QUIC connections in each state (Connection terminating due to error) | | txsend\_​connections\_​state
{quic\_​conn\_​state="close\_​pending"} | gauge | Number of QUIC connections in each state (Connection is closing) | | txsend\_​connections\_​state
{quic\_​conn\_​state="dead"} | gauge | Number of QUIC connections in each state (Connection about to be freed) | | txsend\_​connections\_​created | counter | Total count of QUIC connections created | | txsend\_​connections\_​closed | counter | Total count of QUIC connections closed | | txsend\_​connections\_​aborted | counter | Total count of QUIC connections aborted | | txsend\_​connections\_​timed\_​out | counter | Total count of QUIC connections timed out | | txsend\_​connections\_​retried | counter | Total count of QUIC connections retried | | txsend\_​connection\_​error\_​no\_​slots | counter | Total count of connection errors due to no slots | | txsend\_​connection\_​error\_​retry\_​fail | counter | Total count of connection retry failures | | txsend\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="initial"} | counter | Total count of packets with crypto failures (Initial) | | txsend\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="early"} | counter | Total count of packets with crypto failures (Early data) | | txsend\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="handshake"} | counter | Total count of packets with crypto failures (Handshake) | | txsend\_​pkt\_​crypto\_​failed
{quic\_​enc\_​level="app"} | counter | Total count of packets with crypto failures (App data) | | txsend\_​pkt\_​no\_​key
{quic\_​enc\_​level="initial"} | counter | Total count of packets with no key (Initial) | | txsend\_​pkt\_​no\_​key
{quic\_​enc\_​level="early"} | counter | Total count of packets with no key (Early data) | | txsend\_​pkt\_​no\_​key
{quic\_​enc\_​level="handshake"} | counter | Total count of packets with no key (Handshake) | | txsend\_​pkt\_​no\_​key
{quic\_​enc\_​level="app"} | counter | Total count of packets with no key (App data) | | txsend\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="initial"} | counter | Total count of packets with no connection (Initial) | | txsend\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="retry"} | counter | Total count of packets with no connection (Retry) | | txsend\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="handshake"} | counter | Total count of packets with no connection (Handshake) | | txsend\_​pkt\_​no\_​conn
{quic\_​pkt\_​handle="one\_​rtt"} | counter | Total count of packets with no connection (1-RTT) | | txsend\_​pkt\_​tx\_​alloc\_​fail | counter | Total count of packet TX allocation failures | | txsend\_​pkt\_​net\_​header\_​invalid | counter | Total count of packets with invalid network headers | | txsend\_​pkt\_​quic\_​header\_​invalid | counter | Total count of packets with invalid QUIC headers | | txsend\_​pkt\_​undersz | counter | Total count of undersized packets | | txsend\_​pkt\_​oversz | counter | Total count of oversized packets | | txsend\_​pkt\_​verneg | counter | Total count of version negotiation packets | | txsend\_​pkt\_​retransmissions
{quic\_​enc\_​level="initial"} | counter | Total count of QUIC packet retransmissions (Initial) | | txsend\_​pkt\_​retransmissions
{quic\_​enc\_​level="early"} | counter | Total count of QUIC packet retransmissions (Early data) | | txsend\_​pkt\_​retransmissions
{quic\_​enc\_​level="handshake"} | counter | Total count of QUIC packet retransmissions (Handshake) | | txsend\_​pkt\_​retransmissions
{quic\_​enc\_​level="app"} | counter | Total count of QUIC packet retransmissions (App data) | | txsend\_​handshakes\_​created | counter | Total count of QUIC handshakes created | | txsend\_​handshake\_​error\_​alloc\_​fail | counter | Total count of handshake allocation failures | | txsend\_​handshake\_​evicted | counter | Total count of handshakes evicted | | txsend\_​stream\_​received\_​events | counter | Total count of stream events received | | txsend\_​stream\_​received\_​bytes | counter | Total bytes received via streams | | txsend\_​received\_​frames
{quic\_​frame\_​type="unknown"} | counter | Total count of QUIC frames received (Unknown frame type) | | txsend\_​received\_​frames
{quic\_​frame\_​type="ack"} | counter | Total count of QUIC frames received (ACK frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="reset\_​stream"} | counter | Total count of QUIC frames received (RESET\_STREAM frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="stop\_​sending"} | counter | Total count of QUIC frames received (STOP\_SENDING frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="crypto"} | counter | Total count of QUIC frames received (CRYPTO frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="new\_​token"} | counter | Total count of QUIC frames received (NEW\_TOKEN frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="stream"} | counter | Total count of QUIC frames received (STREAM frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="max\_​data"} | counter | Total count of QUIC frames received (MAX\_DATA frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="max\_​stream\_​data"} | counter | Total count of QUIC frames received (MAX\_STREAM\_DATA frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="max\_​streams"} | counter | Total count of QUIC frames received (MAX\_STREAMS frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="data\_​blocked"} | counter | Total count of QUIC frames received (DATA\_BLOCKED frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="stream\_​data\_​blocked"} | counter | Total count of QUIC frames received (STREAM\_DATA\_BLOCKED frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="streams\_​blocked"} | counter | Total count of QUIC frames received (STREAMS\_BLOCKED(bidi) frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="new\_​conn\_​id"} | counter | Total count of QUIC frames received (NEW\_CONN\_ID frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="retire\_​conn\_​id"} | counter | Total count of QUIC frames received (RETIRE\_CONN\_ID frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="path\_​challenge"} | counter | Total count of QUIC frames received (PATH\_CHALLENGE frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="path\_​response"} | counter | Total count of QUIC frames received (PATH\_RESPONSE frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="conn\_​close\_​quic"} | counter | Total count of QUIC frames received (CONN\_CLOSE(transport) frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="conn\_​close\_​app"} | counter | Total count of QUIC frames received (CONN\_CLOSE(app) frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="handshake\_​done"} | counter | Total count of QUIC frames received (HANDSHAKE\_DONE frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="ping"} | counter | Total count of QUIC frames received (PING frame) | | txsend\_​received\_​frames
{quic\_​frame\_​type="padding"} | counter | Total count of QUIC frames received (PADDING frame) | | txsend\_​frame\_​fail\_​parse | counter | Total count of frame parse failures | | txsend\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="success"} | counter | Results of attempts to acquire QUIC frame metadata (Success) | | txsend\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="fail\_​empty\_​pool"} | counter | Results of attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) | | txsend\_​frame\_​tx\_​alloc
{frame\_​tx\_​alloc\_​result="fail\_​conn\_​max"} | counter | Results of attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) | | txsend\_​ack\_​tx
{quic\_​ack\_​tx="noop"} | counter | Total count of ACK frames transmitted (Non-ACK-eliciting packet) | | txsend\_​ack\_​tx
{quic\_​ack\_​tx="new"} | counter | Total count of ACK frames transmitted (New ACK range) | | txsend\_​ack\_​tx
{quic\_​ack\_​tx="merged"} | counter | Total count of ACK frames transmitted (Merged into existing ACK range) | | txsend\_​ack\_​tx
{quic\_​ack\_​tx="drop"} | counter | Total count of ACK frames transmitted (Out of buffers) | | txsend\_​ack\_​tx
{quic\_​ack\_​tx="cancel"} | counter | Total count of ACK frames transmitted (ACK suppressed by handler) | | txsend\_​service\_​duration\_​seconds | histogram | Duration spent in service | | txsend\_​receive\_​duration\_​seconds | histogram | Duration spent processing packets | Diag Tile [​](https://docs.firedancer.io/api/metrics.html#diag-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | diag\_​bundle\_​health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy bundle subsystem means at least one bundle tile currently zhas an active connection to the block engine server | | diag\_​vote\_​health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy vote subsystem means the client has cast at least one vote in both the last 60 seconds and last 150 slots (before the currently replay slot) | | diag\_​replay\_​health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy replay subsystem means that the largest fully-processed replay slot on the chosen consensus fork is within 12 slots of the current turbine slot | | diag\_​turbine\_​health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy turbine subsystem means that the largest slot associated with received turbine shreds has not stalled for 12 seconds, and also that the average replay ingress traffic exceeds the average ingress repair traffic over the past 12 seconds | Event Tile [​](https://docs.firedancer.io/api/metrics.html#event-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | event\_​connection\_​state | gauge | 0 = disconnected, 1 = connecting, 2 = connected | | event\_​event\_​queue\_​count | gauge | The current number of events in the event queue waiting to be sent to the event service | | event\_​event\_​queue\_​drops | counter | The total number of events dropped because the event queue was full | | event\_​event\_​queue\_​bytes\_​used | gauge | The current number of bytes used in the event queue | | event\_​event\_​queue\_​bytes\_​capacity | gauge | The total capacity in bytes of the event queue | | event\_​events\_​sent | counter | The total number of events sent to the event service | | event\_​events\_​acked | counter | The total number of events acknowledged by the event service | | event\_​bytes\_​written | counter | The total number of bytes written to the event service | | event\_​bytes\_​read | counter | The total number of bytes read from the event service | Gui Tile [​](https://docs.firedancer.io/api/metrics.html#gui-tile) ------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | gui\_​connection\_​count | gauge | The number of active http connections to the GUI service, excluding connections that have been upgraded to a WebSocket connection | | gui\_​websocket\_​connection\_​count | gauge | The number of active websocket connections to the GUI service | | gui\_​websocket\_​frames\_​sent | counter | The total number of websocket frames sent to all connections to the GUI service | | gui\_​websocket\_​frames\_​received | counter | The total number of websocket frames received from all connections to the GUI service | | gui\_​bytes\_​written | counter | The total number of bytes written to all connections to the GUI service | | gui\_​bytes\_​read | counter | The total number of bytes read from all connections to the GUI service | Metric Tile [​](https://docs.firedancer.io/api/metrics.html#metric-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | metric\_​boot\_​timestamp\_​nanos | gauge | Timestamp when validator was started (nanoseconds since epoch) | | metric\_​connection\_​count | gauge | The number of active http connections to the Prometheus endpoint | | metric\_​bytes\_​written | counter | The total number of bytes written to all responses on the Prometheus endpoint | | metric\_​bytes\_​read | counter | The total number of bytes read from all requests to the Prometheus endpoint | Rpc Tile [​](https://docs.firedancer.io/api/metrics.html#rpc-tile) ------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | Resolh Tile [​](https://docs.firedancer.io/api/metrics.html#resolh-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | resolh\_​no\_​bank\_​drop | counter | Count of transactions dropped because the bank was not available | | resolh\_​stash\_​operation
{resolve\_​stash\_​operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) | | resolh\_​stash\_​operation
{resolve\_​stash\_​operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) | | resolh\_​stash\_​operation
{resolve\_​stash\_​operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) | | resolh\_​stash\_​operation
{resolve\_​stash\_​operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​lookup\_​index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="account\_​uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​account\_​data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="invalid\_​account\_​owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="account\_​not\_​found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) | | resolh\_​lut\_​resolved
{lut\_​resolve\_​result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) | | resolh\_​blockhash\_​expired | counter | Count of transactions that failed to resolve because the blockhash was expired | | resolh\_​transaction\_​bundle\_​peer\_​failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed | Bank Tile [​](https://docs.firedancer.io/api/metrics.html#bank-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | bank\_​transaction\_​sanitize\_​failure | counter | Number of transactions that failed to sanitize | | bank\_​transaction\_​not\_​executed\_​failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="invalid\_​lookup\_​index"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The transaction referenced an index in a LUT that didn't exist) | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="account\_​uninitialized"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT hasn't been initialized) | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="invalid\_​account\_​data"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be parsed) | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="invalid\_​account\_​owner"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT wasn't owned by the ALUT program ID) | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="account\_​not\_​found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be found) | | bank\_​transaction\_​load\_​address\_​tables
{lut\_​resolve\_​result="success"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (Resolved successfully) | | bank\_​transaction\_​result
{transaction\_​error="success"} | counter | Result of loading and executing a transaction (Success) | | bank\_​transaction\_​result
{transaction\_​error="account\_​in\_​use"} | counter | Result of loading and executing a transaction (An account is already being processed in another transaction in a way that does not support parallelism) | | bank\_​transaction\_​result
{transaction\_​error="account\_​loaded\_​twice"} | counter | Result of loading and executing a transaction (A `Pubkey` appears twice in the transaction's `account_keys`. Instructions can reference `Pubkey`s more than once but the message must contain a list with no duplicate keys) | | bank\_​transaction\_​result
{transaction\_​error="account\_​not\_​found"} | counter | Result of loading and executing a transaction (Attempt to debit an account but found no record of a prior credit) | | bank\_​transaction\_​result
{transaction\_​error="program\_​account\_​not\_​found"} | counter | Result of loading and executing a transaction (Attempt to load a program that does not exist) | | bank\_​transaction\_​result
{transaction\_​error="insufficient\_​funds\_​for\_​fee"} | counter | Result of loading and executing a transaction (The fee payer `Pubkey` does not have sufficient balance to pay the fee to schedule the transaction) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​account\_​for\_​fee"} | counter | Result of loading and executing a transaction (This account may not be used to pay transaction fees) | | bank\_​transaction\_​result
{transaction\_​error="already\_​processed"} | counter | Result of loading and executing a transaction (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its `recent_blockhash`, or as a double-spend attack) | | bank\_​transaction\_​result
{transaction\_​error="blockhash\_​not\_​found"} | counter | Result of loading and executing a transaction (The bank has not seen the given `recent_blockhash` or the transaction is too old and the `recent_blockhash` has been discarded) | | bank\_​transaction\_​result
{transaction\_​error="instruction\_​error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) | | bank\_​transaction\_​result
{transaction\_​error="call\_​chain\_​too\_​deep"} | counter | Result of loading and executing a transaction (Loader call chain is too deep) | | bank\_​transaction\_​result
{transaction\_​error="missing\_​signature\_​for\_​fee"} | counter | Result of loading and executing a transaction (Transaction requires a fee but has no signature present) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​account\_​index"} | counter | Result of loading and executing a transaction (Transaction contains an invalid account reference) | | bank\_​transaction\_​result
{transaction\_​error="signature\_​failure"} | counter | Result of loading and executing a transaction (Transaction did not pass signature verification) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​program\_​for\_​execution"} | counter | Result of loading and executing a transaction (This program may not be used for executing instructions) | | bank\_​transaction\_​result
{transaction\_​error="sanitize\_​failure"} | counter | Result of loading and executing a transaction (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked) | | bank\_​transaction\_​result
{transaction\_​error="cluster\_​maintenance"} | counter | Result of loading and executing a transaction (Transactions are currently disabled due to cluster maintenance) | | bank\_​transaction\_​result
{transaction\_​error="account\_​borrow\_​outstanding"} | counter | Result of loading and executing a transaction (Transaction processing left an account with an outstanding borrowed reference) | | bank\_​transaction\_​result
{transaction\_​error="would\_​exceed\_​max\_​block\_​cost\_​limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Block Cost Limit) | | bank\_​transaction\_​result
{transaction\_​error="unsupported\_​version"} | counter | Result of loading and executing a transaction (Transaction version is unsupported) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​writable\_​account"} | counter | Result of loading and executing a transaction (Transaction loads a writable account that cannot be written) | | bank\_​transaction\_​result
{transaction\_​error="would\_​exceed\_​max\_​account\_​cost\_​limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max account limit within the block) | | bank\_​transaction\_​result
{transaction\_​error="would\_​exceed\_​account\_​data\_​block\_​limit"} | counter | Result of loading and executing a transaction (Transaction would exceed account data limit within the block) | | bank\_​transaction\_​result
{transaction\_​error="too\_​many\_​account\_​locks"} | counter | Result of loading and executing a transaction (Transaction locked too many accounts) | | bank\_​transaction\_​result
{transaction\_​error="address\_​lookup\_​table\_​not\_​found"} | counter | Result of loading and executing a transaction (Address lookup table not found) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​address\_​lookup\_​table\_​owner"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an account owned by the wrong program) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​address\_​lookup\_​table\_​data"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an invalid account) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​address\_​lookup\_​table\_​index"} | counter | Result of loading and executing a transaction (Address table lookup uses an invalid index) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​rent\_​paying\_​account"} | counter | Result of loading and executing a transaction (Transaction leaves an account with a lower balance than rent-exempt minimum) | | bank\_​transaction\_​result
{transaction\_​error="would\_​exceed\_​max\_​vote\_​cost\_​limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Vote Cost Limit) | | bank\_​transaction\_​result
{transaction\_​error="would\_​exceed\_​account\_​data\_​total\_​limit"} | counter | Result of loading and executing a transaction (Transaction would exceed total account data limit) | | bank\_​transaction\_​result
{transaction\_​error="duplicate\_​instruction"} | counter | Result of loading and executing a transaction (Transaction contains a duplicate instruction that is not allowed) | | bank\_​transaction\_​result
{transaction\_​error="insufficient\_​funds\_​for\_​rent"} | counter | Result of loading and executing a transaction (Transaction results in an account with insufficient funds for rent) | | bank\_​transaction\_​result
{transaction\_​error="max\_​loaded\_​accounts\_​data\_​size\_​exceeded"} | counter | Result of loading and executing a transaction (Transaction exceeded max loaded accounts data size cap) | | bank\_​transaction\_​result
{transaction\_​error="invalid\_​loaded\_​accounts\_​data\_​size\_​limit"} | counter | Result of loading and executing a transaction (LoadedAccountsDataSizeLimit set for transaction must be greater than 0) | | bank\_​transaction\_​result
{transaction\_​error="resanitization\_​needed"} | counter | Result of loading and executing a transaction (Sanitized transaction differed before/after feature activation. Needs to be resanitized) | | bank\_​transaction\_​result
{transaction\_​error="program\_​execution\_​temporarily\_​restricted"} | counter | Result of loading and executing a transaction (Program execution is temporarily restricted on an account) | | bank\_​transaction\_​result
{transaction\_​error="unbalanced\_​transaction"} | counter | Result of loading and executing a transaction (The total balance before the transaction does not equal the total balance after the transaction) | | bank\_​transaction\_​result
{transaction\_​error="program\_​cache\_​hit\_​max\_​limit"} | counter | Result of loading and executing a transaction (The total program cache size hit the maximum allowed limit) | | bank\_​transaction\_​result
{transaction\_​error="commit\_​cancelled"} | counter | Result of loading and executing a transaction (The process for committing the transaction was cancelled internally) | | bank\_​transaction\_​result
{transaction\_​error="bundle\_​peer"} | counter | Result of loading and executing a transaction (Transaction is part of a bundle and one of the peer transactions failed) | | bank\_​processing\_​failed | counter | Count of transactions for which the processing stage failed and won't land on chain | | bank\_​fee\_​only\_​transactions | counter | Count of transactions that will land on chain but without executing | | bank\_​executed\_​failed\_​transactions | counter | Count of transactions that execute on chain but failed | | bank\_​successful\_​transactions | counter | Count of transactions that execute on chain and succeed | | bank\_​cost\_​model\_​undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to | Pohh Tile [​](https://docs.firedancer.io/api/metrics.html#pohh-tile) --------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | pohh\_​begin\_​leader\_​delay\_​seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. | | pohh\_​first\_​microblock\_​delay\_​seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. | | pohh\_​slot\_​done\_​delay\_​seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. | | pohh\_​bundle\_​initialize\_​delay\_​seconds | histogram | Delay in starting the slot caused by loading the information needed to generate the bundle crank transactions | Store Tile [​](https://docs.firedancer.io/api/metrics.html#store-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | store\_​transactions\_​inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far | Backt Tile [​](https://docs.firedancer.io/api/metrics.html#backt-tile) ----------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | backt\_​final\_​slot | gauge | The slot after which the backtest will complete | | backt\_​start\_​slot | gauge | The slot at which the backtest started | Benchs Tile [​](https://docs.firedancer.io/api/metrics.html#benchs-tile) ------------------------------------------------------------------------- | Metric | Type | Description | | --- | --- | --- | | benchs\_​transactions\_​sent | counter | Number of benchmark packets sent | --- # Firedancer WebSocket API | Firedancer [Skip to content](https://docs.firedancer.io/api/websocket.html#VPContent) On this page Firedancer WebSocket API [​](https://docs.firedancer.io/api/websocket.html#firedancer-websocket-api) ===================================================================================================== Firedancer provides an optional [HTTP websockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API for consumers to subscribe to validator information. It primarily exists in the current form to support the Firedancer GUI. WARNING The API is not currently stable, is not versioned, may not exist for long, may break or start producing incorrect data at any moment, and should not generally be used for anything without extreme caution. Connecting [​](https://docs.firedancer.io/api/websocket.html#connecting) ------------------------------------------------------------------------- To connect to the API, create a WebSocket client from the language of your choice, for example in JavaScript client.jsconfig.toml js client = new WebSocket("ws://localhost:80/websocket"); toml [tiles] [tiles.gui] listen_port = 80 The port to connect to is specified in the validator configuration TOML file. The API is split into various topics which will be streamed to any and all connected clients. Compression [​](https://docs.firedancer.io/api/websocket.html#compression) --------------------------------------------------------------------------- If configured properly, the server can optionally compress messages larger than 200 bytes. In order to enable this feature, the client must specify the `compress-zstd` subprotocol in the opening websocket handshake. client.jstxt js client = new WebSocket("ws://localhost:80/websocket", protocols=['compress-zstd']); client.binaryType = "arraybuffer"; ws.onmessage = function onmessage(ev: MessageEvent) { if (typeof ev.data === 'string') { ... parse string } else if (ev.data instanceof ArrayBuffer) { ... decompress then parse } }; In order to distinguish between compressed and non-compressed messages, the server will send compressed messages as a binary websocket frame (i.e. opcode=0x2) and regular messages as a text websocket frame (i.e. opcode=0x1). Keeping Up [​](https://docs.firedancer.io/api/websocket.html#keeping-up) ------------------------------------------------------------------------- The server does not drop information, slow down, or stop publishing the stream of information if the client cannot keep up. A client that is reading too slow and cannot keep up with incoming data stream will have its connection forcibly closed by the server. Most data updates are streamed in real time as the changes occur except certain updates (performance counters like packet counters) which would change too quickly, which are instead republished on a regular frequency described below. Each message is published with frequency described in the documentation below. The meaning of these frequencies are: | Frequency | Meaning | | --- | --- | | _Once_ | The message is published only once, immediately after a connection is established | | _Live_ | The message is published live, immediately after the underlying data in the validator is changed | | _Request_ | The message is published in response to a specific client request | | _1s_ | The message is republished at regular one second intervals | | _Once_ + _Live_ | The message is published immediately after a connection is established, and then republished whenever the data is changed | Most information related to the state of the validator is sent both `Once` when the connection is established, and then live whenever it is updated. All data is encoded in JSON, with a containing envelope as follows: json { "topic": "summary", "key": "cluster", "value": "mainnet-beta", } Queries [​](https://docs.firedancer.io/api/websocket.html#queries) ------------------------------------------------------------------- Some messages are published on-demand in response to a request, and are marked with a frequency of _Request_. To issue a query, send a websocket frame to the server with an envelope like: json { "topic": "slot", "key": "query", "id": 42, "params": { "slot": 285291521 } } The `topic` and `key` correspond to the request method you wish to call. The `id` value is an unsigned integer (must fit in `u64`) that will be echoed back in the envelope of the response object. `params` are request specific parameters documented for each on-demand query. If the client issues a malformed request, it will be forcibly disconnected. If the client issues a well-formed request for data that the validator does not have (for example, an old slot), the query will receive a response with a value of `null`. json { "topic": "slot", "key": "query", "id": 42, "value": null } Forks [​](https://docs.firedancer.io/api/websocket.html#forks) --------------------------------------------------------------- The Solana network may occasionally fork, in which case there will be more than one active chain. When showing information derived from the chain, the API will (unless specified otherwise) show information reflecting the current fork choice of this validator. The current fork choice of this validator might not be the newest, or the heaviest (most voted on, or most likely to be chosen) fork. For example, when showing the transactions per second (TPS) rate under `summary.estimated_tps`, it will be calculated using the transactions and block timings observed in the current fork. Similarly, the `completed_slot` is the last completed slot on the current fork choice. When the validator switches fork choice, certain of this information will be republished to make sure it reflects the new fork choice. Topics [​](https://docs.firedancer.io/api/websocket.html#topics) ----------------------------------------------------------------- ### summary [​](https://docs.firedancer.io/api/websocket.html#summary) A set of high level informational fields about the validator. #### `summary.ping` [​](https://docs.firedancer.io/api/websocket.html#summary-ping) | frequency | type | example | | --- | --- | --- | | _Request_ | `null` | below | Sends a ping to the server, which will respond with a pong. This is an application level ping/pong and not a WebSocket control frame. Example json { "topic": "summary", "key": "ping", "id": 42, } json { "topic": "summary", "key": "ping", "id": 42, "value": null } #### `summary.version` [​](https://docs.firedancer.io/api/websocket.html#summary-version) | frequency | type | example | | --- | --- | --- | | _Once_ | `string` | `"0.106.11814"` | The current version of the running validator. #### `summary.cluster` [​](https://docs.firedancer.io/api/websocket.html#summary-cluster) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `string` | `"mainnet-beta"` | One of `mainnet-beta`, `devnet`, `testnet`, `pythtest`, `pythnet`, `development`, or `unknown`. Indicates the cluster that the validator is likely to be running on. The cluster is guessed by looking at the genesis hash of the chain and comparing it to known cluster genesis hashes. The cluster cannot change once the validator is running, but because it may not be known when the validator first starts, you might get two cluster messages. One `unknown` immediately when the validator is booted, and then a message with `mainnet` (or other known cluster) when the validator learns its cluster from a downloaded snapshot. #### `summary.commit_hash` [​](https://docs.firedancer.io/api/websocket.html#summary-commit-hash) | frequency | type | example | | --- | --- | --- | | _Once_ | `string` | `"78eefec7c779ef138aaaf4afe76cd6eaf4807006"` | The commit hash used to build the validator. #### `summary.identity_key` [​](https://docs.firedancer.io/api/websocket.html#summary-identity-key) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `string` | `"Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF"` | The public identity key assigned to the running validator, encoded in base58. Firedancer supports changing the identity key of the validator while it is running through a `set-identity` command, and if this happens a new `identity_key` will be published. Summary information in this API is tied to the validator instance and not the identity key, for example, the skip rate is the skip rate of all blocks produced by this validator, regardless of what identity key they were published with. The `mine` field of blocks similarly indicates if this validator published the block, not whether it had the same identity key as the validator has now. Because of this, when changing identity key, no other information will be republished. It will simply continue counting for blocks published with the new key. #### `summary.vote_state` [​](https://docs.firedancer.io/api/websocket.html#summary-vote-state) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `string` | `voting` | One of `voting`, `non-voting`, or `delinquent`, indicating the current vote status of the validator. The validator considers itself delinquent if the last vote it has landed on its own currently chosen fork is 150 or more slots behind that fork. #### `summary.vote_distance` [​](https://docs.firedancer.io/api/websocket.html#summary-vote-distance) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `2` | A number showing the distance between the highest slot the validator has landed a vote for, and the current highest replayed slot on the validators fork choice. This value excludes skipped slots, unless the distance is larger than 2 epochs worth of slots (NOTE: skipped slots are not excluded on Frankendancer). A distance of more than 150 means the validator is considered delinquent. #### `summary.turbine_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-turbine-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number\|null` | `100` | The largest slot that is known by the validator to have been published to the blockchain. This is typically going to be the largest slot we've seen in a received turbine shred, but can also be a slot for which we were just leader. During boot, the max known slot may not be known yet if we haven't received any shreds. In this case this message will publish `null`. It is worth noting that `turbine_slot` might be momentarily inaccurate (too large). If this happens, it should self-correct after about 4.8 seconds. This happens because `turbine_slot` is derived from the header on incoming shreds. In the worst case, a malicious leader shred can create an arbitrarily large slot on a new fork. All slot numbers received from shreds, including any malicious shreds, are forgotten after 4.8 seconds. This ensures our estimate self-corrects over time. NOTE: this message is only supported on the Firedancer client, the Frankendancer client will always publish `null` for this message #### `summary.repair_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-repair-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number\|null` | `100` | The largest slot for which the validator sent out a repair request. This slot has the same problem as `summary.turbine_slot` (it might sporadically become unboundedly large) and provides the same guarantees. #### `summary.vote_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-vote-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number\|null` | `100` | The most recent slot this node has landed a vote for. Will typically be one slot behind the current slot on the leader schedule. #### `summary.caught_up_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-caught-up-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number\|null` | `100` | The slot when this validator caught up to the tip of the blockchain. This slot is recorded when replay slot is within 4 slots (one leader rotation) of `summary.turbine_slot`. If the WebSocket client connects before the validator has caught up, then this message will be published with `null`. The message would then be published once when the validator actually catches up. Since `summary.turbine_slot` can be sometimes arbitrarily larger than the ground truth, that affects the accuracy of the catch-up slot as well. If a maliciously large shred arrives within 3 leader rotations of the validator catchup event, then `summary.turbine_slot` will be wrong, and `summary.slot_caught_up` will not be recorded until after 4.8 seconds when the malicious slot is forgotten. Functionally, this means that `summary.slot_caught_up` could be arbitrarily larger than the true catchup slot. The likelihood of this is low, and decreases for larger errors. NOTE: this message is only supported on the Firedancer client, the Frankendancer client will always publish `null` for this message #### `summary.catch_up_history` [​](https://docs.firedancer.io/api/websocket.html#summary-catch-up-history) | frequency | type | example | | --- | --- | --- | | _Once_ | `CatchUpHistory` | see below | This validator records a history of all slots that were received from turbine or repair responses, as well as shred events that occurred while catching up. After catching up, slots are no longer recorded in this history. For repair and turbine slots, the history is available for the lifetime of the validator. Shred events are only available if the validator is in the catching up phase. Example json { "topic": "summary", "key": "catch_up_history", "value": { "repair": [11, 12, 13, ...], "turbine": [21, 22, 23, ...], "shreds": { "reference_slot": 289245044, "reference_ts": "1739657041588242791", "slot_delta": [0, 0], "shred_idx": [1234, null], "event": [0, 1], "event_ts_delta": ["1000000", "2000000"] } } } **`CatchUpHistory`** | Field | Type | Description | | --- | --- | --- | | repair | `number[]` | A list of all slots for which a repair shred was received that are older than `summary.caught_up_slot` | | turbine | `number[]` | A list of all slots for which a turbine shred was received that are older than `summary.caught_up_slot` | | shreds | `SlotShreds` | A list of shred events which have occurred for this validator in the past 15 seconds. If the validator has already caught up, or has not yet started catching up, then `null` | #### `summary.startup_time_nanos` [​](https://docs.firedancer.io/api/websocket.html#summary-startup-time-nanos) | frequency | type | example | | --- | --- | --- | | _Once_ | `string` | `"1719910299914232"` | A UNIX timestamp in nanoseconds of the validator's startup. The timestamp is taken by the gui tile during boot, so it occurs before the validator downloads a snapshot and fully catches up to the cluster. #### `summary.server_time_nanos` [​](https://docs.firedancer.io/api/websocket.html#summary-server-time-nanos) | frequency | type | example | | --- | --- | --- | | _Once_ | `string` | `"1719910299914232"` | #### `summary.startup_progress` [​](https://docs.firedancer.io/api/websocket.html#summary-startup-progress) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `StartupProgress` | below | Information about the validator's progress in starting up. There are various stages of starting up which the validator goes through in order before it is ready. Typically, the phase will only move forward except for a special case: the validator can go from `downloading_full_snapshot` back to `searching_for_full_snapshot` if the snapshot peer found is downloading too slow and we would like to try a different one (and the same applies for the incremental snapshot). The phases are, | Phase | Description | | --- | --- | | initializing | The validator has just booted and has not yet started looking for RPC services to download snapshots from | | searching\_for\_full\_snapshot | The validator is searching over gossip for an RPC node to download a full snapshot from | | downloading\_full\_snapshot | The validator has found an RPC peer to download a full snapshot from and the download is now in progress | | searching\_for\_incremental\_snapshot | The validator is searching over gossip for an RPC node to download an incremental snapshot from | | downloading\_incremental\_snapshot | The validator has found an RPC peer to download an incremental snapshot from and the download is now in progress. The incremental snapshot is a smaller snapshot taken more regularly, which builds on top of a full snapshot | | cleaning\_blockstore | Removes stale data from the blockstore | | cleaning\_accounts | Removes stale data from the accounts database | | loading\_ledger | Loads the ledger data from disk into memory | | processing\_ledger | The blockstore is replayed to get to the current slot | | starting\_services | RPC, the leader TPU, the replay TVU, snapshots, and all other services are being started | | halted | The validator is halted at a specific slot because of a development option provided at boot, and will not proceed further | | waiting\_for\_supermajority | The validator is waiting at a specific slot for a supermajority of stake to come online on the gossip network so it can proceed. This is used during cluster restarts | | running | The validator is fully booted and running normally | Example json { "topic": "summary", "key": "startup_progress", "value": { "phase": "downloading_full_snapshot", "downloading_full_snapshot_peer": "145.40.125.99:8899", "downloading_full_snapshot_slot": 291059318, "downloading_full_snapshot_elapsed_secs": 24.01, "downloading_full_snapshot_remaining_secs": 254.26, "downloading_full_snapshot_throughput": 17193374.00, "downloading_full_snapshot_total_bytes": 4746970624, "downloading_full_snapshot_current_bytes": 375455480, "downloading_incremental_snapshot_peer": null, "downloading_incremental_snapshot_slot": null, "downloading_incremental_snapshot_elapsed_secs": null, "downloading_incremental_snapshot_remaining_secs": null, "downloading_incremental_snapshot_throughput": null, "downloading_incremental_snapshot_total_bytes": null, "downloading_incremental_snapshot_current_bytes": null, "ledger_slot": null, "ledger_max_slot": null, "waiting_for_supermajority_slot": null, "waiting_for_supermajority_stake_percent": null } } **`StartupProgress`** | Field | Type | Description | | --- | --- | --- | | phase | `string` | One of `initializing`, `searching_for_full_snapshot`, `downloading_full_snapshot`, `searching_for_incremental_snapshot`, `downloading_incremental_snapshot`, `cleaning_blockstore`, `cleaning_accounts`, `loading_ledger`, `processing_ledger`, `starting_services`, `halted`, `waiting_for_supermajority`, or `running` | | downloading\_full\_snapshot\_slot | `number\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the slot that is being (or was) downloaded from the snapshot provider. Otherwise it is `null` | | downloading\_full\_snapshot\_peer | `string\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the peer RPC address that the snapshot is being downloaded from. Otherwise it is `null` | | downloading\_full\_snapshot\_elapsed\_secs | `number\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the duration, in seconds that the validator has been downloading the snapshot for. Otherwise it is `null` | | downloading\_full\_snapshot\_remaining\_secs | `number\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the estimated duration, in seconds that the validator has left to download the snapshot. Otherwise it is `null` | | downloading\_full\_snapshot\_throughput | `number\|null` | If the phase is currently `downloading_full_snapshot`, this is the current download throughput in bytes per second. Otherwise it is `null` | | downloading\_full\_snapshot\_total\_bytes | `number\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the total size of the snapshot being downloaded in bytes. Otherwise it is `null` | | downloading\_full\_snapshot\_current\_bytes | `number\|null` | If the phase is at least `downloading_full_snapshot` or later, this is the current size of the snapshot that has been downloaded in bytes. Otherwise it is `null` | | downloading\_incremental\_snapshot\_slot | `number\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the slot that is being (or was) downloaded from the snapshot provider. Otherwise it is `null` | | downloading\_incremental\_snapshot\_peer | `string\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the peer RPC address that the snapshot is being downloaded from. Otherwise it is `null` | | downloading\_incremental\_snapshot\_elapsed\_secs | `number\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the duration, in seconds that the validator has been downloading the snapshot for. Otherwise it is `null` | | downloading\_incremental\_snapshot\_remaining\_secs | `number\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the estimated duration, in seconds that the validator has left to download the snapshot. Otherwise it is `null` | | downloading\_incremental\_snapshot\_throughput | `number\|null` | If the phase is currently `downloading_incremental_snapshot`, this is the current download throughput in bytes per second. Otherwise it is `null` | | downloading\_incremental\_snapshot\_total\_bytes | `number\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the total size of the snapshot being downloaded in bytes. Otherwise it is `null` | | downloading\_incremental\_snapshot\_current\_bytes | `number\|null` | If the phase is at least `downloading_incremental_snapshot` or later, this is the current size of the snapshot that has been downloaded in bytes. Otherwise it is `null` | | ledger\_slot | `number\|null` | If the phase is at least `processing_ledger` or later, this is the current slot that we have replayed up to in the ledger. Otherwise it is `null` | | ledger\_max\_slot | `number\|null` | If the phase is at least `processing_ledger` or later, this is the maximum slot we need to replay up to in the ledger. Otherwise it is `null` | | waiting\_for\_supermajority\_slot | `number\|null` | If the phase is at least `waiting_for_supermajority` or later, and we are stopped waiting for supermajority, this is the slot that we are stopped at. Otherwise it is `null` | | waiting\_for\_supermajority\_stake\_percent | `number\|null` | If the phase is at least `waiting_for_supermajority` or later, and we are stopped waiting for supermajority, this is the percentage of stake that is currently online and gossiping to our node. Otherwise it is `null`. The validator will proceed with starting up once the stake percent reaches 80 | #### `summary.boot_progress` [​](https://docs.firedancer.io/api/websocket.html#summary-boot-progress) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `BootProgress` | below | Information about the validator's progress in starting up. There are various stages of starting up which the validator goes through in order before it is ready. The phases form a state machine, and the validator can progress through them in interesting ways, txt +--+ +------------------------------+ | v | v joining_gossip -> loading_full_snapshot -> catching_up -> running v ^ ^ loading_incremental_snapshot --+ Some interesting transitions are, * The validator may skip joining gossip and go straight to loading a snapshot if it was instructed to load from a specific file or source * The full snapshot may be restarted many times, if a snapshot is corrupt or fails to download * The incremental snapshot may be skipped if the full snapshot is sufficient * The incremental snapshot may be abandoned and the phase returns to a new full snapshot, if the incremental snapshot is corrupt or fails to download * The validator may skip the catching up phase if the snapshot brings it fully up to date, although this is extremely rare and unlikely to happen on mainnet except if the chain is halted or restarting | Phase | Description | | --- | --- | | joining\_gossip | The validator has just booted and has started looking for RPC services to download snapshots from | | loading\_full\_snapshot | The validator has found an RPC peer to download a full snapshot, or a local snapshot to read from disk. The snapshot is being downloaded, decompressed, and inserted into the account database | | loading\_incremental\_snapshot | The validator has found an RPC peer to download an incremental snapshot. The snapshot is being downloaded, decompressed, and inserted into the client database | | catching\_up | The validator is replaying / repairing any missing slots up to the tip of the chain | | running | The validator is fully booted and running normally | Example json { "topic": "summary", "key": "boot_progress", "value": { "phase": "waiting_for_supermajority", "joining_gossip_elapsed_seconds": 5, "loading_full_snapshot_elapsed_seconds": 7.8, "loading_full_snapshot_reset_count": 0, "loading_full_snapshot_slot": 359396820, "loading_full_snapshot_total_bytes_compressed": "5004677960", "loading_full_snapshot_read_bytes_compressed": "960692224", "loading_full_snapshot_read_path": "/path/to/snapshot-359396820-EuXH88VnugHwoeusjHFXAg1Fp1VucJp2Z3SSjmrpBcam.tar.zst", "loading_full_snapshot_decompress_bytes_decompressed": "4961009663", "loading_full_snapshot_decompress_bytes_compressed": "826495323", "loading_full_snapshot_insert_bytes_decompressed": "4864409599", "loading_full_snapshot_insert_accounts": 10634591, "loading_incremental_snapshot_elapsed_seconds": null, "loading_incremental_snapshot_reset_count": null, "loading_incremental_snapshot_slot": null, "loading_incremental_snapshot_total_bytes_compressed": null, "loading_incremental_snapshot_read_bytes_compressed": null, "loading_incremental_snapshot_read_path": null, "loading_incremental_snapshot_decompress_bytes_decompressed": null, "loading_incremental_snapshot_decompress_bytes_compressed": null, "loading_incremental_snapshot_insert_bytes_decompressed": null, "loading_incremental_snapshot_insert_accounts": null, "wait_for_supermajority_bank_hash": "2CeCyRoYmcctDmbXWrSUfTT4aQkGVCnArAmbdmQ5dGFi", "wait_for_supermajority_shred_version": "37500", "wait_for_supermajority_attempt": 1, "wait_for_supermajority_total_stake": "1", "wait_for_supermajority_connected_stake": "1", "wait_for_supermajority_total_peers": 1, "wait_for_supermajority_connected_peers": 1, "catching_up_elapsed_seconds": null, "catching_up_first_replay_slot": null, } } **`BootProgress`** | Field | Type | Description | | --- | --- | --- | | phase | `string` | One of `joining_gossip`, `loading_full_snapshot`, `loading_incremental_snapshot`, `catching_up`, `waiting_for_supermajority`, or `running`. This indicates the current phase of the boot process | | joining\_gossip\_elapsed\_seconds | `number` | If the phase is `joining_gossip`, this is the duration, in seconds, spent joining the gossip network | | loading\_{full\|incremental}\_snapshot\_elapsed\_seconds | `number` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the elapsed time, in seconds, spent reading (either downloading or reading from disk) the snapshot since the last reset | | loading\_{full\|incremental}\_snapshot\_reset\_count | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot` or later, this is the number of times the load for the snapshot failed and the phase was restarted from scratch. A snapshot load may fail due to an unreliable or underperforming network connection. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_slot | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot` or later, this is the slot of the snapshot being loaded. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_total\_bytes\_compressed | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the (compressed) total size of the snapshot being loaded, in bytes. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_read\_bytes\_compressed | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the (compressed) total number of bytes read from disk for the snapshot. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_read\_path | `string\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is either the remote url or local file path from which the snapshot is being read. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_decompress\_bytes\_decompressed | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the (decompressed) number of bytes processed by decompress from the snapshot so far. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_decompress\_bytes\_compressed | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the (compressed) number of bytes processed by decompress from the snapshot so far. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_insert\_bytes\_decompressed | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the (decompressed) number of bytes processed from the snapshot by the snapshot insert time so far. Otherwise, `null` | | loading\_{full\|incremental}\_snapshot\_insert\_accounts | `number\|null` | If the phase is at least `loading_{full\|incremental}_snapshot`, this is the current number of inserted accounts from the snapshot into the validator's accounts database. Otherwise, `null` | | wait\_for\_supermajority\_bank\_hash | `string\|null` | If the client was configured to include the `waiting_for_supermajority` phase at startup, this is the expected bank hash of the snapshot bank. This ensures all validators join the cluster with the same starting state. `null` if wait for supermajority is not enabled | | wait\_for\_supermajority\_shred\_version | `string\|null` | If the client was configured to include the `waiting_for_supermajority` phase at startup, this is the expected shred version it was configured with. Shred version is functionally a hash of (genesis\_hash, cluster\_restart\_history) which ensures only nodes which explicitly agree on the restart slot and restart attempt count can communicate with each other. `null` if wait for supermajority is not configured | | wait\_for\_supermajority\_attempt | `number\|null` | If the client was configured to include the `waiting_for_supermajority` phase at startup, this is the number of times this cluster has been restarted onto the snapshot slot, including the current attempt. `null` if wait for supermajority is not configured | | wait\_for\_supermajority\_total\_stake | `string\|null` | If the phase is at least `waiting_for_supermajority`, this is the total network stake in lamports used to determine the 80% restart threshold | | wait\_for\_supermajority\_connected\_stake | `string\|null` | If the phase is at least `waiting_for_supermajority`, this is the network stake in lamports that is currently active on gossip and waiting for the restart threshold | | wait\_for\_supermajority\_total\_peers | `number\|null` | If the phase is at least `waiting_for_supermajority`, this is the total number of peers with an active stake | | wait\_for\_supermajority\_connected\_peers | `number\|null` | If the phase is at least `waiting_for_supermajority`, this is the number of peers with an active stake currently active on gossip and waiting for the restart threshold | | catching\_up\_elapsed\_seconds | `number` | If the phase is `catching_up`, this is the duration, in seconds, the validator has spent catching up to the current slot | | catching\_up\_first\_replay\_slot | `number` | If the phase is `catching_up`, this is the first slot that exited the replay pipeline after booting | The `wait_for_supermajority_*` fields will be `null` if the client is not configured to wait for a cluster restart, which is the case for typical client usage. The `wait_for_supermajority_*_stake` stake fields are derived differently from the `gossip.network_stats.health` activated stake (which is from the start of the epoch). These fields account for any stake that is activating/deactivating in the current epoch and any stake that was explicitly undelegated prior to restart (e.g. inactive testnet participants or bad actors). During the `waiting_for_supermajority` phase, per-peer offline status is available via the `wait_for_supermajority.peer_{add|remove}` message. #### `summary.schedule_strategy` [​](https://docs.firedancer.io/api/websocket.html#summary-schedule-strategy) | frequency | type | example | | --- | --- | --- | | _Once_ | `string` | below | A description of the configured operational mode of the transaction scheduler. The following modes are possible: * "perf" * "balanced" * "revenue" The scheduler mode determines how eager / greedy the scheduler is when filling a block. "perf" means the scheduler tries to fill the block as quickly as possible while "revenue" means the scheduler will wait as long as possible before filling the block. "balanced" is somewhere in the middle. Example json { "topic": "summary", "key": "schedule_strategy", "value": "balanced" } #### `summary.tiles` [​](https://docs.firedancer.io/api/websocket.html#summary-tiles) | frequency | type | example | | --- | --- | --- | | _Once_ | `Tile[]` | below | Information about the tile topology of Firedancer. This is a list of tiles in the system. In Firedancer, available tiles are * netlnk: Helps the net tile perform network-stack related functions. Separated from the net tile for security/performance reasons. * net: Handles all ingress/egress network traffic. * metric: Serves system-wide metrics from shared memory to an HTTP Prometheus endpoint. * ipecho: Obtains shred version from cluster. * gossvf: "Gossip verify". Performs preliminary validation of untrusted gossip messages arriving from the network. * gossip: Handles trusted, parsed messages from gossvf tile. * shred: Parses, verifies, and reconstructs untrusted shred payloads from the network. * repair: Consumes parsed shreds from shred tile, issues repair requests for any missing shreds, and reconstructs the block. * replay: Consumes block shreds from repair and schedules execution and validation of block transactions. * exec: Executes replay transactions. * tower: Maintains consensus-related data structures and helps replay vote. * send: Sends vote transactions originating from this validator into our own TPU as well as to other leaders' TPU. * quic: Implements QUIC network protocol for receiving transactions. * verify: Verifies transaction signatures and performs preliminary deduplication. * dedup: Performs second deduplication pass for verified transactions. * resolv: Resolves transaction address lookup tables (i.e., tables of account addresses which transactions use but must be retrieved from the accounts database). * pack: Functions as transaction scheduler. * bank: Helps pack execute scheduled transactions. * poh: Generates block "ticks" and manages leader status. * sign: Generates signatures for various tiles which require them (e.g. repair, gossip). * rpc: Supports a subset of the Solana RPC API. * gui: Serves the GUI, which includes the WebSocket API described in this document. ##### short-lived [​](https://docs.firedancer.io/api/websocket.html#short-lived) * snapct: Manages the snapshot loading state machine. * snapld: Loads snapshots from the network or from the file system. * snapdc: Decompresses snapshot data. * snapin: Inserts snapshot data into the accounts database. * genesi: Handles cluster bootstrapping if validator is booting a new cluster. If booting into an existing cluster, fetches cluster info (e.g. genesis hash). **`Tile`** | Field | Type | Description | | --- | --- | --- | | kind | `string` | What kind of tile it is. In Firedancer, one of the above tiles. In Frankendancer, might be one of `net`, `sock`, `quic`, `verify`, `dedup`, `pack`, `bank`, `poh`, `shred`, `store`, `sign`, `plugin`, or `http` | | kind\_id | `number` | The index of the tile in its kind. For example, if there are four `verify` tiles they have `kind_id` values of 0, 1, 2, and 3 respectively | | pid | `number` | The process id of the tile | Example json { "topic": "summary", "key": "tiles", "value": [\ { "tile": "net", "kind_id": 0 },\ { "tile": "quic", "kind_id": 0 },\ { "tile": "verify", "kind_id": 0 },\ { "tile": "verify", "kind_id": 1 },\ { "tile": "verify", "kind_id": 2 },\ { "tile": "verify", "kind_id": 3 },\ { "tile": "dedup", "kind_id": 0 },\ { "tile": "pack", "kind_id": 0 },\ { "tile": "bank", "kind_id": 0 },\ { "tile": "bank", "kind_id": 1 },\ { "tile": "poh", "kind_id": 0 },\ { "tile": "shred", "kind_id": 0 },\ { "tile": "store", "kind_id": 0 },\ { "tile": "sign", "kind_id": 0 },\ { "tile": "plugin", "kind_id": 0 }\ { "tile": "http", "kind_id": 0 }\ ] } #### `summary.identity_balance` [​](https://docs.firedancer.io/api/websocket.html#summary-identity-balance) | frequency | type | example | | --- | --- | --- | | _Once_ + _60s_ | `string` | `"21125572"` | Account balance of this validator's identity account in lamports. The balance is on the highest slot of the currently active fork of the validator. #### `summary.vote_balance` [​](https://docs.firedancer.io/api/websocket.html#summary-vote-balance) | frequency | type | example | | --- | --- | --- | | _Once_ + _60s_ | `string` | `"21125572"` | Account balance of this validator's vote account in lamports. The balance is on the highest slot of the currently active fork of the validator. #### `summary.root_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-root-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The last slot that was rooted. Rooted slots are fully confirmed and irreversible, and the rooted slot will never decrease as switching fork cannot unroot the slot. The number will not always increase by one, as skipped slots do not update the root slot. For example, if the root slot goes from `1001` to `1003` it means slot `1002` was skipped. #### `summary.optimistically_confirmed_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-optimistically-confirmed-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The highest slot on the current fork that was optimistically confirmed. Optimistic confirmation means that over two-thirds of stake have voted to confirm the slot, and it is unlikely (although still possible, if validators switch vote) to not become rooted. Although rare, the `optimistically_confirmed_slot` could decrease if a validator switches to another fork that does not have this slot. #### `summary.completed_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-completed-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The highest completed slot on the current fork choice of the validator. The completed slot may decrease if the validator is switching forks, or could stay the same for much more than the slot production time (400 milliseconds) if leaders are offline and not producing blocks. #### `summary.estimated_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-estimated-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The estimated slot is the same as the completed slot, except it still progresses forward even if the current leaders are skipping (not producing) their slot. For example, if the last completed slot was `1001` and it has been 800 milliseconds since that slot, the estimated slot is likely to be `1003`. #### `summary.reset_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-reset-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The slot corresponding to the head of the fork we've most recently chosen to vote for. A fork choice is triggered by the completion of a replay slot, so the publish interval for this message is approximately one slot duration. #### `summary.storage_slot` [​](https://docs.firedancer.io/api/websocket.html#summary-storage-slot) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `275138349` | The oldest active rooted slot across all banks in the bank pool. Active here means that the bank has a positive reference count, which means there is some consumer which is still using it. This slot is less than or equal to the current consensus root and is always on the canonical consensus fork. Banks for slots before this slot or slots on a non-canonical fork will have a reference count of zero. #### `summary.estimated_slot_duration_nanos` [​](https://docs.firedancer.io/api/websocket.html#summary-estimated-slot-duration-nanos) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | `450267129` | The estimated duration of each slot on the network. This is a moving average from the prior 750 slots, or around five minutes. Live here means the estimate is republished whenever it changes, which is when a new slot is confirmed on the currently active fork. #### `summary.skip_rate` [​](https://docs.firedancer.io/api/websocket.html#summary-skip-rate) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `SkipRate` | `{"epoch": 522, "skip_rate": 0.456172}` | The skip rate of an epoch is the ratio of `skipped_slots/total_slots` for our leader slots in that epoch. The skip rate is only known for slots that have happened since the validator was started, and we do not incorporate slots from before boot, as we cannot know if they were skipped or not. If this validator has not had any leader slots since it was booted, the skip rate reported will be zero. The skip rate is specific to this running validator, and not any given identity key. If the validator identity is changed with `set-identity`, the skip rate will remain the same at first, and then start incorporating skips for the new identity key. **`SkipRate`** | Field | Type | Description | | --- | --- | --- | | epoch | `number` | The epoch that the skip rate is being published for | | skip\_rate | `number` | The updated skip rate for the provided epoch | #### `summary.tps_history` [​](https://docs.firedancer.io/api/websocket.html#summary-tps-history) | frequency | type | example | | --- | --- | --- | | _Once_ | `number[][]` | `[[5492.2,4578.841,914.24,0],[6134.44419,5149.23,985,0]]` | A list of the last 150 TPS samples taken by the validator. Currently the spacing between samples is poorly defined, but it's roughly one sample per slot. Each element in the outer array represents a sample, and the outer array will have up to 150 samples. Each sample will have 4 elements, which are `total_tps`, `vote_tps`, `nonvote_success_tps`, and `nonvote_failed_tps` as defined below. Samples are listed from oldest first. #### `summary.estimated_tps` [​](https://docs.firedancer.io/api/websocket.html#summary-estimated-tps) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `number` | below | The estimated number of transactions per second the network is running at. This includes total, vote, non-vote successful, and non-vote failed transactions. This is a moving average from the prior 150 slots, or around one minute. For a more precise view of transactions per second, the client can calculate it from the stream of new slot data. The sum of the non-vote successful and the non-vote failed transactions represent the number of non-vote transactions. The sum of the estimated vote and non-vote transactions will be equal to the estimated total transactions per second. Example json { "topic": "summary", "key": "estimated_tps", "value": { "total": 8348, "vote": 6875, "nonvote_success": 1473, "nonvote_failed": 0 } } #### `summary.live_network_metrics` [​](https://docs.firedancer.io/api/websocket.html#summary-live-network-metrics) | frequency | type | example | | --- | --- | --- | | _Once_ + _100ms_ | `NetworkMetrics` | below | Live network metrics provides a live view of network bandwidth utilization across the various protocols used in the client. The `protocols` list contains various different protocols the client uses to communicate with the internet. json [\ "turbine",\ "gossip",\ "tpu",\ "repair",\ "metrics"\ ] * turbine: the protocol used to disseminate blockchain data, which contains primarily executable transactions. * gossip: the protocol used to disseminate node metadata, including node IP addresses used to help nodes find each other on the network * tpu: "transaction processing unit", refers to the various subsystems in a client used to consume and forward incoming Solana transactions for their next leader slot. * repair: a client subsystem which requests any missing block data needed by the replay pipeline which may have been lost over the network * metrics: refers to the Firedancer metrics tile, which serves an http Prometheus metrics endpoint json { "topic": "summary", "key": "live_network_metrics", "value": { "ingress": [12345432, 5431234, 92345, ...], "egress": [12345432, 5431234, 92345, ...], } } **`NetworkMetrics`** | Field | Type | Description | | --- | --- | --- | | ingress | `number[]` | `ingress[i]` is the total number of ingress network bytes for `protocols[i]` | | egress | `number[]` | `egress[i]` is the total number of egress network bytes for `protocols[i]` | #### `summary.live_txn_waterfall` [​](https://docs.firedancer.io/api/websocket.html#summary-live-txn-waterfall) | frequency | type | example | | --- | --- | --- | | _Once_ + _100ms_ | `LiveTxnWaterfall` | below | Example json { "topic": "summary", "key": "live_txn_waterfall", "value": { "next_leader_slot": 285228774, "waterfall": { "in": { "pack_cranked": 1, "pack_retained": 2011, "resolv_retained": 13, "quic": 66767, "udp": 1054, "gossip": 517 }, "out": { "net_overrun": 1, "quic_overrun": 44, "quic_frag_drop": 13, "quic_abandoned": 15, "tpu_quic_invalid": 16, "tpu_udp_invalid": 17, "verify_overrun": 2059, "verify_parse": 14, "verify_failed": 4092, "verify_duplicate": 128, "dedup_duplicate": 87, "resolv_lut_failed": 4, "resolv_expired": 0, "resolv_ancient": 2, "resolv_no_ledger": 0, "resolv_retained": 0, "pack_invalid": 6, "pack_expired": 2, "pack_retained": 1985, "pack_overrun": 54, "pack_priority": 58422, "bank_invalid": 14, "block_success": 2976, "block_fail": 419 } } } } **`LiveTxnWaterfall`** | Field | Type | Description | | --- | --- | --- | | next\_leader\_slot | `number\|null` | The next leader slot that the transactions are being accumulated for | | waterfall | `TxnWaterfall` | A waterfall of transactions received since the end of the previous leader slot | A transaction waterfall describes the transactions that are received before and during a leader slot, and what happened to them. A typical waterfall is that we acquire transactions from QUIC or gossip in the lead up to (before) our leader slot, drop a few of them that fail to verify, drop a few duplicates, drop some low priority ones that won't fit into our block, and then successfully place some transactions into a block. Transactions can also be received and dropped during the leader slot, but it's important to note: the waterfall shows statistics for all transactions since the end of our last leader slot. These are transactions that are now eligible for placement into the next one. The waterfall is typically useful when viewing what happened in a past leader slot: we want to know where transactions came from, and for what reasons they didn't make it into the block. For example, if we received 100,000 transactions leading up to the slot, but only 6000 made it in, what happened to the other 94,000? The live waterfall is a special case: it's for the next slot of the validator, rather than one that is in the past. Because the slot hasn't happened yet, we know certain information: how many transactions we have received so far from users that we could pack into our next block, how many have expired, how many failed to verify, and so on, but we probably won't know how many made it into the block yet, as we do when looking at the waterfall for a block that has been published. The waterfall should generally be balanced: total transactions in and total transactions out will be the roughly the same, but not always strictly. Transactions in could be more or less than transactions out due to sampling jitter. When subtracting, be sure to account for potential underflow. #### `summary.live_tile_primary_metric` [​](https://docs.firedancer.io/api/websocket.html#summary-live-tile-primary-metric) | frequency | type | example | | --- | --- | --- | | _Once_ + _100ms_ | `LiveTilePrimaryMetric` | below | Example json { "topic": "summary", "key": "live_tile_primary_metric", "value": { "next_leader_slot": 285228774, "tile_primary_metric": { "quic": 3, "bundle_rtt_smoothed_millis": 30, "bundle_rx_delay_millis_p90": 101, "net_in": 37803082, "net_out": 4982399, "verify": 0, "dedup": 0, "bank": 89407, "pack": 0, "poh": 0, "shred": 0, "store": 0 } } } **`LiveTilePrimaryMetric`** | Field | Type | Description | | --- | --- | --- | | next\_leader\_slot | `number\|null` | The next leader slot | | tile\_primary\_metric | `TilePrimaryMetric` | Per-tile-type primary metrics. Some of these are point-in-time values (P), and some are 1-second moving window averages (W) | **`TilePrimaryMetric`** | Field | Type | Description | | --- | --- | --- | | net\_in | `number` | Network ingress bytes per second (W) | | quic | `number` | Active QUIC connections (P) | | bundle\_rtt\_smoothed\_millis | `number` | The round-trip time for grpc messages sent to the bundle server. These are mostly ping-pong messages. An exponential moving average ( avg = 1/8 val + 7/8 avg ) is used to filter the signal (W) | | bundle\_rx\_delay\_millis\_p90 | `number` | An estimate of the 90th percentile of the one-way delay of a bundle dispatched from the bundle server (W) | | verify | `number` | Fraction of transactions that failed sigverify (W) | | dedup | `number` | Fraction of transactions deduplicated (W) | | pack | `number` | Fraction of pack buffer filled (P) | | bank | `number` | Execution TPS (W) | | net\_out | `number` | Network egress bytes per second (W) | #### `summary.live_tile_timers` [​](https://docs.firedancer.io/api/websocket.html#summary-live-tile-timers) | frequency | type | example | | --- | --- | --- | | _Once_ + _25ms_ | `number[]` | below | Live tile timers is an array, one entry per tile, of how idle the tile was in the preceding 10 millisecond sampling window. A value of `-1` indicates no sample was taken in the window, typically because the tile was context switched out by the kernel or it is hung. The tiles appear in the same order here that they are reported when you first connect by the `summary.tiles` message. Example json { "topic": "summary", "key": "live_tile_timers", "value": [\ 44.972112412,\ 90.12,\ 5.42148,\ 6.24870,\ 5.00158,\ 8.1111556,\ 76.585,\ 44.225,\ 12.98,\ 16.2981,\ 43.857,\ 14.1,\ 3.15716,\ 93.2456,\ 87.998\ ] } #### `summary.live_tile_metrics` [​](https://docs.firedancer.io/api/websocket.html#summary-live-tile-metrics) | frequency | type | example | | --- | --- | --- | | _Once_ + _25ms_ | `TileMetrics` | below | Live tile metrics is a live feed of various metrics related to tile health and resource utilization. The timers field is a matrix of percentages, where entry on row `i`, column `j` is the percentage of time tile `i` spent in `regimes[j]` over the previous 10 millisecond sampling window. A value of `-1` indicates no sample was taken in the window, typically because the tile was context switched out by the kernel or it is hung. The regimes array contains the processing states that a tile can exist in. Tile regimes are the cartesian product of the following two state vectors: State vector 1: * running: means that at the time the run loop executed, there was no upstream message I/O for the tile to handle. * processing: means that at the time the run loop executed, there was one or more messages for the tile to consume. * stalled: means that at the time the run loop executed, a downstream consumer of the messages produced by this tile is slow or stalled, and the message link for that consumer has filled up. This state causes the tile to stop processing upstream messages. State Vector 2: * maintenance: the portion of the run loop that executes infrequent, potentially CPU heavy tasks * routine: the portion of the run loop that executes regularly, regardless of the presence of incoming messages * handling: the portion of the run loop that executes as a side effect of an incoming message from an upstream producer tile ##### `regimes` [​](https://docs.firedancer.io/api/websocket.html#regimes) json [\ "running_maintenance",\ "processing_maintenance",\ "stalled_maintenance",\ "running_routine",\ "processing_routine",\ "stalled_routine",\ "running_handling",\ "processing_handling",\ ] "stalled\_handling" is an impossible state, and is therefore excluded. The sched\_timers field is structured the same as the timers field, but represents a different set of regimes that together make up the total scheduling and execution time for a tile. Like the timers field, these regimes when combined make up the wallclock time elapsed since a previous sample. Unlike timers field, these regimes are designed to highlight context switches and kernel task scheduling overhead. The timers field also includes context switches but just bookkeeps it to whatever regime was running when the tile got switched out. ##### `sched_regimes` [​](https://docs.firedancer.io/api/websocket.html#sched-regimes) json [\ "wait",\ "idle",\ "user",\ "system",\ ] The regimes mean the following * wait: the time a tile's process spent waiting in the runqueue before being dispatched * user: the time a tile's process spent executing in user mode * system: the time a tile's process spent executing in kernel mode * idle: Any remaining wallclock time not accounted for by the other 3 regimes The tile indices `i` appear in the same order here that they are reported when you first connect by the `summary.tiles` message. Example json { "topic": "summary", "key": "live_tile_metrics", "value": { "timers": [\ [10.1, 0, 0, 15.3, 17, 58, 0, 0],\ [10, 0, 0, 15, 17, 58, 0, 0],\ ...\ ], "sched_timers": [\ [20.5, 29.5, 49.0, 1.0],\ [10.5, 39.5, 39.0, 11.0],\ ...\ ], "in_backp": [\ 0,\ 0,\ ...\ ], "backp_msgs": [\ 0,\ 10,\ ...\ ], "alive": [\ 1,\ 1,\ ...\ ], "nvcsw": [\ 0,\ 1234,\ ...\ ], "nivcsw": [\ 0,\ 3,\ ...\ ], "minflt": [\ 1,\ 3,\ ...\ ], "majflt": [\ 0,\ 0,\ ...\ ], "last_cpu": [\ 23,\ 12,\ ...\ ] } } **`TileMetrics`** | Field | Type | Description | | --- | --- | --- | | timers | `(number[]\|null)[]` | `timers[i]` is `null` if no sample was taken in the window, typically because the tile was context switched out by the kernel or it is hung. Otherwise, `timers[i][j]` is the percentage of time from the last 10ms tile `i` spent in regime `regimes[j]` | | sched\_timers | `(number\|null)[]` | `sched_timers[i]` is the percentage of time from the last 10ms tile `i` spent in regime `sched_regimes[j]` | | alive | `number[]` | `alive[i]` is `2` if tile `i` has permanently shut down, `1` if tile `i` has updated its heartbeat timer any time in the last 100ms, and `0` otherwise | | in\_backp | `boolean[]` | `in_backp[i]` is `true` if tile `i` is currently backpressured and `false` otherwise. | | backp\_msgs | `number[]` | `backp_msgs[i]` is the number of times since startup that tile `i` has had to wait for one or more consumers to catch up to resume publishing | | nvcsw | `number[]` | `nvcsw[i]` is the number of voluntary context switches that occurred for tile `i` since startup | | nivcsw | `number[]` | `nivcsw[i]` is the number of involuntary context switches that occurred for tile `i` since startup | | minflt | `number[]` | `minflt[i]` is the number of minor page faults that occurred for tile `i` since startup. Minor page faults occur for requested pages already in memory, but not in the page table | | majflt | `number[]` | `majflt[i]` is the number of major page faults that occurred for tile `i` since startup. Major page faults occur for requested pages not in memory or the page table | | last\_cpu | `number[]` | `last_cpu[i]` is the CPU index that tile `i` was last recorded executing on | Note that a `null` entry in `timers` field indicates that the tile has not published new information about the following fields * timers * alive * in\_backp * backp\_msgs Similarly, a `null` entry in `sched_timers` means that updates for that tile have not been published for the following fields * nvcsw * nivcsw * minflt * majflt * cpu\_idx Since the fields are not nullable, they contain the previously held value. The client should ignore / interpolate them where applicable. #### `summary.live_program_cache` [​](https://docs.firedancer.io/api/websocket.html#summary-live-program-cache) | frequency | type | example | | --- | --- | --- | | _Once_ + _100ms_ | `ProgramCacheMetrics` | below | Live program cache metrics provides a live view of program cache performance and utilization across the replay and execution tiles. The program cache is a fixed-size, fork-aware, thread-concurrent cache over programs in the account database. It avoids re-loading and re-validating sBPF ELF binaries on every transaction execution. Example json { "topic": "summary", "key": "live_program_cache", "value": { "hits": 482910, "lookups": 483200, "insertions": 145, "insertion_bytes": 18743296, "evictions": 12, "eviction_bytes": 1048576, "spills": 0, "spill_bytes": 0, "free_bytes": 104857600, "size_bytes": 134217728 } } **`ProgramCacheMetrics`** | Field | Type | Description | | --- | --- | --- | | hits | `number` | Total number of program cache hits across all execution tiles since startup. Misses can be derived as `lookups - hits` | | lookups | `number` | Total number of program cache lookups across all execution tiles since startup | | insertions | `number` | Total number of program cache insertions across all execution tiles since startup | | insertion\_bytes | `number` | Total number of bytes inserted into the program cache since startup | | evictions | `number` | Total number of program cache evictions across all execution tiles since startup | | eviction\_bytes | `number` | Total number of bytes evicted from the program cache since startup | | spills | `number` | Total number of times a compiled program was not cached because the cache was full and no entry could be evicted. The program is executed from a temporary buffer instead, which is slower | | spill\_bytes | `number` | Total number of bytes used by spilled programs since startup | | free\_bytes | `number` | Current number of free bytes in the program cache heap | | size\_bytes | `number` | Total size in bytes of the program cache heap | The hit rate can be derived as `hits / lookups`. Heap utilization can be derived as `1 - (free_bytes / size_bytes)`. A healthy validator should have a high hit rate (close to 1.0) and low spill counts. ### block\_engine [​](https://docs.firedancer.io/api/websocket.html#block-engine) Block engines are providers of additional transactions to the validator, which are configurable by the operator. The validator may not be configured to use any block engines, in which case no update will be provided. For now, at most one block engine can be configured, and the name and url will not change during the lifetime of the validator. #### `block_engine.update` [​](https://docs.firedancer.io/api/websocket.html#block-engine-update) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `BlockEngine` | below | Example json { "topic": "block_engine", "key": "update", "value": { "name": "jito", "url": "https://mainnet.block-engine.jito.wtf", "ip": "141.98.216.97", "status": "connected" } } **`BlockEngine`** | Field | Type | Description | | --- | --- | --- | | name | `string` | A short, descriptive name for the block engine | | url | `string` | An HTTP URL for the block engine which the validator client connects to | | ip | `string` | The resolved IP address of the block engine | | status | `string` | One of `disconnected`, `connecting`, or `connected` indicating the state of the connection to the block engine | ### epoch [​](https://docs.firedancer.io/api/websocket.html#epoch) Information about an epoch. Epochs are never modified once they have been determined, so the topic only publishes a continuous stream of new epochs as they are known. When connecting, the current and next epoch are known, unless the validator has recently booted in which case they may not be known and no epochs will be sent until the snapshot is loaded. Epochs become known one epoch in advance, and will only be published once they are confirmed (the prior epoch has fully rooted). #### `epoch.new` [​](https://docs.firedancer.io/api/websocket.html#epoch-new) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `EpochSchedule` | below | Example json { "epoch": 636, "start_time_nanos": "12412481240412", "end_time_nanos": "1719910299914232", "start_slot": 274752000, "end_slot": 275183999, "excluded_stake_lamports": "0", "staked_pubkeys": [\ "Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF",\ "2CeCyRoYQcctDmbXWrSUfTT4aQkGVCnArAmbdmQ5QGFi",\ "6JPDr4URdEDP5MqPgmDT6jk2nToyMUzNU27qsGxrRgKz",\ "8ri9HeWZv4Dcf4BD46pVPjmefzJLpbtfdAtyxyeG4enL",\ // ... many more ...\ ], "staked_lamports": [\ "360",\ "240",\ "180",\ "9991",\ // ... many more ...\ ], "leader_slots": [\ 15,\ 1542,\ 761,\ // ... many more ...\ ] } **`EpochSchedule`** | Field | Type | Description | | --- | --- | --- | | epoch | `number` | An identity counter for each epoch, starting at zero for the first epoch and going up | | start\_time\_nanos | `string` | A UNIX timestamp, in nanoseconds, of when the epoch started. This is the time the last non-skipped block of the prior epoch finished replaying locally on this validator, if the validator was online when that happened, otherwise it is null | | end\_time\_nanos | `string` | A UNIX timestamp, in nanoseconds, of when the epoch ended. This is the time the last non-skipped block of the epoch finished replaying locally on this validator, if the validator was online when that happened, otherwise it is null | | start\_slot | `number` | The first slot (inclusive) in the epoch | | end\_slot | `number` | The last slot (inclusive) in the epoch | | excluded\_stake\_lamports | `string` | This number is almost always zero. Firedancer has a limit of 40,200 for the number of staked peer validators it can keep track of. In the unlikely event that this number is exceeded, the lowest staked peers will be forgotten, and their stake will not appear in the below lists. But it is useful to know the total stake in the epoch, so this value represents the leftover/excluded ("poisoned") amount of stake that we do not know which validator it belongs to | | staked\_pubkeys | `string[]` | A list of all of validator identity keys for validators which are staked in this epoch. There will be at most 40,200 staked keys, after which lower staked keys will not be included | | staked\_lamports | `string[]` | A list with the same length as the `staked_pubkeys` field. `stake_lamports[ i ]` is the number of lamports staked on the pubkey `staked_pubkeys[ i ]` as of this epoch | | leader\_slots | `number[]` | An array, one entry per four slots, of which pubkey in the `leader_pubkeys` array is leader for those slots. On `mainnet-beta` this array will always have a length of 108,000, which is the number of slots in an epoch divided by four. Leader slots are in groups of four because the leader schedule is generated in such a way as to guarantee each leader gets at least four consecutive slots. For example, to find the pubkey of the leader in slot 1000 of the epoch, it is `staked_pubkeys[ leader_slots[ 1000/4 ] ]` | On establishing a connection two epochs are sent to the client. The current epoch that the cluster is in, and the next epoch. From then on, new epochs are published live as they are calculated by the validator. For epoch T, it is published as `end_slot` in epoch T-2 is rooted. The epoch is speculatively known as soon as `end_slot` in epoch T-2 is completed, rather than rooted, but no speculative epoch information is published until the epoch is finalized by rooting the slot. ### gossip [​](https://docs.firedancer.io/api/websocket.html#gossip) Information about the validator's connection to the gossip network. Gossip is a distributed database which maintains a single underlying store called the Cluster Replicated Data Store (CRDS), which this documentation will simply call the "Gossip Table". The Gossip Table has "CRDS values", which are store entries that take the form of one of several different structured variants specified in the protocol. This documentation will simply call these "table entries". Note that "Gossip messages" and "Gossip Table entries" are distinct measures and therefore cannot be compared coherently. A typical Gossip message may contain several table entries, or none at all. The server maintains a table called the "Peer Table" with per-peer Gossip connection metrics. This table is large and updates frequently. Instead of sending a new copy of the table to every client periodically, the node maintains a viewport of a sorted instance of the Peer Table. The `gossip.query_scroll`, `gossip.query_sort`, `gossip.peers_size_update`, and `gossip.view_update` allow the client to synchronize with and update their viewport. A viewport is parameterized by the following attributes: * sort\_key: a list of (column, direction) tuples which describe a possible ordering of the Peer Table. Earlier columns in the sort key have higher precedence, meaning they are "stable sorted" later. This increases the visual impact of their ordering. * start\_row: the Peer Table index of the first row in the viewport * row\_cnt: the number of rows in the viewport The server imposes a limit of a maximum of 200 rows per viewport. When a client first connects, the server assigns them a default viewport state, which is specified below. #### `gossip.network_stats` [​](https://docs.firedancer.io/api/websocket.html#gossip-network-stats) | frequency | type | example | | --- | --- | --- | | _Once_ + _300ms_ | `GossipNetworkStats` | below | Example json { "health": { "num_push_messages_rx_success": 1234, "num_push_messages_rx_failure": 0, "num_push_entries_rx_success": 0, "num_push_entries_rx_failure": 0, "num_push_entries_rx_duplicate": 0, "num_pull_response_messages_rx_success": 0, "num_pull_response_messages_rx_failure": 0, "num_pull_response_entries_rx_success": 0, "num_pull_response_entries_rx_failure": 0, "num_pull_response_entries_rx_duplicate": 0, "total_stake": "411123000000000000", "total_staked_peers": "911", "total_unstaked_peers": "5334", "connected_stake": "123456789", "connected_staked_peers": 623, "connected_unstaked_peers": 1432, }, "ingress": { "total_throughput": 131204210, "peer_names": ["Coinbase 02", "Figment", "Jupiter", ... ], "peer_identities": ["FDpbCBMxVnDK7maPM5tGv6MvB3v1sRMC86PZ8okm21FD", "FD7btgySsrjuo25CJCj7oE7VPMyezDhnx7pZkj2v69FD", "FDXWcZ7T1wP4bW9SB4XgNNwjnFEJ982nE8aVbbNuwFD", ... ], "peer_throughput": [15121541, 11697591, 9131124 ] }, "egress": { "total_throughput": 131204210, "peer_names": ["Coinbase 02", "Figment", "Jupiter", ... ], "peer_identities": ["FDpbCBMxVnDK7maPM5tGv6MvB3v1sRMC86PZ8okm21FD", "FD7btgySsrjuo25CJCj7oE7VPMyezDhnx7pZkj2v69FD", "FDXWcZ7T1wP4bW9SB4XgNNwjnFEJ982nE8aVbbNuwFD", ... ], "peer_throughput": [15121541, 11697591, 9131124 ] }, "storage": { "capacity": 2097152, "expired_total": 1234, "evicted_total": 0, "count": [0, 10608, 95, ...], "count_tx": [0, 10608, 95, ...], "bytes_tx": [0, 9827342, 9723, ...], }, "messages": { "num_bytes_rx": [857419, 8839524, 43480758, ...], "num_bytes_tx": [28938, 2416123, 72351557, ...], "num_messages_rx": [1364, 20477, 456094, ...], "num_messages_tx": [26, 2490, 73599, ...], } } **`GossipNetworkStats`** | Field | Type | Description | | --- | --- | --- | | health | `GossipNetworkStake` | Aggregate statistics related to the health of the gossip network and the amount of connected peers / stake | | ingress | `GossipNetworkTraffic` | Ingress network traffic and peer metrics | | egress | `GossipNetworkTraffic` | Egress network traffic and peer metrics | | storage | `GossipStorageStats` | Storage statistics showing the storage utilization for the Gossip Table. Inner arrays are ordered according to the following `tables_entries` array `["ContactInfoV1","Vote","LowestSlot","SnapshotHashes","AccountsHashes","EpochSlots","VersionV1","VersionV2","NodeInstance","DuplicateShred","IncrementalSnapshotHashes","ContactInfoV2","RestartLastVotedForkSlots","RestartHeaviestFork"]` | | messages | `GossipMessageStats` | Message statistics showing the message traffic for the Gossip Table. Inner arrays are ordered according to the following `message_types` array \["pull\_request","pull\_response","push","ping","pong","prune"\] | **`GossipNetworkHealth`** | Field | Type | Description | | --- | --- | --- | | num\_{push\|pull\_response}\_entries\_rx\_ | `number` | The number of Gossip Table entries that this node has ever received. `success` means only entries that were fully received and included in the Table are counted. `failure` means only entries that were dropped for any reason, including parsing failures or invariant violations, are counted. `duplicate` refers to entries that were dropped as duplicates. {push\|pull\_request} means that only entries received via Gossip {push\|pull\_request} messages are counted | | num\_{push\|pull\_response}\_messages\_rx\_ | `number` | The number of Gossip messages that this node has ever received. `success` means only messages that were fully valid, even if any entries they contain were dropped. `failure` means only messages that were dropped for any reason, including parsing failures or invariant violations, are counted. `duplicate` refers to messages that were dropped as duplicates. {push\|pull\_request} is the type of Gossip message counted | | total\_stake | `number` | The total active stake on the Solana network for the current epoch. The information is derived from the getLeaderSchedule rpc call at startup and is fixed for the duration of the epoch | | total\_staked\_peers | `number` | The total number of peers on the current epoch leader schedule also active on Gossip. This information is derived from `getClusterNodes` and `getLeaderSchedule` rpc calls at startup | | total\_unstaked\_peers | `number` | The total number of peers active on gossip, not including peers on the leader schedule. This information is derived from `getClusterNodes` and `getLeaderSchedule` rpc calls at startup | | connected\_stake | `number` | The sum of active stake across all peers with a ContactInfo entry in the Gossip Table. The stake quantity is taken from the leader schedule, and reflects the activate stake at the start of the current epoch | | connected\_staked\_peers | `number` | The number of currently connected peers that have nonzero active stake | | connected\_unstaked\_peers | `number` | The number of currently connected peers without any stake currently active | **`GossipNetworkTraffic`** | Field | Type | Description | | --- | --- | --- | | total\_throughput | `number` | The Gossip network throughput in bytes per second | | peer\_names | `string[]` | The names of the 64 peers on the Gossip network with the largest contribution to our traffic | | peer\_identities | `string[]` | The base58 identity pubkey of the 64 peers on the Gossip network with the largest contribution to our traffic | | peer\_throughput | `number[]` | A list of network throughputs in bytes per second. The peer name for each entry is the corresponding entry in `peer_names` | **`GossipStorageStats`** | Field | Type | Description | | --- | --- | --- | | capacity | `number` | The total number of entries that can be stored in the Gossip Table before old entries start being evicted | | expired\_total | `number` | The cumulative number of Gossip Table entries that have expired and been removed | | evicted\_total | `number` | The cumulative number of Gossip Table entries that have been evicted due to insufficient space | | count | `number[]` | `count[i]` is the number of currently active `table_entries[i]` entries currently in the Gossip Table | | count\_tx | `number[]` | `count_tx[i]` is the number of egress `table_entries[i]` entries transmitted until now | | bytes\_tx | `number[]` | `bytes_tx[i]` is the number of egress `table_entries[i]` bytes transmitted until now | **`GossipMessageStats`** | Field | Type | Description | | --- | --- | --- | | num\_bytes\_rx | `number[]` | `num_bytes_rx[i]` is the ingress cumulative byte amount received as `message_types[i]` messages | | num\_bytes\_tx | `number[]` | `num_bytes_tx[i]` is the egress cumulative byte amount sent for `message_types[i]` messages | | num\_messages\_rx | `number[]` | `num_messages_rx[i]` is the ingress cumulative message count received as `message_types[i]` messages | | num\_messages\_tx | `number[]` | `num_messages_tx[i]` is the egress cumulative message count sent for `message_types[i]` messages | #### `gossip.query_scroll` [​](https://docs.firedancer.io/api/websocket.html#gossip-query-scroll) | frequency | type | example | | --- | --- | --- | | _Request_ | `GossipViewData` | below | | param | type | description | | --- | --- | --- | | start\_row | `number` | The first row in the contiguous chunk of rows from the Peer Table in the client's view | | end\_row | `number` | The last row in the contiguous chunk of rows from the Peer Table in the client's view | The client's view of the peer table changes when they scroll. This request includes the bounds for the updated view, which lets the server respond with the view's data. If the requested rows are outside the bounds of the table, only the active rows are included in the response. When a client first connects, before a `query_scroll` request has been made, their viewport state will be initialized with `start_row=0` and `row_cnt=0` (i.e. an empty viewport), meaning they will get no updates until their viewport grows to a non-zero size. Note that the default client view is an empty viewport, meaning no updates will be published to the client until after the first `query_scroll` received by the server. Example json { "topic": "gossip", "key": "query_scroll", "id": 16, "params": { "start_row": 10, "row_cnt": 12, } } json { "topic": "gossip", "key": "query_scroll", "id": 16, "value": { "10": {"IP Address": "192.168.0.1", "col2": 2}, "11": {"IP Address": "192.168.0.2", "col2": 4}, "12": {"IP Address": "192.168.0.3", "col2": 6} } } **`GossipViewData`** The tabular data in the clients view, as a 2D dictionary. The dictionary is keyed by row index (object keys are always strings). Each value is a dictionary that represents a table row. Each row is keyed by column name, and each row value is the value of the cell for the corresponding (rowIndex, column\_name) #### `gossip.query_sort` [​](https://docs.firedancer.io/api/websocket.html#gossip-query-sort) | frequency | type | example | | --- | --- | --- | | _Request_ | `GossipViewData` | below | | param | type | description | | --- | --- | --- | | col | `string[]` | `col[ i ]` is the name of the column with the `i`th sort precedence in the requested view | | dir | `number[]` | `dir[ i ]` is sort direction `col[ i ]` in the requested view | The server maintains a copy of each client's active sort key. This message allows clients to change their sort key which will in change the ordering of their view. Since updating the sort key changes the client's view completely, the response will be a fresh copy of all the data in the client's new view. When a client first connects, the start with the following sort key by default, until an update is made. * ("Stake", desc) * ("Pubkey", desc) * ("Name", desc) * ("Country", desc) * ("IP Addr", desc) * ("Ingress Push", desc) * ("Ingress Pull", desc) * ("Egress Push", desc) * ("Egress Pull", desc) The provided sort key is a list of column names and a corresponding list of column directions. Directions are provided as signed integers with the following meanings: * ascending: 1 * descending: -1 * no sort / ignore: 0 All columns in the table must be present in the provided sort key. If the column doesn't affect the ordering of the view, it should have a direction of `0`. Note that the relative ordering of columns with `dir==0` can be arbitrary as it does not change the view ordering. Example json { "topic": "gossip", "key": "query_sort", "id": 32, "params": { "col": ["IP Addr", "Pubkey", "Name", "Country", "Stake", "Egress Pull", "Egress Push", "Ingress Pull", "Ingress Push"], "dir": [1, 0, 0, 0, 0, 0, 0, 0, 0], } } json { "topic": "gossip", "key": "query_sort", "id": 32, "value": { "10": {"IP Address": "192.168.0.1", ...}, "11": {"IP Address": "192.168.0.2", ...}, "12": {"IP Address": "192.168.0.3", ...} } } #### `gossip.peers_size_update` [​](https://docs.firedancer.io/api/websocket.html#gossip-peers-size-update) | frequency | type | example | | --- | --- | --- | | _Live_ | `number` | below | The latest known count of the number of rows in the gossip peer table. This is sent every time the total number of rows in the gossip peer table changes. Example json { "topic": "gossip", "key": "peers_size_update", "value": 1234 } #### `gossip.view_update` [​](https://docs.firedancer.io/api/websocket.html#gossip-view-update) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `GossipPeerViewUpdate` | below | Sent every time the content of the client's view changes (i.e. cell values). Example json { "topic": "gossip", "key": "view_update", "value": { "changes": [\ {\ "row_index": 10,\ "column_name": "IP Address",\ "new_value": "192.168.0.1"\ },\ {\ "row_index": 10,\ "column_name": "Port",\ "new_value": 12345\ }\ ] } } **`GossipPeerViewUpdate`** | Field | Type | Description | | --- | --- | --- | | changes | `GossipPeerViewCellUpdate[]` | An list of cells in the client's view that changed values since the last `GossipPeerViewUpdate` message | **`GossipPeerViewCellUpdate`** | Field | Type | Description | | --- | --- | --- | | row\_index | `number` | The index of the updated cell's row | | column\_name | `string` | The name of the updated cell's column | | new\_value | `any` | The new value display in the cell | ### peers [​](https://docs.firedancer.io/api/websocket.html#peers) Information about validator peers from the cluster. Peer data is sourced from gossip, the accounts database, and the on-chain configuration program. All peer information is authenticated meaning it can only be reported from the holder of the private key, however not all peer data is validated or checked for correctness. In particular, data from the gossip network and the config program is self reported by the validator and could be empty, corrupt, filled with garbage, or malicious. Peer information is keyed by the validator identity key. Multiple vote accounts could in theory use the same identity keypair, although it is not likely. Not all identities reported will have gossip data, a vote account, or validator information published to the config program, but all identities will have at least one of these fields reported. Once an identity is no longer in these three data sources, it will be removed. #### `peers.update` [​](https://docs.firedancer.io/api/websocket.html#peers-update) | frequency | type | example | | --- | --- | --- | | _Once_ + _60s_ | `PeerUpdate` | below | Example json { "update": [\ {\ "identity_pubkey": "Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF",\ "gossip": {\ "version": "1.18.15",\ "feature_set": 4215500110,\ "wallclock": 0,\ "shred_version": 0,\ "sockets": {\ "gossip": "93.119.195.160:8001",\ "tpu": "192.64.85.26:8000",\ // ... other sockets ...\ },\ "country_code": "CN",\ "city_name": "Beijing"\ },\ "vote": [\ {\ "vote_account": "8ri9HeWZv4Dcf4BD46pVPjmefzJLpbtfdAtyxyeG4enL",\ "activated_stake": "5812",\ "last_vote": 281795801,\ "root_slot": 281795770,\ "epoch_credits": 5917,\ "commission": 5,\ "delinquent": false\ }\ ],\ "info": {\ "name": "ExampleStake Firedancer 🔥💃",\ "details": "A longer description of the validator, perhaps describing the team behind it or how the node is operated",\ "website": "https://github.com/firedancer-io/firedancer",\ "icon_url": "https://docs.firedancer.io/fire.svg",\ "keybase_username": ""\ }\ }\ ], "remove": [\ { "identity_pubkey": "8ri9HeWZv4Dcf4BD46pVPjmefzJLpbtfdAtyxyeG4enL" }\ ] } **`PeerUpdateGossip`** | Field | Type | Description | | --- | --- | --- | | wallclock | `number` | Not entirely sure yet TODO | | shred\_version | `number` | A `u16` representing the shred version the validator is configured to use. The shred version is changed when the cluster restarts, and is used to make sure the validator is talking to nodes that have participated in the same cluster restart | | client\_id | `number\|null` | The client id broadcast by the validator on Gossip. Refer to [https://github.com/solana-foundation/solana-validator-client-ids/blob/main/client-ids.csv](https://github.com/solana-foundation/solana-validator-client-ids/blob/main/client-ids.csv)
for an official mapping of id to name. Will be `null` on Frankendancer. | | version | `string\|null` | Software version being advertised by the validator. Might be `null` if the validator is not gossiping a version, or we have received the contact information but not the version yet. The version string, if not null, will always be formatted like `major`.`minor`.`patch` where `major`, `minor`, and `patch` are `u16`s | | feature\_set | `number\|null` | First four bytes of the `FeatureSet` hash interpreted as a little endian `u32`. Might be `null` if the validator is not gossiping a feature set, or we have received the contact information but not the feature set yet | | sockets | `[key: string]: string` | A dictionary of sockets that are advertised by the validator. `key` will be one of gossip `serve_repair_quic`, `rpc`, `rpc_pubsub`, `serve_repair`, `tpu`, `tpu_forwards`, `tpu_forwards_quic`, `tpu_quic`, `tpu_vote`, `tvu`, `tvu_quic`, `tpu_vote_quic`, or `alpenglow`. The value is an address like `:`: the location to send traffic to for this validator with the given protocol. Address might be either an IPv4 or an IPv6 address | | country\_code | `string\|null` | ISO 3166-1 alpha-2 country code of where the validator is located, determined by GeoIP lookup on the gossip IP address. This information is powered by `DB-IP.com`. Country code may not be correct and is a best estimate. If no country code could be determined, will be `null`. | | city\_name | `string\|null` | The name of the city where the validator is located, determined by GeoIP lookup on the gossip IP address. This information is powered by `DB-IP.com`. City name may not be correct and is a best estimate. If no city name could be determined, will be `null`. | **`PeerUpdateVoteAccount`** | Field | Type | Description | | --- | --- | --- | | vote\_account | `string` | The public key of vote account, encoded in base58 | | activated\_stake | `string` | The amount of stake in lamports that is activated on this vote account for the current epoch. Warming up or cooling down stake that was delegating during this epoch is not included | | last\_vote | `number\|null` | The last vote by the vote account that was landed on chain, as seen by this validator. If the vote account has not yet landed any votes on the chain this will be `null` | | root\_slot | `number\|null` | The last slot that was rooted by the vote account, based on the vote history. If the vote account has not yet rooted any slots this will be `null` | | epoch\_credits | `number` | The number of credits earned by the vote account during the current epoch | | delinquent | `boolean` | Whether the vote account is delinquent or not. A vote account is considered delinquent if it has not had a vote land on chain for any of the last 127 (inclusive) confirmed slots, according to this validator. If there have been less than 128 confirmed slots on the chain (it is a new chain), a validator is considered delinquent only if it has not voted yet at all | **`PeerUpdateInfo`** | Field | Type | Description | | --- | --- | --- | | name | `string` | Self reported name of the validator, could be any string or empty string if there is no name set | | details | `string` | Self reported detailed description of the validator, could be any string or empty string if there is no details set | | website | `string` | Self reported website of the validator, could be any string and need not be a valid URI, or could be empty string if there is no website set | | icon\_url | `string` | Self reported URL of the validator icon, could be any string and need not be a valid URI, or could be empty string if there is no icon URI set | | keybase\_username | `string` | Self reported keybase username of the validator, could be any string or empty string if there is no username set. Keybase is a public, legacy storage for icon images. Although this method for publicizing an icon is deprecated, it is included for completeness as many validators have not migrated to `iconUrl` | **`PeerUpdate`** | Field | Type | Description | | --- | --- | --- | | identity | `string` | Identity public key of the validator, encoded in base58 | | gossip | `PeerUpdateGossip\|null` | Information reported for the validator identity over the gossip network. This is authenticated and the gossip node must have been in possession of the private key to publish gossip data as this identity. Gossip information is not validated or checked for correctness and could be set to any values by the peer | | vote | `PeerUpdateVoteAccount[]` | Information about the vote account(s) associated with this identity key, if there are any. It is extremely unusual for multiple vote accounts to report the same identity key. Vote account information like stake and commission is derived from the accounts on chain and cannot be corrupt, invalid, or incorrect | | info | `PeerUpdateInfo\|null` | If the validator has published self reported identifying information to the chain. This is authenticated and the operator must have been in possession of the private key to publish info as this identity. Information is not validated or checked for correctness and could be set to any values by the peer | **`PeerRemove`** | Field | Type | Description | | --- | --- | --- | | identity | `string` | Identity public key of the validator, encoded in base58 | **`PeersUpdate`** | Field | Type | Description | | --- | --- | --- | | add | `GossipPeerUpdate[]` | List of peer validators that were added since the last update, or all of the peers for the first update after connecting | | update | `GossipPeerUpdate[]` | List of peer validators that were changed since the last update | | remove | `GossipPeerRemove[]` | List of peer validators that were removed since the last update | The `gossip.update` message is republished every five seconds, with a list of gossip peers added, removed, or updated. The list of peers is full and includes this node itself, nodes with a different `shred_version`, nodes publishing corrupt or bad information, and so on. ### wait\_for\_supermajority [​](https://docs.firedancer.io/api/websocket.html#wait-for-supermajority) Messages published during the wait-for-supermajority phase. These messages are only published if the client is configured to boot with the `waiting_for_supermajority` phase enabled. #### `wait_for_supermajority.stakes` [​](https://docs.firedancer.io/api/websocket.html#wait-for-supermajority-stakes) | frequency | type | example | | --- | --- | --- | | _Once_ | `WaitForSupermajorityEpoch` | below | Sent once per connection, after the snapshot is fully loaded and validator info has been parsed from the ConfigProgram accounts in the snapshot. Example json { "topic": "wait_for_supermajority", "key": "stakes", "value": { "staked_pubkeys": [\ "Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF",\ "2CeCyRoYQcctDmbXWrSUfTT4aQkGVCnArAmbdmQ5QGFi",\ "6JPDr4URdEDP5MqPgmDT6jk2nToyMUzNU27qsGxrRgKz"\ ], "staked_lamports": [\ "360",\ "240",\ "180"\ ], "infos": [\ null,\ null,\ {\ "name": "Validator",\ "details": "",\ "website": "",\ "icon_url": "",\ "keybase_username": ""\ }\ ] } } **`WaitForSupermajorityEpoch`** | Field | Type | Description | | --- | --- | --- | | staked\_pubkeys | `string[]` | Identity pubkeys of all staked validators in the snapshot epoch, base58 encoded | | staked\_lamports | `string[]` | A list with the same length as `staked_pubkeys`. `staked_lamports[ i ]` is the number of lamports staked on `staked_pubkeys[ i ]` | | infos | `(PeerUpdateInfo\|null)[]` | A list with the same length as `staked_pubkeys`. Each element is a `PeerUpdateInfo` object if the validator has published self-reported info via ConfigProgram in the snapshot, or `null` otherwise | #### `wait_for_supermajority.peer_add` [​](https://docs.firedancer.io/api/websocket.html#wait-for-supermajority-peer-add) | frequency | type | example | | --- | --- | --- | | _Once_ + _Live_ | `string[]` | below | Example json { "topic": "wait_for_supermajority", "key": "peer_add", "value": [\ "Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF",\ "2CeCyRoYQcctDmbXWrSUfTT4aQkGVCnArAmbdmQ5QGFi"\ ] } Value is a flat array of base58-encoded identity pubkeys that have come online since the last message (or all currently-online peers on initial connect). #### `wait_for_supermajority.peer_remove` [​](https://docs.firedancer.io/api/websocket.html#wait-for-supermajority-peer-remove) | frequency | type | example | | --- | --- | --- | | _Live_ | `string[]` | below | Example json { "topic": "wait_for_supermajority", "key": "peer_remove", "value": [\ "9aE6Bp1hbDpMFKqnWGUMbfxfMPXswPbkNwNrSjhpFiSN"\ ] } Value is a flat array of base58-encoded identity pubkeys that have gone offline (activity timeout expired) since the last message. ### slot [​](https://docs.firedancer.io/api/websocket.html#slot) Slots are opportunities for a leader to produce a block. A slot can be in one of five levels, and in typical operation a slot moves through them in normal order, starting as `incomplete` and finishing as `finalized`. **`SlotLevel`** | level | description | | --- | --- | | `incomplete` | The slot does not exist, either because the chain has not yet reached the slot or because it is still in the process of being replayed by our validator | | `completed` | The slot has been fully received and successfully replayed by our validator | | `optimistically_confirmed` | The slot has been finished and successfully replayed by our validator, and more than two-thirds of stake have voted to confirm the slot | | `rooted` | Our validator has rooted the slot and considers the slot final. This occurs when 32 subsequent slots have been built on top of it | | `finalized` | Our validator has rooted the slot, and more than two-thirds of stake has rooted the slot, the network considers it final | Slots are `incomplete` by default as most slots exist far in the future, and the `incomplete` level update is not typically published. A slot will only be explicitly marked as `incomplete` if it exists on the currently active fork, and we switch to a fork that is slower, which has not yet reached the slot so it no longer exists. A slot that has become `rooted` or `finalized` cannot go backwards, and will not become `incomplete`, `completed`, or `optimistically_confirmed` (nor will a `finalized` slot become `rooted`), but otherwise all transitions are valid. An `optimistically_confirmed` slot, for example, could become `incomplete` if the validator switches to a fork where the slot has not yet been received, although it should be exceedingly rare. Switching from `completed` to `incomplete` is more common. Levels can also be skipped, for example going from `incomplete` straight to `optimistically_confirmed` (if we switched from a fork that was running behind, to the cluster majority fork including the slot). In addition to a level, a slot can also be either skipped or included. All levels can be in either the skipped or included state, for example a skipped slot that is `rooted` means that the slot is skipped and cannot be changed. A slot in the future could be skipped, because we know it does not build on top of the fork we have currently selected, in which case it would be both `incomplete` and `skipped`. Slots are either `mine` (created by this validator), or not, in which case we are replaying a block from another validator. Slots that are `mine` contain additional information about our performance creating the block for that slot. The `mine` field means that this specific validator published the block. It might happen that a block is published by a leader with our identity key, but not this specific validator (for example, if the block was published by another computer, and then this validator took over the identity key with a `set-identity` operation) in which case the `mine` field will be set to false, even though the block has our key. Some information is only known for blocks that have been replayed successfully (reached the `completed` state), for example the number of transactions in the block. This number can still be known even if we are on a fork which skips that slot. It's possible that we are on a fork where the slot does not yet exist, a slot could be both `skipped`, and have a status of `incomplete`, and yet we still know a valid number of `transactions` in the slot. Once we know information like `transactions` it does not typically change, although can in extremely rare cases where a leader publishes two different blocks for their leader slot, and we initially replay one but the cluster votes on the other one. **`SlotPublish`** | Field | Type | Description | | --- | --- | --- | | slot | `number` | Identity of the slot, counting up from zero for the first slot in the chain | | mine | `boolean` | True if this validator was the leader for this slot. This will never change for a slot once it has been published, and will be aligned with the epoch information, except in cases where the validator identity is changed while the validator is running | | start\_timestamp\_nanos | `string` | A UNIX timestamp, in nanoseconds, representing the time that the validator is first aware that it is leader. At this point the poh tile will signal the pack tile to begin filling the block for this slot with transactions | | target\_end\_timestamp\_nanos | `string` | A UNIX timestamp, in nanoseconds, representing the target time in nanoseconds that the pack tile should stop scheduling transactions for the slot. Transactions might still finish executing after this end time, if they started executing before it and ran over the deadline. In rare cases, transactions may also appear to begin after this timestamp due to slight clock drift between execution cores | | skipped | `boolean` | True if the slot was skipped. The skipped state is the state in the currently active fork of the validator. The skipped state can change if the validator switches active fork | | duration\_nanos | `number\|null` | A duration in nanoseconds of how long it took us to receive and replay the slot. This is the time as measured since we completed replay of the parent slot locally on this validator, til the time we replayed this slot locally on this validator | | completed\_time\_nanos | `string\|null` | UNIX timestamp in nanoseconds of when this validator finished replaying the slot locally. If the slot was skipped, this may be `null` which indicates the block for this slot did not finish replaying on this validator. In some cases, a skipped slot will still have a completed time, if we received the data for the block, replayed it, and then decided to use a different fork | | level | `string` | One of `incomplete`, `completed`, `optimistically_confirmed`, `rooted`, or `finalized` as described above. The state is the state in the currently active fork of this validator. The state can change normally (for example, a completed slot becoming optimistically confirmed or rooted), or also because the validator switched forks | | success\_nonvote\_transaction\_cnt | `number\|null` | Total number of successfully executed non-vote transactions in the block. If the slot is not skipped, this will be non-null, but in some cases it will also be non-null even if the slot was skipped. That's because we replayed the block but selected a fork without it, but we still know how many transactions were in it | | failed\_nonvote\_transaction\_cnt | `number\|null` | Total number of failed non-vote transactions in the block. If the slot is not skipped, this will be non-null, but in some cases it will also be non-null even if the slot was skipped. That's because we replayed the block but selected a fork without it, but we still know how many transactions were in it | | success\_vote\_transaction\_cnt | `number\|null` | Total number of successfully executed vote transactions in the block | | failed\_vote\_transaction\_cnt | `number\|null` | Total number of failed vote transactions in the block. This should be near-zero in a healthy cluster | | max\_compute\_units | `number\|null` | The maximum number of compute units that can be packed into the slot. This limit is one of many consensus-critical limits defined by the solana protocol, and helps keeps blocks small enough for validators consume them quickly. It may grow occasionally via on-chain feature activations | | compute\_units | `number\|null` | Total number of compute units used by the slot. Compute units are a synthetic metric that attempt to capture, based on the content of the block, the various costs that go into processing that block (i.e. cpu, memory, and disk utilization). They are based on certain transaction features, like the number of included signatures, the number of included signature verification programs, the number of included writeable accounts, the size of the instruction data, the size of the on-chain loaded account data, and the number of computation steps. NOTE: "compute units" is an overloaded term that is often used in misleading contexts to refer to only a single part of the whole consensus-critical cost formula. For example, the getBlock RPC call includes a "computeUnitsConsumed" which actually only refers only the execution compute units associated with a transaction, but excludes other costs like signature costs, data costs, etc. This API will always use compute units in a way that includes ALL consensus-relevant costs, unless otherwise specified | | shreds | `number\|null` | Total number of shreds in the successfully replayed block. Note value is only available in the Firedancer client and will be 0 or null in the Frankendancer client | | transaction\_fee | `string\|null` | Total amount of transaction fees that this slot collects in lamports after any burning | | priority\_fee | `string\|null` | Total amount of priority fees that this slot collects in lamports after any burning | | tips | `string\|null` | Total amount of tips that this slot collects in lamports, across all block builders, after any commission to the block builder is subtracted | | vote\_slot | `number\|null` | The most recent slot for which this validator had landed a vote as of the time that this slot was replayed. This is equivalent to the largest voted-for slot in this validator's on-chain vote account after the execution of `slot`. `vote_slot` will typically be one less than `slot`, though `vote_slot` may be arbitrarily small if the last successfully landed vote from this validator was long before `slot`. May be `null` if the vote account for this node does not exist | | vote\_latency | `number\|null` | The distance in slots between this slot and the slot which contains our vote for this slot. This field is `null` if we have not yet landed a vote for this slot, and this message will be re-published once our vote lands. Due to forking or votes not landing, this field may be updated arbitrarily many times, or never | #### `slot.skipped_history` [​](https://docs.firedancer.io/api/websocket.html#slot-skipped-history) | frequency | type | example | | --- | --- | --- | | _Once_ | `number[]` | `[286576808, 286576809, 286576810, 286576811, 286625025, 286625026, 286625027]` | A list of leader slot of the validator from the current epoch which were skipped. The skipped slots include unrooted and unconfirmed slots of ours which are skipped on the currently active fork. #### `slot.skipped_history_cluster` [​](https://docs.firedancer.io/api/websocket.html#slot-skipped-history-cluster) | frequency | type | example | | --- | --- | --- | | _Once_ | `number[]` | `[286576808, 286576809, 286576810, 286576811, 286625025, 286625026, 286625027]` | A list of all of the leader slots which were skipped in the current epoch. Recent non-rooted slots may be included, and included skipped slots will not become unskipped as a later slot has rooted. #### `slot.late_votes_history` [​](https://docs.firedancer.io/api/websocket.html#slot-late-votes-history) | frequency | type | example | | --- | --- | --- | | _Once_ | `number[]` | below | A collection of slots from this epoch for which this validator voted late or not at all. Specifically, the following slots are included * rooted slots with a vote latency > 1 * rooted slots that were never voted for that were not skipped The slot array is run-length encoded. For example [ a, b, c, d, e, f ] means that slots `[a, b]`, slots `[c, d]`, and slots `[e, f]` all received late or non-existent votes. The latency array is not run-length encoded. That means if the decoded slots array has `n` slots, then the length of `latency` will be `n`. Example json { "topic": "slot", "key": "late_votes_history", "value": { "slot": [286576808, 286576808, 286576810, 286576811, 286625025, 286625026], "latency": [2, 3, null, 2, 2] } } #### `slot.live_shreds` [​](https://docs.firedancer.io/api/websocket.html#slot-live-shreds) | frequency | type | example | | --- | --- | --- | | _50ms_ | `SlotShreds` | below | The validator sends a continuous stream of update messages with detailed information about the time and duration of different shred state transitions (i.e. shred events). A given event is only ever sent once and is broadcast to all WebSocket clients. Example json { "topic": "slot", "key": "live_shreds", "value": { "reference_slot": 289245044, "reference_ts": "1739657041588242791", "slot_delta": [0, 0], "shred_idx": [1234, null], "event": [0, 1], "event_ts_delta": ["1000000", "2000000"] } } **`SlotShreds`** | Field | Type | Description | | --- | --- | --- | | reference\_slot | `number` | The smallest slot number across all the shreds in a given message | | reference\_ts | `number` | The smallest UNIX nanosecond event timestamp number across all the events in a given message | | slot\_delta | `number[]` | `reference_slot + slot_delta[i]` is the slot to which shred event `i` belongs | | shred\_idx | `(number\|null)[]` | `shred_idx[i]` is the slot shred index of the shred for shred event `i`. If null, then shred event `i` applies to all shreds in the slot (i.e. this is used for `slot_complete`) | | event | `number[]` | `event[i]` is the enum value for shred event `i`. Possible values are `repair_request` (0), `shred_received_turbine` (1), `shred_received_repair` (2), `shred_replay_exec_done` (3), `shred_replay_exec_start` (4), and `slot_complete` (5) | | event\_ts\_delta | `string[]` | `reference_ts + event_ts_delta[i]` is the UNIX nanosecond timestamp when shred event `i` occurred | #### `slot.query_shreds` [​](https://docs.firedancer.io/api/websocket.html#slot-query-shreds) | frequency | type | example | | --- | --- | --- | | _Request_ | `SlotShreds\|null` | below | | param | type | description | | --- | --- | --- | | slot | `number` | The requested slot for which the response will provide shred timing data | WebSocket clients may request historical shred metadata on a per-slot basis. For slots that are too old (i.e. they've been expired from an in-memory store) or too new (i.e. they haven't been finalized yet), the response value will be `null`. Example json { "topic": "slot", "key": "query_shreds", "id": 32, "params": { "slot": 289245044 } } json { "topic": "slot", "key": "query_shreds", "id": 32, "value": { "reference_slot": 289245044, "reference_ts": "1739657041588242791", "slot_delta": [0, 0], "shred_idx": [1234, null], "event": [0, 1], "event_ts_delta": ["1000000", "2000000"] } } #### `slot.update` [​](https://docs.firedancer.io/api/websocket.html#slot-update) | frequency | type | example | | --- | --- | --- | | _Live_ | `SlotUpdate` | below | Example **`SlotUpdate`** | Field | Type | Description | | --- | --- | --- | | publish | `SlotPublish` | General information about the slot. Contains several nullable fields in case a future slot is queried and the information is not known yet | | waterfall | `TxnWaterfall\|null` | If the slot is not `mine`, will be `null`. Otherwise, a waterfall showing reasons transactions were acquired since the end of the prior leader slot | | tile\_primary\_metric | `TilePrimaryMetric\|null` | If the slot is not `mine`, will be `null`. Otherwise, max value of per-tile-type primary metrics since the end of the prior leader slot | #### `slot.query_rankings` [​](https://docs.firedancer.io/api/websocket.html#slot-query-rankings) | frequency | type | example | | --- | --- | --- | | _Request_ | `SlotRankings` | below | | param | type | description | | --- | --- | --- | | mine | `bool` | If `mine` is true, only include slots produced by this validator in the result. Otherwise, any slot from the current epoch may be included | Example json { "topic": "slot", "key": "query_rankings", "id": 32, "params": { "mine": false } } json { "topic": "slot", "key": "query_rankings", "id": 32, "value": { "slots_largest_tips": [1, 2, 3], "vals_largest_tips": [12345678, 1234567, 123456], "slots_largest_fees": [1, 2, 3], "vals_largest_fees": [12345678, 1234567, 123456], "slots_largest_rewards": [1, 2, 3], "vals_largest_rewards": [12345678, 1234567, 123456], "slots_largest_duration": [1, 2, 3], "vals_largest_duration": [450000000, 440000000, 430000000], "slots_largest_compute_units": [1, 2, 3], "vals_largest_compute_units": [47000000, 46000000, 45000000], "slots_largest_skipped": [7, 8, 9], "vals_largest_skipped": [7, 8, 9], "slots_smallest_tips": [1, 2, 3], "vals_smallest_tips": [0, 0, 0], "slots_smallest_fees": [1, 2, 3], "vals_smallest_fees": [0, 0, 0], "slots_smallest_rewards": [1, 2, 3], "vals_smallest_rewards": [0, 0, 0], "slots_smallest_duration": [1, 2, 3], "vals_smallest_duration": [100000000, 120000000, 160000000], "slots_smallest_compute_units": [1, 2, 3], "vals_smallest_compute_units": [15000000, 16000000, 17000000], "slots_smallest_skipped": [4, 5, 6], "vals_smallest_skipped": [4, 5, 6] } } **`SlotRankings`** | Field | Type | Description | | --- | --- | --- | | {slots\|vals}\_{smallest\|largest}\_tips | `number[]` | Rankings for the {smallest\|largest} tips this epoch | | {slots\|vals}\_{smallest\|largest}\_fees | `number[]` | Rankings for the {smallest\|largest} fees this epoch | | {slots\|vals}\_{smallest\|largest}\_rewards | `number[]` | Rankings for the {smallest\|largest} rewards this epoch | | {slots\|vals}\_{smallest\|largest}\_rewards\_per\_cu | `number[]` | Rankings for the {smallest\|largest} rewards/cu ratio this epoch | | {slots\|vals}\_{smallest\|largest}\_duration | `number[]` | Rankings for the {smallest\|largest} slot durations this epoch | | {slots\|vals}\_{smallest\|largest}\_compute\_units | `number[]` | Rankings for the {smallest\|largest} compute units this epoch | | {slots\|vals}\_{smallest\|largest}\_skipped | `number[]` | Rankings for the {earliest\|latest} skipped slots this epoch | Each metric in this message will have four associated arrays. * vals\_smallest\_metric: Metric value for the lowest ranked slots (sorted ascending) * slots\_smallest\_metric: Slot numbers for vals\_smallest\_metric source slots * slots\_largest\_metric: Slot numbers for vals\_largest\_metric source slots * vals\_largest\_metric: Metric value for the highest ranked slots (sorted descending) Slots before boot time are not included in these rankings. Unless explicitly mentioned, skipped slots are not included. #### `slot.query` [​](https://docs.firedancer.io/api/websocket.html#slot-query) | frequency | type | example | | --- | --- | --- | | _Request_ | `SlotResponse` | below | | param | type | description | | --- | --- | --- | | slot | `number` | The slot to query for information about | Example json { "topic": "slot", "key": "query", "id": 32, "params": { "slot": 289245044 } } json { "topic": "slot", "key": "query", "id": 32, "value": { "publish": { "slot": 289245044, "mine": true, "start_timestamp_nanos": null, "target_end_timestamp_nanos": null, "skipped": false, "duration_nanos": 400000000, "completed_time_nanos": "1739657041688342791", "level": "rooted", "success_nonvote_transaction_cnt": 6821, "failed_nonvote_transaction_cnt": 6746, "success_vote_transaction_cnt": 3703, "failed_vote_transaction_cnt": 0, "max_compute_units": 48000000, "compute_units": 0, "shreds": 123, "transaction_fee": 12345, "priority_fee": 123456, "tips": 0, "vote_slot": 289245043 } } } #### `slot.query_detailed` [​](https://docs.firedancer.io/api/websocket.html#slot-query-detailed) | frequency | type | example | | --- | --- | --- | | _Request_ | `SlotResponse` | below | | param | type | description | | --- | --- | --- | | slot | `number` | The slot to query for information about | Example json { "topic": "slot", "key": "query_detailed", "id": 32, "params": { "slot": 289245044 } } json { "topic": "slot", "key": "query_detailed", "id": 32, "value": { "publish": { "slot": 289245044, "mine": true, "start_timestamp_nanos": null, "target_end_timestamp_nanos": null, "skipped": false, "duration_nanos": 400000000, "completed_time_nanos": "1739657041688342791", "level": "rooted", "success_nonvote_transaction_cnt": 6821, "failed_nonvote_transaction_cnt": 6746, "success_vote_transaction_cnt": 3703, "failed_vote_transaction_cnt": 0, "max_compute_units": 48000000, "compute_units": 0, "shreds": 123, "transaction_fee": 12345, "priority_fee": 123456, "tips": 0, "vote_slot": 289245043 }, "waterfall": { "in": { "pack_cranked": 1, "pack_retained": 0, "resolv_retained": 0, "quic": 28159, "udp": 14323, "gossip": 4646, "block_engine": 13 }, "out": { "net_overrun": 0, "quic_overrun": 0, "quic_frag_drop": 0, "quic_abandoned": 0, "tpu_quic_invalid": 0, "tpu_udp_invalid": 0, "verify_overrun": 0, "verify_parse": 0, "verify_failed": 0, "verify_duplicate": 114, "dedup_duplicate": 19384, "resolv_lut_failed": 3, "resolv_expired": 0, "resolv_ancient": 0, "resolv_retained": 0, "resolv_no_ledger": 0, "pack_invalid": 0, "pack_expired": 0, "pack_already_executed": 0, "pack_retained": 2225, "pack_wait_full": 0, "pack_leader_slow": 0, "bank_invalid": 10253, "block_success": 3101, "block_fail": 3720 } }, "tile_primary_metric": { "quic": 3, "net_in": 37803082, "net_out": 4982399, "verify": 0, "dedup": 0, "bank": 89407, "pack": 0, "poh": 0, "shred": 0, "store": 0 }, "tile_timers": [\ {\ "timestamp_nanos": "1739657041688242791",\ "tile_timers": [\ 44.972112412,\ 90.12,\ 5.42148,\ 6.24870,\ 5.00158,\ 8.1111556,\ 76.585,\ 44.225,\ 12.98,\ 16.2981,\ 43.857,\ 14.1,\ 3.15716,\ 93.2456,\ 87.998\ ]\ },\ {\ "timestamp_nanos": "1739657041688342791",\ "tile_timers": [\ 44.972112412,\ 90.12,\ 5.42148,\ 6.24870,\ 5.00158,\ 8.1111556,\ 76.585,\ 44.225,\ 12.98,\ 16.2981,\ 43.857,\ 14.1,\ 3.15716,\ 93.2456,\ 87.998\ ]\ },\ // ... many more ...\ ], "scheduler_counts": [\ {\ "timestamp_nanos": "1739657041688242791",\ "regular": 123,\ "votes": 123,\ "conflicting": 123,\ "bundles": 123\ },\ {\ "timestamp_nanos": "1739657041688342791",\ "regular": 124,\ "votes": 120,\ "conflicting": 123,\ "bundles": 123\ }\ // ... many more ...\ ] } } #### `slot.query_transactions` [​](https://docs.firedancer.io/api/websocket.html#slot-query-transactions) | frequency | type | example | | --- | --- | --- | | _Request_ | `SlotTransactionsResponse` | below | | param | type | description | | --- | --- | --- | | slot | `number` | The slot to query for information about | Example json { "topic": "slot", "key": "query_transactions", "id": 32, "params": { "slot": 289245044 } } json { "topic": "slot", "key": "query_transactions", "id": 32, "value": { "publish": { "slot": 289245044, "mine": true, "start_timestamp_nanos": "1739657041688346791", "target_end_timestamp_nanos": "1739657042088346880", "skipped": false, "duration_nanos": 400000000, "completed_time_nanos": "1739657041688342791", "level": "rooted", "success_nonvote_transaction_cnt": 6821, "failed_nonvote_transaction_cnt": 6746, "success_vote_transaction_cnt": 3703, "failed_vote_transaction_cnt": 0, "max_compute_units": 48000000, "compute_units": 0, "shreds": 123, "transaction_fee": 12345, "priority_fee": 123456, "tips": 0, "vote_slot": 289245043 }, "limits": { "used_total_block_cost": 10000000, "used_total_vote_cost": 5000000, "used_account_write_costs": 1000000, "used_total_bytes": 123456, "used_total_microblocks": 12345, "max_total_block_cost": 100000000, "max_total_vote_cost": 12000000, "max_account_write_cost": 40000000, "max_total_bytes": 987654321, "max_total_microblocks": 32768 }, "scheduler_stats": { "end_slot_reason": "timeout", "block_hash": "9cjSTpZ82xeHouc3sDEzsQ8yZHKjvc8o46EstxfrprD1", "slot_schedule_counts": [123, 123, 123, 123, 123, 123, 123], "end_slot_schedule_counts": [0, 10, 0, 0, 0, 0, 0], "pending_smallest_cost": 3000, "pending_smallest_bytes": 1232, "pending_vote_smallest_cost": 3000, "pending_vote_smallest_bytes": 1232 }, "transactions": { "start_timestamp_nanos": "1739657041688346791", "target_end_timestamp_nanos": "1739657042088346880", "txn_arrival_timestamps_nanos": ["1754409729593613895"], "txn_bank_idx": [0], "txn_compute_units_consumed": [3428], "txn_compute_units_requested": [3428], "txn_end_timestamps_nanos": ["1754409729594477657"], "txn_error_code": [0], "txn_from_bundle": [false], "txn_is_simple_vote": [true], "txn_landed": [true], "txn_load_end_timestamps_nanos": ["1754409729594455631"], "txn_mb_end_timestamps_nanos": ["1754409729594625003"], "txn_mb_start_timestamps_nanos": ["1754409729594431327"], "txn_microblock_id": [0], "txn_preload_end_timestamps_nanos": ["1754409729594432846"], "txn_priority_fee": ["0"], "txn_signature": ["2BfWBnhTP1ZZwFZutwThj5VT1hX71X9otbgFr21W2XJfcppXakbPCvJ2eCh8eBcS74Lfjar5AuowuppAjsEceSuW"], "txn_start_timestamps_nanos": ["1754409729594451074"], "txn_transaction_fee": [0], "txn_tips": ["0"], "txn_source_ipv4": ["123.123.123.123"], "txn_source_tpu": ["gossip"], } } } **`SlotResponse`** | Field | Type | Description | | --- | --- | --- | | publish | `SlotPublish` | General information about the slot. Contains several nullable fields in case a future slot is queried and the information is not known yet | | waterfall | `TxnWaterfall\|null` | If the slot is not `mine`, will be `null`. Otherwise, a waterfall showing reasons transactions were acquired since the end of the prior leader slot | | tile\_primary\_metric | `TilePrimaryMetric\|null` | If the slot is not `mine`, will be `null`. Otherwise, max value of per-tile-type primary metrics since the end of the prior leader slot | | tile\_timers | `TsTileTimers[]\|null` | If the slot is not `mine`, will be `null`. Otherwise, an array of `TsTileTimers` samples from the slot, sorted earliest to latest. We store this information for the most recently completed 4096 leader slots. This will be `null` for leader slots before that | | scheduler\_counts | `SchedulerCounts[]\|null` | If the slot is not `mine`, will be `null`. Otherwise, an array of `SchedulerCounts` samples from the slot, sorted earliest to latest. We store this information for the most recently completed 4096 leader slots. This will be `null` for leader slots before that | **`SlotTransactionsResponse`** | Field | Type | Description | | --- | --- | --- | | publish | `SlotPublish` | General information about the slot. Contains several nullable fields in case a future slot is queried and the information is not known yet | | transactions | `Transactions\|null` | If the slot is not `mine`, will be `null`. Otherwise, metrics for the transactions in this slot. Arrays have a separate entry for each scheduled transaction that was packed in this slot, and are ordered in the same order the transactions appear in the block. Note that not all scheduled transactions will land in the produced block (e.g. failed bundles are ignored), but these arrays nonetheless include metrics for excluded transactions | | limits | `SlotLimits` | The various protocol-derived resource limits and their corresponding utilization for this block. If `mine` is false, then this value is `null` | | scheduler\_stats | `SlotScheduleStats` | Various metrics tracked by the transaction scheduler and collected at the end of a leader slot. If `mine` is false, then this value is `null` | **`TxnWaterfall`** | Field | Type | Description | | --- | --- | --- | | in | `TxnWaterfallIn` | Transactions received into the waterfall | | out | `TxnWaterfallOut` | Transactions sent out of the waterfall | **`TxnWaterfallIn`** | Field | Type | Description | | --- | --- | --- | | pack\_cranked | `number` | Transactions were created as part of an initializer bundle. Initializer bundles are special bundles created by pack that manage block engine state on the chain. They contain crank transactions, which create and update tip distribution accounts. There is typically one crank transaction per leader rotation | | pack\_retained | `number` | Transactions were received during or prior to an earlier leader slot, but weren't executed because they weren't a high enough priority, and were retained inside the validator to potentially be included in a later slot | | resolv\_retained | `number` | Transactions were received during or prior to an earlier leader slot, but weren't executed because we did not know the blockhash they referenced. They were instead kept in a holding area in case we learn the blockhash later | | quic | `number` | A QUIC transaction was received. The stream does not have to successfully complete | | udp | `number` | A non-QUIC UDP transaction was received | | gossip | `number` | A gossipped vote transaction was received from a gossip peer | | block\_engine | `number` | A transaction received from a block engine, for example Jito. The transaction might or might not have been part of a bundle | **`TxnWaterfallOut`** | Field | Type | Description | | --- | --- | --- | | net\_overrun | `number` | Transactions were dropped because the net tile couldn't keep up with incoming network packets. It is unclear how many transactions would have been produced by the packets that were dropped, and this counter (along with the corresponding counter for the `in` side) assumes one transaction per dropped packet | | quic\_overrun | `number` | Transactions were dropped because the QUIC tile couldn't keep up with incoming network packets. It is unclear how many transactions would have been produced by the fragments from net that were overrun, and this counter (along with the corresponding counter for the `in` side) assumes one transaction per dropped packet | | quic\_frag\_drop | `number` | Transactions were dropped because there are more ongoing receive operations than buffer space | | quic\_abandoned | `number` | Transactions were dropped because a connection closed before all bytes were received | | tpu\_quic\_invalid | `number` | Transactions were dropped because the QUIC tile decided that incoming QUIC packets were not valid. It is unclear how many transactions would have been produced by the packets that were invalid, and this counter (along with the corresponding counter for the `in` side) assumes one transaction per invalid packet | | tpu\_udp\_invalid | `number` | Transactions were dropped because the QUIC tile decided that incoming non-QUIC (regular UDP) packets were not valid | | verify\_overrun | `number` | Transactions were dropped because the verify tiles could not verify them quickly enough | | verify\_parse | `number` | Transactions were dropped because they were malformed and failed to parse | | verify\_failed | `number` | Transactions were dropped because signature verification failed | | verify\_duplicate | `number` | Transactions were dropped because the verify tiles determined that they had already been processed | | dedup\_duplicate | `number` | Transactions were dropped because the dedup tile determined that they had already been processed | | resolv\_retained | `number` | Transactions were retained inside the validator memory because they referenced a blockhash we do not yet know. We might include the transactions in a future block, if we learn about the blockhash they reference | | resolv\_lut\_failed | `number` | Transactions were dropped because they contained invalid address lookup tables (LUTs) | | resolv\_expired | `number` | Transactions were dropped because they contained a transaction that was already expired | | resolv\_no\_ledger | `number` | Transactions were dropped because they contained a LUT but we didn't yet have a ledger to look them up in | | resolv\_ancient | `number` | Transactions were dropped because they referenced a blockhash we didn't recognize, and while waiting to see if the blockhash would arrive, the buffer became full | | pack\_invalid | `number` | Transactions were dropped because pack determined they would never execute. Reasons can include the transaction requested too many compute units, or was too large to fit in a block | | pack\_expired | `number` | Transactions were dropped because pack determined that their TTL expired | | pack\_already\_executed | `number` | Transactions dropped from pack because they were already executed (in either the replay or leader pipeline) | | pack\_retained | `number` | Transactions were retained inside the validator memory because they were not high enough priority to make it into a prior block we produced, but have not yet expired. We might include the transactions in a future block | | pack\_leader\_slow | `number` | Transactions were dropped while leader because the bank tiles could not execute them quickly enough, pack will drop the lowest priority transactions first | | pack\_wait\_full | `number` | Transactions were dropped while we were waiting for our leader slot because we ran out of memory to store them. All incoming transactions are dropped without regard for the priority | | bank\_invalid | `number` | Transactions were dropped because a bank tile could not execute them enough to charge fees. Failed transactions can still pay fees and be included in a block, but invalid transactions do not make it to a block. Reasons can include insufficient fee payer balance, or invalid address lookup tables | | block\_success | `number` | Transactions made it into a block, and execution succeeded | | block\_fail | `number` | Transactions made it into a block, but execution failed | **`SchedulerCounts`** | Field | Type | Description | | --- | --- | --- | | timestamp\_nanos | `string` | A UNIX nanosecond timestamp representing the time when these counts were sampled by the gui tile. | | regular | `number` | The number of transactions stored in the "regular" treap (i.e. the primary buffer) at `timestamp_nanos` | | votes | `number` | The number of transactions stored in the "votes" treap (i.e. the buffer dedicated for vote transactions) at `timestamp_nanos` | | conflicting | `number` | The number of transactions stored in the "conflicting" treap (i.e. the buffer for transactions with perceived account write conflicts, which receive slightly less priority) at `timestamp_nanos` | | bundles | `number` | The number of transactions stored in the "bundles" treap (i.e. the buffer dedicated for bundle transactions) at `timestamp_nanos` | **`TsTileTimers`** | Field | Type | Description | | --- | --- | --- | | timestamp\_nanos | `string` | A timestamp of when the tile timers were sampled, nanoseconds since the UNIX epoch | | tile\_timers | `TileTimer[]` | A list of all tile timing information at the given sample timestamp | **`WriteAcctCost`** | Field | Type | Description | | --- | --- | --- | | account | `string` | The pubkey for this writeable account | | cost | `number` | The total compute units for all transactions that list `account` as a writeable account | **`SlotLimits`** | Field | Type | Description | | --- | --- | --- | | used\_total\_block\_cost | `number` | The total block cost in compute units | | used\_total\_vote\_cost | `number` | The compute units from vote transactions consumed for this block | | used\_account\_write\_costs | `WriteAcctCost[]` | The top 5 writeable account costs for this block | | used\_total\_bytes | `number` | The number of bytes from transaction payloads and microblock headers consumed in total for this block | | used\_total\_microblocks | `number` | The total number of microblocks included in this block | | max\_total\_block\_cost | `number` | The maximum possible value for `used_total_block_cost` | | max\_total\_vote\_cost | `number` | The maximum possible value for `used_total_vote_cost` | | max\_account\_write\_cost | `number` | The maximum possible value for `used_account_write_cost` | | max\_total\_bytes | `number` | The maximum possible value for `used_total_bytes` | | max\_total\_microblocks | `number` | The maximum possible value for `used_total_microblocks` | **`SlotScheduleStats`** | Field | Type | Description | | --- | --- | --- | | block\_hash | `string` | The final POH hash in the block as a base58 encoded string | | end\_slot\_reason | `string` | The reason pack ended packing for this leader slot. One of `"timeout"`, `"microblock_limit"`, or `"leader_switch"` | | slot\_schedule\_counts | `number[]` | `slot_schedule_counts[i]` is the number of transactions across the leader slot that had `["success", "fail_cu_limit", "fail_fast_path", "fail_byte_limit", "fail_alloc_limit", "fail_write_cost", "fail_slow_path", "fail_defer_skip"][i]` as the outcome after being scheduled. "success" means the transaction was successfully scheduled to a bank. "fail\_cu\_limit" means Pack skipped the transaction because it would have exceeded the block CU limit. "fail\_fast\_path" means Pack skipped the transaction because of account conflicts using the fast bitvector check. "fail\_byte\_limit" means Pack skipped the transaction because it would have exceeded the block data size limit. "fail\_alloc\_limit" means Pack skipped the transaction because it would have exceeded the block account allocation limit. "fail\_write\_cost" means Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit. "fail\_slow\_path" means Pack skipped the transaction because of account conflicts using the full slow check. "fail\_defer\_skip" means Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times | | end\_slot\_schedule\_counts | `number[]` | `end_slot_schedule_counts` has the same meaning as `slot_schedule_counts` except only transactions that occur after the last successfully scheduled transaction in the slot are counted | | pending\_smallest\_cost | `number` | The cost in compute units of the smallest eligible non-vote transaction in pack's transaction buffer at the end of the slot. If the buffer is empty, then this is `null` | | pending\_smallest\_bytes | `number` | The size in bytes of the smallest eligible non-vote transaction in pack's transaction buffer at the end of the slot. If the buffer is empty, then this is `null` | | pending\_vote\_smallest\_cost | `number` | The cost in compute units of the smallest eligible vote transaction in pack's transaction buffer at the end of the slot. If the buffer is empty, then this is `null` | | pending\_vote\_smallest\_bytes | `number` | The size in bytes of the smallest eligible vote transaction in pack's transaction buffer at the end of the slot. If the buffer is empty, then this is `null` | **`Transactions`** | Field | Type | Description | | --- | --- | --- | | start\_timestamp\_nanos | `string` | A UNIX timestamp, in nanoseconds, representing the time that the validator is first aware that it is leader. At this point the poh tile will signal the pack tile to begin filling the block for this slot with transactions | | target\_end\_timestamp\_nanos | `string` | A UNIX timestamp, in nanoseconds, representing the target time in nanoseconds that the pack tile should stop scheduling transactions for the slot. Transactions might still finish executing after this end time, if they started executing before it and ran over the deadline. In rare cases, transactions may also appear to begin after this timestamp due to slight clock drift between execution cores | | txn\_arrival\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_arrival_timestamps_nanos[i]` is the time when the `i`\-th transaction in the slot arrived at the transaction scheduler (i.e. pack) | | txn\_mb\_start\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_mb_start_timestamps_nanos[i]` is the time when the microblock for the `i`\-th transaction in the slot was successfully scheduled for execution by pack. At this point, the microblock was sent off to a bank tile for execution. Since a microblock may contain multiple transactions (e.g. a bundle), all transactions from the same microblock will share the same start timestamp | | txn\_preload\_end\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_preload_end_timestamps_nanos[i]` is the time when the `i`\-th transaction in the slot was successfully dispatched into an execution environment and is about to start validation checks, which include a final deduplication check as well as an expiration check | | txn\_start\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_start_timestamps_nanos[i]` is the time when the `i`\-th transaction in the slot started loading | | txn\_load\_end\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_load_end_timestamps_nanos[i]` is the time when the `i`\-th transaction in the slot finished loading and started executing. At this point, relevant on-chain data has been loaded for the transaction and it is ready to be fed into the Solana Virtual Machine (SVM) | | txn\_end\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_end_timestamps_nanos[i]` is the time when the `i`\-th transaction in the slot finished executing | | txn\_mb\_end\_timestamps\_nanos | `string[]` | An array of UNIX timestamps, in nanoseconds. `txn_mb_end_timestamps_nanos[i]` is the time when the microblock for the `i`\-th transaction in the slot completed executing. At this point, the bank tile for this microblock was ready to communicate the execution result back to the pack. pack uses this result to track the progress of the growing block and also repurposes any unused compute units for other microblocks. The current implementation splits microblocks which originally contained multiple transactions (i.e. bundles) apart so that consumers always receive one transaction per microblock, so unlike `txn_mb_start_timestamps_nanos` this timestamp may be unique for a given transaction | | txn\_compute\_units\_requested | `number[]` | `txn_compute_units_requested[i]` is a strict upper bound on the total cost for the `i`\-th transaction in the slot. The transaction cannot have succeeded if its incurred cost (known after execution) exceeds this bound. This bound is used by the pack tile to estimate the pace at which the block is being filled, and to filter out transactions that it knows will fail ahead of time | | txn\_compute\_units\_consumed | `number[]` | `txn_compute_units_consumed[i]` is the actual post-execution cost of the `i`\-th transaction in the slot. While some transactions costs are known from the transaction payload itself (such as the cost incurred by the amount of instruction data), other costs (like execution costs or the cost due to loaded on-chain account data) are a function of the state of the blockchain at the time of execution. This value represents the actual cost after a transaction is executed. Consensus requires that all validators agree on this value for a given transaction in a slot. There are two special cases to consider for scheduled transactions that were not added to the produced block. Failed bundle transactions that successfully executed up to the point of failure will show actual consumed CUs. Subsequent failed bundle transactions will show 0 cus consumed. Non-bundle transactions that were not added to the block will also show 0 cus consumed | | txn\_transaction\_fee | `string[]` | `txn_transaction_fee[i]` is the signature fee for the `i`\-th transaction in the slot. Currently, this is the number of signatures in the transaction times 5000 lamports. This fee used to (and may in the future) include rewards from other parts of the transaction, which is why a more general name is used. 50% of this fee is burned and the other 50% is included in validator block rewards. The provided values reflect the fee balance after burning | | txn\_priority\_fee | `string[]` | `txn_priority_fee[i]` is the priority fee in lamports for the `i`\-th transaction in the slot. The priority fee is a static metric computed by multiplying the requested execution cost (derived from a provided computeBudget instruction, or from a protocol defined default) by the compute unit price (derived from a separate computeBudget instruction) | | txn\_tips | `string[]` | `txn_tips[i]` is the total tip in lamports for the `i`\-th transaction in the slot. The tip is the increase (due to this transaction) in the total balance of all tip payment accounts across all block builders after any commission to the block builder is subtracted. This implies that both the validator and staker portions of the tip are included in this value. Non-bundle transactions may have a non-zero tip. Tips for transactions in failed bundles are included up to the point of failure | | txn\_error\_code | `number[]` | `txn_error_code[i]` is the error code that explains the failure for the `i`\-th transaction in the slot. See below for more details | | txn\_from\_bundle | `boolean[]` | `txn_from_bundle[i]` is `true` if the `i`\-th transaction in the slot came from a bundle and `false` otherwise. A bundle is a microblock with 1-5 transactions that atomically fail or succeed. It is sent to the validator from a compatible block engine (e.g. jito) that can additionally collect MEV rewards that are distributed to stakers (i.e. tips) | | txn\_is\_simple\_vote | `boolean[]` | `txn_is_simple_vote[i]` is `true` if the `i`\-th transaction in the slot is a simple vote and `false` otherwise | | txn\_landed | `boolean[]` | `txn_landed[i]` is `true` if the `i`\-th transaction in the slot was included in the produced block. A scheduled transaction may not be included in the block for any number of reasons (e.g. a failed bundle, a duplicate transaction, invalid fee-payer) | | txn\_bank\_idx | `number[]` | `txn_bank_idx[i]` is the index of the bank tile that executed the `i`\-th transaction in the slot | | txn\_microblock\_id | `string[]` | `txn_microblock_id[i]` is the index of the microblock for the `i`\-th transaction in the slot. Microblocks are collections of 1+ transactions. All of the transactions from a bundle share the same microblock. Microblock ids are monotonically increasing in the order they appear in the block and start at 0 for each slot | | txn\_signature | `string[]` | `txn_signature[i]` is the base58 signature of the `i`\-th transaction in the slot | | txn\_source\_ipv4 | `string[]` | `txn_source_ipv4[i]` is the source ipv4 address for the `i`\-th transaction in the slot | | txn\_source\_tpu | `string[]` | `txn_source_tpu[i]` is the transaction processing unit (TPU) which handled the `i`\-th transaction in the slot | The source tpu for a transaction can be one of the following | TPU | Description | | --- | --- | | quic | the primary ingress tpu for user transactions. Utilizes the quic protocol to receive packets | | udp | ingress transactions received as simple UDP packets | | gossip | vote transactions received from the gossip network | | bundle | bundle transactions received by the bundle tile from a block builder. Utilizes a grpc connection to receive packets | | send | vote transactions produced by this validator received from the send tile. These transactions are meant for the active cluster leader | These are the possible error codes that might be included in `txn_error_code` and their meanings. | Code Name | Code | Description | | --- | --- | --- | | Success | 0 | The transaction successfully executed | | AccountInUse | 1 | Includes a writable account that was already in use at the time this transaction was executed | | AccountLoadedTwice | 2 | Lists at least one account pubkey more than once | | AccountNotFound | 3 | Lists at least one account pubkey that was not found in the accounts database | | ProgramAccountNotFound | 4 | Could not find or parse a listed program account | | InsufficientFundsForFee | 5 | Lists a fee payer that does not have enough SOL to fund this transaction | | InvalidAccountForFee | 6 | Lists a fee payer that may not be used to pay transaction fees | | AlreadyProcessed | 7 | This transaction has been processed before (e.g. the transaction was sent twice) | | BlockhashNotFound | 8 | Provides a block hash of a `recent` block in the chain, `b`, that this validator has not seen yet, or that is so old it has been discarded | | InstructionError | 9 | Includes an instruction that failed to process | | CallChainTooDeep | 10 | Includes a cross program invocation (CPI) chain that exceeds the maximum depth allowed | | MissingSignatureForFee | 11 | Requires a fee but has no signature present | | InvalidAccountIndex | 12 | Contains an invalid account reference in one of its instructions | | SignatureFailure | 13 | Includes a signature that did not pass verification | | InvalidProgramForExecution | 14 | Includes a program that may not be used for executing transactions | | SanitizeFailure | 15 | Failed to parse a portion of the transaction payload | | ClusterMaintenance | 16 | Cluster is undergoing an active maintenance window | | AccountBorrowOutstanding | 17 | Transaction processing left an account with an outstanding borrowed reference | | WouldExceedMaxBlockCostLimit | 18 | Exceeded the maximum compute unit cost allowed for this slot | | UnsupportedVersion | 19 | Includes a transaction version that is not supported by this validator | | InvalidWritableAccount | 20 | Includes an account marked as writable that is not in fact writable | | WouldExceedMaxAccountCostLimit | 21 | Exceeded the maximum per-account compute unit cost allowed for this slot | | WouldExceedAccountDataBlockLimit | 22 | Retrieved accounts data size exceeds the limit imposed for this slot | | TooManyAccountLocks | 23 | Locked too many accounts | | AddressLookupTableNotFound | 24 | Loads an address table account that doesn't exist | | InvalidAddressLookupTableOwner | 25 | Loads an address table account with an invalid owner | | InvalidAddressLookupTableData | 26 | Loads an address table account with invalid data | | InvalidAddressLookupTableIndex | 27 | Address table lookup uses an invalid index | | InvalidRentPayingAccount | 28 | Deprecated | | WouldExceedMaxVoteCostLimit | 29 | Exceeded the maximum vote compute unit cost allowed for this slot | | WouldExceedAccountDataTotalLimit | 30 | Deprecated | | DuplicateInstruction | 31 | Contains duplicate instructions | | InsufficientFundsForRent | 32 | Deprecated | | MaxLoadedAccountsDataSizeExceeded | 33 | Retrieved accounts data size exceeds the limit imposed for this transaction | | InvalidLoadedAccountsDataSizeLimit | 34 | Requested an invalid data size (i.e. 0) | | ResanitizationNeeded | 35 | Sanitized transaction differed before/after feature activation. Needs to be resanitized | | ProgramExecutionTemporarilyRestricted | 36 | Execution of a program referenced by this transaction is restricted | | UnbalancedTransaction | 37 | The total accounts balance before the transaction does not equal the total balance after | | ProgramCacheHitMaxLimit | 38 | The program cache allocated for transaction batch for this transaction hit its load limit | | CommitCancelled | 39 | This transaction was aborted during the commit stage | | BundlePeer | 40 | This transaction was part of a bundle that failed | | BlockhashNonceAlreadyAdvanced | 50 | This transaction references a nonce account that is already advanced | | BlockhashNonceAdvanceFailed | 51 | This transaction is a nonce transaction but the advance instruction was not valid or failed | | BlockhashNonceWrong | 52 | This transaction is a nonce transaction but the blockhash is not the correct one | ---