Hardening Firedancer

Introduction

Starting today and for the next month, we are doubling all bounty rewards and tripling them for findings related to Alpenglow, Solana’s upcoming network upgrade.

Firedancer is a fully independent validator client for Solana, built for security and performance. The validator is a high-value target. If an attacker compromises a supermajority of the network’s validators, all user funds are at risk. To defend against such threats, we pair a layered security architecture with continuous fuzzing, regular internal and external audits, and public bounties.

Following three years of development and testing, we ran a $1,000,000 public audit competition in May. Since then, Firedancer has launched on mainnet and now secures 15% of the network’s stake (at the time of writing). We used our experience from the competition to build our own bug bounty platform to handle community submissions at scale.

Security Model

The Firedancer validator is structured as a set of sandboxed processes, called tiles. Each isolated process is in its own address space, typically pinned to a dedicated CPU core. All memory used by the validator is pre-allocated in a collection of shared-memory regions. Specific allocations are placed in different regions at boot to maintain the principle of least privilege. Tiles can access only memory that they need. Dynamic memory allocation is not allowed. IPC between tiles happens mostly through shared-memory queues, sitting inside this pre-allocated memory. These queues, and other objects like tile-local memory and shared data structures, are defined statically in the code that specifies the system’s topology. This is both a performance and a security decision: with no heap, dynamic allocation bugs are less likely, all memory use is intentional, and the validator can never crash due to running out of memory.

After each tile boots up and maps in its memory, it enters a sandbox, applying a series of restrictions including:

• Seccomp BPF Filtering: Each tile must pre-specify the list of syscalls that it’s allowed to make, shrinking the kernel attack surface by limiting the interface to kernel space.
• Restricted Filesystem Access: An initial allow-list of file descriptors is verified, and then any additional access to the filesystem is restricted via Landlock. This prevents a compromised tile from accessing files like the validator’s keypair.
• rlimit Resource Limits: Per-process limits are explicitly set on core dumps, stack growth, file descriptor counts, locked memory, etc. This prevents tiles from consuming kernel resources beyond what is allowed.
• Root Pivot: Unmounts the existing filesystem and changes the visible root to an empty directory, preventing access to the rest of the host filesystem.
• Namespace Isolation: Each tile is unshared into a fresh namespace, and then is blocked from creating new ones, which isolates it from the host and prevents namespace-based privilege escalation.
• Drop Capabilities: Tiles are prevented from regaining privileged kernel permissions, blocking privileged operations like loading kernel modules.

The motivation for these features is to limit the blast radius of a compromised tile. For example, let’s assume an attacker has remote code execution in the gossip tile. The Solana gossip protocol is responsible for discovering peers and sharing cluster metadata with other nodes. Something the attacker would want is unrestricted access to the validator’s private key. Due to the seccomp filter and process isolation in the gossip tile, the attacker cannot write, let alone read, the memory or disk where the private key is stored. Landlock and rlimit provide further defense in depth, restricting which files the tile can access and capping the file descriptors it can keep open. The validator’s private key is only mapped into the sign tile and nowhere else. If the gossip tile needs information signed, it must send a well-formed signing request to the sign tile via the gossip_sign link and the signed result is published back on the sign_gossip link. Because the sign tile scopes requests per link, the gossip tile can only obtain signatures on well-formed gossip messages, not on votes or transactions. The attacker is therefore limited to sending signed but malicious gossip, which would cause the node to be pruned by its peers and become unreachable over gossip. What could have been private-key access becomes a loss of gossip connectivity, while consensus and replay continue to make forward progress.

Running The Contest

A bug bounty contest is a time-boxed event where security researchers hunt for vulnerabilities in a codebase. The rewards for each finding are scaled to its severity and paid from a fixed pool. We ran a similar contest for Frankendancer in 2024 and ran an open bug bounty program for Frankendancer for two years. Now, we run our own program for Firedancer.

Our release plan was to soak-test Firedancer in production settings across testnet, devnet, and mainnet-beta while we continued development and ran audits with security firms. We ran the Frankendancer contest right before our public mainnet release. For Firedancer, we wanted to start this contest earlier in the development process to get community feedback sooner. We like the bug bounty format because it gives us access to more fresh eyes than traditional audits would, and it aligns incentives: we pay rewards for bugs found, not time spent.

For the contest, we classified findings into five buckets:

• Insight: performance improvements or non-exploitable software bugs, such as an out-of-bounds read with no reachable impact.
• Low: narrowly triggerable vulnerabilities, such as a hard-to-trigger data race causing a validator crash or a crash during boot.
• Medium: vulnerabilities that could cause an individual validator to crash or produce an invalid block.
• High: vulnerabilities that could cause an execution/consensus divergence or a cluster-wide loss of availability for Firedancer nodes.
• Critical: vulnerabilities with direct, deterministic economic impact, such as an infinite mint or total loss of funds.

We ran the contest for one month from April 9 to May 9 through Immunefi. The prize pool unlocked in tranches based on the highest-severity bug found; the minimum was \$100,000, at least one medium would unlock \$250,000, one high would bring the pool to \$500,000, and a critical bug would unlock the full \$1,000,000. Duplicate submissions were rewarded, with the primary finder receiving a larger share. 10% of the pool was reserved for the top 3 submitters. To incentivize quality submissions over a spray-and-pray approach, we also introduced a $100 fee per submission alongside a platform-enforced rate limit of one submission per researcher per day. Any KYC’d researcher was allowed to participate, and any mainnet-triggerable vulnerability was in scope. Researchers were expected to submit a writeup of their findings along with a reproducible PoC.

Results

In 30 days, the contest received 809 total reports from 162 researchers, resulting in 127 unique, non-insight security submissions. Nearly half of valid submissions were duplicates. That’s roughly 20x more volume than we received for the Frankendancer contest, which ran for 2 weeks longer than the Firedancer contest.

This is in line with a broader, AI-enabled trend that has led to a tremendous spike in submissions across similar public bounty programs in the industry. An automated researcher found 21 zero-days in FFmpeg, some of which were introduced nearly two decades ago. Mozilla announced that it made 423 security fixes in April, around 20x its prior monthly average. Google Chrome patched more bugs in June and July than it did in the past two years combined. More recently, Debian fixed 1,313 CVEs in its latest release. Due to increased and unmanageable submission volumes, notable bug bounty programs have lowered payouts for bugs, completely removed payouts, or even halted their programs.

Participation followed a familiar pattern seen in large contests. The top 20 researchers by submissions accounted for more than half of all reports, with the rest forming a long tail of a handful of submissions each. Top researchers dominated even more if you just consider confirmed findings. The mean number of submissions was ~5 per researcher, but the median dropped to 2, indicating a top-heavy skew.

As expected, the top submitters by volume had a better confirmation rate than the field, implying that the most productive researchers have a meaningful edge over raw token spend. This performance varied depending on what a “top” submitter is. The gap with the rest of the field widened when ranking submitters by the number of unique, accepted bugs instead of total confirmed findings.

Reports have also gotten much longer over time. Excluding formatting and code snippets, the average report was 1,064 words; counting everything, 2,610 words. By contrast, reports submitted to our bounty programs before 2025 averaged just 406 words. This sharp increase in length indicates a shift toward AI-generated submissions.

Findings Deep Dive

We want to highlight some examples of interesting reports to show a variety of bug classes and areas of code that submissions covered:

Low: permanent leak in the hard fork detector. After a validator processes a block, it sends a vote containing a bank hash to the rest of the network. The bank hash is computed from the result of executing the block. The hard fork detector uses votes (replayed and gossiped) to detect whether a bank hash mismatch occurred or Firedancer marked a valid block as invalid (or vice versa). A bank hash mismatch means that the node has diverged from the majority of the cluster.

Replayed blocks are allocated a blk_t object, which stores metadata about a block including its state and bank hash. These blk_t objects are allocated out of a pool unconditionally for any block that is replayed or marked dead. Every time a vote is recorded, a vote entry object, vte_t, is allocated from a separate pool, added to an LRU deque, and used to increment the corresponding blk_t reference count. When the deque is full, the oldest vte_t is freed and the reference count for its blk_t is decremented. If the reference count drops to zero while the vte_t is being freed, then the corresponding blk_t is also freed. However, if there are no vte_t objects recorded for a block, the blk_t is never freed, leading to a permanent leak. Once every blk_t has leaked, the validator crashes.

We classified this as low severity because it would not cause a network-wide loss of availability. All nodes leak blk_t objects at the same rate, but validators don’t start simultaneously, so they would crash at different times. In practice, exhausting the pool would either take months of continuous runtime to happen naturally or require an attacker with substantial stake to coordinate the production of hundreds of thousands of invalid zero-vote blocks.

Medium: stale sibling pointer in fd_forest_t. The repair tile keeps incoming shreds in a left-child, right-sibling tree (fd_forest_t) where each tree element corresponds to a block. Because the tree is built up as shreds arrive (potentially out of order), it’s sometimes necessary to update the parent of a given block if the parent arrives after the child block’s shreds. When the correct parent arrives, verified_parent_update() is called. The child node should be disconnected from the tree and any references to the old parent should be removed. However, the repair tile doesn’t clear the sibling pointers from that node’s children. This is a problem only if the re-parented slot has multiple children. In the same function, these child nodes get re-added to the tree via a call to the helper link_sibling(). Then the stale sibling pointers can be partially overwritten, leading to a cycle. Any later traversal of these nodes in the forest tree causes an infinite loop, stalling the repair tile.

We classified this bug as medium severity because it could stall an individual repair tile but could not be reliably triggered across the network. Remote triggering depends on the order in which each validator receives network traffic and reattaches child nodes, so an attacker could not deterministically cause a cluster-wide loss of availability.

High: transaction scheduling invariant violation. Firedancer’s replay dispatcher (fd_rdisp_t) schedules transactions from a block to different executor tiles. Transactions reference accounts to read or modify on-chain data and each account can be referenced as read-only or read-write. Blocks contain a serial list of transactions, but these transactions can be executed in parallel if there is no conflicting reference to an account. Multiple transactions that reference an account read-only can be executed at the same time, but if one transaction references the account read-write, then the transaction becomes a serializing point in the block.

Each transaction is assigned a “score” which is used for scheduling priority. Transactions are dispatched based on a min-queue: lower-score transactions are executed first. The score is computed as is_txn_behind_serialized + priority_heuristic. The heuristic involves an exponential moving average (EMA) of how often an account is referenced. The range of the EMA is [0,1). A higher EMA means that the account gets used frequently and that transactions referencing the account should be dispatched quickly to unblock future transactions (and vice versa for a lower EMA). The priority_heuristic starts at 1.0 and is calculated for each writable account by:

           priority_heuristic *= (1 - EMA(account)) 

The EMA gets updated lazily when an account is re-referenced as writable by this formula: 

            computed_delta = (global_insert_cnt - last_ref) & 0xFFFFFF

           new_ema = 0.005 + 0.995 ^ (computed_delta) * old_ema

However, the last_ref is represented by 24 bits and the computed delta is masked. So if global_insert_cnt - last_ref is a multiple of 2^24, it gets masked to 0. This leads to an EMA that can be >1 if the old_ema is high. Now it’s possible to have a negative priority heuristic. As a result, the score when the transaction is behind a serialized one can be <1 since the priority heuristic is negative. If the serializing transaction has a relatively high score (from low EMAs), then the transaction that should’ve been executed after it is dispatched first. This results in a different execution order for blocks and consequently a consensus mismatch, since different validators vote on different results from replaying a block.

We classified this bug as high severity because it would cause Firedancer validators to deterministically execute a block out of order, leading to a consensus divergence with other clients. This would count as a loss of availability and it could have been feasibly triggered by a remote attacker. The report did not qualify for a critical because even though the bug made it possible to execute transactions out of order, the attack couldn’t be precise enough to cause predictable, serious harm.

Insights

Going into the contest, we expected an uptick in submission volume and a roughly even mix of human and AI-assisted reports. Instead, we saw a major increase in volume, and we believe nearly every submission was either discovered or written by an agent. AI-generated reports are all too recognizable by their walls of text and redundant background information. We also saw far more duplicate submissions than we expected, likely because agents using the same models converged on the same findings. While we can’t know which issues agents failed to find, it’s clear to our team that there are classes of bugs that AI is excellent at identifying and classes where it consistently generates false positives.

Agents are great at catching subtle divergences between components in Firedancer and Agave that map closely to each other. For example, if Firedancer were missing a validation check that Agave performs, an agent would quickly flag the discrepancy without much coaxing from a prompter; a human auditor could easily miss it in a tedious side-by-side comparison. Similarly, AI excels at identifying places where code deviates from explicitly documented invariants. And once a bug is discovered and the agent has it in its context window, the agent can easily identify variants and other bugs in the same class. The common thread is that agents do their best work when there is a clear reference to compare against, whether it is another implementation, a specification, or a bug report.

On the other hand, the vast majority of false positives were reports that failed to understand implicit invariants about the system and how different components interact. A large portion of the Solana protocol is based on implicit business logic that provides guarantees about ordering and upstream invariants. This logic is often obvious to engineers and researchers who work closely with the code, but agents can miss this context when evaluating components in isolation. As an example, several reports were based on the assumption that a running Firedancer validator could have multiple epoch boundaries’ worth of unrooted state (network state that a supermajority hasn’t agreed to). In the worst case, Firedancer bounds the max amount of unrooted state it retains to 2,048 slots (~10 minutes), and at the protocol level, an epoch boundary happens every 432,000 slots (~1 per day). To a Solana developer, it’s pretty clear that this means there can be at most one epoch boundary in an unrooted window, but it is not to agents and researchers with limited Solana-specific knowledge.

Learnings

We ran one of the first contests at the start of the AI-saturated security research era, and the results surprised us. The biggest bottleneck was managing the volume of reports. Our main goal going forward is to reduce the amount of time we spend triaging. The value of a contest like this comes from bounty hunters surfacing bugs more cost-effectively than we could internally, and that value holds only if triage is also efficient. Two of the main levers we can pull here are increasing the signal-to-noise ratio of submissions and reducing the time spent on each one. The lessons below are some of the changes we’ve made and the practices we’d recommend to anyone running bounties and contests today:

• Charge submission fees. Going into the contest, we were worried that the $100 submission fee would discourage serious security researchers from participating. The 809 submissions show that the fee was obviously not a problem. With no fee or a lower one, there would have been even more reports, and they would almost certainly have been of lower quality. The fee is a disincentive to spray-and-pray researchers who hope that they can earn back their brute-force AI token spend multiple times over through rewards while shifting the validation effort onto the triagers. With this in mind, our new platform enforces a 1 SOL burn per submission.

• Document invariants aggressively. We noticed that agents perform best when they have reference information they can quickly absorb. Implicit invariants lead to submissions that look valid but don’t hold up under further scrutiny. Our team has transitioned from documenting only for other engineers to also giving key context to automated auditors. One rule of thumb is to treat report findings as continuous feedback. If reports keep gravitating toward the same area of code, then it makes sense to document that area more thoroughly. Another rule is to state explicitly how data flows through the system as a whole and what invariants can be expected from upstream components.

• Enforce a submission length limit. Our triagers spent too long trying to understand the key points of thousand-word, AI-generated reports with buried ledes. These reports could almost certainly have been summarized in a few bullet points. As mentioned above, submissions have more than doubled in length since 2025. Length limits push researchers and their agents to be more concise. Shorter reports mean shorter triage times, which in turn leads to less wasted developer time. We now enforce a 3,000-character limit per section on the Firedancer bug bounty platform.

• Stay ahead of the queue. We saw far fewer reports in the previous contest, so there was never a need to constantly monitor incoming submissions. We learned the hard way that this passive approach doesn't scale. With roughly 26 daily submissions, falling behind even for a day created a huge backlog (peaking at 299 open reports). We noticed a pattern where, once a submission came in, a wave of similar reports would follow over the next few days as different agents converged on the same bug. This created a vicious cycle: as untriaged bugs piled up, incoming volume increased because the chief findings hadn't been fixed or publicly disclosed yet, so other agents kept rediscovering and resubmitting them. In fact, around 80% of all duplicates were submitted more than 3 days after the primary report. We were slow to triage in the first half of the contest, which caused duplicates to peak midway through; they declined once our triage efforts caught up. Had we kept our time to triage low from the start, we would have likely seen a much lower duplicate rate throughout the contest.

• Automate first-pass filtering. Our team spent hours reading and pasting each report by hand into a separate agent window for initial triage. We never saw an LLM discard a valid finding. A big, easy win would be to run an agent on every submission to generate a summary and filter out obviously invalid ones.

Conclusion

LLMs have collapsed the cost of finding bugs, but not the cost of judging them. In the past few months, the open-source software community has seen order-of-magnitude increases in submissions to public bug bounty programs. We believe the solution isn't to simply suppress or ignore this volume, but to handle it at scale. To that end, we have built in-house AI security tools, both to aid our triage efforts and to act as agentic auditors. We invite researchers to join us in the next phase of hardening Firedancer, with doubled and tripled rewards at bounty.firedancer.io.