Huckleberry: IBC Event Hallucinations

This blog post describes a vulnerability in ibc-go, the reference implementation of the Interblockchain Communication Protocol (IBC) used by most Cosmos blockchains. The issue can lead to the incorrect emission of Cosmos events triggered by a rolled-back IBC transfer. We privately disclosed the vulnerability to the Cosmos team, the fix with the codename Huckleberry got released in May and there is no indication that any malicious exploitation took place.

As part of our efforts as builders, researchers, and collaborators in the crypto space, one of our goals at Jump Crypto is to improve security assurance across the entire ecosystem. Our specialized security team has ongoing research efforts dedicated to discovering and patching vulnerabilities across projects via coordinated disclosure. This announcement once again brings these efforts to light.

Cosmos Events

Similar to Ethereum log events, Cosmos events offer an easy way for offchain applications to monitor the execution of a Cosmos chain. Although the blockchain state is the authoritative source of information, events enable efficient monitoring.

For instance, consider a token transfer on a Cosmos chain: A successful transaction that contains a MsgSend message will initiate a transfer of Coins from the sender's account to the recipient's. The primary outcome of this transfer is the modification of the two account states. In addition, a bank transfer event will be emitted.

Events emitted during the processing of a transaction are included in TxResponse. Moreover, the Cosmos RPC API makes it possible to subscribe to specific events using a fully-featured query language.  Offchain applications can passively ingest onchain activity using these mechanisms. Centralized exchanges and bridges are the most interesting users from a security perspective, as they can use events to monitor interactions with their deposit wallets or bridging contracts.

It is important that state changes and event emissions are always in sync and that failed transactions should not emit any events. Otherwise, an offchain application can be tricked into processing an operation that was never persisted.

As it turns out, this invariant was broken in all Cosmos chains with IBC support, due to an incorrect error handling mechanism in ibc-go’s OnRecvPacket function.

IBC Packet Handling

IBC packets are the basic building block of IBC. When a relayer submits a packet originating from a connected chain to the target chain by sending a MsgRecvPacket message, its data is first authenticated and then delivered to the corresponding application module via the OnRecvPacket callback. Unlike most other Cosmos functions, OnRecvPacket does not handle issues by returning an optional Golang error.

Instead, it returns an IBC acknowledgment. A successful acknowledgment means that the IBC packet was processed correctly, while a failed acknowledgment means the opposite. However, regardless of the result, the overall transaction will still succeed. This makes it possible to persist the acknowledgment result on-chain and relay it back to the sender chain.

As it turns out, this invariant was broken in all Cosmos chains with IBC support, due to an incorrect error handling mechanism in ibc-go’s OnRecvPacket function.

//https://github.com/cosmos/ibc-go/blob/9c1a81d09025014e1e4bb6fe369f76f0bc463102/modules/core/keeper/msg_server.go#L410
func (k Keeper) RecvPacket(goCtx context.Context, msg *channeltypes.MsgRecvPacket) (*channeltypes.MsgRecvPacketResponse, error) {
  
  [...]
  // Perform application logic callback
	//
	// Cache context so that we may discard state changes from callback if the acknowledgement is unsuccessful.
	cacheCtx, writeFn = ctx.CacheContext()
	ack := cbs.OnRecvPacket(cacheCtx, msg.Packet, relayer)
	if ack == nil || ack.Success() {
		// write application state changes for asynchronous and successful acknowledgements
		writeFn()
	} else {
		// NOTE: The context returned by CacheContext() refers to a new EventManager, so it needs to explicitly set events to the original context.
		// Events should still be emitted from failed acks and asynchronous acks
		ctx.EventManager().EmitEvents(cacheCtx.EventManager().Events())
	}

In the happy path, a successful acknowledgment is returned, state changes are committed, and events are emitted. In the error path, no state changes are committed but all generated events are still emitted.

This means that events generated during an IBC packet receive callback can end up in the final transaction results, even though their corresponding state changes are not persisted on chain.

Exploiting Hallucinations

An attacker who wishes to exploit these incorrect event emissions must identify a target that trusts events for sensitive actions. Then, they need to find a way to trigger the relevant events during the processing of an IBC packet, which will return a negative acknowledgment.

Triggering a negative acknowledgment after emitting security sensitive events is easy on chains that support arbitrary Cosmwasm contracts: An attacker can create a malicious smart contract that implements the IBC interface. Whenever the contract receives an IBC packet, it creates sub messages to perform sensitive actions such as triggering a bridge transfer. Regardless of this, it always return an Error acknowledgment. Exploitation without a WASM runtime is more difficult, but still possible if a chain uses IBC modules that can trigger security sensitive events like a Bank transfers before returning an error acknowledgment.

As mentioned earlier, two security sensitive examples of offchain applications that rely on Cosmos events are centralized exchanges and bridges.

Exchanges can use events to monitor for transfers to their deposit wallets. Thanks to Cosmos instant finality and the normal guarantees around Event emissions, a single transfer event to an exchange wallet would normally be sufficient to confirm a deposit. By abusing the bug described above an attacker could trigger repeated deposit events without losing control of the tokens.  As CEX codebases are closed source and an exchange might implement additional verification steps that protect against this attack, we were not able to verify which exchanges were vulnerable to this attack. However, we are aware of multiple exchanges halting deposits on Cosmos chains around the time of this disclosure.

Fortunately, multiple bridges that support Cosmos networks have partially open-source offchain implementations, which allows us to verify the impact of this bug without performing any visible onchain experiments. cBridge, Multichain and Wormhole’s legacy Cosmos integration (replaced by the IBC-native Wormhole Gateway) all depend on Cosmos events for detecting interactions with their smart contracts or deposit wallets, making them vulnerable to this issue.

Patch

The patch for Huckleberry is simple. Don’t emit events for failed acknowledgments.

Conclusion

Vulnerabilities such as Huckleberry, which break core assumptions of offchain components, can lead to a wide range of unexpected security vulnerabilities in various products. These vulnerabilities clearly demonstrate the need for robust defense-in-depth mitigations in web3 applications.