Shelby: Decentralized Storage Designed to Serve

Samir Islam | @dragonfrond

Decentralized, byzantine fault-tolerant technology (blockchain) promised us a shared world computer, unshackled from any single gatekeeper, and accessible to all. This distributed, 24/7, high performance ledger has already created a revolution in how we interact with each other. Blockchains enable trustless settlement, and this has resulted in a vibrant world of Decentralized Finance (DeFi).

An incredible amount of engineering talent and technology has gone into developing blockchains and applications deployed on them, but utility outside of DeFi remains limited. Given its massive potential, blockchain should have a much bigger piece of the cloud computing market than it currently does.

This post shares why Jump Crypto is teaming up with Aptos Labs to build the Shelby Storage Protocol.

Someone Else’s Computer

General purpose cloud computing has captured huge value creation ecosystems. These markets have effectively been trapped by the high-walled gardens of centralized providers. Just name a few examples:

• Data analytics
• Social media
• AI inference and training
• Ad markets & serving

Blockchain technology has attempted to break into these use cases several times, with limited success, and sometimes relying on those same web2 services they seek to replace. Generally, using blockchains for serious computation is costly and inefficient, severely limiting the product that can be built. Unlocking high-performance compute within the blockchain ecosystem will empower a new wave of on-chain applications, capturing tasks currently running on centralized providers.

So, we need to build efficient compute, but before we get started on that, we must first figure out information storage.

Why Storage?

Storage is essential for non-trivial computation. While blockchains today have mostly solved the ability to execute code, they cannot access large datasets. All blockchains can only see the latest state when executing and are unable to view any past context. Yet historical data is crucial to run any kind of interesting business. In the examples above, high-performance data storage and retrieval is a key ingredient:

• Data analytics: Data lakes/bases/warehouses.
• Social media: Media storage & sharing, graph analysis.
• AI inference and training: Models are huge, training data sets are even larger.
• Ad markets & serving: Demographic & targeting tables, media for serving.

Back in 2006 Amazon launched their Simple Storage Service (S3). This basically created the cloud computing market; before then every organization had to figure out infrastructure themselves. This also changed the role of the internet from convenient access to essential utility. Cloud storage, and its cloud computing sibling, let businesses to focus on their native strengths, without having to worry where to run it.

Today, these markets are dominated by a handful of technology giants, mostly based in the US. This creates centralization risk, customers are faced with the choice of reduced performance or living with walled gardens. High capital requirements make it difficult for new (centralized) entrants to effectively compete, meaning incumbents, in the aggregate, control access and pricing.

Why Decentralize?

We believe in the future of decentralized infrastructure, where control, ownership and monetization shift from system gatekeepers to contributors and consumers.  This future gives builders the option to avoid lock in, maintains the decentralization ethos, and ultimately lowers the cost to deliver their content to users. Our goal is to give people the tools to build something more efficient, more accessible, and more valuable than they can now.

Decentralization spreads costs across participants, keeping extraction rates competitive even within a protocol. Whereas centralized providers benefit from scale, decentralization brings diversity: each contributor maximizes their value and lowers their costs, reducing overall expenses. For data that you want to use, Shelby has the potential to deliver centralized storage quality at a fraction of the cost. With this technology, high-performance decentralized storage is feasible and delivers real value beyond just finance.

The challenge is building a community that lets the whole be greater than the sum of its parts. Shelby is designed to enable individual providers to contribute small-scale storage instances with accessible hardware requirements, fostering a more comprehensive and resilient system.

As an individual, you can rent storage nodes for less than $2/TB/month [1] - far below centralized provider costs, although you must implement your own version of durability and distribution. Using low level networking protocols, erasure coding and orchestration, the Shelby protocol is designed to combine multiple low-cost instances into a highly performant, highly durable system, matching centralized providers at lower cost.

Shelby’s cost model, including cryptographic replication and performance expectations, is based on real data and working models, which we will share in a future blog post.

Why Now?

We believe in the promise of blockchain to power a new way of organizing, building a new wave of decentralized applications. We are working to build and support the infrastructure for these new use cases.

We have seen many projects recently that recognize the same limitation and are attempting to build some form of non-replicated compute; however, they are all missing the crucial component of high-performance data storage to truly succeed.

Our experience taught us that high-performance data requires a high-performance network. Through our work on Firedancer, we became convinced that a high-performance network was also required for current blockchains. The identification of this constraint led to the launch of Double Zero, which is exactly that: a high-performance decentralized network for high bandwidth applications. Shelby is designed to leverage that foundation to deliver high-performance storage, a critical step towards true blockchain compute.  

The goal is to allow blockchain to effectively compete with centralized providers. Network + Data + Compute = The Decentralized Cloud.

Why Jump?

Jump Crypto’s engineering approach is to solve immediate, tangible problems where constraints exist today, rather than chasing long-term moonshots. As we discussed in our recent post, we have focused on building decentralized consortiums to solve limitations with existing oracle, moving messages, scaling Solana, and a decentralized global fiber-optic network for distributed systems.

In each case, we applied our extensive expertise and technology stack to address practical gaps in the decentralized ecosystem. Each project was built open source, with no one, including Jump Crypto, singularly in control of the protocol. That's been our guiding principle from day one.

Over the past 25 years Jump has seen traditional electronic financial markets evolve, growing and learning from that experience. To remain competitive, we have developed high-performance technologies and a deep bench of engineering talent that can solve novel and difficult problems. We see real value in taking these capabilities and bringing them to blockchain.

For over a decade, Jump has handled massive data volumes in technical operations to train and test our trading models. We process and store hundreds of petabytes of market data, which feeds the tiered archive (our data system) into the grid (our analytics system). We routinely capture over a hundred terabytes of new data daily. To handle this scale, we first had to build a computer network that allowed us to quickly move data around our infrastructure. From there we built layered software systems that can store vast amounts of data in relatively slow (and cost-effective) warm storage and feed it up to the grid, itself a collection of thousands of nodes, at many terabytes a second.

This network sparked the founding of Double Zero, and the tiered archive is was an inspiration for Shelby. Just as with Firedancer - where we used our high-speed trading technology to improve blockchain transaction processing - Shelby applies our high-performance storage expertise to deliver best-in-class decentralized storage. Jump’s engineering stack and experience uniquely position us to deliver high-performance decentralized storage at scale.

Why Aptos?

We fully intend for Shelby to work across blockchains, but it needs a home base for coordination. We wanted to avoid the burden of building and maintaining a blockchain exclusively for Shelby.

While Jump Crypto has extensive experience in high-performance networking, storage and compute, Aptos Labs is a demonstrable leader in low latency, high throughput blockchain technology. The Aptos Blockchain is an ideal environment for real world use cases and consumer applications.

Even more important than the technology are the people who turn it into reality. The Aptos Labs team shares a common mindset of high-performance engineering and the vision to grow blockchain for value creation. Working together, each of our teams can focus on their strengths, with the conviction of a shared vision. Combining these talents makes it possible to build Shelby into a truly extraordinary system.

Shelby exists to make decentralized storage fast, affordable, and reliable - and to unlock blockchain’s value for the real-world.

Let us build the future of blockchain together.

[1] https://www.ovhcloud.com/en/bare-metal/high-grade/hgr-stor-2/ with 102 102x14TB drives for $1342 / month ($0.93 / TB / month) or Hetzner https://www.hetzner.com/dedicated-rootserver/sx295 with 14x22 TB drives for €384 / month (€1.2 / TB / Month)