Blog: Blog HomeCompanyEcosystemIncrementProduct
Niall Yorke
Product

Open Sourcing the Refiner; Phase 2 of the Covalent Network

TL;DR:

  • Phase 2 of the Covalent Network is here! The Refiner revolutionizes granular data access, allowing anyone to retrieve one-to-one representations of blocks without relying on JSON-RPC calls, simplifying data access and enhancing scalability in Web3 data infrastructure.

  • Given Covalent Networks' distributed environment, The Refiner is capable of concurrent tracing of small block sections for faster and more efficient data processing and synchronization.

  • Operations with the Refiner are cryptographically secure, ensuring the integrity and accuracy of data on the Covalent Network fostering trust among developers, researchers, and enterprises.

  • Today's release is the Whitepaper along with the reference implementation of the Refiner specification under the Apache License 2.0.

The Refiner has Arrived, Revolutionizing Granular Blockchain Data Access

The second phase of the Covalent Network is here; The Refiner. The Refiner builds upon the innovation of the Block Specimen, addressing the ongoing challenge of accessing deep, granular blockchain data, which is essential for unlocking the full potential of blockchains.

New Data, Same Old Problems

While accessing "high-level" blockchain data like historical transactions and balances has become easier in recent years, granular data remains inaccessible.

Any developer or data provider is familiar with the pain that comes with accessing this kind of blockchain data. If one wishes to examine internal transactions, contract interactions or perform transaction debugging, there is a myriad of tasks associated with doing such. It could require re-executing block after block from Genesis, waiting for your node to sync, or querying each transaction individually using JSON-RPC methods.

Given the further adoption the space has witnessed in recent years, enterprises and institutions are now demanding granular blockchain data too. This data could be used for taxation, AI data modelling, enhanced analytics, and auditing. Yet they cannot access it.

The bottom line is that current clients are inadequate for querying blockchain data and should not be used. They are designed to execute transactions and reach consensus, not read them. Hence, data providers cannot simply offer this data.

That is about to change.

The Refiner Simplifies Access to Granular Data

The Refiner approach revolutionizes granular data access in the Web3 space. The Refiner takes Block Specimens as input and performs transformations concurrently with other Network Operators. The output is a Block Result, a cryptographically verifiable one-to-one representation of block data returned from an RPC call to a blockchain node with optional and additional informative fields.

Similar to an oil refinery that takes crude oil and manufactures various useful products simultaneously, the Refiner transforms Block Specimens into Block Results and, in the future, other outputs based on performing concurrent transformations on any input provided. The Block Results can be retrieved and indexed by anyone on the Covalent Network.

This allows for the re-execution of Ethereum blocks and transactions, completely independent of their source node client software, a capability unmatched by any blockchain node clients currently available.

With the Refiner live, the Covalent Network empowers users to pull historical and live blocks complete with the data retrieved from the JSON-RPC layer but without the associated bottlenecks.

What Does This Mean for Web3 Data?

With the Covalent Network powered by the Block Specimen and Refiner, data providers can bid easily retrieve any blockchain point they desire with ease. More importantly, this unparalleled access to blockchain data translates to a better user experience on the application level as well. Developers will be able to build better data-driven DApps. Analysts will be able to construct dashboards with unmatched visibility into blockchain data.

But unlocking easy access to granular blockchain data is part of the magic with the Refiner.

Distributed Processing

The Refiner operates in a distributed environment that is independent of storage, execution, or consensus nodes. This capability revolutionizes tracing on blockchains by enabling independent tracing or retracing of small sections of blocks, significantly speeding up the process. Unlike traditional approaches, computations for extraction and indexing are completed in parallel, resulting in greater efficiency, reduced overhead on the network, and faster, more flexible chain data synchronization. This distributed processing capability brings unprecedented scalability and performance to the Web3 data infrastructure space.

No More JSON-RPC Calls

With the Refiner and its Block Results, users can retrieve one-to-one representations of blocks without relying on the JSON-RPC layer. This eliminates the need for developers and applications to make expensive and time-consuming JSON-RPC calls, enabling the Covalent Network and other protocols built on top of it to scale freely without the constraint of running nodes. By removing the burden of read scalability from nodes, they can focus on their primary purpose of executing transactions on the blockchain. This advancement simplifies and accelerates data access, opening up new possibilities for Web3 data infrastructure.

Cryptographically Secure

Similar to the production of Block Specimens, all operations performed by Network Operators with the Refiner are cryptographically secure. For every Block Result created, a corresponding proof is generated and validated, ensuring the integrity and accuracy of the data. This cryptographic security guarantees the trustworthiness of the data available on the Covalent Network. Network Operators are rewarded in CQT for publishing valid proofs, incentivizing their commitment to maintaining the network's security and reliability. This robust security mechanism enhances the overall integrity of Web3 data infrastructure, providing a foundation of trust for developers, researchers, and enterprises.

Between both the Block Specimen and the Refiner, the Covalent Network is radically changing how blockchain data is accessed, processed, and analyzed.

What's Next?

As the Covalent Network continues to evolve, there are exciting plans on the horizon. While the Refiner goes live on August 1st producing Blocking Result for Ethereum Mainnet, the next phase will see an expansion in the capabilities of the Refiner, enabling even more advanced data transformation and processing. Additionally, efforts are underway to enhance the Covalent Network's infrastructure with the much-anticipated network migration that will ensure scalability and reliability as the demand for blockchain data continues to grow.

The Refiner is the latest unlock that Covalent Network brings to granular data access, unlocking the true potential of blockchain technology by scaling verified data access to the application layer.

Learn More about the Refiner

About Covalent:

Covalent provides the industry-leading Unified API bringing visibility to billions of Web3 data points. Developers and analysts use Covalent to build exciting multi-chain applications like crypto wallets, NFT galleries, and investor dashboard tools utilizing data from 229 + blockchains. Covalent is trusted by a community of 40,000+ developers and powers data for 5,000+ applications, including 0x, Zerion, Rainbow Wallet, Rotki, Bitski, and many others.