The Refiner
The Refiner is a concurrent Block Specimen data processing framework designed for scalable and verified data transformations. The output, one of many possible, is a Block Result; a one-to-one representation of block data returned from an RPC call to a blockchain node with optional and additional informative fields. And importantly, the Refiner can perform arbitrary transformations over any input Block Specimens concurrently with other transformations, which essentially means that the Refiner can concurrently re-execute ethereum blocks and transactions completely external to their source blockchain node client software, something that no blockchain node clients can currently do. By leveraging the Refiner's concurrent processing capability, one can effortlessly perform customizable data indexing with minimal overhead.
The Block Results consequently indexed will power products such as Covalent's Unified API, and is the first of many steps needed to build our vision of a truly decentralized data infrastructure aimed at solving Ethereum's long term data availability and read scalability problem.
Enrichment
With the Refiner live, Block Specimens can be enriched and transformed into Block Results by multiple operators on the Covalent Network. These verified Block Results can then be retrieved and indexed by anyone utilizing the Covalent Network for decentralized data access without the overhead of running full blockchain client nodes.
The net result is that anyone will be able to pull historical and live blocks that are complete with the data one could retrieve by calling the JSON-RPC layer but without the bottlenecks associated with the latter method. Furthermore, anyone wishing to analyze this data or build a web3 data application on top of the Covalent Network, such as block explorers, can pull this type of granular data without running a node.
An Ethereum transaction tracing engine is also on its way to unlock deeper insights into an Ethereum transaction. This work is iterative and research-oriented in nature (unlike Block Specimen to Block Result transformation, which is stable). The concurrent nature of Refiner's Block Specimen execution means that the network can quickly run all Ethereum blocks through the tracing engine.
Hardware Requirements
The recommended configurations will make sure the sync is not lag prone and the node is able to keep up with the Ethereum network.
Minimum
- 2 vCPUs (cores)
- 3GB RAM
- 200GB HDD free storage (mostly storing artifacts on local ipfs; can be pruned periodically)
- 8 MBit/sec download Internet service
Recommended
- 4 vCPUs
- 8 GB+ RAM
- SSD with >= 500GB storage space
- 25+ MBit/sec download Internet service
Software Requirements (Docker Setup)
- 64-bit Linux, Mac OS 13+
- SSL certificates
- docker, docker-compose, direnv