Core Components

Core components are the basic building block for building your own crawler. They include:

• Bus Components to manage the process of crawling

• RPC Components to communicate with RPC APIs

• Stats Components for tracking the performance of individual components or the crawler as a whole

• Data Client Components for communicating with HTTP/ IPFS, and Arweave metadata providers

• Storage Client Components for storing metadata