Introduction

The Polkadot ecosystem, envisioned as a "heterogeneous multi-chain framework," presents a paradigm of interoperable, specialized blockchains operating in parallel.¹ While this architecture fosters innovation and scalability, it simultaneously introduces a significant challenge: the accessibility of on-chain data. For the vibrant ecosystem of decentralized applications (dApps), analytics platforms, and monitoring tools to thrive, they require a method to query and process blockchain data that is both performant and reliable. Standard node Remote Procedure Call (RPC) interfaces, designed for simple, state-based lookups, are fundamentally ill-equipped to handle the complex, historical, and aggregated queries that sophisticated applications demand, creating a critical infrastructure gap.²

This report presents a comprehensive technical blueprint for a high-performance, adaptable blockchain indexer specifically designed for the Polkadot Substrate ecosystem and implemented in the Rust programming language. The objective is to architect a system that transcends the limitations of a simple data pipeline, instead serving as a foundational piece of infrastructure that embodies the core principles of the ecosystem it serves: performance, adaptability, resilience, and decentralization. The proposed design is a modular, metadata-driven architecture that prioritizes a clean separation of concerns. This approach allows for the independent optimization of each system layer—from high-velocity data ingestion to flexible API presentation—while providing a holistic and robust solution to the unique challenges posed by Substrate's dynamic nature, particularly its capacity for forkless runtime upgrades and the potential for chain reorganizations. By leveraging the unparalleled safety, concurrency, and performance guarantees of Rust, this architecture aims to deliver a system that surpasses existing solutions in both speed and operational reliability.⁴

I. The Substrate Data Access Paradigm: Navigating a Dynamic On-Chain World

To architect an effective indexer for Substrate-based chains, one must first comprehend the foundational design principles that differentiate Substrate from other blockchain frameworks. These principles, while enabling unprecedented flexibility and evolvability for the chains themselves, impose a unique set of requirements on any external tooling that aims to interpret their data. The entire architecture of the indexer is a direct response to these core Substrate characteristics.

The Client-Runtime Dichotomy

The most critical architectural feature of Substrate is its deliberate separation of the blockchain node into two distinct components: the Client and the Runtime.⁴

• The Client: This is the native binary, compiled for a specific machine architecture. It is responsible for the core, non-deterministic functions of the blockchain: peer-to-peer networking, transaction pool management, consensus mechanisms (e.g., BABE, GRANDPA), and managing the underlying database (typically RocksDB).⁴
• The Runtime: This is the State Transition Function (STF), which contains all the business logic of the blockchain—how balances are transferred, how governance votes are tallied, and how smart contracts are executed. Crucially, the Runtime is compiled to a platform-agnostic WebAssembly (Wasm) blob, which is stored on-chain as part of the state itself.⁴

This separation is not merely a technical implementation detail; it is a profound design philosophy. It prioritizes the long-term evolvability of the blockchain's logic over the short-term simplicity of external tool development. While frameworks for more static blockchains allow tool builders to make long-standing assumptions about data structures, Substrate's design intentionally invalidates this approach. It shifts the burden of adaptability from the core protocol, which can now evolve seamlessly, to the surrounding ecosystem of dApps, wallets, and indexers. Consequently, an indexer built for Substrate cannot be a rigid, static application; its architecture must be fundamentally dynamic and adaptive by design, mirroring the evolvability of the chains it serves.

SCALE Encoding and Metadata

All dynamic data within a Substrate runtime—including storage items, transaction payloads (extrinsics), and events—is encoded using the SCALE (Simple Concatenating Aggregated Little Endians) codec.⁹ SCALE is designed to be compact and performant in resource-constrained environments like Wasm, but it is not a self-describing format. A raw, SCALE-encoded blob of bytes is opaque without a schema to interpret it.

This schema is provided by the chain's metadata. The metadata is a comprehensive data structure, itself SCALE-encoded, that the runtime exposes via an RPC call (state_getMetadata). It serves as a complete, machine-readable blueprint of the runtime's capabilities, detailing every pallet, extrinsic, event, storage item, constant, and their corresponding data types.¹⁰ The indexer's ability to comprehend and decode on-chain data is entirely dependent on its ability to fetch, parse, and correctly utilize the appropriate version of this metadata for any given block.

The Imperative of Adaptability: Runtime Upgrades

The primary benefit of the Client-Runtime dichotomy and the Wasm meta-protocol is the ability to perform forkless runtime upgrades.⁸ A new version of the runtime logic can be compiled to Wasm and submitted to the chain via a special transaction (typically

system.setCode). Once this transaction is executed, the Wasm blob stored in the chain's state is replaced, and all nodes immediately begin using the new logic for subsequent blocks.¹³

For an indexer, this presents the ultimate challenge. A runtime upgrade can introduce breaking changes: the structure of an event can be altered, the parameters of an extrinsic can be modified, or a storage item can be migrated to a new format. An indexer with hardcoded assumptions about these structures will fail catastrophically upon such an upgrade, either by crashing, halting its progress, or silently ingesting corrupted and meaningless data. Therefore, the single most important non-functional requirement for a Substrate indexer is the ability to handle runtime upgrades gracefully and autonomously. This involves detecting the upgrade event, fetching the new metadata, and dynamically adjusting its decoding and processing logic to match the new on-chain reality. The upcoming Metadata V16 specification will provide even richer information, such as associated types and deprecation flags, further increasing the need for a sophisticated, metadata-aware processing pipeline.¹⁵

II. A Blueprint for a Modular Indexer Architecture

To manage the complex and often conflicting requirements of a high-performance Substrate indexer—the need for raw ingestion speed, logical adaptability, storage robustness, and API flexibility—a monolithic design is untenable. A change in one area, such as optimizing a database query, could inadvertently create a bottleneck in the real-time ingestion pipeline. The only viable approach is a modular, layered architecture where each component has a single, well-defined responsibility and interacts with other layers through stable, explicit interfaces. This separation of concerns allows each component to be developed, optimized, and scaled independently.

The proposed architecture consists of four distinct layers: Ingestion, Processing, Storage, and Presentation.

Layer 1: The Ingestion Layer (The Collector)

• Responsibility: This layer is the indexer's sole interface with the blockchain network. Its primary function is to establish and maintain a durable, resilient connection to one or more Substrate nodes, efficiently fetch raw block data, and pass this opaque data to the Processing Layer. It is optimized for network I/O and throughput, not for data interpretation.
• Key Components:
• RPC Client Manager: Manages a pool of connections to Substrate node RPC endpoints.
• Real-Time Subscription Service: Listens for new finalized blocks via a WebSocket subscription.
• Historical Fetch Service: Fetches ranges of historical blocks in parallel during the initial sync.

Layer 2: The Processing Layer (The Transformer)

• Responsibility: This is the cognitive core of the indexer. It receives raw, SCALE-encoded block data from the Ingestion Layer and transforms it into structured, meaningful information. It is responsible for managing metadata versions, decoding data, applying user-defined filters, and mapping the decoded data to the storage schema.
• Key Components:
• Metadata Registry: A persistent cache of all historical runtime metadata versions, keyed by spec_version.
• SCALE Decoder Engine: Dynamically decodes raw extrinsics and events using the correct metadata version for the given block height.
• Filtering Engine: Applies user-defined rules to discard irrelevant data before it reaches the storage layer.
• Data Mapper: Transforms the decoded, filtered data into the relational models defined by the storage schema.

Layer 3: The Storage Layer (The Database)

• Responsibility: This layer provides a durable, queryable, and transactionally-consistent persistence mechanism for the processed data. Its design prioritizes data integrity, query performance, and scalability.
• Key Components:
• Database Connection Pool: Manages a pool of connections to the underlying database.
• Data Modeling Schema: Defines the relational tables and indexes for storing blocks, extrinsics, events, and other user-defined entities.
• Transaction Manager: Ensures that all data related to a single block is written to the database atomically.

Layer 4: The Presentation Layer (The API)

• Responsibility: This layer exposes the indexed data to end-user applications. It provides a flexible, performant, and developer-friendly query interface.
• Key Components:
• GraphQL Server: Hosts a GraphQL endpoint for handling queries and mutations.
• Query Resolvers: Contains the logic to translate incoming GraphQL queries into efficient SQL queries against the storage layer.
• Subscription Handler: Manages real-time data updates to clients via GraphQL subscriptions over WebSockets.

This layered design directly translates the user's multifaceted requirements into a coherent and scalable engineering strategy. By decoupling these concerns, the system can be optimized at each level without compromise, resulting in an architecture that is simultaneously fast, adaptable, and robust.

III. High-Velocity Ingestion: Real-Time and Parallel Historical Sync

The performance of an indexer is most acutely perceived during two phases: its ability to keep up with the live chain head and the time it takes to perform the initial historical sync. The Ingestion Layer is designed to excel at both, using a combination of a low-latency subscription model for real-time data and a massively parallel approach for backfilling historical data.

Real-Time Block Subscription via subxt

To achieve near-instantaneous processing of new blocks, a polling-based approach (periodically calling chain_getBlock) is inefficient and introduces unnecessary latency. The optimal strategy is to use a WebSocket subscription to the node's RPC endpoint. The subxt library, Parity's official Rust client for Substrate nodes, provides a clean, asynchronous interface for this purpose.¹

The implementation will leverage subxt's OnlineClient to connect to a node and then call the api.blocks().subscribe_finalized().await? method.¹⁶ This returns a Rust

Stream that yields new Block objects as soon as they are finalized by the chain's GRANDPA consensus mechanism. This event-driven approach ensures the indexer receives data with minimal delay, fulfilling the requirement to "listen for blocks as soon as they are produced."

Massively Parallel Backfilling

The initial synchronization of a chain's full history, which can consist of millions of blocks, is a significant performance bottleneck for any indexer. To "quickly traverse the blockchain," a parallel processing strategy is essential. The design will partition the entire historical block range (e.g., block 1 to the current finalized head) into smaller, independent chunks.

This task is a perfect fit for data parallelism and will be implemented using the rayon crate in Rust.¹⁷ A master controller will determine the block ranges to be processed. A

rayon thread pool will then execute the fetching and processing of these ranges in parallel. Each worker thread will be equipped with its own RPC client instance and will be responsible for fetching the raw data for its assigned block range. This strategy is designed to saturate the available CPU cores and network bandwidth, dramatically reducing the time required for the initial sync.

However, a naive parallelization strategy can be counterproductive. Public RPC providers and even self-hosted nodes have connection and rate limits.¹ Spawning an excessive number of threads that all make simultaneous requests can lead to dropped connections, rate-limiting errors, and a net decrease in throughput. The true optimization lies in finding the ideal balance between the number of worker threads and the number of blocks fetched per batch RPC call. Therefore, the ingestion layer must be configurable, allowing operators to tune these parameters. It must also implement robust error handling with exponential backoff and retry logic to gracefully handle transient network issues or rate-limiting responses from RPC nodes. This transforms the backfilling process from a simple CPU-bound task into a sophisticated, network-aware distributed computing problem.

Resilience to Chain Reorganizations (Reorgs)

Blockchains are not immutable linear histories until a certain point of finality. Short-lived forks are a normal part of the consensus process, and an indexer must be able to handle them to avoid storing data from blocks that are eventually orphaned.²⁰ While GRANDPA provides strong finality, reorgs can and do happen on unfinalized chains.

The ingestion layer will be designed to be "reorg-aware".²¹ It will maintain an in-memory buffer of the most recent N block hashes. For each new block it receives, it will perform a critical check: does the new block's

parentHash field match the hash of the previously received block?.²²

If a mismatch is detected, a reorganization has occurred. The handling process is as follows:

1. Detection: The parentHash mismatch signals the reorg.
2. Find Common Ancestor: The service will walk backward from the new block's parent hash via RPC calls until it finds a block hash that exists in its buffer. This block is the common ancestor.
3. Signal Rollback: The ingestion layer will issue a command to the Storage Layer, instructing it to atomically delete all indexed data (blocks, extrinsics, events) for all block numbers greater than the common ancestor.
4. Re-ingest: The ingestion layer will then begin fetching and processing blocks from the new canonical chain, starting from the block after the common ancestor.

This automated detection and rollback mechanism is crucial for data integrity, ensuring that the database always reflects the true canonical history of the chain.

IV. The Chameleonic Core: A Strategy for Runtime Upgrade Adaptability

The central challenge of building a Substrate indexer is its ability to adapt to forkless runtime upgrades. The Processing Layer is the "chameleonic core" designed to solve this problem through a dynamic, metadata-driven approach. It must be capable of correctly decoding data from any point in the chain's history, regardless of how many times the runtime has evolved.

A Multi-Version Metadata Registry

The foundation of this adaptability is a persistent registry of all runtime metadata versions the chain has ever had. The indexer will maintain a simple database table mapping a runtime's spec_version to its full, SCALE-encoded metadata blob.

When processing a block at a given height, the indexer must use the metadata that was active at that height. It determines the correct spec_version by querying the System::LastRuntimeUpgrade storage item or, more efficiently, by observing the system.CodeUpdated event, which signals a runtime upgrade has occurred. If the spec_version for a block is not present in its local registry, the indexer will make a state_getMetadata RPC call to fetch the corresponding metadata blob and persist it for future use.⁹

The Power of subxt: Static and Dynamic Clients

The subxt library is uniquely suited to this challenge because it offers two distinct modes of operation: static and dynamic.²³ This dual capability allows the architecture to resolve the inherent tension between the need for raw performance and the need for absolute flexibility.

• Static, Type-Safe Decoding (The "Hot Path"): For real-time indexing of the latest blocks, performance is paramount. Here, the indexer can use subxt's static mode. Using the subxt-cli tool, the metadata for the current runtime version can be downloaded, and the #[subxt::subxt] procedural macro can be used to generate a complete, type-safe Rust interface at compile time.²³ This allows the Rust compiler to generate highly optimized code for decoding and interacting with the chain, providing the best possible performance and developer experience.
• Dynamic, Runtime-Aware Decoding (The "Cold Path"): When backfilling historical data or processing blocks immediately following a runtime upgrade (before the static types can be regenerated and the indexer redeployed), the static approach is not viable. For these scenarios, the indexer will switch to subxt's dynamic client.²⁵ The dynamic client does not rely on compile-time generated code. Instead, it accepts a metadata blob at runtime and uses it to interpret and decode on-chain data. This provides the essential flexibility to handle any historical data structure the chain has ever used. Under the hood, this relies on powerful decoding libraries like
  scale-decode, which can traverse SCALE-encoded byte streams and map them to dynamic value representations based on the provided metadata types.²⁶

The Automated Upgrade Handler Service

To ensure seamless, zero-downtime operation across runtime upgrades, a dedicated service within the Processing Layer will monitor the chain for upgrade events.

1. Trigger: The service will subscribe to finalized blocks and specifically filter for the system.CodeUpdated event.
2. Action: Upon detecting this event, the service will automatically trigger the following workflow:

• Fetch the new spec_version and runtime metadata from the node.
• Validate the fetched metadata to ensure it's well-formed.
• Atomically insert the new (spec_version, metadata_blob) pair into the metadata registry.
• Clear any in-memory caches related to the old metadata.

1. Transition: The processing pipeline, upon encountering blocks with the new spec_version, will now automatically load the new metadata from the registry and use the dynamic client to decode the data, ensuring continuous and correct indexing without any manual intervention.

This hybrid architectural pattern provides a robust and elegant solution, leveraging the strengths of both static and dynamic dispatch to create an indexer that is both deeply optimized for real-time performance and universally adaptable to the entire evolutionary history of a Substrate chain.

V. The Persistence Layer: Selecting and Modeling the Optimal Storage Engine

The choice of a storage engine is a critical architectural decision that profoundly impacts the indexer's query performance, data integrity, and operational complexity. The data must be stored in a way that is not only efficient for writing but also, and more importantly, highly optimized for the complex read patterns of dApps and analytics tools.

Storage Technology Analysis: Relational vs. Key-Value

Two primary categories of databases are viable for this use case: low-level key-value stores and high-level relational databases.

• RocksDB (Key-Value): This is the default storage engine used by the Substrate client itself, renowned for its exceptional write performance and efficiency.⁷ Projects like
  substrate-archive leverage a secondary RocksDB instance to achieve very high ingestion speeds.²⁷ However, this approach comes with significant drawbacks. RocksDB is a library, not a standalone server, and it offers no high-level query language like SQL. All complex queries, joins, and aggregations must be implemented in the application logic, which is complex and error-prone. Furthermore, its operational tuning can be challenging.²⁸
• PostgreSQL (Relational): As a mature, open-source object-relational database, PostgreSQL offers unparalleled query flexibility through SQL, strong ACID transactional guarantees, and a vast ecosystem of tools and extensions.³⁰ It provides a robust foundation for building complex data models.
• TimescaleDB (Time-Series on Postgres): TimescaleDB is a PostgreSQL extension specifically designed to handle time-series data at scale.³¹ Since blockchain data is fundamentally a time-series of events, TimescaleDB is an ideal fit. Its core feature,
  hypertables, automatically partitions large tables by a time dimension (like a block timestamp or block number). This partitioning allows the query planner to prune irrelevant partitions (or "chunks"), leading to dramatic performance improvements for time-bound queries.³¹ It also offers specialized time-series functions and features like continuous aggregates for pre-computing analytical views.

The clear architectural choice is PostgreSQL with the TimescaleDB extension. This combination offers the transactional integrity and query power of a leading relational database while providing the specialized performance optimizations required for handling massive volumes of blockchain data.

Table: Comparison of Storage Engines

Feature	PostgreSQL with TimescaleDB	RocksDB
Data Model	Relational, Time-Series	Key-Value
Query Language	Full SQL, specialized time-series functions	Get/Put/Scan API
Indexing Capabilities	B-tree, GIN, BRIN, Hash, GiST	Prefix-based, Bloom filters
Transactional Guarantees	Full ACID compliance	Atomic Writes, Snapshots
Operational Complexity	Moderate (standalone server)	High (embedded library, tuning required)
Ecosystem & Tooling	Vast and mature	Limited, library-specific
Suitability for Analytics	Excellent (SQL, aggregations, joins)	Poor (requires application-level logic)

Data Modeling for Substrate in a Relational Schema

A normalized relational schema is essential for maintaining data integrity and enabling efficient queries. The core of the schema will consist of the following tables, which will be configured as TimescaleDB hypertables:

• blocks: Contains one row per block, storing essential header information.
  • block_number (BIGINT, PRIMARY KEY)
  • hash (BYTEA, UNIQUE)
  • parent_hash (BYTEA)
  • timestamp (TIMESTAMPTZ NOT NULL)
  • spec_version (INTEGER)
• extrinsics: Stores information about each transaction.
  • block_number (BIGINT, FOREIGN KEY -> blocks)
  • extrinsic_index (INTEGER)
  • hash (BYTEA, UNIQUE)
  • signer (BYTEA, NULLABLE)
  • pallet_name (TEXT)
  • call_name (TEXT)
  • params (JSONB)
  • success (BOOLEAN)
  • fee (NUMERIC)
• events: Captures every event emitted by the runtime.
  • block_number (BIGINT, FOREIGN KEY -> blocks)
  • extrinsic_index (INTEGER, NULLABLE)
  • event_index (INTEGER)
  • pallet_name (TEXT)
  • event_name (TEXT)
  • data (JSONB)

The use of the JSONB data type for extrinsic parameters and event data is a crucial design choice. It provides the flexibility needed to store data whose structure changes with runtime upgrades, without requiring schema migrations for every upgrade. PostgreSQL's powerful JSONB indexing and query operators allow for efficient filtering and extraction of data from these semi-structured fields.

Ensuring Atomicity: Database Transactions and the Outbox Pattern

Data integrity is paramount. To ensure the database never enters a partially indexed or inconsistent state, two patterns will be employed:

1. Block-Level Transactions: All database write operations for a single block—inserting the block record, all of its extrinsics, and all of its associated events—will be wrapped within a single, atomic database transaction. If any single operation fails (e.g., due to a constraint violation or a connection drop), the entire transaction is rolled back, leaving the database in the state it was in before that block was processed. This guarantees that a block is either indexed completely or not at all.³³
2. Transactional Outbox Pattern: For reliably propagating indexed data to downstream systems (like a real-time API or message queue), the indexer must avoid dual-write problems. A naive approach of "write to DB, then send message" can fail after the DB write but before the message is sent, leading to inconsistency. The transactional outbox pattern solves this by making the notification part of the same atomic operation. An outbox table is created in the database. When processing a block, the processor writes the indexed data and a record representing the notification to the outbox table within the same transaction. A separate, simple poller process then reads from this table and sends the messages. This guarantees that a notification is only sent if and only if the underlying data has been successfully and durably committed.³⁵

VI. User-Defined Data Scopes: A Declarative Configuration Framework

A powerful indexer is only useful if it is accessible to developers. A key requirement is to provide a "simple way to specify what data a user wants to index." This prevents developers from wasting compute and storage resources on data irrelevant to their application. The solution is a declarative configuration file coupled with an intuitive Domain-Specific Language (DSL) for filtering.

Configuration via TOML

The indexer will be configured via a single config.toml file. TOML is chosen for its clear syntax and excellent support within the Rust ecosystem. This file will serve as the single source of truth for an indexer's behavior, defining node endpoints, database connection details, and, most importantly, the data selection rules.

A DSL for Data Filtering

To provide fine-grained control over data selection, a simple DSL will be designed within the TOML configuration structure. This pattern is common in Rust tooling for creating expressive and readable configurations.³⁷ The DSL will allow users to specify which pallets, calls, and events they are interested in, and even apply basic filters to their fields.

At startup, the indexer will parse this TOML file and construct an in-memory representation of the filter rules. The Processing Layer will then consult these rules for every extrinsic and event it decodes, efficiently discarding any data that does not match the user's defined scope.

Table: Configuration DSL Syntax and Examples

The following table specifies the proposed syntax for the filtering DSL, demonstrating how users can express rules from broad to highly specific.

Syntax	Description	Example
pallets = ["*"]	Index all data from all pallets. This is the default behavior if no rules are specified.	pallets = ["*"]
pallets =	Index data only from the specified list of pallets. All other pallets will be ignored.	pallets =
`` include =	Within the Balances pallet, only index the Transfer and Deposit events. All other Balances events are ignored.	`` include =
`` exclude =	Within the Balances pallet, index all events except for the Reserved event.	`` exclude =
`` where = "value > 1000000000000"	For the Balances.Transfer event, only index instances where the value field is greater than 1,000 KSM/DOT (assuming 12 decimals).	`` where = "asset_id == 1984"
`` include = ["remark_with_event"]	Index only the remark_with_event call from the System pallet.	[pallets.Utility.calls] include = ["batch", "batch_all"]

This DSL provides a powerful yet intuitive interface. It allows developers to precisely target the data they need, significantly reducing the indexer's storage footprint and processing load, leading to a more efficient and cost-effective system.

VII. Engineering for Resilience: Fault Tolerance and High Availability

A production-grade indexer cannot be a fragile, single-process application. It must be engineered to withstand hardware failures, network partitions, and software crashes without data loss or significant downtime. This requires a multi-faceted approach to resilience, encompassing stateful recovery, comprehensive monitoring, and a redundant deployment architecture.

Stateful Recovery and Persistent Cursors

The most fundamental requirement for fault tolerance is the ability to restart after a failure and resume work from the exact point where it left off. Re-indexing the entire chain after every crash is operationally infeasible.

To solve this, the indexer will employ a persistent cursor. After each block is successfully processed and its data committed to the database within a single transaction, the indexer will update a specific record in the database (e.g., in a _indexer_status table) with the block number it just completed. Upon startup, the indexer's first action will be to query this table to retrieve the last known good block number. It will then begin its ingestion process from the very next block, ensuring no data is missed and no work is duplicated. This guarantees an at-least-once processing semantic.

Health Checks and Monitoring

To integrate with modern DevOps practices and orchestration systems like Kubernetes, the indexer must be observable. It will expose a standard HTTP health check endpoint (e.g., /healthz). This endpoint will perform a series of internal checks—such as verifying the connection to the node RPC and the database—and return a 200 OK status if the service is healthy, or a 503 Service Unavailable if it is not. Orchestrators can use this endpoint to automatically restart or replace unhealthy instances.

Furthermore, the indexer will export a rich set of metrics in a Prometheus-compatible format. This will be achieved using Rust crates like prometheus or metrics. Key exported metrics will include:

• indexer_processed_blocks_total: A counter for the total number of blocks processed.
• indexer_latest_processed_block: A gauge indicating the current block height of the indexer.
• indexer_rpc_requests_duration_seconds: A histogram of RPC request latencies.
• indexer_db_transaction_duration_seconds: A histogram of database transaction latencies.

These metrics provide critical visibility into the indexer's performance and health, enabling operators to set up alerting and diagnose issues proactively.

Redundant, Highly-Available Deployment

To achieve high availability and eliminate single points of failure, the indexer should be deployed in a redundant configuration. However, a naive active-active deployment where multiple indexer instances write to the same database concurrently would lead to race conditions and data corruption.

The recommended architecture is an active-passive model enforced by a distributed lock.

1. Deployment: Two or more identical indexer instances are deployed.
2. Leader Election: Upon startup, each instance attempts to acquire a distributed lock (e.g., using a consensus system like etcd, or a database-level advisory lock in PostgreSQL). Only the instance that successfully acquires the lock becomes the "active" or "leader" instance. All other instances enter a "passive" or "standby" state.
3. Operation: The active instance performs all indexing operations. The passive instances periodically attempt to acquire the lock.
4. Failover: The active instance maintains its lock with a lease or heartbeat. If the active instance crashes or becomes unresponsive, its lease on the lock expires. One of the passive instances will then succeed in acquiring the lock, promote itself to active, and resume indexing from the last known good state recorded in the database.

This architecture, combined with a clustered and highly-available database deployment (e.g., using Patroni for self-hosted PostgreSQL or a managed service like Amazon Aurora ³⁹), ensures that the indexing service can survive the failure of any single component with minimal interruption.

VIII. The Path to Decentralization

While a centralized, highly-available indexer provides significant value, the ultimate goal for critical blockchain infrastructure is to achieve a degree of decentralization, enhancing censorship resistance and resilience. This can be achieved through a pragmatic, phased approach.

Phase 1: A Federated Network

The first step towards decentralization is to move from a single-operator model to a federated one. In this model, a consortium of trusted, independent entities (e.g., prominent staking providers, infrastructure companies, or dApp development teams) each run their own instance of the highly-available indexer architecture. While each indexer is centrally managed by its operator, the network of indexers as a whole is decentralized. This provides users with data source redundancy; if one operator's API is compromised, censored, or suffers an outage, applications can seamlessly failover to another operator's endpoint.

Phase 2: A Fully Decentralized Protocol

The final phase is the creation of a fully decentralized, permissionless network of indexer operators. This architecture draws inspiration from battle-tested protocols like The Graph, which have established a viable economic model for decentralized indexing.⁴⁰ This protocol would consist of several key participants and on-chain components:

• Indexers: Independent node operators who run the indexer software. They must stake a native network token as collateral, which can be slashed for misbehavior (e.g., serving incorrect data). In return for providing indexing and query services, they earn query fees from consumers and issuance rewards from the protocol.⁴³
• Consumers: dApps and end-users who pay query fees in the native token to retrieve data from Indexers.
• Curators: Participants who use the native token to signal which on-chain data sources (e.g., data from a specific parachain or smart contract) are valuable and high-quality. Indexers use this curation signal to decide which data to index. Curators earn a portion of the query fees for the data they signal on.
• On-Chain Logic: A set of smart contracts or a dedicated Substrate pallet that governs the network. This on-chain component manages indexer registration, staking and slashing, the curation market, and the flow of payments between participants.

Proof of Indexing (PoI): The Verifiability Layer

In a trustless, decentralized network, a consumer cannot blindly trust the data returned by an anonymous indexer. The network needs a mechanism to ensure data is correct and verifiable. This is achieved through a Proof of Indexing (PoI).⁴⁴

The PoI is a cryptographic commitment, typically a Merkle root, that represents a digest of the indexer's data state at a specific block. The process works as follows:

1. Commitment: Indexers periodically compute a PoI for their indexed data and publish it on-chain.
2. Query & Proof: When a consumer queries an indexer, the indexer returns the requested data along with a Merkle proof.
3. Verification: The consumer (or a lightweight client acting on their behalf) can use the Merkle proof to verify that the returned data is consistent with the PoI that was published on-chain.

This mechanism creates a trust-minimized data market. Consumers can cryptographically verify the integrity of the data they receive, and the on-chain logic can slash any indexer who is caught publishing an invalid PoI.

Table: Roles and Tokenomics in a Decentralized Indexer Network

The following table outlines the economic model that aligns incentives and secures the network. This cryptoeconomic design is essential for the long-term sustainability and security of a decentralized protocol.⁴⁷

Role	Primary Action	Incentive Mechanism	Slashing Condition (Penalty)
Indexer	Runs indexer software, processes queries, stakes tokens.	Query Fees, Issuance Rewards.	Serving incorrect data (invalid PoI), significant downtime.
Curator	Stakes tokens on specific data sources to signal their value.	Share of query fees from the data sources they signal on.	Signaling on malicious or low-quality data sources (potential loss of stake).
Delegator	Stakes tokens on behalf of an Indexer to increase their stake-weight.	A share of the Indexer's earnings (query fees and rewards).	If the chosen Indexer is slashed, a portion of the delegated stake is also slashed.
Consumer	Pays query fees to retrieve data for their dApp or analysis.	Access to fast, reliable, and verifiable on-chain data.	N/A (Pays for service).

Conclusion and Future Horizons

This report has laid out a comprehensive architectural blueprint for a Substrate indexer in Rust, designed to meet the stringent demands of the Polkadot ecosystem. The proposed four-layer, metadata-driven architecture directly addresses the core challenges of performance, adaptability to runtime upgrades, and resilience. By leveraging the power of Rust, the subxt library's dual static/dynamic capabilities, and a robust PostgreSQL/TimescaleDB persistence layer, this design provides a solid foundation for building a best-in-class indexing solution.

The path to implementation can be phased, beginning with the development of the core single-node indexer, focusing on the critical logic for handling runtime upgrades and reorgs. From there, high-availability features can be layered on, followed by the gradual transition towards a federated and ultimately a fully decentralized network.

Looking forward, this architecture opens up several exciting possibilities. The structured and queryable data it produces is an ideal substrate for advanced analytics and the training of AI models, positioning it as a key enabler for the emerging generation of data-hungry AI agents in the Web3 space.⁴⁹ The framework can be extended to support cross-chain indexing, correlating XCM messages and events across multiple parachains to provide a unified view of the entire Polkadot network.⁴⁹ Finally, by integrating with light-client technologies like

smoldot and Substrate Connect, the ingestion layer could evolve to become even more decentralized, removing its reliance on trusted RPC nodes and instead verifying chain data directly through cryptographic proofs.⁵² Ultimately, the indexer described herein is not merely a tool for data retrieval but a foundational component for unlocking the full potential of a decentralized, multi-chain future.

Works cited

1. A curated list of awesome projects and resources related to the Substrate blockchain development framework. - GitHub, accessed July 19, 2025, https://github.com/polkadot-developers/awesome-substrate
2. Subsquid: Making the Next Generation of Web3 Possible - Cyber Academy, accessed July 19, 2025, https://cyberacademy.dev/blog/69-subsquid-making-the-next-generation-of-web3-possible
3. Chronicle: Blockchain Indexer Built in Rust [Blazing Fast ] | by Developer Uche - Medium, accessed July 19, 2025, https://developeruche.medium.com/chronicle-blockchain-indexer-built-in-rust-blazing-fast-fff45ba60a97
4. Introduction to Polkadot SDK | Polkadot Developer Docs, accessed July 19, 2025, https://docs.polkadot.com/develop/parachains/intro-polkadot-sdk/
5. The Most Complete Introduction to Substrate Development Tools for Developers | by OneBlock+ | Medium, accessed July 19, 2025, https://medium.com/@OneBlockplus/the-most-complete-introduction-to-substrate-development-tools-for-developers-9584a7b8361
6. Auditing Substrate Based Systems in Rust - CertiK, accessed July 19, 2025, https://www.certik.com/resources/blog/auditing-substrate-based-systems-in-rust
7. Internal Workings of Substrate Lesson | Rise In, accessed July 19, 2025, https://www.risein.com/courses/polkadot-fundamentals-and-substrate-development/internal-workings-of-substrate
8. What are some differences between Substrate and Cosmos SDK? - Polkadot Forum, accessed July 19, 2025, https://forum.polkadot.network/t/what-are-some-differences-between-substrate-and-cosmos-sdk/1354
9. Node Interaction · Guide - Kusama Network, accessed July 19, 2025, https://guide.kusama.network/docs/build-node-interaction
10. substrate_parser - Rust - Docs.rs, accessed July 19, 2025, https://docs.rs/substrate_parser/
11. Basics & Metadata - polkadot{.js}, accessed July 19, 2025, https://polkadot.js.org/docs/api/start/basics/
12. polkadot_sdk_docs::polkadot_sdk::substrate - Rust - Parity, accessed July 19, 2025, https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/polkadot_sdk/substrate/index.html
13. Deprecation and Removal of Substrate Native Runtime Optimization · Issue #7288 - GitHub, accessed July 19, 2025, https://github.com/paritytech/substrate/issues/7288
14. [Guide] How to upgrade your runtime to the latest version of Polkadot SDK and not die trying, accessed July 19, 2025, https://forum.polkadot.network/t/guide-how-to-upgrade-your-runtime-to-the-latest-version-of-polkadot-sdk-and-not-die-trying/13016
15. Upcoming Metadata V16 - Features to include in V16 - Tech Talk - Polkadot Forum, accessed July 19, 2025, https://forum.polkadot.network/t/upcoming-metadata-v16-features-to-include-in-v16/8153
16. Getting started using Rust and subxt for Polkadot data extraction - Tech Talk, accessed July 19, 2025, https://forum.polkadot.network/t/getting-started-using-rust-and-subxt-for-polkadot-data-extraction/7652
17. Parallel Processing in Rust - Medium, accessed July 19, 2025, https://kartik-chauhan.medium.com/parallel-processing-in-rust-d8a7f4a6e32f
18. Optimization adventures: making a parallel Rust workload 10x faster with (or without) Rayon, accessed July 19, 2025, https://gendignoux.com/blog/2024/11/18/rust-rayon-optimized.html
19. Polkadot via Substrate Sidecar API - Blockdaemon Docs, accessed July 19, 2025, https://docs.blockdaemon.com/docs/polkadot-via-substrate-sidecar-api
20. Indexing & Reorgs. In this article, we unpack the… | by Envio - Medium, accessed July 19, 2025, https://medium.com/@envio_indexer/indexing-reorgs-326f7b6b13ba
21. Introducing Chainhook: a Reorg-Aware Transaction Indexer for Bitcoin and Stacks, accessed July 19, 2025, https://www.hiro.so/blog/introducing-chainhook-a-reorg-aware-transaction-indexer-for-bitcoin-and-stacks
22. Chain Reorganization Strategy · Issue #408 · blockscout/blockscout - GitHub, accessed July 19, 2025, https://github.com/poanetwork/blockscout/issues/408
23. subxt - Rust, accessed July 19, 2025, https://tidelabs.github.io/tidext/subxt/index.html
24. Subxt Rust API | Polkadot Developer Docs, accessed July 19, 2025, https://docs.polkadot.com/develop/toolkit/api-libraries/subxt/
25. subxt::dynamic - Rust, accessed July 19, 2025, https://tidelabs.github.io/tidext/subxt/dynamic/index.html
26. scale-decode - Encoding - Lib.rs, accessed July 19, 2025, https://lib.rs/crates/scale-decode
27. paritytech/substrate-archive: Blockchain Indexing Engine - GitHub, accessed July 19, 2025, https://github.com/paritytech/substrate-archive
28. RocksDB: Not A Good Choice for a High-Performance Streaming Platform : r/rust - Reddit, accessed July 19, 2025, https://www.reddit.com/r/rust/comments/1e9rmxv/rocksdb_not_a_good_choice_for_a_highperformance/
29. Embedded Key-value database - 2024. : r/rust - Reddit, accessed July 19, 2025, https://www.reddit.com/r/rust/comments/1dsmj9d/embedded_keyvalue_database_2024/
30. Compare PostgreSQL vs. RocksDB vs. Yugabyte in 2025 - Slashdot, accessed July 19, 2025, https://slashdot.org/software/comparison/PostgreSQL-vs-RocksDB-vs-Yugabyte/
31. Building Blockchain Apps on Postgres - TigerData, accessed July 19, 2025, https://www.tigerdata.com/blog/building-blockchain-apps-on-postgres
32. timescale/timescaledb: A time-series database for high-performance real-time analytics packaged as a Postgres extension - GitHub, accessed July 19, 2025, https://github.com/timescale/timescaledb
33. Transaction in postgres - Rust - Docs.rs, accessed July 19, 2025, https://docs.rs/postgres/latest/postgres/struct.Transaction.html
34. rust-postgres/tokio-postgres/src/transaction.rs at master · sfackler/rust-postgres - GitHub, accessed July 19, 2025, https://github.com/sfackler/rust-postgres/blob/master/tokio-postgres/src/transaction.rs
35. Transactional Outbox Pattern Benefits - Apiumhub, accessed July 19, 2025, https://apiumhub.com/tech-blog-barcelona/transactional-outbox-pattern/
36. Implementing the Outbox Pattern from Scratch by Following DDD - Stackademic, accessed July 19, 2025, https://blog.stackademic.com/implementing-the-outbox-pattern-from-scratch-by-following-ddd-9972eae4f1ab
37. DSL (Domain Specific Languages) - Rust By Example, accessed July 19, 2025, https://doc.rust-lang.org/rust-by-example/macros/dsl.html
38. DSL (rust_by_example_src/Domain Specific Languages) - The Rust Programming Language, accessed July 19, 2025, https://kuanhsiaokuo.github.io/the-rust-programming-book-khk/rust_by_example_src/macros/dsl.html
39. Ask HN: Have you used SQLite as a primary database? - Hacker News, accessed July 19, 2025, https://news.ycombinator.com/item?id=31152490
40. Blockchain Indexing Protocol: How It Works & Benefits - Webisoft, accessed July 19, 2025, https://webisoft.com/articles/blockchain-indexing-protocol/
41. Blockchain Indexer Protocol: How it Works? - IdeaUsher, accessed July 19, 2025, https://ideausher.com/blog/blockchain-indexer-protocol-how-it-works/
42. The Graph Whitepaper: Streamlining Data Processing Across Storage Networks, accessed July 19, 2025, https://www.cryptopolitan.com/the-graph-whitepaper-streamlining-data/
43. Tokenomics di The Graph Network | Docs, accessed July 19, 2025, https://thegraph.com/docs/it/resources/tokenomics/
44. Indexing Overview | Docs | The Graph, accessed July 19, 2025, https://thegraph.com/docs/en/indexing/overview/
45. thegraph.com, accessed July 19, 2025, https://thegraph.com/docs/en/indexing/overview/#:~:text=What%20is%20a%20proof%20of,be%20eligible%20for%20indexing%20rewards.
46. Do indexers sign their indexed data for verification? - The Graph - Expert Q&A, accessed July 19, 2025, https://thegraph.peera.ai/experts/1-2217/do-indexers-sign-their-indexed-data-for-verification
47. Understanding DeFi Tokenomics: Revolutionizing Finance, accessed July 19, 2025, https://www.findas.org/blogs/defi-tokenomics
48. SubQuery Network Explained: Decentralised Indexing for the Future of Web3 Indexing, accessed July 19, 2025, https://subquery.medium.com/subquery-network-explained-decentralised-indexing-for-the-future-of-web3-indexing-1e58dbf4255d
49. SubQuery Example Project — Multi-Chain Indexing in Polkadot, accessed July 19, 2025, https://subquery.medium.com/subquery-example-project-multi-chain-indexing-in-polkadot-e175c1e023fb
50. Subsquid, accessed July 19, 2025, https://www.sqd.ai/
51. SubQuery Indexer - Archway Docs, accessed July 19, 2025, https://docs.archway.io/developers/developer-tools/subquery
52. Light Clients | Polkadot Developer Docs, accessed July 19, 2025, https://docs.polkadot.com/develop/toolkit/parachains/light-clients/