Build a Resumable Ethereum Log Indexer with eth_getLogs

The difficult part of a log indexer often appears after its first successful run. A process crashes after writing half a range. A retry inserts duplicates. A recent block is replaced while the checkpoint still says its height is complete.

For a backend engineer building a durable event history, the unit of work should include the query, persisted results, block identity, and committed progress. eth_getLogs supplies a read interface. Recovery is a responsibility of the system around it.

Define exactly which events belong in the index

Begin with a network identifier, contract set, event schema, and starting block. Version that definition. A checkpoint for one filter cannot safely stand in for a different filter added later.

The Solidity ABI event specification distinguishes indexed parameters in topics from non-indexed values in data. For a non-anonymous event, topic zero identifies its signature. Build filters from the contract ABI; a field encoded only in data cannot simply be moved into a topic filter.

This request illustrates a narrow, 256-block query. The contract address is synthetic; replace it with the contract you intend to index. The topic is the signature of Transfer(address,address,uint256), which by itself does not identify a particular token standard or contract:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "eth_getLogs",
  "params": [{
    "fromBlock": "0x1000",
    "toBlock": "0x10ff",
    "address": "0x0000000000000000000000000000000000000001",
    "topics": [
      "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
    ]
  }]
}

The execution API specification defines inclusive range bounds for eth_getLogs. The next adjacent chunk therefore begins at 0x1100. A chunk size of 256 is an example for developing the recovery loop, not a universal provider limit. Event density, response-size limits, retained history, and endpoint policy all affect useful chunk sizes.

Commit progress with the data it describes

Use a checkpoint that means every block through a particular point has been processed for a particular filter version. Do not define it as the highest block containing a matching event: quiet blocks also need a completed position.

A practical record includes chainId, filterVersion, completedBlockNumber, and completedBlockHash. Store raw log identity and payload before applying derived business calculations. An identity such as chain ID, block hash, and log index makes replay idempotent while preserving the distinction between competing blocks.

The following is storage pseudocode; database.transaction, upsertLog, and saveCheckpoint are application-specific operations:

query one bounded range
validate the RPC envelope and every returned log
reconcile the range against the selected chain view
begin database transaction
  upsert each log by (chainId, blockHash, logIndex)
  save completed range end and its block hash
commit database transaction
advance the next range only after commit succeeds

If the process dies before commit, replay the same range. If it dies after commit, reload the checkpoint. When publishing downstream messages, use a transactional outbox or another deliberate coordination mechanism so database progress cannot outrun required downstream work.

An empty successful result can complete a range. A timeout, malformed result, pruned-history error, or interrupted response cannot. Never turn an exception into an empty array just to keep the loop moving.

Adapt the query without losing the cursor

Start with a modest span and a bounded number of in-flight requests. On an endpoint's range or response-size error, reduce the span and retry the same start block. On throttling, respect the documented retry behavior and slow the worker. Authentication errors need configuration repair; repeatedly halving a range will not resolve them.

Keep concurrency and checkpoint advancement separate. Workers may finish out of order, but the durable checkpoint should advance only through a contiguous prefix of completed ranges. Otherwise a later success can skip an earlier failure after restart.

Record range size, response bytes, elapsed time, returned log count, and error class. Increase the span cautiously only when your own workload supports it. If even a one-block query is too large or unavailable, stop that partition with a visible failure rather than skipping it. Splitting a filter into smaller contract or topic partitions may help, provided their union preserves the intended event set.

Treat recent history as replaceable

Choose a chain-progress policy for the scan target, resolve it to a concrete header, and keep that target stable for the run. A numeric height alone does not preserve block identity through a reorganization.

EIP-234 provides a blockHash filter as an alternative to fromBlock and toBlock. Where supported, querying each selected block by hash removes ambiguity about which block supplied its logs, including an empty result. Hash selection does not itself establish that the block is still canonical.

For range scans near the head, maintain and reconcile block headers, validate each returned log's block hash against that view, and recheck progress as the chain advances. Separate RPC calls are not an atomic snapshot. A recent-window reconciliation loop must revisit already processed provisional blocks and invalidate derived rows when their hashes change.

A chosen rollback window is an operational bound, not proof that deeper changes cannot happen. If the common ancestor lies beyond retained recovery information, pause and rebuild from a verified checkpoint or follow an explicit incident procedure. Avoid quietly keeping a mixture of old and replacement history.

Exercise recovery before increasing throughput

Test a crash before commit, a crash after commit, duplicate delivery, an empty range, a failed middle chunk, and a replaced recent block. Verify both the raw rows and the derived totals after replay. A count that looks plausible is weaker evidence than matching stored block identities and completed ranges.

The getLogs documentation introduces the request shape. Durable indexing adds a clear answer to three operational questions: what is complete, which chain view it belongs to, and how the application repairs it after interruption.