-
Notifications
You must be signed in to change notification settings - Fork 1.1k
perf: remove blocking calls from POI queries #6300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
8045272 to
b6cd094
Compare
Remove synchronous `block_on` call in `resolve_proof_of_indexing` which was blocking tokio worker threads while waiting for database queries. Before this change, each POI query blocked an entire tokio worker thread. After this change, POI queries properly yield to the async runtime while waiting for database I/O, allowing the connection pool to be fully utilized. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Change the publicProofsOfIndexing resolver to process all POI requests in parallel using `future::join_all` instead of sequentially in a for loop. Co-Authored-By: Claude Opus 4.5 <[email protected]>
b6cd094 to
888a12e
Compare
|
The first commit is great and something that absolutely needs to happen. For the second commit, I am not quite clear on what the motivation is - are PoI requests slow? The worry I have is that when |
|
The motivation is POI verification tooling for dispute investigation. When investigating POI discrepancies (e.g., for arbitration disputes), we need to compute POIs across large block ranges to identify where divergence occurred. With sequential processing, throughput is bottlenecked at ~2k POIs/second even though the database connection pool has capacity. The publicProofsOfIndexing endpoint accepts batch requests (up to 10), but processing them sequentially means we're not utilizing available parallelism. Checking every block sequentially on a network with 100m blocks takes 13+ hours at ~2k blocks/second. |
Pre-fetch all block hashes in a single batch query before parallel POI processing, reducing database round-trips from 10+ to 1-2 per batch. - Add block_hashes_by_block_numbers batch method to ChainStore trait - Add get_public_proof_of_indexing_with_block_hash to StatusStore trait - Modify resolver to group requests by network and batch-fetch hashes - Pass pre-fetched hashes to avoid redundant lookups during parallel POI Co-Authored-By: Claude Opus 4.5 <[email protected]>
8b2a3fc to
4f89b3f
Compare
The third commit addresses this, now pre-fetch all block hashes in a single batch query before parallel processing, reducing DB round-trips. |
Have you tried a binary search for PoI's? I assume you are looking for the point where the PoI's for some subgraph diverge between two indexers. Instead of a binary search, you could actually request multiple PoI's across some range (start with |
|
Binary search works well for finding divergence points between two indexers. This PR speeds up computations for finding which block a given POI actually corresponds to if it doesnt correspond to the block that it is claimed to (can be used to prove an indexer claimed rewards for block N, but only synced to block X). Under the assumption that an indexer force closes with a valid POI for a block x < n, but submits that block x POI for block n. |
|
Thanks for the explanation - if you use this to find the block for which a given POI was generated, how about we add a route in the resolver that just does that? That might be much faster than just speeding up POI lookup. |
Summary
Two optimizations for POI query handling in the index-node resolver:
Make
resolve_proof_of_indexingasync - Removes synchronousblock_oncall that was blocking tokio worker threads while waiting for database queries. POI queries now properly yield to the async runtime.Parallelize
publicProofsOfIndexingrequests - Changes sequential for-loop tofuture::join_allfor concurrent execution of batch POI requests.Changes
server/index-node/src/resolver.rsresolve_proof_of_indexing:fn→async fn,block_on(fut)→fut.awaitresolve_public_proofs_of_indexing: sequential loop →future::join_allfutureimport fromgraph::futures03