Notice: Polkadot has migrated to AssetHub. Balances, data, referenda, and other on-chain activity has moved to AssetHub.Learn more
Subsquid - Public data indexing Infrastructure for Polkadot and Kusama (Q4 2023)
Proponent: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Beneficiary: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Contact Details:
Subsquid Labs GmbH 6302 Zug, Switzerland
Short description: Ongoing development and maintenance costs for public Subsquid Archives and the Giant Squid API
Archive raw data: Archive Infrastructure Metrics - October, November, December
Giant Squid raw data: Giant Squid Infrastructure Metrics - October, November, December
Requested DOT: 37,174.81 DOT + 12,082.84 DOT = 49 257.65 DOT = $288,157.30
Previous proposals: https://polkadot.polkassembly.io/referenda/372
Motivation
A) Archives
Archives are an important piece of Polkadot and Kusama data infrastructure provided by Subsquid. They provide access to on-chain data in GraphQL format for all major parachains on Kusama and Polkadot, with more chains getting support as the ecosystem grows. They are being used as a data source for Squids and for efficient data exploration and ad-hoc queries. An up-to-date list of Archives is kept in a public Archive Registry
The Archives, as performant data sources of historical on-chain data, are critical to the operation of backend APIs run by major projects in the Polkadot ecosystem (the list is not exhaustive):
- Polkassembly
- Talisman
- RMRK
- SubWallet (+ Giant Squid API)
- KodaDot
- ChainSafe
- Multix
- Phala dApp
- Subsocial + Polkaverse
- Stellaswap
- Beamswap
- Polkascan (+ Giant Squid API)
- Calamar (+ Giant Squid API)
- Moonsama/Exosama
- Reef
- Manta Wallet
- ZooDAO
- Zenlink
All archives can be accessed and queried via an web intuitive GraphQL interface and are often used as developer friendly replacement of a block explorer.
For example, here’s a query for the last ten transfers on Kusama blockchain run against the Kusama Archive endpoint
Over the fourth Quarter of 2023 we've provided a free public data indexing service for Substrate blockchains (some archives have been dismissed, new ones have been added), so that everyone interested in the projects can publicly access indexed data on Substrate Events, Extrinsics, Storage Items and EVM logs. Here are projects (parachains and mainnets aiming for future parachains slots) we currently support:
- acala
- aleph-zero
- aleph-zero-testnet
- altair
- amplitude
- astar
- bajun
- basilisk
- bifrost
- bifrost-polkadot
- bitcountry-pioneer
- calamari
- centrifuge
- cere
- collectives
- composable-finance
- crab
- crust
- darwinia
- elysium
- elysium-testnet
- equilibrium
- exosama
- foucoco
- frequency
- frequency-testnet
- gear-testnet
- gmordie
- hashed
- heiko
- hydradx
- integritee-network
- interlay
- invarch-tinkernet
- joystream
- kabocha
- karura
- khala
- kilt
- kintsugi
- kusama
- kylin
- litentry
- litmus
- manta
- matrix
- moonbase
- moonbeam
- moonriver
- moonsama
- myriad
- myriad-testnet
- origin-trail
- pangolin
- pangoro
- parallel
- peaq
- pendulum
- phala
- picasso
- polkadex
- polkadot
- polymesh
- reef
- reef-testnet
- robonomics
- rococo
- shibuya
- shiden
- soonsocial
- sora
- sora-testnet
- statemine
- statemint
- t0rn
- tanganika
- ternoa
- tidechain
- turing
- vara
- westend-collectives
- xsocial
- xx-network
- zeitgeist
B) Giant Squid
Giant Squid is a set of open-source GraphQL APIs built with Squid SDK and maintained by Subsquid Labs. The APIs cover the most commonly used data for Pokadot, Kusama and a growing list of major parachains and is used by explorers (such as Calamar and Polkascan) and wallets (e.g. SubWallet).
As of Q1 2024, Subsquid has sunsetted support for Giant Squid, with LimeChain taking over its further development, detailed in the proposal at https://polkadot.polkassembly.io/treasury/758
Achievements
- Enhanced Query Packing: Due to optimization and better filtering, archive queries have become more accurate and stable. Now, significantly fewer queries are needed to retrieve the necessary chain data, reducing the load on the infrastructure. It is worth noting that the response size has increased as fewer queries are required to obtain the same data.
- ArrowSquid release: The main feature introduced by the ArrowSquid update is the new ability of the processor to ingest unfinalized blocks directly from a network node, instead of waiting for the archive to ingest and serve it first. The processor can now handle forks and rewrite the contents of its database if it happens to have indexed orphaned blocks. This allows Subsquid-based APIs to become near real-time and respond to the on-chain activity with subsecond latency. Another major feature introduced by ArrowSquid is the support for transaction execution receipts, traces and state diffs.
- Decentralizing Archives: launched public testnet on CoinList with over 40.000 participants (https://coinlist.co/subsquid-testnet). Several thousand indexers in the Subsquid testnet were deployed to index Moonbeam, a Polkadot parachain.
- Subsquid is powering ≈ 650 live projects in Q4 2023
- The biggest Polkadot projects have fully migrated or are in the process of finishing the migration from FireSquid to ArrowSquid, including: Polkassembly, Subsocial, HydraDX, KodaDot, Talisman, Multix Multisig , RMRK, Apillon, StellaSwap
- Support continues to be added for new and upcoming Polkadot projects, including new and burgeoning partnerships and Archive implementations with Polimec Protocol, Acurast, Watr, and others.
Statistics and data
Over the course of Q4 2023:
- Served a total of 1,325,237,833 (~1.32 billion) queries to the substrate based archives in Q4 2023
- Served ~500m monthly archive requests for substrate based archives
- The highest daily total responses have been 19.15m in a single 24 hour period for substrate
- The decrease in queries reflects an ongoing shift to Firesquid, largely attributed to a substantially improved query structure that optimizes performance.
Giant Squid is a service we introduced in Q1 2023, and has been launched in mid-January, here are two charts showing its quick adoption by the community, as an index of its success:
Costing
This proposal includes the combined costs for running Giant Squid and Archives for Polkadot, Kusama and their Parachains,which amounts to a total of 49 257.65 DOT. Raw data for Archives can be seen here. Raw data for Giant Squid can be seen here. The proposal is submitted towards the Polkadot treasury.
Here is a summary of the running costs for Archives, in a tabular format:
Giant Squid running costs:
Comments (4)
Requested
Proposal Passed
Big fan of subsquid.
Can you describe your revenue sources for the polkadot eco side of the business?
Like, is the treasury your only source of funding for it, or are you also raising from fees?
And will this change going forward?
Will access to your data lake continue to be free way into the future?
BTW, you've got a bad SSL cert in the link you posted and have done since last week at least :/
Worse, behind the bad cert page is a 404.
@Mork
Hey! Thanks for reaching out.
This is a retroactive proposal for Q4 2023, and the GraphQL endpoint you mention was sunset in February 2024 as soon as all the teams migrated to the new version (ArrowSquid) and ArrowSquid SDK. In particular, the GraphQL endpoint above is disable since February 2024.
The community agreement we had discussing the retroactive treasury funding for Q2 2023 and Q3 2023 boils down to the following points:
Thus, starting from Q1 2024 the archive data for parachains is funded externally, and we expect that only the infra for public goods chains + relay chains (Kusama + Polkadot) will be funded by the treasury proposals.
Subsquid has been doing an excellent job in the ecosystem. Keep up the good work.