Proponent: <a href="https://polkadot.subscan.io/account/13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE">Subsquid Lab Official</a> - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Beneficiary: <a href="https://polkadot.subscan.io/account/13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE">Subsquid Lab Official</a> - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Contact Details: Subsquid Labs GmbH, 6302 Zug, Switzerland - <a href="mailto:mf@subsquid.io">mf@subsquid.io</a>
Short description: Ongoing development and maintenance costs for public Subsquid Archives and the Giant Squid API
Archive raw data: <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQqax-cS0HO8iECupffOlhQnarfMO4ry_ML1arn_4kif6Ykei-bs-LqUpmdWHXT1c13IVfVB-MednjO/pubhtml">Archive Infrastructure Metrics - April, May, June</a>
Giant Squid raw data: <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vSzBVpkSUqr9vof92Th7xdbQiHYpuqhnSLM1Rpa8P_ABwZzGYcNm3I7QTWKE5bw4lIZA3YtOATg55cV/pubhtml">Giant Squid Infrastructure Metrics - April, May, June</a>
Requested DOT: 43995.35 DOT + 14258.31 DOT = 58253.66 DOT = $ 233,014.67
Previous proposals: <a href="15">https://polkadot.polkassembly.io/referenda/15</a>
<h3 id="motivation">Motivation</h3>
A) Archives
Archives are an important piece of Polkadot and Kusama data infrastructure provided by Subsquid. They provide access to on-chain data in GraphQL format for all major parachains on Kusama and Polkadot, with more chains getting support as the ecosystem grows. They are being used as a data source for <a href="https://app.subsquid.io/squids">Squids</a> and for efficient data exploration and ad-hoc queries. An up-to-date list of Archives is kept in a public <a href="https://github.com/subsquid/archive-registry/blob/main/archives.json">Archive Registry</a>.
The Archives, as performant data sources of historical on-chain data, are critical to the operation of backend APIs run by major projects in the Polkadot ecosystem (the list is not exhaustive):
<ul>
<li>Polkassembly</li>
<li>Talisman</li>
<li>RMRK</li>
<li>SubWallet (+ GiantSquid API)</li>
<li>KodaDot</li>
<li>Phala dApp</li>
<li>Subsocial + Polkaverse</li>
<li>Stellaswap</li>
<li>Beamswap</li>
<li>Polkascan (+ GiantSquid API)</li>
<li>Calamar (+ GiantSquid API)</li>
<li>Moonsama/Exosama</li>
<li>Reef</li>
<li>Manta Wallet</li>
<li>ZooDAO</li>
<li>Zenlink</li>
</ul>
All archives can be accessed and queried via an web intuitive GraphQL interface and are often used as developer friendly replacement of a block explorer.
For example, here&rsquo;s a query for the last ten transfers on Kusama blockchain run against the <a href="https://kusama.explorer.subsquid.io/graphql">Kusama Archive endpoint</a>.
<img src="https://lh3.googleusercontent.com/u/0/docs/ADP-6oF28QRWrAQx4x6nJNPA_fSg4c73ukxfq-ATwlJOaeMkFcveUEN3abJyXlkg0M5tgxMSU9_hY00ovl7JB0PIaZV25wlpNy9QiYTah8HYt1QcfnmUiLPz61vWbAjWe6yEsIrl_dVS68yi_skDsIhpTHtk0NqWhZ2HakawVgzv13cB0Zz2GFF1uD4WduGKh1cT71w21kf8XhbV9NU8bYkYjJht4UPz5CEmrc-J8V-1g6zjdMK0TWcbRtZE469tyZurbsR6hrxtz3AgcT-yBnIaFSR1RjuDt9cU4ON1EuhAYFfbt_hxKS5eJAmkJnoRN9b98lm3z-ENAW4F4lGD4eaefmLjW0Ni96jVwyx2-0qpj_FCU8M4Jv-ADArSOogQU6E4WMjJ0IXO0G0wvqcpJritTjnVfaMg1BWVP4coevffowL396wjU7-php3xXPT1cYJHRi9hob4LQQUYXeBz_oGA2mcZ-VTw8fonH_Q5H3MWRqaaPttS618JzrkVaeEGScCMF9qJ5tMwBeMFQ9rcG9njN1frHKgaco97oSsZsSygw-OsuCAyBta90ZmHWeHo3ZJqtRNBdE33rEzJlBvlVo4uCVbMnjE_64s5bz2-OQczY8bEdjD6v7g4awrLXZXwx1zdRnHkc5TZsnG65LhbYXzLus-n-drRCSuk-67GihaJrMul5IoKRgHaUJekuQVaFNnL4YhKAGOotb7tvvNCLdDtpfnwZdg_bhGrALNf8b4hGBfJL1_cHPeTeKj2IrsCBVz1cm7EQp6fx5VTXD3oJnwLkx09vmauHqlftCgqfzYjSvL5E2yWRHI5NZa5igjdV9UNPUvj_LB4-NvVx_RCofmUNjC62qj8AUYJH8z7RhIvGmpvpdesVlFgFqlB-VMhPd8emdb6l1089w7jrc3bODqhQuNtEbM" alt="" />
Over the seconf Quarter of 2023 we've provided a free public data indexing service for ~95 Substrate blockchains (some archives have been dismissed, new ones have been added), so that everyone interested in the projects can publicly access indexed data on Substrate Events, Extrinsics, Storage Items and EVM logs. Here are projects (parachains and mainnets aiming for future parachains slots) we currently support:
<ul>
<li>acala</li>
<li>agung</li>
<li>aleph-zero</li>
<li>aleph-zero-testnet</li>
<li>altair</li>
<li>amplitude</li>
<li>astar</li>
<li>astar-evm</li>
<li>bajun</li>
<li>basilisk</li>
<li>bifrost</li>
<li>bifrost-polkadot</li>
<li>bitcountry-pioneer</li>
<li>calamari</li>
<li>centrifuge</li>
<li>cere</li>
<li>clover</li>
<li>collectives</li>
<li>composable-finance</li>
<li>crab</li>
<li>crust</li>
<li>darwinia</li>
<li>debionetwork</li>
<li>dorafactory</li>
<li>efinity</li>
<li>elysium</li>
<li>elysium-testnet</li>
<li>equilibrium</li>
<li>exosama</li>
<li>foucoco</li>
<li>frequency</li>
<li>frequency-testnet</li>
<li>gear-testnet</li>
<li>gemini-2a-testnet</li>
<li>gmordie</li>
<li>hashed</li>
<li>heiko</li>
<li>hydradx</li>
<li>imbue</li>
<li>integritee-network</li>
<li>interlay</li>
<li>invarch-tinkernet</li>
<li>joystream</li>
<li>kabocha</li>
<li>karura</li>
<li>khala</li>
<li>kilt</li>
<li>kintsugi</li>
<li>kusama</li>
<li>kylin</li>
<li>litentry</li>
<li>litmus</li>
<li>mangata-x</li>
<li>manta</li>
<li>moonbase</li>
<li>moonbase-evm</li>
<li>moonbeam</li>
<li>moonbeam-evm</li>
<li>moonriver</li>
<li>moonriver-evm</li>
<li>origin-trail</li>
<li>pangolin</li>
<li>parallel</li>
<li>peaq</li>
<li>pendulum</li>
<li>phala</li>
<li>picasso</li>
<li>pichiu</li>
<li>polkadex</li>
<li>polkadot</li>
<li>polymesh</li>
<li>reef</li>
<li>reef-testnet</li>
<li>robonomics</li>
<li>rococo</li>
<li>shibuya</li>
<li>shiden</li>
<li>snow</li>
<li>soonsocial</li>
<li>sora</li>
<li>sora-testnet</li>
<li>statemine</li>
<li>statemint</li>
<li>subsocial</li>
<li>t0rn</li>
<li>tanganika</li>
<li>ternoa</li>
<li>tidechain</li>
<li>turing</li>
<li>vara</li>
<li>westend-collectives</li>
<li>xsocial</li>
<li>xx-network</li>
<li>zeitgeist</li>
</ul>
B) Giant Squid Giant Squid is a set of open-source GraphQL APIs built with Squid SDK and maintained by Subsquid Labs. The APIs cover the most commonly used data for Pokadot, Kusama and a growing list of major parachains and is used by explorers (such as <a href="https://calamar.app/">Calamar</a> and <a href="https://explorer.polkascan.io/">Polkascan</a>) and wallets (e.g. <a href="https://www.subwallet.app/">SubWallet</a>).
The Giant Squid currently consists of two families of endpoints outlined below. The API endpoints for each network and the current status are available at the <a href="https://docs.subsquid.io/giant-squid-api/statuses/">Status page</a>.
Explorer API. The endpoint URLs follow the convention <a href="https://squid.subsquid.io/gs-explorer-${network}/graphql.">https://squid.subsquid.io/gs-explorer-${network}/graphql.</a> Support queries on:
<ul>
<li>Blocks</li>
<li>Events</li>
<li>Extrinsics</li>
<li>Calls</li>
</ul>
Statistics API. The endpoint URLs follow the convention <a href="https://squid.subsquid.io/gs-stats-${network}/graphql.">https://squid.subsquid.io/gs-stats-${network}/graphql.</a> Support queries on:
<ul>
<li>Validator/Collator data</li>
<li>Account balances</li>
<li>Staking statistics</li>
<li>Nomination pools statistics</li>
<li>Transfers statistics</li>
<li>Extrinsic statistics</li>
</ul>
<h3 id="achievements">Achievements</h3>
<ul>
<li>Subsquid Network in private testnet</li>
<li>Powering &asymp; 550 live projects in Q1 2023 ( &asymp; 650 as of August 2023) including substrate and evm</li>
<li>Archive requests increased by 6.5x (from ~160 million to ~1.05 billion)</li>
<li>Arrow Squid release: The main feature introduced by the ArrowSquid update is the new ability of the processor to ingest unfinalized blocks directly from a network node, instead of waiting for the archive to ingest and serve it first. The processor can now handle forks and rewrite the contents of its database if it happens to have indexed orphaned blocks. This allows Subsquid-based APIs to become near real-time and respond to the on-chain activity with subsecond latency.Another major feature introduced by ArrowSquid is the support for transaction execution receipts, traces and state diffs.</li>
</ul>
<h3 id="statisticsanddata">Statistics and data</h3>
<img src="https://lh4.googleusercontent.com/gifFP-DbmhdP_A-8mJ9wnLpL5BhMSEPFZcskNZVswYTSDNcbYF5eKEBBq4soKcb70-HDxbbYi4gZOrsUUEjA5SSEtXVcELRIMq_Rj74y76zmzr5y9JKse0KtSX-0vCMdoXJwrTiSeQhoJaXlOHfFPCM" alt="" />
<img src="https://lh3.googleusercontent.com/dUxinCvs3irjanC5b16Out7ZPF_YHl53oVSLDMDAQn_w3qP2m5mzdPxvYJRMn4qF73IhlsJypK5vMGDiwy2njv21ktJRr1OunRyfEkobURM2dx7HsclaKBkKgRutiZHdAhkk_SbwnyB62qMwUVqjtRk" alt="" />
Over the course of Q2 2023:
Over the course of Q2 2023: Served a total of 1,048,730,936 (~1 billion) queries to the substrate based archives. The highest daily total responses have been 13.6m in a single 24 hour period for substrate. The daily queries confirm a growing trend.
Giant Squid is a service we introduced in Q1 2023, and has been launched in mid-January, here are two charts showing its quick adoption by the community, as an index of its success:
<img src="https://lh3.googleusercontent.com/cziWhB9mPDdGxzn3f1lTxQunm-tHifzRttcXBbwQXBVKpDVwa2qiAZu1nL-tnPxgwbID98QNrz1unkL0lqI3E3HwqQPQyAZK2qwbStK-01RsKczomSv__f6-0aVEp8bK8Z5fJE1yCr7yDhusbD-vjl0" alt="" />
<img src="https://lh4.googleusercontent.com/SY8LpEMYcJO99eGJzX0myyqUivNa2V6Q5cQDazv1u0Ml4Ysy9iYPL-U3dh7qlb2Zk0yt5qRDFqPeqDBxhrjoUV1g9hdXATyDyhiucIV7RxR-ZGQvzQTB7s72QWskrOhNrVOFFWBNQZwBFmflRHk2Smo" alt="" />
<h3 id="costing">Costing</h3>
This proposal includes the combined costs for running Giant Squid and Archives for Polkadot, Kusama and their Parachains,which amounts to a total of 58253.66 DOT. Raw data for Archives can be seen <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQqax-cS0HO8iECupffOlhQnarfMO4ry_ML1arn_4kif6Ykei-bs-LqUpmdWHXT1c13IVfVB-MednjO/pubhtml">here.</a> Raw data for Giant Squid can be seen <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vSzBVpkSUqr9vof92Th7xdbQiHYpuqhnSLM1Rpa8P_ABwZzGYcNm3I7QTWKE5bw4lIZA3YtOATg55cV/pubhtml">here</a>.
The proposal is submitted towards the Polkadot treasury. Here is a summary of the running costs for Archives, in a tabular format:
<img src="https://lh4.googleusercontent.com/0W89VtHm9oPhU-MYCztyFb2jZJm80bKnQIhUz6VGZ7yW5iukf_92fnVQ6q7gxoESKl49xvFYd43IknvvqN_Lyqm544pprHGmwrZKYt3HiTOYS5bWnloW9290lVzjl6DUr_Q56NuZmPzpblKE-tGSr3Q" alt="" />
Giant Squid running costs:
<img src="https://lh4.googleusercontent.com/Vbg9RdfW_V1EiKWmzQ5k9Tl8HbGHGvIlWOKbEzON8lRVPG0a59ejiW2W8JlkK7UOzzk0klZYTCZikGfJiCWMQoTFi9XYPevHDwLZO6N9BhQBz7p02LqbFOckvlNoqvFW-UGDw9YgGLOhZZHucllPfAQ" alt="" />
You can read the full report<a href="https://docs.google.com/document/d/e/2PACX-1vRJlrLGnUVlaXnMBUGbbD450pOcockxe5Ld5jhfM1fo-K8Pxt4jE6FuHSy_nOKXRyyR_W81r_x6b1PX/pub"> here</a>.

- Problem Statement: The Polkadot and Kusama data infrastructure, provided by Subsquid Labs, requires ongoing development and maintenance for public Subsquid Archives and the Giant Squid API. The intended audience includes major projects in the Polkadot ecosystem that rely on these services.
- Significance: Subsquid Archives and the Giant Squid API are critical for the operation of backend APIs run by major projects in the Polkadot ecosystem. They provide access to on-chain data in GraphQL format for all major parachains on Kusama and Polkadot.
- Proposed Solution: Subsquid Labs is requesting a total of 58253.66 DOT ($233,014.67) from the Polkadot treasury to cover the combined costs for running Giant Squid and Archives for Polkadot, Kusama, and their Parachains.
- Expected Outcomes: The funding will ensure the continued operation and development of these essential services, supporting the growth of the Polkadot ecosystem. The services have already served a total of ~1 billion queries to the substrate-based archives in Q2 2023, indicating their significant value to the community.

#420 Subsquid - Public data indexing Infrastructure for Polkadot and Kusama (Q2 2023)

No Active Proposals