<h2 id="context">Context</h2>
<h3 id="motivation">Motivation</h3>
<p>Several months ago, <a href="https://github.com/subcoin-project/subcoin">Subcoin</a>, a Rust Bitcoin Full Node implementation, received a grant from the Web3 Foundation as a showcase of the <code>polkadot-sdk</code> framework. The project aimed to implement a decentralized Bitcoin fast sync mechanism, enabling Bitcoin users to quickly sync with the network tip by directly downloading the Bitcoin state (UTXO set) from the Subcoin P2P network in a decentralized manner.</p>
<p>However, limitations in the Substrate framework prevented the realization of this vision. The primary issue was Substrate's inability to import large states (several gigabytes or more) downloaded during state sync without encountering Out-of-Memory (OOM) errors. As a result, we could only demonstrate the Bitcoin fast sync functionality by syncing to an intermediate block height, falling short of the original objective.</p>
<h3 id="background">Background</h3>
<p>Since completing the Subcoin w3f grant, I have been actively working to address the challenges related to state sync. Through extensive investigation, we identified the root cause of the OOM issue: Substrate attempts to construct the entire trie in memory during the state import process (<a href="https://github.com/paritytech/polkadot-sdk/issues/5053#issuecomment-2325449167">Issue #5053</a>).</p>
<p>With guidance from the <code>polkadot-sdk</code> maintainers, a rough plan to resolve the OOM issue and introduce persistent state sync support has been formulated. Preliminary efforts have already yielded encouraging results, laying the groundwork for a comprehensive solution.</p>
<p>Based on the progress made so far, I submitted a follow-up grant proposal to the w3f grants program. However, after careful consideration, the reviewers recommended the treasury funding route to better support this work (<a href="https://github.com/w3f/Grants-Program/pull/2436#issuecomment-2479329146).">https://github.com/w3f/Grants-Program/pull/2436#issuecomment-2479329146).</a></p>
<h2 id="problem">Problem</h2>
<p>The current fast sync feature in Substrate suffers from significant limitations, making it unusable or impractical in certain scenarios:</p>
<ol>
<li>
<p><strong>OOM issue on importing the large state:</strong></p>
<p>Substrate fails to import states downloaded via state sync due to the OOM issue when the chain state is large. This limitation affects not only Subcoin but any project using Substrate, including Polkadot itself. For example, the Astar network has already encountered this issue (<a href="https://github.com/paritytech/polkadot-sdk/issues/5053">Issue #5053</a>).</p>
</li>
<li>
<p><strong>Lack of persistent state sync support:</strong></p>
<p>State sync in Substrate currently lacks the ability to persist partial progress, requiring the entire process to restart in case of interruptions.</p>
</li>
</ol>
<p>These challenges must be addressed to unlock the full potential of Substrate's advanced sync strategies, including warp sync and fast sync, particularly for projects with large chain states.</p>
<p><strong>Additional Notes</strong></p>
<p>In addition to the above issues, I have identified and addressed several other fast sync-related bugs during my work, with some fixes already implemented and others still in progress. These improvements aim to enhance the overall reliability and functionality of Substrate's sync mechanisms.</p>
<ul>
<li><a href="https://github.com/paritytech/polkadot-sdk/pull/5774">https://github.com/paritytech/polkadot-sdk/pull/5774</a></li>
<li><a href="https://github.com/paritytech/polkadot-sdk/pull/5635">https://github.com/paritytech/polkadot-sdk/pull/5635</a></li>
<li><a href="https://github.com/paritytech/polkadot-sdk/pull/5703">https://github.com/paritytech/polkadot-sdk/pull/5703</a></li>
</ul>
<h2 id="objective">Objective</h2>
<p>The goal of this proposal is to enhance Substrate's capabilities by addressing the key state sync obstacles listed in <a href="https://github.com/subcoin-project/subcoin/issues/56">Subcoin Issue #56</a>. Specifically, this work will focus on resolving the OOM issue and implementing persistent state sync, ensuring that Substrate-based projects can reliably utilize advanced synchronization strategies, regardless of chain state size.</p>
<p><a href="https://docs.google.com/document/d/1ppZQTst4k2QPjN4MKfJI0goZ88L2_nW5fHQJ3PUfLSk/edit?usp=sharing">Full proposal</a></p>

Context
-------

### Motivation

Several months ago, [Subcoin](https://github.com/subcoin-project/subcoin), a Rust Bitcoin Full Node implementation, received a grant from the Web3 Foundation as a showcase of the `polkadot-sdk` framework. The project aimed to implement a decentralized Bitcoin fast sync mechanism, enabling Bitcoin users to quickly sync with the network tip by directly downloading the Bitcoin state (UTXO set) from the Subcoin P2P network in a decentralized manner.

However, limitations in the Substrate framework prevented the realization of this vision. The primary issue was Substrate's inability to import large states (several gigabytes or more) downloaded during state sync without encountering Out-of-Memory (OOM) errors. As a result, we could only demonstrate the Bitcoin fast sync functionality by syncing to an intermediate block height, falling short of the original objective.

### Background

Since completing the Subcoin w3f grant, I have been actively working to address the challenges related to state sync. Through extensive investigation, we identified the root cause of the OOM issue: Substrate attempts to construct the entire trie in memory during the state import process ([Issue #5053](https://github.com/paritytech/polkadot-sdk/issues/5053#issuecomment-2325449167)).

With guidance from the `polkadot-sdk` maintainers, a rough plan to resolve the OOM issue and introduce persistent state sync support has been formulated. Preliminary efforts have already yielded encouraging results, laying the groundwork for a comprehensive solution.

Based on the progress made so far, I submitted a follow-up grant proposal to the w3f grants program. However, after careful consideration, the reviewers recommended the treasury funding route to better support this work ([https://github.com/w3f/Grants-Program/pull/2436#issuecomment-2479329146).](https://github.com/w3f/Grants-Program/pull/2436#issuecomment-2479329146\).)

Problem
-------

The current fast sync feature in Substrate suffers from significant limitations, making it unusable or impractical in certain scenarios:

1.  **OOM issue on importing the large state:**
    
    Substrate fails to import states downloaded via state sync due to the OOM issue when the chain state is large. This limitation affects not only Subcoin but any project using Substrate, including Polkadot itself. For example, the Astar network has already encountered this issue ([Issue #5053](https://github.com/paritytech/polkadot-sdk/issues/5053)).
    
2.  **Lack of persistent state sync support:**
    
    State sync in Substrate currently lacks the ability to persist partial progress, requiring the entire process to restart in case of interruptions.
    

These challenges must be addressed to unlock the full potential of Substrate's advanced sync strategies, including warp sync and fast sync, particularly for projects with large chain states.

**Additional Notes**

In addition to the above issues, I have identified and addressed several other fast sync-related bugs during my work, with some fixes already implemented and others still in progress. These improvements aim to enhance the overall reliability and functionality of Substrate's sync mechanisms.

*   [https://github.com/paritytech/polkadot-sdk/pull/5774](https://github.com/paritytech/polkadot-sdk/pull/5774)
*   [https://github.com/paritytech/polkadot-sdk/pull/5635](https://github.com/paritytech/polkadot-sdk/pull/5635)
*   [https://github.com/paritytech/polkadot-sdk/pull/5703](https://github.com/paritytech/polkadot-sdk/pull/5703)

Objective
---------

The goal of this proposal is to enhance Substrate's capabilities by addressing the key state sync obstacles listed in [Subcoin Issue #56](https://github.com/subcoin-project/subcoin/issues/56). Specifically, this work will focus on resolving the OOM issue and implementing persistent state sync, ensuring that Substrate-based projects can reliably utilize advanced synchronization strategies, regardless of chain state size.

[Full proposal](https://docs.google.com/document/d/1ppZQTst4k2QPjN4MKfJI0goZ88L2_nW5fHQJ3PUfLSk/edit?usp=sharing)

Treasury Proposal: Funding for Enhancing Substrate Fast Sync for Large Chain States

Treasury

Propose and approve a spend of treasury funds.

## Dispatch Origin

Must be [`Config::SpendOrigin`] with the `Success` value being at least
`amount` of `asset_kind` in the native asset. The amount of `asset_kind` is converted
for assertion using the [`Config::BalanceConverter`].

## Details

Create an approved spend for transferring a specific `amount` of `asset_kind` to a
designated beneficiary. The spend must be claimed using the `payout` dispatchable within
the [`Config::PayoutPeriod`].

### Parameters
- `asset_kind`: An indicator of the specific asset class to be spent.
- `amount`: The amount to be transferred from the treasury to the `beneficiary`.
- `beneficiary`: The beneficiary of the spend.
- `valid_from`: The block number from which the spend can be claimed. It can refer to
  the past if the resulting spend has not yet expired according to the
  [`Config::PayoutPeriod`]. If `None`, the spend can be claimed immediately after
  approval.

## Events

Emits [`Event::AssetSpendApproved`] if successful.

- **Problem Statement:** Substrate's current fast sync feature faces challenges like OOM errors when importing large states and lacks persistent state sync support, impacting projects like Subcoin and Polkadot.
- **Significance:** Resolving these issues is crucial to unlock Substrate's advanced sync strategies for projects with large chain states.
- **Proposed Solution:** Address the OOM error and introduce persistent state sync in Substrate to enhance synchronization capabilities.
- **Expected Outcomes:** Improved reliability and functionality of Substrate's sync mechanisms, enabling seamless utilization of advanced synchronization strategies.

#1301 Treasury Proposal: Funding for Enhancing Substrate Fast Sync for Large Chain States

No Active Proposals