Treasury Proposal: QMDB Core Integration for Substrate Storage
Enhancement Proponent: 12K97oseeBxoWnyYRU3DJkxKTiDGFBh1pK4jLyuUJzYHkqdD
Date: [29.01.2025]
Requested DOT: 5 000
Short description: Initial integration of QMDB (<a href="https://github.com/LayerZero-Labs/qmdb)">https://github.com/LayerZero-Labs/qmdb)</a> core functionality into Substrate (<a href="https://github.com/paritytech/polkadot-sdk).">https://github.com/paritytech/polkadot-sdk).</a> Project Category/Type: Storage and Performance Optimization
Previous Treasury Proposals:
Context of the Proposal This proposal focuses on the first phase of QMDB integration, specifically implementing core storage functionality without iterators and prefixes with limited support or non-optimal support. This foundational work will enable future enhancements while maintaining compatibility with existing Substrate systems. 
<h3 id="projectdetails">Project Details</h3>
Technology stack:
<ul>
<li>RUST language for QMDB integration with Substrate</li>
<li>DevOps tools for deployment and monitoring</li>
</ul>
Documentation:
<ul>
<li>Core components and architecture documentation to be developed</li>
<li>Integration specifications and API documentation</li>
<li>Performance benchmarking methodology</li>
</ul>
Research and PoC:
<ul>
<li><a href="https://github.com/QuantumFusion-network/spec/issues/16">https://github.com/QuantumFusion-network/spec/issues/16</a></li>
<li><a href="https://qfnetwork.xyz/litepaper">https://qfnetwork.xyz/litepaper</a></li>
</ul>
What this project will not provide:
<ul>
<li>Full prefix search functionality</li>
<li>Production-ready indexing solution</li>
<li>Advanced iterator implementation</li>
</ul>
<h2 id="teammembers">Team members</h2>
<ul>
<li>Team Lead: Vas Soshnikov</li>
<li>Team Member: Andrew Orlov</li>
</ul>
<h3 id="teamsexperience">Team's experience</h3>
Previous Polkadot grants:
<ul>
<li><a href="https://github.com/kaonone/polkahub-monorepo">https://github.com/kaonone/polkahub-monorepo</a></li>
<li><a href="https://substrate.io/ecosystem/projects/polkahub/">https://substrate.io/ecosystem/projects/polkahub/</a></li>
</ul>
<h3 id="teamcoderepos">Team Code Repos</h3>
<ul>
<li><a href="https://github.com/QuantumFusion-network">https://github.com/QuantumFusion-network</a></li>
<li><a href="https://github.com/andor0">https://github.com/andor0</a></li>
<li><a href="https://github.com/dedok">https://github.com/dedok</a></li>
</ul>
Problem Statement Current Substrate storage implementation faces several fundamental challenges that this initial integration phase aims to address: Current crate &lsquo;sp-trie&rsquo; implementation requires optimisation Inefficient key-based operations affecting basic transaction processing Suboptimal transaction verification process
Key Limitations of QMDB
<ul>
<li>Does not support iterators and prefix searches</li>
<li>Non-limitation, mostly its design of QMDN, requires additional memory (RAM, Disk) for indexing (indexer)</li>
</ul>
Proposal Objectives &amp; Solutions Objectives
<ul>
<li>Integrate QMDB as the proof storage engine within Substrate</li>
<li>Enable Merkle Proof-based transaction verification</li>
<li>Benchmark QMDB inside our DevNet (<a href="https://dev.qfnetwork.xyz).">https://dev.qfnetwork.xyz)</a></li>
</ul>
Milestones, Timelines, and Budgets Milestone 1. Implement QMDB storage traits Working prototype integrated with Substrate 3 weeks 2 000 DOT 2. Develop a storage indexing mechanism In-memory B-Tree or sorted sparse list 2 weeks 1 500 DOT 3. Conduct full system testing All Substrate tests pass 1 week 750 DOT 4. Deploy, benchmark and optimise Live on DevNet with monitoring 1 week 750 DOT
Risks and Mitigation Risk Mitigation Strategy
<ul>
<li>QMDB may not be production-ready</li>
<li>Extensive DevNet testing before MainNet deployment</li>
<li>Substrate&rsquo;s evolving architecture may cause compatibility issues</li>
<li>Freeze the Substrate version until integration is stable</li>
<li>Increased resource consumption due to indexers</li>
<li>Optimise RAM/disk usage and implement caching mechanisms</li>
</ul>
Resources Required
<ol style="list-style-type: decimal;">
<li>Senior Rust Developer: Expertise in Substrate and low-level systems</li>
<li>Senior Rust Developer: Specializing in database architecture</li>
<li>DevNet Environment: Dedicated nodes for rigorous testing</li>
</ol>
Success Criteria 
<ul>
<li>Substrate nodes use QMDB for transaction storage</li>
<li>All Substrate and pallet-specific tests pass successfully</li>
<li>Performance benchmarks demonstrate improved transaction processing times</li>
<li>DevNet deployment runs stably with real-time monitoring and alerting</li>
</ul>
&nbsp;
Payment Conditions 
Payments will be made in instalments based on milestone completion. Each instalment will be subject to the successful delivery of agreed milestones. A detailed progress report will be submitted on Polkassembly/Subsquare.
Why Polkadot Network? 
Polkadot&rsquo;s focus on modular and scalable blockchain infrastructure makes it an ideal candidate for advanced storage optimisations like QMDB integration. By enhancing storage efficiency, this proposal aligns with Polkadot&rsquo;s long-term vision of high-performance, decentralized systems.
Additional References <a href="https://layerzero.network/publications/QMDB_13Jan2025_v1.0.pdf">QMDB Technical Paper </a> <a href="https://github.com/LayerZero-Labs/qmdb">QMDB GitHub Repository</a>

Propose and approve a spend of treasury funds.

## Dispatch Origin

Must be [`Config::SpendOrigin`] with the `Success` value being at least `amount`.

### Details
NOTE: For record-keeping purposes, the proposer is deemed to be equivalent to the
beneficiary.

### Parameters
- `amount`: The amount to be transferred from the treasury to the `beneficiary`.
- `beneficiary`: The destination account for the transfer.

## Events

Emits [`Event::SpendApproved`] if successful.

- **Problem Statement:** Current Substrate storage faces challenges like inefficient key-based operations and suboptimal transaction verification.
- **Significance:** QMDB integration aims to address these challenges by enabling Merkle Proof-based transaction verification and optimizing transaction processing.
- **Proposed Solution:** Integrate QMDB into Substrate, develop a storage indexing mechanism, conduct testing, and optimize for deployment.
- **Expected Outcomes:** Improved transaction processing times, successful Substrate node usage of QMDB, and stable DevNet deployment.

#1404 QMDB Core Integration for Substrate Storage Enhancement

No Active Proposals