Notice: Polkadot has migrated to AssetHub. Balances, data, referenda, and other on-chain activity has moved to AssetHub.Learn more
Treasury Proposal by Rust Syndicate: Uptest funding proposal
Proposal: Uptest Funding Proposal
Beneficiary: Rust Syndicate LLC 16Ziip8mK44sh7uKFkZgHbapxoKrRxriZaDdzqNAPW9Wr6x4
Date: 31.10.2022
Requested DOT: ($60000) 9254 DOT
Contact: Filip Kalebo (blockchain@firosolutions.com)
Short description:
Uptest aims to be the tool for testing Pallet functionality before and after runtime upgrades! Making it easier for developers to quickly pinpoint needed runtime migrations and test pallet functionality before and after runtime upgrades.
Google docs full proposal link:
https://docs.google.com/document/d/1dJ43Bl3jYJ7LDA96_Hskyelqf9K2Zw4qJvdD53w6TwE/edit?usp=sharing
This is a Polkadot treasury proposal by Rust Syndicate LLC, a software company that aims to take on the responsibility to develop Uptest. A tool that enables developers to test upgrade chain functionality throughout a runtime upgrade.
All milestones will be published as releases on Github and crates.io under the MIT license.
Background:
During previous work, part of our team was tasked with upgrading a 16-month older forked substrate chain to a newer polkadot-v0.9.x release.
This was a massive upgrade and during this journey, a lot of time was spent dry-running runtime upgrades, going through migrations and testing pallet upgrades. After utilizing the majority of the available tools in the substrate space, no tool filled our use case.
Problem Statement:
Performing runtime upgrades for substrate developers is a necessary step in keeping modern chains up to date with dependencies and implementations. By previously working on massive dependency upgrades, a lot of substrate tooling was tested and evaluated. The majority of the tools that are available are either end of life / simple proof of concept code. We want to have a tool that quickly lets us test upgraded pallet functionality before and after a runtime upgrade happens. There is currently no tool available in the substrate ecosystem that lets the user simply write custom extrinsic tests, connect to a live chain, fork the chain to a local instance, launch the first tests, submit the runtime upgrade, track the changes throughout the runtime upgrade, verify that the storage migrations kicked in, run second load of tests and give us a detailed verdict.
Proposed Solution:
We propose “Uptest”. A tool that allows you to simply write tests, have Uptest submit the tests using ws/rpc, perform the runtime upgrade with the new changes, verify that the chain is running and perform after chain upgrade tests for the pallet tests. Uptest aims to be the “curl” for runtime upgrades. Making it a lot easier to perform runtime upgrades and dry-run new pallet features in local environments with real on-chain data. With Uptest we want to give the substrate community a universal tool in the form of a command line tool testing the upgraded code changes before and after the runtime upgrade has been submitted.
Testing storage migrations:
When upgrading a pallet to a newer version that changes functionality and storage types, we need to verify that the storage migrations are executed successfully and that our chain keeps
With Uptest we could write a test functionality to interact with the chain throughout the runtime
(first test)Interact with the pallet and submit data to be stored in the old format
write the storage migration for the new pallet and compile the new runtime to wasm
Uptest submits the runtime upgrade runtime upgrade
Uptest analyzes the changes made to the chain
(second test) Check that we can interact with upgraded pallet functionality and verify that the earlier submitted data was migrated in a correct way
Display a detailed changelog and give verdicts of the test results
Uptest allows you to simply let you write a test file, give Uptest the test file, point it to the live chain and have Uptest handle all the heavy lifting, giving developers a smooth experience testing new functionality throughout a runtime upgrade.
Comments (4)
As a runtime developer, I have felt the pain your proposal tries to solve. So thanks a lot in advance!
Could you please differentiate Uptest from the
try-runtimeCLI, Centrifuge's FUDGE and Substrate SimNode? I think this will be quite helpful to understand the whole picture.E.g., you can write tests for
try-runtimefeature to check for type changes etc. but it would not be feasible to do/maintain this for every pallet a runtime implements for every runtime upgrade. IIUC, Uptest still requires a manual setup for each pallet, otherwise it does not detect type changes and recommends migrations?Great questions!
This is true that extended tests would be needed to be added for each pallet. However, we want to make the integration with uptest and libuptest as high level as possible. Allowing
tests to be created with a simple, short high-level syntax.
Let me address this one by one:
try-runtime CLI:
Try-runtime requires the chain to be using a newer version of substrate and has
added the try-runtime implementation into the running node. You need to add it to the node
implementation, add it as a feature, and then it's ready to be used. Try-runtime is great if the chain has support for the implementation for testing the migrations that kick in. However, it
doesn't fit a use case where a specific pallet behavior needs to be tested with pre-submitted data before. We want to build uptest to be a more universal solution.
Centrifuge's FUDGE:
I think FUDGE is awesome and mustermeiszer has helped me personally solve a de-couple problem a certain pallet had in the past. Fudge is super useful for manually parsing a node's db to get a clear vision of what's happening in the storage area. Fudge is a great library for
"messing around with the storage". However, we want to build a command line tool that let's
you simply create a test file, testing let's say a pallet using extrinsic and other logic. Fork a live chain, replace the keys, runs the first set of tests, execute a runtime upgrade, run the second set of tests, and give us the output.
With a syntax like this:
uptest -c mychain -t tests/check_balance_transfers.rs --rpc mychain --fork -u runtime.wasm
Fudge is also a library where libuptest is the library and uptest is the cli tool that utilizes its functionality. So we have a focus on building a smooth cli with extended functionality that Fudge does not provide.
Maybe Uptest could spread its wings to a db parse to quickly extract events and similar data, then we could implement fudge in a future release as a side feature.
SimNode:
Simnode is a full node environment that can be used to simulate a live environment where
runtime upgrade migrations could be tested. The scope of Simnode is to simulate a node, meanwhile, the scope for Uptest is to be a cli tool that lets you dry-run pallet tests before and after a runtime migration. Uptests seek to be a clean way of forking live chains, submitting tests before and after a runtime upgrade happens, and calculating the useful output.
Extra note:
Uptest aims to be the tool that handles the heavy lifting of keeping track the phases of a runtime upgrade, out of the box support for just giving the rpc endpoint to uptest and it will interact with it, fork the chain and at the same time keep track of.
I was just wondering if the testing is mostly concerned with storage types being changed and if not, could you clarify what would be the main focus? Because, at least from my point of view, logical changes would be covered by unit-test on the pallet side, right?
Thanks for the feedback @mustermeiszer. Storage types being changed is a crucial thing to keep track on. One of the biggest use cases for uptest is to allow developers to test that new pallet functions through out the runtime upgrade. Pallet tests could be created with mock interfaces and run as unit tests and than executed with cargo test. This does not let us interact with live chain data, with Uptest we want to be able to test pallet functionality with real live chain data and verify that our code upgrades handles the chain well after a runtime upgrade has taken place.