View All Discussion

Patract Hub's treasury report for Europa (Sandbox) v0.1

lipeng
5 years ago

Patract Hub (https://patract.io) develops local open source toolkits and one-stop cloud smart IDE, committed to provide free development toolkits and infrastructure services for the entire smart contract ecosystem. 8 weeks ago, we applied a treasury proposal #15 for Europa v0.1 , and now we have finished the development (https://github.com/patractlabs/europa) and recorded a [YouTube demo] (https://www.youtube.com/watch?v=z1SWq0vfgYY). We are delayed behind the required schedule for 2 weeks and we are very sorry about that.

Introduction

Summary of Europa's future plan:

v0.1: Have an independent runtime environment to facilitate more subsequent expansion directions.


The independent runtime environment of excluded nodes can be expanded more without the constraints of the node environment and WASM compilation, and can be easily integrated with other components. In this version, it is more like simulating the Ganache project in Ethereum ecosystem, enabling contract developers to develop without having to build a contract blockchain. Developers can quickly fire up a personal Substrate chain, which can be used to run tests, execute commands, and inspect state while controlling how the chain operates.

How to verify v0.1: Youtube demo & Github source

  • Build and run in an independent executable file
  • Can start from different workspaces, and a new workspace is an empty chain
  • Can receive extrinsics to set_code, deploy and call contracts directly
  • Can interact with Redspot
  • Can use RPC to jump over some blocks and revert to a specified height to revert states

Europa is kind of another implementation for Substrate client in our design. We know that the runtime of a blockchain is the business logic that defines its behavior, and the Substrate runtime need to run by an executor and environment. So that we design the executor and environment more like a "sandbox" to run a Substrate runtime.

In v0.1, the primary target is to establish this "sandbox" environment to run a runtime. Then only in this way, we can start to fork the wasmi and other components to extends the features or the contracts execution environment.

Features

For this purpose, we implement following features for a sandbox framework:

1. Europa sandbox framework is another implementation for Substrate client.

In the process of implementation, we move parts of the requirements in v0.3 to v0.1. We regard Europa as a "framework" at start, so that all substrate runtime could integrate Europa client directly.

Same as Substrate, Europa use bin/europa as an example for blockchain projects which based on Substrate to show how to use Europa, just like what bin/node and bin/node-template do in Substrate. Developers just need to create a new runtime directory, remove the pallets about consensus ("pallet-babe", "pallet-grandpa" and "pallet-session”), because Europa doesn't have the consensus process. Developers need also remove build.rs for runtime crate, and implement bin which contains service.rs and other parts. Then the sandbox can run for this blockchain runtime.

Thus, we can find that the structure of bin/europa is as same as bin/node in Substrate. And on the other hand, bin/europa is also our particular implementation for pallet-contracts smart contracts platform sandbox. It can be used directly to debug smart-contracts for ink!.

2. Sandbox doesn’t need consensus, and can produce block directly.

Sandbox is just used to execute the runtime, so that we remove all components about consensus. Thus, we use sc-consensus-manual-seal crate to produce block. This crate have a good abstract, so we do not need to fork it. We just modify the part of commands stream for manual-seal. The async stream could receive information from different place (e.g. transaction pool, RPC and others) to drive seal-engine to produce blocks.

3. Sandbox doesn’t need WASM.

As sandbox need to be easily debugged, we remove all WASM components. As we currently know, WASM causes many problems when debugging in details. At first, We need to extend many features for low level libraries, so if we base on the runtime to do something which need to be compiled into WASM, we will meet many unexpected problems.

Europa runtime should remove build.rs in runtime crate and remove [build-dependencies] in runtime crate (cargo.toml file). On the other hand, a sandbox doesn’t need WASM features.

4. Provide state-kv database.

In Substrate, there is a way to provide all existing state under a block, but there is no way to export the modified state-kvs for a block. However, the developers are concerned mostly about the state changes between blocks. So that, they can check whether the changes match their expectations for debugging.

Thus, in europa sandbox, we store the mapping of blockhash and state kvs in the state-kv database, so that developers could export the state changes to look up the details.

You can run the export command while the europa node is running:

# in a shell window
$ ./target/debug/europa
Nov 12 17:10:14.524  INFO Europa Dev Node    
Nov 12 17:10:14.524  INFO ✌️  version 0.1.0-7b4463c-x86_64-linux-gnu    
Nov 12 17:10:14.524  INFO ❤️  by patract labs <https://github.com/patractlabs>, 2020-2020    
Nov 12 17:10:14.524  INFO 📋 Chain specification: Development    
# ...
Nov 12 17:21:23.544  INFO Accepted a new tcp connection from 127.0.0.1:44210.    
Nov 12 17:21:32.238  INFO 🙌 Starting consensus session on top of parent 0xc7e1ce585807b34b7fecabe1242cafb2628c958b984ec0aee7727cdd34117529    
Nov 12 17:21:32.252  INFO 🎁 Prepared block for proposing at 1 [hash: 0x0109608217316a298c88135cf39a87cc31c37729fbe567b4a1a9f8dcdb81ebeb; parent_hash: 0xc7e1…7529; extrinsics (2): [0x2194…baf8, 0x0931…58bb]]    
Nov 12 17:21:32.267  INFO Instant Seal success: CreatedBlock { hash: 0x0109608217316a298c88135cf39a87cc31c37729fbe567b4a1a9f8dcdb81ebeb, aux: ImportedAux { header_only: false, clear_justification_requests: false, needs_justification: false, bad_justification: false, needs_finality_proof: false, is_new_best: true } }    

# open another shell window, the following command would print the changes for block height 1
$ ./target/debug/europa state-kv 1 
Nov 12 15:53:27.699  INFO modified state for block:0x6c119a8f7de42e330aca8b9d3587937aacbbc203cc21650b60644c2f2d33e7fb    
Nov 12 15:53:27.699  INFO       key:26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac|value:[DELETED]   
Nov 12 15:53:27.699  INFO       key:26aa394eea5630e07c48ae0c9558cef70a98fdbe9ce6c55837576c60c7af3850|value:05000000
# ...

And Europa also has a RPC method to export the state-kvs while running:

You can call Europa’s RPC like this:

{
	"id":100, 
	"jsonrpc":"2.0", 
	"method":"europa_modifiedStateKvs", 
	"params":[ 1 ]
}

will return:

{
    "jsonrpc": "2.0",
    "result": {
        "0x26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac": null,
        "0x26aa394eea5630e07c48ae0c9558cef70a98fdbe9ce6c55837576c60c7af3850": "0x05000000",
        // ...
    }
}

5. Provide some special RPC method for sandbox.

One feature of sandbox is that the developers can operate blockchain state easily. Assuming a situation like this:

A developer is developing a module which has a call of foo, while the call foo needs other basic states (e.g. An account could be a validator only if he had registered his information on chain). Then, when this developer does a test and the result doesn’t match his expectation (had modified related states), something must be wrong for foo. He found the reason and fixed it, and wanted to try it again to test, however the current state is not the old state.

So the sandbox provides a way to "go back" to the old state, allowing developers to do a rollback operation. Thus we provide a RPC method, named: europa_backwardToHeight. It allows developers to recover to any old state as the current best state, then the following operations are based on that old state.

By contrast, with this "backward” RPC method, we also provide a "forward" method, named europa_forwardToHeight. It sends a command to let Europa to produce a batch of blocks to the height which is assigned by this RPC call. This is very useful for the developments which are related to the block number. Many DeFi DApps need to allow users to claim something after a specific block. So, using this RPC to do integration test is very suitable.

And for the feature #4, we also provide a RPC, named europa_modifiedStateKvs to export the modified state kvs.

As above, in current version, Europa provides 3 RPC methods:

  • europa_forwardToHeight: allow Europa to produce empty blocks to reach to the assigned height.
  • europa_backwardToHeight: allow Europa to revert to the assigned height and state.
  • europa_modifiedStateKvs: allow Europa to export modified state kvs and child state kvs for a block.

6. Provide workspace to isolate different execution environments.

Substrate could use the command -d/--base-path to assign a directory to store data, thus different environment could use different directory to distinguish. But -d directory is not recorded, so that developers should manage those different paths by themselves. Europa provide a concept of "workspace" to do this work based on the "base path". Developers could use command -w/--worksace to assign different workspace directory to run different Europa node instance. Europa also provides commands to operate "workspaces”, like setting default workspace, listing all existed workspace, or deleting a workspace.

In our workspace management, developers can switch different environments by setting different workspace. By using this feature, developers can switch between different environments to do test easily. On the other hand, this feature provides a friendly way for integration testing.

Start Europa on your own computer

Build

The building process for this project is as same as Substrate.

When building finished, current executable file in target directory is named europa.

Run

Run Europa

The following is builded in debug mode. If you want to build in release mode, using release to replace debug.

$ ./target/debug/europa 
# if you what to specify a directory, add `-d` or `--base-path`
$ ./target/debug/europa -d database

Access Europa
Now, you can use the Official Portal (https://polkadot.js.org/apps/) to access Europa:

  • click the left tab and switch to DEVELOPMENT - Local Node.

  • click Settings - Developer, and paste:

{
  "Address": "AccountId",
  "LookupSource": "AccountId"
}
  • click "save" . Then, you can do a transfer as normal.

Other third parties can access Europa like accessing any Substrate node.

Export modified state kvs

$ ./target/debug/europa state-kv 1
# if you have specified a directory, add `-d` or `--base-path`
$ ./target/debug/europa state-kv -d database 1

RPC calls

RPC call is same as other RPCs in Substrate. You can do POST requests by rpc/ws like this:

{
	"id":1, 
	"jsonrpc":"2.0", 
	"method":"europa_forwardToHeight", 
	"params":[  5  ]
}

Just replace "method" and "params" to corresponding parameters.

Workspaces

Specify another workspace

$ ./target/debug/europa -w another-workspace
# if you have specify a directory, add `-d` or `--base-path`
$ ./target/debug/europa -d database -w another-workspace
Set default workspace

Stop the Europa, then execute:

# another-workspace is the workspace name that we want to set as default.
$ ./target/debug/europa workspace default another-workspace
Nov 12 17:28:41.980  INFO Current default workspace:    
Nov 12 17:28:41.981  INFO       default    
Nov 12 17:28:41.981  INFO     
Nov 12 17:28:41.981  INFO Set [another-workspace] as default workspace.

Then, start Europa. Europa will use "another-workspace" as default workspace.

$ ./target/debug/europa
# ...
Nov 12 17:29:33.862  INFO 💾 Database: RocksDb at .sub/another-workspace/chains/dev/db    
Nov 12 17:29:33.862  INFO 📖 Workspace: another-workspace | Current workspace list: ["default", "another-workspace"]    
Nov 12 17:29:33.862  INFO ⛓  Native runtime: europa-1 (europa-1.tx1.au1)

Delete workspace

$ ./target/debug/europa workspace delete another-workspace
Nov 12 17:30:49.549  INFO Current default workspace:    
Nov 12 17:30:49.549  INFO       another-workspace    
Nov 12 17:30:49.549  INFO     
Nov 12 17:30:49.550  INFO Delete workspace [another-workspace].    
Nov 12 17:30:49.550  INFO       delete default record: [another-workspace]    
Nov 12 17:30:49.550  INFO       delete workspace:[another-workspace] from workspace list

What we have implemented for v0.1

As I describe above, we have implemented all features required in proposal.

  • Build and run in an independent executable file

Europa can be compiled and run directly.

  • Can start from different workspaces, and a new workspace is an empty chain

Europa can start from different workspaces and do management.

  • Can receive extrinsics to set_code, deploy and call contracts directly

Europa can run any Substrate runtime, and currently Europa has already integrated pallet-contracts

  • Can interact with Redspot

Europa can interact with any third parties, including “Official Portal”. Thus Redspot can also access Europa as a normal node.

  • Can use RPC to jump over some blocks and revert to a specified height to revert states

Europa has provided some RPCs to support this.

That's all, we think Europa can become a very useful tool for Substrate developers. We need more time to do the research for Europa v0.2, then we will post the detailed proposal in about a week.

Comments (0)

PleaseLogin to comment

Comments (0)

PleaseLogin to comment
    Polkassembly - Patract Hub's treasury report for Europa (Sandbox) v0.1 #166