### The Kapex Parachain has been producing blocks since winning and parachain slot on Polkadot, but cannot be upgraded due to an error.

> [SRLabs](https://www.srlabs.de/) conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code that impacted the Transaction Payment pallet adapted by the team. Unfortunately the team won a slot on Polkadot before the audit could have been conducted and so the issue was not discovered until the audit completed. 

## Background
The standard Transaction Payment Pallet had been forked and adapted to record transaction fees into the accounting engine of the KAPEX Parachain. the original version of this code was changed by a new team developer to use a function reserved for off-chain workers instead of the original code but as the code compiled it was not spotted until the audit. It was thought that an upgrade would fix the issue.

## Upgrade failure
As the Transaction Payment Pallet is called during an upgrade to reserve funds (even though the actual upgrade is free of charge), the illegal code was unfortunately also called to record this reservation of fees causing an error, preventing the upgrade from taking place and therefore not allowing the parachain to be fixed.

**_In addition this also meant that the XCM security vulnerability discovered in late September also cannot be fixed either at this point._**

All attempts were made to try to fix this issue without having to go to the Council and Referendum process, with help provided with gratitude from the Kilt.io team and Parity team Bastian, Alejandro and Santiago.

The new [Manual Parachain Lock](https://github.com/paritytech/polkadot/pull/5451) cannot fix our chain because [the unlocking mechanism does not allow for centrally controlled chains to unlock the parachain from the relaychain side](https://github.com/paritytech/polkadot/issues/6196). 

The fix that worked was to apply a validation code substitution using `paras.forceSetCurrentCode()` executed on the relaychain once the parachain `chain spec` had been updated accordingly and applied to the collators in our test networks. This is the subject of this proposal.

## Testing the fix

- [X] Tested the fix (`paras.forceSetCurrentCode()`) on our [Relaychain development network](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frelaychain.totem.live#/parachains) connected to the Lego development parachain.

- [X] Upgraded the Lego Parachain to version [`v1.2.0`](https://github.com/totem-tech/totem-parachains/tree/kapex-v1.2.0-audit-fix) 

- [X] Upgrade Lego to [`v1.3.0`](https://github.com/totem-tech/totem-parachains/tree/kapex-v1.3.0-polkadot-v0.9.28-alignment)

- [X] Tested the fix on the [Rococo chain](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-rpc.polkadot.io#/accounts) connected to our [Stagex](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fs-ui.kapex.network#/explorer) production staging parachain with the help of Parity devs.

- [X] Upgraded the Stagex Parachain to version `v1.2.0`

- [X] Then upgrade Stagex to `v1.3.0`

- [X] Upgraded [docker Stagex nodes](https://hub.docker.com/layers/totemlive/totem-parachain-collator/staging/images/sha256-bc6fe0e61a347e2f64150aa2fd739bb5e0b12b0702c1b8a4f7da262b39328897?context=explore) to run version `v1.3.0`

## Production preparation
- [X] Decide the block number for the transition in Kapex [chainspec](https://github.com/totem-tech/totem-parachains/blob/main/res/kapex/kapex-parachain-raw.json) - Block number `400000`

- [X] Apply the [new production validation code](https://github.com/totem-tech/totem-parachains/blob/main/res/parachain-code-substitution-wasm.wasm) to Kapex `chainspec`

- [X] Tag the code repo from the main branch in the repo as [`kapex-v1.1.0-codeSub`](https://github.com/totem-tech/totem-parachains/releases/tag/kapex-v1.1.0-codeSub)

- [X] Build a new [docker image](https://hub.docker.com/layers/totemlive/totem-parachain-collator/production/images/sha256-21857f3efd8496a57bd41646d728086173b622ac085f5204f48a015dd9e02310?context=explore) with the new Kapex `chainspec`.

- [X] Restart the Kapex Collators with the new `chainspec`.

- [X] Transfer funds to request code Substitution on Polkadot

- [X] Create Preimage hash on Polkadot `0x1ca2532cbb04d421f1b216f354354524feffd8540e7ca74b0938e58d2cdfc923` 

- [X] Create a proposal

- [X] Create Preimage on Polkadot

We are unfortunately unable to distribute our funds from the Crowdloan until this issue is fixed, so we look forward to the community supporting this proposal through the referendum as soon as possible. Thanks in advance.

The Totem Kapex Team

Set the storage for the parachain validation code immediately.

:
1. SRLabs conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code which prevented the parachain from being upgraded
2. A Manual Parachain Lock failed owing to the illegal code
3. The fix that worked was to apply a validation code substitution using `paras.forceSetCurrentCode()` executed on the relaychain once the parachain `chain spec` had been updated
4. This proposal, if passed, will allow the KAPEX Parachain to upgrade and fix the issue, allowing the team to distribute funds from the Crowdloan.

#85 Kapex Parachain - Validation Code Substitution to fix broken upgrade mechanism and permit XCM Security Vulnerability upgrade.