5 weeks ago, [Patract Hub](https://patract.io) applied a [treasury proposal](https://polkadot.polkassembly.io/motion/33) for Elara v0.2. Now, we have finished all the development work on time and recorded a Youtube [demo video](https://www.youtube.com/watch?v=KFowqts8S94). Please review our design, work, and enjoy the separated workspace free services (1M requests per day) online at https://elara.patract.io. We have also provided a shared limited bandwidth node RPC service for Polkadot ([wss://polkadot.elara.patract.io](wss://polkadot.elara.patract.io) and https://polkadot.elara.patract.io), you can add it directly into your project and https://polkadot.js.org/apps.

> ### Recap of the detailed design of Elara v0.2
> 
> - Implement the account space features, supporting developers to use Github as a third-party login system to create an account space.
> - Support the creating and management of multiple projects under the developer’s account space
> - Provide developers with detailed access statistics features on the single project dimension, showing statistical indicators such as daily and weekly requests, calling methods, and source of user groups.
> - Officially launch https://elara.patract.io, providing the community with public access services to the Polkadot mainnet

Now, let's show you the design and implementation of v0.2, and how to run and verify the work. With this micro-service architecture described blow, we can provide ** 5K requests per second ** and 500M requests per day for now. In the future, we will upgrade this architecture for ** 10 times improvements. **

## Architecture of Elara v0.2

We had learned from Infura's work model, so that each development account can apply for multiple `Projects` to divide different `Workspace`, and then assign different `Provider` to access chain nodes. Therefore, the user's smallest resource unit is a `Project`. We have assigned an independent `PID` for each project. In the index of all resources, the developer can use this `PID` to find the information corresponding to this `Project`, such as detailed information, daily or weekly access statistics, etc.

Therefore, the architecture of Elara v0.2 is roughly as follows:

![elara](https://user-images.githubusercontent.com/5023721/100614013-6a5fde00-3350-11eb-882f-43efe02e1ecd.jpg)

As can be seen from the above, in the backend management platform of Elara v0.2, the entire system is divided into 3 main modules:

* `Developer-Account`: The module to manage the user's account login status.
* `Stat`: The module for management and data statistics of `Project`.
* `API`: The `Route` module to manage user's `Requests`.

In the overall design, the three modules are designed into a microservice architecture system, which can be independently extended. Because these three modules all run in independent processes after they are started, in the future, individual modules can be upgraded in parallel, such as expansion, reconstruction, and replacement. Among them, the `Developer-Account` module and the `Stat` module share data through a shared database, and the `Stat` and `API` are decoupled through the message queue `kafka`. In the future, it will be easy to add more complex functional modules based on this design to make Elara play a more powerful effect.

These three modules are responsible for different functions:

### 1. Developer-Account module

This module manages user account information and maintains login status. This module currently uses `Redis` to cache user information and maintain the user's login status. This login status is shared with the `Stat` module. Currently, this module uses Github's `AccountID` information to request authorization to access Github to create users.

Therefore, the external access interfaces provided by the module after running are:
* GET `/auth/login`: Get login status information
* GET `/auth/logout`: logout, remove the user's login information.
* GET `/auth/github`: request authorization information from github
* GET `/auth/github/callback`: processing of callbacks authorized to github

### 2. Stat module

This module is mainly responsible for the management of `Project`, and counts the request information initiated by users from the `API` module in the dimensions of Project and users. This module is `Consumer` of `kafka` Message Middleware, which accepts request information sent from the `API` module and performs processing statistics. At the same time, the module shares the user's login status information with `Developer-Account` to check whether the user is still connected to Elara's webpage. In this module, each user can apply for the allocation of multiple `Projects`, representing a restricted resource unit in Elara, such as limiting the amount of requests and total bandwidth that a Project can accept per day.

The access interface of the `Stat` module uses `PID` as the basic element, and the `PID` of a project will be used as the specific node `Provider` of this project in the `API` module. In the process of using this `PID` to raise a node access request to the `API`, the developer will consume the resources managed under the project corresponding to this `PID` and accumulate the statistical information of this project.

Based on the above discussion, the `Stat` module is designed as follows:

* GET `/project/list`: List all project information under the currently logged-in user, and return:

```json
{
   "code": 0,
   "message": "",
   "data": [
       {
           "id": "b78a79f98a2bb1a991a357504a5b04c1",
           "status": "Active",
           "chain": "substrate",
           "name": "thedao",
           "uid": "...",
           "secret": "...",
           "createtime": "1600398327",
           "lasttime": "1600398327",
           "allowlist": "false"
       },
      ...
   ]
}
```

The `id` in the returned information is the `PID` of a `Project`.

* GET `/project/<PID>`: Returns the detailed information associated with this `Project`.
* POST `/project/create`: Create a new `Project` based on the information of the currently logged in user

Parameters：
```text
chain:<Polkdaot|Kusama>
name:<Validation rules /[a-zA-Z]{4,32}/>
```

* GET `/stat/day/<PID>`: Get the statistics of a `Project` in one day
* GET `/stat/day/<PID>`: Get the statistical information of a `Project` within one week
* See the document for other interface definitions: [Stat README](https://github.com/patractlabs/elara/tree/master/packages/stat)

### 3. API module

After receiving requests from different users, the `API` module distinguishes which request to a `Project` according to the `PID` carried in the routing information, sends the corresponding information to the `kafka` queue, and forwards the user request to the corresponding node. Therefore, for the `API` module, Elara designed the following two access interfaces, among which the parameters of `<chain>` can currently be filled in `Polkadot` and `Kusama`.
   
* HTTPS, POST `/<chain>/<PID>`: Accept the RPC request initiated by the user. The parameters of the request are all RPC interfaces that the node provides external services
* Websocket `/<chain>/<PID>`: Accept the Websocket request initiated by the user. The parameters of the request are all RPC interfaces that the node provides external services

The above three modules work together to provide Elara's back-end components. Adding a front-end kit on top of it can become a complete Elara product to provide external services: https://elara.patract.io/ The sample screenshot is as follows:

This module is used to provide node services. It is mainly responsible to proxy user's node access request (RPC & websocket) to the node, and as the Producer of the `kafka` message queue, it sends the relevant information of the user request to the `Stat` module. In the `API` module, the user composes the `PID` created in the `Stat` module into the URL of the access node `Provider`, and then initiates a node access request to the `API` module.

### 4. Summary

The above three modules work together to provide Elara's back-end components. Adding a front-end kit on top of it can become a complete Elara product to provide external services: https://elara.patract.io/ The sample screenshot is as follows:

![elara1](https://user-images.githubusercontent.com/21072025/100822117-aa2be000-348c-11eb-9198-815ab2a5ebd5.png)
![elara2](https://user-images.githubusercontent.com/21072025/100822212-d5163400-348c-11eb-9e37-cdc3f5dd960c.png)
![elara3](https://user-images.githubusercontent.com/21072025/100822302-042ca580-348d-11eb-9f70-756c2cc9265f.png)

## Run Elara

The processes corresponding to the above 3 modules need to be started **in sequence** in the following order to make Elara work properly: 1, `Developer-Account`, 2, `Stat`, 3.`API`

### 1. Environment preparation

Elara's operating environment requires `redis` and `kafka` support. Therefore, before running the following commands, first ensure that there is a running `redis` instance in the running environment and the executable file of `kafka` has been installed. The 3 processes of Elara are run by `node`, so you need to install the corresponding dependencies with the `yarn` command before running.

### 2. Start Services

1. Start `Developer-Account` Service
   1. Enter `elara/packages/account` directory
   ```bash
     $ cd package/account
   ```
   2. Install dependence
   ```bash
     $ yarn install
   ```
   3. Modify the configuration file of the `Developer-Account` service, including the `redis` field and the `github` field. The default port is set to `7001`, so if the `port` domain is not modified, the service will occupy the `7001` port after startup.
   ```bash
   $ vim ./config/env/dev.env.js
   ```
   4. Start the service, and the log output by the process is located in the `./logs` directory.
   ```bash
      $ node app.js
      # or use pm2 to manage process
      $ pm2 start pm2.json --env dev
   ```
      
2. Start `Stat` Service
   1. Enter `elara/packages/stat` directory（If you are in `account` directory）
   ```bash
      $ cd ../stat
   ```
   2. Install dependence
   ```bash
      $ yarn install
   ```
   3. Switch to the `kafka` directory, start the `kafka` directory and set the `topic`
      1. Start `kafka` and related processes. If the configuration file of `kafka` is not modified, port `9092` will be occupied by default.
      ```bash
         $ cd <the root directory of kafka>
         $ bin/zookeeper-server-start.sh config/zookeeper.properties
         $ bin/kafka-server-start.sh config/server.properties　
      ```
      2. Add `elara-dev` topic to `kafka`
      ```bash
         # add `elara-dev` topic to kafka
         $ bin/kafka-topics.sh --create --topic elara-dev --bootstrap-server localhost:9092
         # check whethere this topic is initted
         bin/kafka-topics.sh --describe --topic elara-dev --bootstrap-server localhost:9092
      ```
   4. Switch to `elara/packages/stat` directory，modify the configration file of `Stat` module.
   ```bash
      $ vim ./config/env/dev.env.js
   ```
      1. Since `Stat` and `Developer-Account` share the same `redis` instance, the configuration of the `redis` field in this configuration file should be consistent with the configuration of `redis` in the `Developer-Account` configuration file .
      2. The `kafka` domain here needs to be consistent with the configuration of starting `kafka` in the previous step.
      3. **If it is only used for local testing, please modify the `test:false` in the default configuration to `test:true`**. After enabling this flag, when accessing the interface provided by `stat`, the login status of the login account will not be verified, and all interfaces can be directly accessed. In a production environment, please ensure that the value of the `test` field is `false`.
      4. The default `port` domain in the configuration file is `7002`, so if you do not modify this value, it will occupy the `7002` port to provide services after startup.
      
   5. Start Services
      1. Start the `stat` service. If you do not modify the port after startup, the `7002` port will be occupied. The following text uses `127.0.0.1:7002` as the service address performed by `stat`. The log output by the process is located in the `./logs` directory.
      ```bash
         $ node app.js
         # or use pm2 to manage process
         $ pm2 start pm2.json --env dev
      ```
      2. Start `kafka` consumer service
      ```bash
         $ node ./kafka/consumer.js
      ```
      3. If you need to see the console information, you need to start the `dashboard` service
      ```bash
         $ node ./timer/dashboard.js
      ```
      At this point, you can open `http://127.0.0.1:7002/dashboard` in the browser to see the statistics information of the monitoring station.

3. Start `API` Service

   1. Enter `elara/packages/api` directory（if you are in `stat`）
   ```bash
      $ cd ../api
   ```
   2. Install dependence
   ```bash
      $ yarn install
   ```
   3. Modify the configuration file of the `API` module
   ```bash
      $ vim ./config/env/dev.env.js
   ```
      * The `chain` field is used to configure the real chain node, where the name of the chain is part of the route provided by the `api` to the developer. For example, if there are the following configurations in the default, the access method of `/Polkadot/<PID>` is supported in the routing supported by the API.
      ```json
         'Polkadot': {
             'rpc': ['****:**'], //configure as node http://IP: RPC port in step 2
             'ws': ['****:**'] //configure as node ws://IP: WS port in step 2
         }
      ```  
      * The `kafka` field is used to configure the information of the `kafka` process started in the `stat` section. Note that the configured topic is the topic `elara-dev` registered to `kafka` when the `stat` part is configured. ,
      * `startServer` filed is used to configure the ip and port of the `stat` process. For example, if `api` and `stat` are on the same host and the default configuration is used (that is, `stat` occupies port 7002 by default), then The value here is `127.0.0.1:7002`
      * `port` field is used to configure the port occupied by the `api` service process, the default configuration is 7003

   4. Start Services
   ```bash
      $ node app.js
      # or use pm2 to manage process
      $ pm2 start pm2.json --env dev
   ```
      At this time the service has been started, if the port is not modified, the port `7003` will be occupied. The following text uses the port `127.0.0.1:7003` as the host for user access. The log output by the process is located in the `./logs` directory.

### 3. Call Elara

Since the configuration of `test:true` is used when starting `stat` in the above case, the process of user login is skipped in the following process, and the process of creating `Project` is directly started.

1. Create Project：

   Send a request to create a project to the service address `127.0.0.1:7002` of `stat`: This project is a request for access to the `Polkadot` chain, so the value of the parameter `chain` is specified as `Polkadot`.

   ```text
   POST http://127.0.0.1:7002/project/create
   
   chain:Polkadot
   name:hello
   ```

   Return：

   ```json
   {
       "code": 0,
       "mssage": "",
       "data": {
           "id": "5d4ebb40b08f127652ad022f5936b9da",
           "status": "Active",
           "chain": "Polkadot",
           "name": "hello",
           "uid": "Only  For Test",
           "secret": "0b89a386965bfe4dcb19f724ecb41890",
           "createtime": "1606581366",
           "lasttime": "1606581366",
           "allowlist": "false"
       }
   }
   ```

   You can get the detailed information of creating `Project` and `PID`, here is `5d4ebb40b08f127652ad022f5936b9da`

2. List all the `Projects` for this user

   ```text
   GET http://127.0.0.1:7002/project/list
   ```

   Return:

   ```json
   {
       "code": 0,
       "message": "",
       "data": [
           {
               "id": "5d4ebb40b08f127652ad022f5936b9da",
               "status": "Active",
               "chain": "Polkadot",
               "name": "hello",
               "uid": "Only  For Test",
               "secret": "0b89a386965bfe4dcb19f724ecb41890",
               "createtime": "1606581366",
               "lasttime": "1606581366",
               "allowlist": "false"
           }
       ]
   }
   ```

   Since testing is currently enabled, all `Project` information will be listed. If the test is not turned on, only the information list of `Project` for the current login will be listed.

3. Use this PID as part of the route to initiate an access request to the node.

    The access request to the node needs to be served through `API`, the current `API` service address is `127.0.0.1:7003`, so we can initiate the following request: The access route is `<api server host>/<project chain>/< PID>` composition, so in this case

    * The access address for rpc is `http://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da`
    * The access address for websocket is `ws://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da`

   ```text
   POST http://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da
   
   {
       "id":1, 
       "jsonrpc":"2.0", 
       "method":"system_chain",
       "params":[ ]
   }
   ```

   `API` Service is normal and return:

   ```json
   {
       "jsonrpc": "2.0",
       "result": "Polkadot",
       "id": 1
   }
   ```

   This request will be counted by Elara normally and recorded that project `5d4ebb40b08f127652ad022f5936b9da` has one request.

4. If you want to test whether the api service can satisfy normal node requests, you can add the websocket access link to [`Polkadot/Substrate Portal`](https://polkadot.js.org/apps/)。

   1. Log in to the Portal web page in the browser
   2. Click on the upper left corner
   3. Click on the `DEVELOPMENT` column and click on the `Custom` column to fill in `ws://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da`
   4. Click "Switch"
    
    ![image](https://user-images.githubusercontent.com/2844215/100872118-4a5d2580-34dc-11eb-8bf2-a15de4af94c8.png)

   At this point the Portal should be able to work normally, and all access requests are recorded by Elara.

5. Access the `Stat` service to get statistics

   Such as listing all the visit statistics of the current project in the day

   ```text
   GET http://127.0.0.1:7002/stat/day/5d4ebb40b08f127652ad022f5936b9da
   ```

   Or list the statistics of this project this week

   ```text
   GET http://127.0.0.1:7002/stat/week/5d4ebb40b08f127652ad022f5936b9da
   ```

6. Visit `http://127.0.0.1:7002/dashboard` and you can see the total statistics from the console.

At this point, the entire system can operate normally.

## Recap of verification of v0.2

- ~~Developers can use Github account to log in to the developer account space from https://elara.patract.io~~
  > Anyone can currently get access to the Polkadot chain through this link.
- ~~Developers can create new projects and access related API keys under the account~~
  > The above process has introduced the concept of project, and you can log in to github with authorization and create a project at https://elara.patract.io.
- ~~Developers can access the Polkadot node in Elara through Http and web-socket protocols~~
  > The `API` service is used to support this function, and the corresponding operations have been shown above
- ~~Developers can view the monitoring statistics of the projects on the day and the recent weeks~~
  > The `/stat/day/<PID>` and `/stat/week/<PID>` interfaces of `Stat` service are used to support this function.
- ~~You can view the relevant monitoring statistics from the Elara dashboard~~
  > You can see the corresponding information from the `http://127.0.0.1:7002/dashboard` console.

- Problem Statement: Patract Hub has completed the development of Elara v0.2, a service that provides public access to the Polkadot mainnet, and is seeking review and feedback from the Polkadot community.
- Significance: Elara v0.2 is designed to support developers in creating and managing multiple projects, providing detailed access statistics and handling up to 500M requests per day. This service is crucial for the scalability and functionality of the Polkadot network.
- Proposed Solution: Elara v0.2 is built on a microservice architecture, divided into three main modules: Developer-Account, Stat, and API. Each module can be independently extended and upgraded. The service also provides a shared limited bandwidth node RPC service for Polkadot.
- Expected Outcomes: The successful implementation of Elara v0.2 will enhance the Polkadot network's capacity to handle requests, improve developer experience, and contribute to the overall growth and development of the Polkadot ecosystem.

#192 Patract Hub’s treasury report for Elara v0.2 (API Services)

No Active Proposals