/
Scaling Experiments with Hyperledger Umbra: Internship Activities - Project Plan

Scaling Experiments with Hyperledger Umbra: Internship Activities - Project Plan

Nov 12, 2020

Overview

This document contains the roadmap of activities to accomplish the topics proposed by the 2020 internship project named Scaling Experiments with Hyperledger Umbra. 

The source code and documentation of this project is going to be sent as a pull request to the main umbra repository https://github.com/hyperledger-labs/umbra. 

During the development of the project a fork of such repository will be used for the activities below.

Activities

  1. Add monitor and agent source code

    1. Add monitor component to umbra - hosts, processes and containers can be monitored (cpu, mem, disk) and network (capture pcaps)

      1. Monitor is triggered by broker events

      2. Monitoring can be continuous or not

    2. Add agent component to umbra - enables containers to run tools that can create disturbances in the network (e.g., play pcaps, iperf, ping, etc)

      1. Agent is triggered by broker events

      2. Different types of agent events can be called (live stream metrics to database, repetitions, etc)

  2. Provide support for HL Fabric 2.0

    1. Read the docs about Fabric 2.0 

    2. Implement modifications to fully support Fabric 2.0

  3. Scale to cluster of servers

    1. Define requirements to scale project

    2. e.g., ssh keys, config files, docker image downloads/builds

    3. Code needed modules/classes

    4. Test all the architecture running with Fabric in scaled experimental scenarios

  4. Build umbra monitoring toolset

    1. Define monitoring interfaces for nodes and links

    2. e.g., capture node cpu percent, link packet statistics

    3. Add methods to capture pcaps in node interfaces

    4. Set database interface to store monitored data (e.g., influxdb, elasticsearch)

    5. Set visualization interface to show stored data (e.g., graphana/kibana)

  5. Build Experimental Analysis Report

    1. Define possible analysis of results to generate test report

    2. Define test report format

    3. Code modules/classes to report test results

    4. Test modules and output report

  6. Document the project

    1. Iterate documentation of source code during project

    2. Refine documentation by the end of project

    3. Publish readthedocs documentation for the whole project

    4. Containing multiple test cases exploring: basic architecture, scaling techniques, analysis report

    5. Write project demo and article describing internship activities


Extra (If time allows):

  • Extend architecture to other Hyperledger project(s)

    • Choose a HL blockchain project

    • Define how architecture can be extended

    • e.g., needed components, interfaces, modules

    • Create the basic architecture to extend project


Schedule

Quarterly Evaluations

Activities

July 10: First 

1, 2

August 21: Second 

3

October 2: Third 

4, 5 

November 13: Fourth 

6

 

Progress

  • 1st Quarterly Evaluation (July 10):

    • Successfully accomplished. The source code produces is in https://github.com/raphaelvrosa/umbra defined by the commit 78ee22b5f73af70d83b8e6f671853a96274c0692. 

    • Observations:

      • The support for Fabric 2.0+ is defined for the topology components, but as the events depend on the Fabric python SDK to support Fabric 2.0+ (work in progress by the internship "Add interactive examples to Fabric Python SDK"). 

      • No major changes were required in the source code to support Fabric 2.0 - but new events need to be defined to support the new defined chaincode lifecycle operations.

      • Two new components were added to umbra, agent and monitor. Both make use of async to monitor and probe fabric nodes in run-time topologies.

  • 2nd Quarterly Evaluation (Aug. 21):

    • Delivered Fabric 2.0 implementation for nodes. Missing fabric-python-sdk (wait for the project to support 2.0+).

  • 3rd Quarterly Evaluation (Oct. 2):

  • 4th Quarterly Evaluation (Nov. 13):

    • Makefile, Vagrant, dockerfile created/upgraded to enable full automation and easy experimentation.

    • Examples commented with functionalities implemented.

    • Using Containernet install stable for Ubuntu 20.04.

    • Merged fuzzing capabilities to scale to multiple environments (servers).

 


Communication

On weekly basis by zoom call with other umbra related projects