NOTE: this is a rough draft of the proposal for our new CI/CD and test net system. Feel free to comment and/or add content.

CI/CD and Test Net Proposal

Summary

Hyperledger would switch to running an instance of Gitlab and allow project maintainers to set up and run their own CI/CD and test net clusters. Hyperledger would run a few "shared" runners to ensure that there is always a machine or two available to run jobs for projects that do not have any volunteer runners. This solution allows for projects to not only handle CI/CD but also test nets. Gitlab runners are capable of running just about anything. Project maintainers can create test net projects with a CI pipeline definition that tells the runners to download a docker image of the latest build of a project, and all of our blockchain platforms publish ready-to-run docker images, they can easily stand up a test net for a scaling or soak test. This proposal reduces the human overhead and the financial overhead of running a CI/CD pipeline as well as increasing the level of control each project will have over their CI/CD pipeline.

TL;DR

Requirements

• Shall be able to clone and build all of the code repositories for all Hyperledger projects. This includes repositories hosted on Github as well as Gerrit.
• Shall present a dashboard for maintainers that reflects the build status and performance of the CI/CD pipeline.
• Shall be "self service" in a few key ways:
  • Maintainers of a project shall be able to exercise full control over what gets accomplished by the runners in their CI/CD cluster.
  • Contributors/volunteers shall be able to set up computers as runners to contribute resources to a CI/CD cluster.
  • Maintainers shall be able to restrict access to which computers contribute to the CD portion of the cluster.
• Shall have integrations with external services such as Github, Gerrit, Confluence, and Jira.

The proposal is to use Gitlab for managing and running the new CI/CD pipelines for all of the Hyperledger projects. Gitlab meets all of these requirements without any special configuration and with a specific setup, we can accomplish all of the self-service goals as well.

External Repositories

Per the documentation, Gitlab supports mirroring external git repositories and triggering CI/CD builds from merges and commits in the external repository. It has full integration with Github but also has the ability to mirror any external repository that is publicly accessible. So our Gerrit hosted code repositories are supported.

Dashboards

Gitlab has a CI/CD "charts" page that shows the statistics from past CI/CD runs over time.

Self Service

Gitlab's real power lies in the self-service nature of setting up the CI/CD configuration by maintainers and the setting up of runners by developers that contribute resources to the CI/CD cluster for each project. Gitlab has four different roles that actions are assigned to; Guest, Developer, Maintainer, and Admin. Gitlab's security model perfectly fits what Hyperledger is looking for. The HL/LF staff will be Admins and will be responsible for site-wide configuration. Maintainers from each project are able to create and configure projects without assistance from HL/LF staff. This gives maximum freedom to each project team in setting up their CI/CD pipeline as well as setting up test net like clusters for soak, scaling and other tests. Developers are able to trigger jobs on the runners cluster.

Integrations

Gitlab integrates with Jira and therefore also integrates with Confluence.