2019 03 28 TSC Minutes

Owned by Ry Jones
Oct 01, 2019
9 min read

 (7:00am - 8:00am PT)

via Zoom

Hyperledger is committed to creating a safe and welcoming community for all. For more information please visit our Code of Conduct: Hyperledger Code of Conduct

TSC Members

Present?

  • Arnaud Le Hors
  • Baohua Yang
  • Binh Nguyen
  • Christopher Ferris
  • Dan Middleton
  • Hart Montgomery
  • Kelly Olson
  • Mark Wagner
  • Mic Bowman
  • Nathan George
  • Silas Davis

Resources:

Announcements

  • Contributors Summit
    • Venue from Hitachi fell through due to lack of wifi access.
    • Dealing with budgeting issues. Budget is half of what was originally thought.
    • Some options for sponsorship to ease the cost.
  • Quilt Reboot at SF office April 4th
    • Quilt will be focused on protocols.
    • Restructuring the project.
    • 3 hour meeting to get it going.
  • Internships
    • Over 100 submissions, expecting 500
    • Mostly Chinese and Indian students
    • Proposed to change the name from "internship" to "mentorship"
      • Kelly: why are we considering the change from internship to mentorship
      • Dave: LF uses the term mentorship on their platforms
      • There are concerns that internship and mentorship mean different things to students
      • Silona: primary concern with SEO
      • Dave: LF is building something that appears to be a year-round mentorship program rather than a summer internship
  • Kelly: Any other questions or comments?
    • Silona: needs help finding a venue for the contributors summit in Japan
    • Binh: do we know how many would be attending?
    • Ry: somewhere around 100 to 150
    • Needs to accommodate that many people with tables and a couple of conference rooms.
    • Chris: re: Contributor's summit, what are we hoping to achieve? It should be about various teams collaborating right?
      • Open stack would hold a contributor summit type meeting where every project would have a room to do face-to-face discussions about their roadmap and actively discussing proposal for improvement and building consensus around the forward direction. And then there was a track for the TSC type group where they would talk about the integration of the different projects.
      • Hoping that it wouldn't be 150 people, but specifically the maintainers and core contributors to build the project agendas followed by another day of meetings talking about interop.
      • It should be much more about the projects rather than bringing newbies up and running on the technology.
      • Dave: I like what Chris is saying because it compliments the bootcamps.
      • Kelly: +1 on what Chris said. Action items: maybe open a page to gather who would come and what topics are of interest.
      • Chris: Would like to see specific proposals about getting to a single API for tx submission, that kind of thing. It can't be an unconference.
      • Silona: Looking at 2 days, 1 day that is architected and a second day that is a curated uncoference. The bootcamp wasn't complete chaos. It was focused on onboarding new contributors. The contributor's summit would be focused on the the interop and roadmaps. We haven't put up a page because we don't have a date or location yet.
    • Kelly: Have we reached out to the premiere board members for help?
      • Hitachi just fell through because they won't give us wifi access.
      • Fujitsu and NEC connections would be helpful.
      • Chris: maybe HL can afford a handful of Myfi devices.
      • Silona: we have a considerable bandwidth demand and not sure that those devices would suffice.
    • Arnaud: we moved from having hackfests to bootcamps and contributor's summit and now we're short of money and the contributor's summit is in danger. Should we have more contributor's summit instead of bootcamps?
    • Silona: certainly a consideration and the we're restructuring our priorities so that the contributor's summit can happen.
    • Mark: thinking about the travel, is it worth it for Americans? What is going to happen to justify the travel expense? It needs to be concrete in-person things to justify the travel.
    • Kelly: next steps
      • reach out to other premier members to get a location with wifi
      • then work on agenda

Items of discussion

  • Vote on moving Indy from incubation to active status.
    • Nathan did a great job getting the details in front of the TSC ahead of time.
    • Questions?
      • None
    • Thoughts or endorsements?
      • None
    • Motion to vote and seconded
      • In Favor: unanimous
      • Abstaining: none
      • Against: none
    • Motion approved
  • CI/CD/Testnets propsals
    • Sent out a proposal to the TSC mailing list as a potential solution for CI/CD/Testnet
    • Drastically reduces our technical, financial, and human overhead.
    • Uses Gitlab central controller run by HL with crowdsourced runners from the community.
    • Like the solutions since it is mostly self-service.
    • The roles in Gitlab match our community structure.
    • The proposal is on the Wiki
    • Questions?
      • Chris: we currently have gerrit and jenkins, not everybody likes them, don't understand why were talking about going to another set of tooling. The work to transition will be hugely difficult.
      • Dave: I don't think the transition will be that difficult.
      • Nathan: not replacing github with gitlab. emphasis is on self-service of the nature.
      • Dave: The Gitlab solution is just going to be CI/CD, we wouldn't require you to leave jenkins right away. We're just trying to to get a system that gives all of the teams more self-service control over their CI/CD pipeline. We have multiple teams have tried to use jenkins and have failed and have built their own because the solution we currently have does not meet their needs. I'm open to discussion and negotiation. Suggesting that the requirements from all of our teams could best be met by standing up a Gitlab coordinator, a few runners, and then let the teams manage it themselves. 
      • Chris: A couple of runners is not going to handle Fabric.
      • Dave: It's something that the teams would have to set up. A lot of our other teams have their own machines.
      • Chris: So you're saying you want it to happen behind the scene.
      • Dave: I have no visibility into the existing CI/CD systems that many of the teams have set up. The few runners would ensure that there is a minimum capability that the smaller teams can use. This would largely reduce our overhead from what we're doing now. How can we mitigate the fact that so many teams are building their own systems that are not public?
      • Chris: Soramitsu already had something in place. Indy is different, they have tried to use ours but built their own. When we set up HL in the first place, we worked with LFIT to get into place tech that LFIT supported. That's Jenkins and Gerrit because tons of projects at LF use that. I just don't see when Fabric is going to have the bandwidth to reconstitute all of its stuff on a platform we don't like. The fabric architects would probably prefer something like Concourse because it gives a view into the full pipeline?
      • Dave: Are those tests run in a docker image?
      • Chris: Everything we have is containerized but we're getting more sophisticated towards spinning up Kubernetes networks.
      • Dave: So gitlab knows how to drive kubernetes.
      • Chris: Where does the kube cluster come from?
      • Dave: We can crowdsource, we can get CNCF resources, companies can sponsor. I'm not telling you you can't keep using gerrit and jenkins for the time being. What I'm concerned with trying to explain to the members of the community that we're not going to change a system that doesn't meet their requirements and forced them to build systems outside of HL.
      • Silas: understands the issues with migration. gitlab does git ops really well. it's generally more reliable than jenkins. the gitlab-ci file is more of a first class citizen than the jenkins file. jenkins has the difficulty of trying to get all of the right plugins installed and the global config right. Gitlab has a lot of flexibility for all of the different testing scenarios.
        • Understands that there aren't a lot of resources, but where are these shared runners coming from?
        • It would be helpful if we have a kubernetes cluster for us to use and possibly for having runners in as well.
        • To get the best out of Gitlab, we may have to do tricky things like pushing code into Gitlab's git hosting to get git ops behavior.
      • Dave: the problem with kubernetes clusters is that it is hard to make them self-service and to control how much resources each team uses. It takes a lot to police the quotas.
      • Silas: maybe can achieve it with namespaces in kubernetes.
      • Dave: i want to make sure that we're capping the money we spend and the human cost and to make it self-service.
      • Chris: we need to get feedback from all of the team maintainers before moving forward. This will make our bill even larger. We're already paying for cloud to do jenkins and now we're going to be paying more.
      • Dave: the implication is that we would be scaling back vexxhost over time.
      • Chris: so you're saying we will have to transition and I don't see how we can do that without a huge cost to the Fabric team. We'd have to drop larges sets of features on the floor. The fabric project is HUGE in comparison to the other projects.
      • Dave: that's a fair criticism.
      • Chris: moving all of it over is not a wave a magic wand kind of thing.
      • Dave: the motivation is that only Fabric team is successfully using the existing CI/CD platform. many other teams have made an attempt to use the existing CI/CD and failed or run into enough roadblocks that it was easier for them to set up their own systems outside of HL. My mission is to make sure that our software delivery is solid and I don't have a visibility into these outside CI/CD systems. The other motivation is reducing our monetary overhead. It's not self-service because each team needs  hands-on help from HL staff to tweak it. New projects coming in struggle with it. There are lots of negatives to our existing system. I have gone out on a limb and proposed Gitlab. Chris has pointed out that there are problems with that proposal. We need to have this discussion.
      • Silas: it sounds like we need a persistent kubernetes cluster. gitlab can run directly inside of a kub cluster straight away. we can also run jenkins. so maybe having a persistent kubernetes cluster would solve this.
      • Dave: if we do a kubernetes cluster it is either/or. They are so expensive that we can only have one or the other. We'd have to turn off Vexxhost to make a kubernetes cluster happen. I'm proposing instead that we keep vexxhost and stand up a small gitlab coordinator and then allow the teams to start using that and crowdsourcing the resources. Using docker runners on gitlab can utilize spare resources of computers owned by contributors. It was straight forward for me to add a spare machine I own to the Ursa cluster. It takes less than five minutes to stand up a runner.
      • Silas: I understand the either/or tradeoff.
      • Silona: another motivator is the testnets.
      • Mark: how much work will be pushed back on small projects? how much work does HL do for smaller projects?
      • Dave: we don't do any work because the smaller projects haven't been able to get it set up.
      • Chris: For Fabric, the Linux Foundation is not running the Fabric CI/CD. They are running Jenkins and Gerrit and Nexus. When we need configuration changes the LFIT have to make them because they control those. But in terms of CI/CD, the Fabric team owns all of that. The Fabric team started with jenkins jjb's but are moving to pipelines. It would roughly be the same as Travis and what you're trying to do with Gitlab. There is a humongous amount of already built stuff that would have to be migrated over and I just don't know how or when that would happen.
      • Silona: LFIT is looking at changing some of their processes for access and upgrades. So some things will be changing soon.
      • Dave: where do the computing resources come from for running Fabric tests?
      • Chris: vexxhost.
      • Dave: so the entire community is subsidizing the running of servers that really only Fabric is using?
      • Kelly: at the end of the day this is ultimately a budget question. if we had unlimited resources we would run both. the situation is difficult because all of the other teams are footing the bill for their own CI/CD instead of leveraging the LF resources. I appreciate the burden of having to move Fabric off of the existing system but we need to also consider that we need a solution for the entire community.
      • Dave: what I'm taking away from this is that we shouldn't take this lightly and that I need to spend more time fully understand the Fabric pipeline so that I can get a sense of what the transition would look like from jenkins to something else. I'm open to suggestions and I'm all ears on the suggestions. I came into this a little bit ignorant about the scope of Fabric's CI/CD and I appreciate that Chris is pushing back. The current system doesn't work for everybody and we need to move towards a solution that does.
      • Silas: can you also take a look at the financial cost of running a kubernetes cluster? kubernetes would be more useful for the burrow team instead of gitlab.
      • Kelly: At the top of the hour. Action items:
        • Maintainers from the projects need to get their CI/CD people to get in touch with Dave so that he can gather all of those.
        • Add a backlog item to discuss the budget allocations for different things like contributor's summit or CI/CD so that we can better discuss it.

Quarterly updates

None

Upcoming items

  • Ambassadors reboot (Silona Bonewald, postponed - waiting on contractor)
  • Hyperledger Indy moving to Active Status (Nathan George, beginning of April)
  • Hyperledger Indy indy-sdk moving to 2.0 (Nathan George, beginning of April)
  • Project Readiness for 1.0 (Dave Huseby, April 11th)
  • Overall Engagement in Projects (Mark Wagner, Ry Jones, mid-April)
  • Election of New Chair for Learning Materials Dev WG (Silona Bonewald and Bobbi TBD ) 
    • What are documentation Best practices?
    • How are all the projects doing documentation wise?

Recordings