Project Plan - Indy Read Replica Implementation

Abstract

A Hyperledger Indy "read replica” is a web service that holds copies of the ledger transactions from one or more Indy ledgers and can be used by clients that trust the replica. While an instance of the service is intended to be accessed by arbitrary agents, an enterprise might run a read replica on their own servers for the use of their own agents – the agents "trust" the deployment of the read replica. In turn, the use of read replicas will reduce the read load on the Indy ledgers it replicates. Each replica must stay up to date with the transactions on the ledgers it supports, and agents using the replica need not connect directly with the ledger.

Mentor and Mentee

Mentors: Sam Curren, Stephen Curran

Timezone: US Mountain and Canadian Pacific

Mentee: Bryan Elee Atonye

Timezone: West Africa

Project repo: TBD

Deliverables

  • JavaScript-based Web Service that maintains a database copy of one or more Indy Ledgers.
  • An API for clients (such as Aries Agents – ACA-Py and Credo) to invoke to receive transaction and objects from those Indy ledgers.
  • Unit and integration tests for the service.
  • GitHub Actions to invoke the tests, and when triggered, release artifacts for published versions of the implementation.
  • Documentation for deploying and configuring a horizontally scalable instance of the web service.

Milestones

Eval 1:

  • Create a new GitHub repository with any useful code from the open source IndyScan project. Include a suitable acknowledgement about the initial codebase. We agree that while there are components of IndyScan useful to this project, forking IndyScan is not appropriate, as the goals of the two projects are quite different.
  • Define and implement an API call that mimics the Indy “GET_TXN” request to allow a client to retrieve transactions from the replica.
  • Get the new implementation running using Docker and document how to run a local instance.

Eval 2:

  • Define and implement an appropriate set of GitHub actions to test Pull Requests, and when triggered, create a version and publish artifacts.
  • Consult with Mentors to decide how to implement the API calls for the object “GET” – NYM, ATTRIB, SCHEMA, CLAIM_DEF, REVOC_REG_DEF, REVOC_REG, REVOC_REG_DELTA
  • Implement the agreed upon design for the “GET” object API calls.
  • Implement an appropriate authorization scheme to protect the web service.

Eval 3:

  • Design and implement an appropriate mechanism for dealing with the “startup issue” – getting all existing transactions without having to do transaction-by-transaction reads of the ledger.  Note that this feature may require updates to the Indy Node project that may or may not be possible as part of this effort.

Eval 4:

  • Design and, if feasible, implement a pub/sub mechanism such that the read replicas need not continually poll the ledger for updates. Note that this feature may require updates to the Indy Node project that may or may not be possible as part of this effort.

Timeline

WeekTask/PlanStatus
June 03 - June 23On boarding/orientation sessions. Meet with the mentors, discuss project implementation details,
deliverables and scope. Prepare the project plan.

COMPLETED    

June 24 - July 7

Design Hyperledger Indy Read Replica web service component and Spec out API endpoints with
input from mentors.


Set up a simple set of tests for verifying correctness of implementation

COMPLETED    

July 8 - July 22

Evolve IndyScan to include and support a client agent API with an HTTP version of the existing
GET_TXN ledger API to handle any ledger read request.

COMPLETED    

July 22 - July 26Prepare docker deployment and instructions.

1ST QUARTER MENTEE EVALUATION

COMPLETED    

July 27 - August 18Consult with the Mentors to design the remainder of the API calls for getting Indy Objects.

COMPLETED

August 19 - September 01Implement agreed upon design.

COMPLETED

September 02 - September 06

Implement GitHub CI/CD Pipeline for testing pull requests and preparing release artifacts.

Implement an appropriate authorization scheme to protect the web service.

MIDTERM EVALUATIONS

IN PROGRESS    


September 08 - September 22

Continue work on GitHub CI/CD pipeline as necessary.

Consult with Mentors on approach to address the “startup issue” – loading a replica without querying the ledger transaction by transaction.


September 23 - October 19

Implementation of the agreed upon replica start strategy.


October 14 - October 183RD QUARTER MENTEE EVALUATION
October 19 - November 10Implement Pub-Sub APIs for notifying read replicas of new transactions using an approach agreed
upon with the mentors.

November 11 - November 29

Buffer for project wrap up.

Revaluate work done so far. Review the code and make changes where necessary; Complete testing.
Preparation of the final report and Submission of the final project report.


Explanation


Methodology