Hi Jayan, I would like to echo Dimuthu and say that this looks great and I appreciate the effort in your pulling this all together. I have some feedback to share.
The high-level architecture diagram shows the replica id being stored in the data catalog. That was an initial idea that we had, but we decided that the replica catalog would store the data product id. That seems reflected in your API design so I think you already know this, but I wanted to point it out since the diagram might be a little confusing for others. In the ReplicaCatalogAPI.proto the name of the data product id field is "parent_data_product_id". I would suggest calling it "data_product_id" instead. "parent_data_product_id" means "the id of the parent data product of this data product" in the data catalog. It might be confusing to use the same name in ReplicaCatalogAPI.proto. Thanks, Marcus > On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana > <jcvidanapathir...@gmail.com> wrote: > > Hi All, > > As a new contributor to the Cybershuttle project, I have been actively > involved in implementing the Data Replica Catalog. This new catalog is > designed to interface with both the Apache Airavata Data Catalog [1] and > Airavata MFT[2]. This replica catalog should be able to store each replica > resource storage details and secret/credential details specific to the > storage type. The proposed high-level architecture will be as follows: > > > > I will mainly work on the highlighted area (red color box) and as an initial > step started defining APIs which communicate with Replica Catalog. This API > calls will be gRPC APIs and following methods will be implement, > > Replica Registration > > • registerReplicaLocation(DataReplicaCreateRequest createRequest) > • updateReplicaLocation(DataReplicaCreateRequest updateRequest) > • DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest > getReplicaRequest) > • removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest) > • getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest) > • removeAllReplicaLocations(AllDataReplicaDeleteRequest > allDataDeleteRequest) > > Storage Registration > > registerSecretForStorage(SecretForStorage request) > deleteSecretsForStorage(SecretForStorageDeleteRequest request) > getSecretForStorage(SecretForStorageGetRequest request) > searchStorages(StorageSearchRequest request) > listStorages(StorageListRequest request) > resolveStorageType (StorageTypeResolveRequest request) > > Storage - Internal APIs > > S3StorageListResponse listS3Storage(S3StorageListRequest request) > Optional<S3Storage> getS3Storage(S3StorageGetRequest request) > S3Storage createS3Storage(S3StorageCreateRequest request) > boolean updateS3Storage(S3StorageUpdateRequest request) > boolean deleteS3Storage(S3StorageDeleteRequest request) > > AzureStorageListResponse listAzureStorage(AzureStorageListRequest request) > Optional<AzureStorage> getAzureStorage(AzureStorageGetRequest request) > AzureStorage createAzureStorage(AzureStorageCreateRequest request) > boolean updateAzureStorage(AzureStorageUpdateRequest request) > boolean deleteAzureStorage(AzureStorageDeleteRequest request) > > GCSStorageListResponse listGCSStorage(GCSStorageListRequest request) > Optional<GCSStorage> getGCSStorage(GCSStorageGetRequest request) > GCSStorage createGCSStorage(GCSStorageCreateRequest request) > boolean updateGCSStorage(GCSStorageUpdateRequest request) > boolean deleteGCSStorage(GCSStorageDeleteRequest request) > > Secret Registration > > registerSecret(SecretRegistrationRequest request) > deleteSecret(SecretDeleteRequest request) > resolveStorageType (StorageTypeResolveRequest request) > > Secret - Internal APIs > > Optional<S3Secret> getS3Secret(S3SecretGetRequest request) > S3Secret createS3Secret(S3SecretCreateRequest request) > boolean updateS3Secret(S3SecretUpdateRequest request) > boolean deleteS3Secret(S3SecretDeleteRequest request) > > Optional<AzureSecret> getAzureSecret(AzureSecretGetRequest request) > AzureSecret createAzureSecret(AzureSecretCreateRequest request) > boolean updateAzureSecret(AzureSecretUpdateRequest request) > boolean deleteAzureSecret(AzureSecretDeleteRequest request) > > Optional<GCSSecret> getGCSSecret(GCSSecretGetRequest request) > GCSSecret createGCSSecret(GCSSecretCreateRequest request) > boolean updateGCSSecret(GCSSecretUpdateRequest request) > boolean deleteGCSSecret(GCSSecretDeleteRequest request) > > > Poc[3] : https://github.com/Jayancv/airavata-replica-catalog (Defining API > calls) > Draft APIs : refer the attachment replicaCatalogAPIsDocumentation.html[4] > which generated using the Poc [3] > > I greatly appreciate your thoughts and feedback on the designs[5], as they > can help us improve and adopt a more generalized approach. Additionally, I > would like to identify any other factors that we should take into account to > minimize potential issues in the future. Are there any other considerations > that we should keep in mind? > > > [1] - https://github.com/apache/airavata-data-catalog > [2] - https://github.com/apache/airavata-mft > [3] - https://github.com/Jayancv/airavata-replica-catalog > [4] - > https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing > [5] - > https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing > > Thanks. > -- > Best Regards > > Jayan Vidanapathirana > > <replicaCatalogAPIsDocumentation.html>
smime.p7s
Description: S/MIME cryptographic signature
