Hi Jayan, Can you contribute a PR to the data catalog repo so we can keep the feedback on that issue?
Thanks for your contribution, Suresh > On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana > <jcvidanapathir...@gmail.com> wrote: > > Hi All, > > I have updated the draft code base[1] with a simple workflow of adding data > to replica catalog. Still services are not yet finalized and will be enhanced > with the workflow. > > [1] https://github.com/Jayancv/airavata-replica-catalog > <https://github.com/Jayancv/airavata-replica-catalog> > > Thanks. > > On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana > <jcvidanapathir...@gmail.com <mailto:jcvidanapathir...@gmail.com>> wrote: > Hi Dimuthu and Marcus, > > Thank you both for checking my PoC and providing valuable feedback. > > Dimuthu, > Im agree with you regarding Replica location categories. It should be a data > catalog level attribute. > To manage replica data access permissions don't we need user information at > Replica catalog level ? I'm a bit confused on the permission management side > of this catalog. > ReplicaListEntry - Added to expose the list DataReplicaLocation s with basic > details in AllDataReplicaGetResponse which provide all the replica items for > the given product_id. However, here I was not considering that hierarchical > structure. ReplicaGroupEntry is actually a one product replica which holds > the file structure of the replica data. According to your suggestion we can > model that AllDataReplicaGetResponse as follows, > message AllDataReplicaGetResponse { > data_product_id = 1 > repeated ReplicaGroupEntry replica_list = 2; > } > > message ReplicaGroupEntry { > string replica_group_id = 1 > repeated ReplicaGroupEntry directories = 2; > repeated DataReplicaLocation files = 3; > } > > Marcus, > Yes, I will remove replica_id from the data catalog diagram. > I added that parent_data_product_id to replica data by considering full > context with replica catalog and data catalog relation. But within replica > catalog context there is no such paranet product relationship. Therefore we > can rename it to data_product_id. Thanks for pointing this out. > > Thanks. > > On Thu, Feb 23, 2023 at 2:48 AM Christie, Marcus Aaron <machr...@iu.edu > <mailto:machr...@iu.edu>> wrote: > Hi Jayan, > > I would like to echo Dimuthu and say that this looks great and I appreciate > the effort in your pulling this all together. I have some feedback to share. > > The high-level architecture diagram shows the replica id being stored in the > data catalog. That was an initial idea that we had, but we decided that the > replica catalog would store the data product id. That seems reflected in your > API design so I think you already know this, but I wanted to point it out > since the diagram might be a little confusing for others. > > In the ReplicaCatalogAPI.proto the name of the data product id field is > "parent_data_product_id". I would suggest calling it "data_product_id" > instead. "parent_data_product_id" means "the id of the parent data product of > this data product" in the data catalog. It might be confusing to use the same > name in ReplicaCatalogAPI.proto. > > > Thanks, > > Marcus > > > On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana > > <jcvidanapathir...@gmail.com <mailto:jcvidanapathir...@gmail.com>> wrote: > > > > Hi All, > > > > As a new contributor to the Cybershuttle project, I have been actively > > involved in implementing the Data Replica Catalog. This new catalog is > > designed to interface with both the Apache Airavata Data Catalog [1] and > > Airavata MFT[2]. This replica catalog should be able to store each replica > > resource storage details and secret/credential details specific to the > > storage type. The proposed high-level architecture will be as follows: > > > > > > > > I will mainly work on the highlighted area (red color box) and as an > > initial step started defining APIs which communicate with Replica Catalog. > > This API calls will be gRPC APIs and following methods will be implement, > > > > Replica Registration > > > > • registerReplicaLocation(DataReplicaCreateRequest createRequest) > > • updateReplicaLocation(DataReplicaCreateRequest updateRequest) > > • DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest > > getReplicaRequest) > > • removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest) > > • getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest) > > • removeAllReplicaLocations(AllDataReplicaDeleteRequest > > allDataDeleteRequest) > > > > Storage Registration > > > > registerSecretForStorage(SecretForStorage request) > > deleteSecretsForStorage(SecretForStorageDeleteRequest request) > > getSecretForStorage(SecretForStorageGetRequest request) > > searchStorages(StorageSearchRequest request) > > listStorages(StorageListRequest request) > > resolveStorageType (StorageTypeResolveRequest request) > > > > Storage - Internal APIs > > > > S3StorageListResponse listS3Storage(S3StorageListRequest request) > > Optional<S3Storage> getS3Storage(S3StorageGetRequest request) > > S3Storage createS3Storage(S3StorageCreateRequest request) > > boolean updateS3Storage(S3StorageUpdateRequest request) > > boolean deleteS3Storage(S3StorageDeleteRequest request) > > > > AzureStorageListResponse listAzureStorage(AzureStorageListRequest request) > > Optional<AzureStorage> getAzureStorage(AzureStorageGetRequest request) > > AzureStorage createAzureStorage(AzureStorageCreateRequest request) > > boolean updateAzureStorage(AzureStorageUpdateRequest request) > > boolean deleteAzureStorage(AzureStorageDeleteRequest request) > > > > GCSStorageListResponse listGCSStorage(GCSStorageListRequest request) > > Optional<GCSStorage> getGCSStorage(GCSStorageGetRequest request) > > GCSStorage createGCSStorage(GCSStorageCreateRequest request) > > boolean updateGCSStorage(GCSStorageUpdateRequest request) > > boolean deleteGCSStorage(GCSStorageDeleteRequest request) > > > > Secret Registration > > > > registerSecret(SecretRegistrationRequest request) > > deleteSecret(SecretDeleteRequest request) > > resolveStorageType (StorageTypeResolveRequest request) > > > > Secret - Internal APIs > > > > Optional<S3Secret> getS3Secret(S3SecretGetRequest request) > > S3Secret createS3Secret(S3SecretCreateRequest request) > > boolean updateS3Secret(S3SecretUpdateRequest request) > > boolean deleteS3Secret(S3SecretDeleteRequest request) > > > > Optional<AzureSecret> getAzureSecret(AzureSecretGetRequest request) > > AzureSecret createAzureSecret(AzureSecretCreateRequest request) > > boolean updateAzureSecret(AzureSecretUpdateRequest request) > > boolean deleteAzureSecret(AzureSecretDeleteRequest request) > > > > Optional<GCSSecret> getGCSSecret(GCSSecretGetRequest request) > > GCSSecret createGCSSecret(GCSSecretCreateRequest request) > > boolean updateGCSSecret(GCSSecretUpdateRequest request) > > boolean deleteGCSSecret(GCSSecretDeleteRequest request) > > > > > > Poc[3] : https://github.com/Jayancv/airavata-replica-catalog > > <https://github.com/Jayancv/airavata-replica-catalog> (Defining API calls) > > Draft APIs : refer the attachment replicaCatalogAPIsDocumentation.html[4] > > which generated using the Poc [3] > > > > I greatly appreciate your thoughts and feedback on the designs[5], as they > > can help us improve and adopt a more generalized approach. Additionally, I > > would like to identify any other factors that we should take into account > > to minimize potential issues in the future. Are there any other > > considerations that we should keep in mind? > > > > > > [1] - https://github.com/apache/airavata-data-catalog > > <https://github.com/apache/airavata-data-catalog> > > [2] - https://github.com/apache/airavata-mft > > <https://github.com/apache/airavata-mft> > > [3] - https://github.com/Jayancv/airavata-replica-catalog > > <https://github.com/Jayancv/airavata-replica-catalog> > > [4] - > > https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing > > > > <https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing> > > [5] - > > https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing > > > > <https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing> > > > > Thanks. > > -- > > Best Regards > > > > Jayan Vidanapathirana > > > > <replicaCatalogAPIsDocumentation.html> > > > > -- > Best Regards > > Jayan Vidanapathirana > <https://lk.linkedin.com/in/jayancv> > > > -- > Best Regards > > Jayan Vidanapathirana > <https://lk.linkedin.com/in/jayancv>
