Hi Marcus, I sincerely appreciate you taking the time to review my changes. Actually, I thought these two services needed to be operated independently when i'm starting the API definition. If we couple both services then users can simply use the Data-Catalog Data Product APIs and Data models. Then I can remove data product APIs and models from the replica catalog.
Can I know if there is any overall deployment plan for these 2 catalogs ? Thanks. On Thu, Apr 6, 2023 at 7:44 PM Christie, Marcus Aaron <machr...@iu.edu> wrote: > Hi Jayan, > > Thanks for sharing. One question, the airavata-data-catalog already has a > DATA_PRODUCT table and a way to store a data product's metadata. Could that > be used instead of adding a new table? > > Or more generally my question is how does this replica catalog API relate > to the data catalog API/data model? > > Thanks, > > Marcus > > > On Mar 31, 2023, at 4:11 PM, Jayan Vidanapathirana < > jcvidanapathir...@gmail.com> wrote: > > > > Hi All, > > > > I have implemented basic flow(simple create and retrieve) of the replica > catalog and drafted a pull request[1] to the Airavata data catalog as a new > module. According to that implementation I have come to the following > database structure for the replica catalog and I greatly appreciate your > thoughts and feedback on the designs[2]. At this stage S3 storage type was > considered as a sample. > > > > <Replica Catalog V2.drawio (1).png> > > > > Also please refer to the following google doc[3] to review the > implemented APIs. > > > > [1] https://github.com/apache/airavata-data-catalog/pull/28 > > [2] > https://drive.google.com/file/d/1KP-8IWdvpPvjSWUG2t41K7WQXW2f9_qN/view?usp=sharing > > [3] > https://docs.google.com/document/d/1U-ok1ICt_EmjjxR9UuACV6g6YYkgoADfm0ECJ9ZDI3k/edit?usp=sharing > > > > Thank you. > > > > On Mon, Mar 20, 2023 at 2:20 AM Suresh Marru <sma...@apache.org> wrote: > > Hi Jayan, > > > > Can you contribute a PR to the data catalog repo so we can keep the > feedback on that issue? > > > > Thanks for your contribution, > > Suresh > > > >> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana < > jcvidanapathir...@gmail.com> wrote: > >> > >> Hi All, > >> > >> I have updated the draft code base[1] with a simple workflow of adding > data to replica catalog. Still services are not yet finalized and will be > enhanced with the workflow. > >> > >> [1] https://github.com/Jayancv/airavata-replica-catalog > >> > >> Thanks. > >> > >> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana < > jcvidanapathir...@gmail.com> wrote: > >> Hi Dimuthu and Marcus, > >> > >> Thank you both for checking my PoC and providing valuable feedback. > >> > >> Dimuthu, > >> • Im agree with you regarding Replica location categories. It > should be a data catalog level attribute. > >> • To manage replica data access permissions don't we need user > information at Replica catalog level ? I'm a bit confused on the permission > management side of this catalog. > >> • ReplicaListEntry - Added to expose the list DataReplicaLocation > s with basic details in AllDataReplicaGetResponse which provide all the > replica items for the given product_id. However, here I was not considering > that hierarchical structure. ReplicaGroupEntry is actually a one product > replica which holds the file structure of the replica data. According to > your suggestion we can model that AllDataReplicaGetResponse as follows, > >> message AllDataReplicaGetResponse { > >> data_product_id = 1 > >> repeated ReplicaGroupEntry replica_list = 2; > >> } > >> > >> message ReplicaGroupEntry { > >> string replica_group_id = 1 > >> repeated ReplicaGroupEntry directories = 2; > >> repeated DataReplicaLocation files = 3; > >> } > >> > >> Marcus, > >> • Yes, I will remove replica_id from the data catalog diagram. > >> • I added that parent_data_product_id to replica data by > considering full context with replica catalog and data catalog relation. > But within replica catalog context there is no such paranet product > relationship. Therefore we can rename it to data_product_id. Thanks for > pointing this out. > >> > >> Thanks. > >> > >> On Thu, Feb 23, 2023 at 2:48 AM Christie, Marcus Aaron <machr...@iu.edu> > wrote: > >> Hi Jayan, > >> > >> I would like to echo Dimuthu and say that this looks great and I > appreciate the effort in your pulling this all together. I have some > feedback to share. > >> > >> The high-level architecture diagram shows the replica id being stored > in the data catalog. That was an initial idea that we had, but we decided > that the replica catalog would store the data product id. That seems > reflected in your API design so I think you already know this, but I wanted > to point it out since the diagram might be a little confusing for others. > >> > >> In the ReplicaCatalogAPI.proto the name of the data product id field is > "parent_data_product_id". I would suggest calling it "data_product_id" > instead. "parent_data_product_id" means "the id of the parent data product > of this data product" in the data catalog. It might be confusing to use the > same name in ReplicaCatalogAPI.proto. > >> > >> > >> Thanks, > >> > >> Marcus > >> > >> > On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana < > jcvidanapathir...@gmail.com> wrote: > >> > > >> > Hi All, > >> > > >> > As a new contributor to the Cybershuttle project, I have been > actively involved in implementing the Data Replica Catalog. This new > catalog is designed to interface with both the Apache Airavata Data Catalog > [1] and Airavata MFT[2]. This replica catalog should be able to store each > replica resource storage details and secret/credential details specific to > the storage type. The proposed high-level architecture will be as follows: > >> > > >> > > >> > > >> > I will mainly work on the highlighted area (red color box) and as an > initial step started defining APIs which communicate with Replica Catalog. > This API calls will be gRPC APIs and following methods will be implement, > >> > > >> > Replica Registration > >> > > >> > • registerReplicaLocation(DataReplicaCreateRequest > createRequest) > >> > • updateReplicaLocation(DataReplicaCreateRequest updateRequest) > >> > • DataReplicaLocationModel > getReplicaLocation(DataReplicaGetRequest getReplicaRequest) > >> > • removeReplicaLocation(DataReplicaDeleteRequest > deleteReplicaRequest) > >> > • getAllReplicaLocations(AllDataReplicaGetRequest > allDataGetRequest) > >> > • removeAllReplicaLocations(AllDataReplicaDeleteRequest > allDataDeleteRequest) > >> > > >> > Storage Registration > >> > > >> > registerSecretForStorage(SecretForStorage request) > >> > deleteSecretsForStorage(SecretForStorageDeleteRequest request) > >> > getSecretForStorage(SecretForStorageGetRequest request) > >> > searchStorages(StorageSearchRequest request) > >> > listStorages(StorageListRequest request) > >> > resolveStorageType (StorageTypeResolveRequest request) > >> > > >> > Storage - Internal APIs > >> > > >> > S3StorageListResponse listS3Storage(S3StorageListRequest request) > >> > Optional<S3Storage> getS3Storage(S3StorageGetRequest request) > >> > S3Storage createS3Storage(S3StorageCreateRequest request) > >> > boolean updateS3Storage(S3StorageUpdateRequest request) > >> > boolean deleteS3Storage(S3StorageDeleteRequest request) > >> > > >> > AzureStorageListResponse listAzureStorage(AzureStorageListRequest > request) > >> > Optional<AzureStorage> getAzureStorage(AzureStorageGetRequest > request) > >> > AzureStorage createAzureStorage(AzureStorageCreateRequest request) > >> > boolean updateAzureStorage(AzureStorageUpdateRequest request) > >> > boolean deleteAzureStorage(AzureStorageDeleteRequest request) > >> > > >> > GCSStorageListResponse listGCSStorage(GCSStorageListRequest request) > >> > Optional<GCSStorage> getGCSStorage(GCSStorageGetRequest request) > >> > GCSStorage createGCSStorage(GCSStorageCreateRequest request) > >> > boolean updateGCSStorage(GCSStorageUpdateRequest request) > >> > boolean deleteGCSStorage(GCSStorageDeleteRequest request) > >> > > >> > Secret Registration > >> > > >> > registerSecret(SecretRegistrationRequest request) > >> > deleteSecret(SecretDeleteRequest request) > >> > resolveStorageType (StorageTypeResolveRequest request) > >> > > >> > Secret - Internal APIs > >> > > >> > Optional<S3Secret> getS3Secret(S3SecretGetRequest request) > >> > S3Secret createS3Secret(S3SecretCreateRequest request) > >> > boolean updateS3Secret(S3SecretUpdateRequest request) > >> > boolean deleteS3Secret(S3SecretDeleteRequest request) > >> > > >> > Optional<AzureSecret> getAzureSecret(AzureSecretGetRequest request) > >> > AzureSecret createAzureSecret(AzureSecretCreateRequest request) > >> > boolean updateAzureSecret(AzureSecretUpdateRequest request) > >> > boolean deleteAzureSecret(AzureSecretDeleteRequest request) > >> > > >> > Optional<GCSSecret> getGCSSecret(GCSSecretGetRequest request) > >> > GCSSecret createGCSSecret(GCSSecretCreateRequest request) > >> > boolean updateGCSSecret(GCSSecretUpdateRequest request) > >> > boolean deleteGCSSecret(GCSSecretDeleteRequest request) > >> > > >> > > >> > Poc[3] : https://github.com/Jayancv/airavata-replica-catalog > (Defining API calls) > >> > Draft APIs : refer the attachment > replicaCatalogAPIsDocumentation.html[4] which generated using the Poc [3] > >> > > >> > I greatly appreciate your thoughts and feedback on the designs[5], as > they can help us improve and adopt a more generalized approach. > Additionally, I would like to identify any other factors that we should > take into account to minimize potential issues in the future. Are there any > other considerations that we should keep in mind? > >> > > >> > > >> > [1] - https://github.com/apache/airavata-data-catalog > >> > [2] - https://github.com/apache/airavata-mft > >> > [3] - https://github.com/Jayancv/airavata-replica-catalog > >> > [4] - > https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing > >> > [5] - > https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing > >> > > >> > Thanks. > >> > -- > >> > Best Regards > >> > > >> > Jayan Vidanapathirana > >> > > >> > <replicaCatalogAPIsDocumentation.html> > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jayan Vidanapathirana > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jayan Vidanapathirana > >> > > > > > > > > -- > > Best Regards > > > > Jayan Vidanapathirana > > > > -- Best Regards Jayan Vidanapathirana [image: https://lk.linkedin.com/in/jayancv] <https://lk.linkedin.com/in/jayancv>
