Hi Jayan, This looks really good. Thanks for putting together this test case. It makes it very clear how the API should be used.
Thanks, Marcus > On Apr 20, 2023, at 1:44 PM, Jayan Vidanapathirana > <jcvidanapathir...@gmail.com> wrote: > > Hi All, > > A new test case [1] has been incorporated into the replica catalog, enabling > the addition and retrieval of replica data through the data catalog pull > request [2]. The current implementation allows users to operate with S3 and > Google Cloud Storage credentials and storage. > > Ongoing implementation > How to model this replica locations as hierarchical system to support replica > groups > > [1] org.apache.airavata.ReplicaCatalogAPIClientTest#testCase1 > [2] https://github.com/apache/airavata-data-catalog/pull/28 > > Thanks. > > On Sat, Apr 8, 2023 at 3:07 AM Christie, Marcus Aaron <machr...@iu.edu> wrote: > Hi Jayan, > > > > I think we can keep them loosely coupled. When a client is searching for > files, it will search the Data Catalog. That will yield data product ids. > Then the client needs to resolve those data product ids to replica locations. > So that means that the Replica Catalog needs to record the data product id > associated with each replica location. > > > > For deployment plans, I haven’t though much about it but I think we’ll deploy > them as separate services. That is, separate JVM processes. They will likely > connect to the same backend database out of convenience, but I don’t think > they would have to do so, they could have separate databases. Does that > answer your question? > > > > Thanks, > > > > Marcus > > > > From: Jayan Vidanapathirana <jcvidanapathir...@gmail.com> > Date: Friday, April 7, 2023 at 2:00 PM > To: dev@airavata.apache.org <dev@airavata.apache.org> > Cc: Dimuthu Upeksha Wannipurage <dimuthu.upeks...@gmail.com>, Christie, > Marcus Aaron <machr...@iu.edu> > Subject: Re: Cybershuttle Replica Catalog API > > Hi Marcus, > > I sincerely appreciate you taking the time to review my changes. Actually, I > thought these two services needed to be operated independently when i'm > starting the API definition. > If we couple both services then users can simply use the Data-Catalog Data > Product APIs and Data models. Then I can remove data product APIs and models > from the replica catalog. > > > > Can I know if there is any overall deployment plan for these 2 catalogs ? > > > > Thanks. > > > > On Thu, Apr 6, 2023 at 7:44 PM Christie, Marcus Aaron <machr...@iu.edu> wrote: > > Hi Jayan, > > Thanks for sharing. One question, the airavata-data-catalog already has a > DATA_PRODUCT table and a way to store a data product's metadata. Could that > be used instead of adding a new table? > > Or more generally my question is how does this replica catalog API relate to > the data catalog API/data model? > > Thanks, > > Marcus > > > On Mar 31, 2023, at 4:11 PM, Jayan Vidanapathirana > > <jcvidanapathir...@gmail.com> wrote: > > > > Hi All, > > > > I have implemented basic flow(simple create and retrieve) of the replica > > catalog and drafted a pull request[1] to the Airavata data catalog as a new > > module. According to that implementation I have come to the following > > database structure for the replica catalog and I greatly appreciate your > > thoughts and feedback on the designs[2]. At this stage S3 storage type was > > considered as a sample. > > > > <Replica Catalog V2.drawio (1).png> > > > > Also please refer to the following google doc[3] to review the implemented > > APIs. > > > > [1] https://github.com/apache/airavata-data-catalog/pull/28 > > [2] > > https://drive.google.com/file/d/1KP-8IWdvpPvjSWUG2t41K7WQXW2f9_qN/view?usp=sharing > > [3] > > https://docs.google.com/document/d/1U-ok1ICt_EmjjxR9UuACV6g6YYkgoADfm0ECJ9ZDI3k/edit?usp=sharing > > > > Thank you. > > > > On Mon, Mar 20, 2023 at 2:20 AM Suresh Marru <sma...@apache.org> wrote: > > Hi Jayan, > > > > Can you contribute a PR to the data catalog repo so we can keep the > > feedback on that issue? > > > > Thanks for your contribution, > > Suresh > > > >> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana > >> <jcvidanapathir...@gmail.com> wrote: > >> > >> Hi All, > >> > >> I have updated the draft code base[1] with a simple workflow of adding > >> data to replica catalog. Still services are not yet finalized and will be > >> enhanced with the workflow. > >> > >> [1] https://github.com/Jayancv/airavata-replica-catalog > >> > >> Thanks. > >> > >> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana > >> <jcvidanapathir...@gmail.com> wrote: > >> Hi Dimuthu and Marcus, > >> > >> Thank you both for checking my PoC and providing valuable feedback. > >> > >> Dimuthu, > >> • Im agree with you regarding Replica location categories. It should > >> be a data catalog level attribute. > >> • To manage replica data access permissions don't we need user > >> information at Replica catalog level ? I'm a bit confused on the > >> permission management side of this catalog. > >> • ReplicaListEntry - Added to expose the list DataReplicaLocation s > >> with basic details in AllDataReplicaGetResponse which provide all the > >> replica items for the given product_id. However, here I was not > >> considering that hierarchical structure. ReplicaGroupEntry is actually a > >> one product replica which holds the file structure of the replica data. > >> According to your suggestion we can model that AllDataReplicaGetResponse > >> as follows, > >> message AllDataReplicaGetResponse { > >> data_product_id = 1 > >> repeated ReplicaGroupEntry replica_list = 2; > >> } > >> > >> message ReplicaGroupEntry { > >> string replica_group_id = 1 > >> repeated ReplicaGroupEntry directories = 2; > >> repeated DataReplicaLocation files = 3; > >> } > >> > >> Marcus, > >> • Yes, I will remove replica_id from the data catalog diagram. > >> • I added that parent_data_product_id to replica data by considering > >> full context with replica catalog and data catalog relation. But within > >> replica catalog context there is no such paranet product relationship. > >> Therefore we can rename it to data_product_id. Thanks for pointing this > >> out. > >> > >> Thanks. > >> > >> On Thu, Feb 23, 2023 at 2:48 AM Christie, Marcus Aaron <machr...@iu.edu> > >> wrote: > >> Hi Jayan, > >> > >> I would like to echo Dimuthu and say that this looks great and I > >> appreciate the effort in your pulling this all together. I have some > >> feedback to share. > >> > >> The high-level architecture diagram shows the replica id being stored in > >> the data catalog. That was an initial idea that we had, but we decided > >> that the replica catalog would store the data product id. That seems > >> reflected in your API design so I think you already know this, but I > >> wanted to point it out since the diagram might be a little confusing for > >> others. > >> > >> In the ReplicaCatalogAPI.proto the name of the data product id field is > >> "parent_data_product_id". I would suggest calling it "data_product_id" > >> instead. "parent_data_product_id" means "the id of the parent data product > >> of this data product" in the data catalog. It might be confusing to use > >> the same name in ReplicaCatalogAPI.proto. > >> > >> > >> Thanks, > >> > >> Marcus > >> > >> > On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana > >> > <jcvidanapathir...@gmail.com> wrote: > >> > > >> > Hi All, > >> > > >> > As a new contributor to the Cybershuttle project, I have been actively > >> > involved in implementing the Data Replica Catalog. This new catalog is > >> > designed to interface with both the Apache Airavata Data Catalog [1] and > >> > Airavata MFT[2]. This replica catalog should be able to store each > >> > replica resource storage details and secret/credential details specific > >> > to the storage type. The proposed high-level architecture will be as > >> > follows: > >> > > >> > > >> > > >> > I will mainly work on the highlighted area (red color box) and as an > >> > initial step started defining APIs which communicate with Replica > >> > Catalog. This API calls will be gRPC APIs and following methods will be > >> > implement, > >> > > >> > Replica Registration > >> > > >> > • registerReplicaLocation(DataReplicaCreateRequest createRequest) > >> > • updateReplicaLocation(DataReplicaCreateRequest updateRequest) > >> > • DataReplicaLocationModel > >> > getReplicaLocation(DataReplicaGetRequest getReplicaRequest) > >> > • removeReplicaLocation(DataReplicaDeleteRequest > >> > deleteReplicaRequest) > >> > • getAllReplicaLocations(AllDataReplicaGetRequest > >> > allDataGetRequest) > >> > • removeAllReplicaLocations(AllDataReplicaDeleteRequest > >> > allDataDeleteRequest) > >> > > >> > Storage Registration > >> > > >> > registerSecretForStorage(SecretForStorage request) > >> > deleteSecretsForStorage(SecretForStorageDeleteRequest request) > >> > getSecretForStorage(SecretForStorageGetRequest request) > >> > searchStorages(StorageSearchRequest request) > >> > listStorages(StorageListRequest request) > >> > resolveStorageType (StorageTypeResolveRequest request) > >> > > >> > Storage - Internal APIs > >> > > >> > S3StorageListResponse listS3Storage(S3StorageListRequest request) > >> > Optional<S3Storage> getS3Storage(S3StorageGetRequest request) > >> > S3Storage createS3Storage(S3StorageCreateRequest request) > >> > boolean updateS3Storage(S3StorageUpdateRequest request) > >> > boolean deleteS3Storage(S3StorageDeleteRequest request) > >> > > >> > AzureStorageListResponse listAzureStorage(AzureStorageListRequest > >> > request) > >> > Optional<AzureStorage> getAzureStorage(AzureStorageGetRequest request) > >> > AzureStorage createAzureStorage(AzureStorageCreateRequest request) > >> > boolean updateAzureStorage(AzureStorageUpdateRequest request) > >> > boolean deleteAzureStorage(AzureStorageDeleteRequest request) > >> > > >> > GCSStorageListResponse listGCSStorage(GCSStorageListRequest request) > >> > Optional<GCSStorage> getGCSStorage(GCSStorageGetRequest request) > >> > GCSStorage createGCSStorage(GCSStorageCreateRequest request) > >> > boolean updateGCSStorage(GCSStorageUpdateRequest request) > >> > boolean deleteGCSStorage(GCSStorageDeleteRequest request) > >> > > >> > Secret Registration > >> > > >> > registerSecret(SecretRegistrationRequest request) > >> > deleteSecret(SecretDeleteRequest request) > >> > resolveStorageType (StorageTypeResolveRequest request) > >> > > >> > Secret - Internal APIs > >> > > >> > Optional<S3Secret> getS3Secret(S3SecretGetRequest request) > >> > S3Secret createS3Secret(S3SecretCreateRequest request) > >> > boolean updateS3Secret(S3SecretUpdateRequest request) > >> > boolean deleteS3Secret(S3SecretDeleteRequest request) > >> > > >> > Optional<AzureSecret> getAzureSecret(AzureSecretGetRequest request) > >> > AzureSecret createAzureSecret(AzureSecretCreateRequest request) > >> > boolean updateAzureSecret(AzureSecretUpdateRequest request) > >> > boolean deleteAzureSecret(AzureSecretDeleteRequest request) > >> > > >> > Optional<GCSSecret> getGCSSecret(GCSSecretGetRequest request) > >> > GCSSecret createGCSSecret(GCSSecretCreateRequest request) > >> > boolean updateGCSSecret(GCSSecretUpdateRequest request) > >> > boolean deleteGCSSecret(GCSSecretDeleteRequest request) > >> > > >> > > >> > Poc[3] : https://github.com/Jayancv/airavata-replica-catalog (Defining > >> > API calls) > >> > Draft APIs : refer the attachment > >> > replicaCatalogAPIsDocumentation.html[4] which generated using the Poc [3] > >> > > >> > I greatly appreciate your thoughts and feedback on the designs[5], as > >> > they can help us improve and adopt a more generalized approach. > >> > Additionally, I would like to identify any other factors that we should > >> > take into account to minimize potential issues in the future. Are there > >> > any other considerations that we should keep in mind? > >> > > >> > > >> > [1] - https://github.com/apache/airavata-data-catalog > >> > [2] - https://github.com/apache/airavata-mft > >> > [3] - https://github.com/Jayancv/airavata-replica-catalog > >> > [4] - > >> > https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing > >> > [5] - > >> > https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing > >> > > >> > Thanks. > >> > -- > >> > Best Regards > >> > > >> > Jayan Vidanapathirana > >> > > >> > <replicaCatalogAPIsDocumentation.html> > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jayan Vidanapathirana > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jayan Vidanapathirana > >> > > > > > > > > -- > > Best Regards > > > > Jayan Vidanapathirana > > > > > > > > -- > > Best Regards > > > > Jayan Vidanapathirana > > > > > > -- > Best Regards > > Jayan Vidanapathirana >
smime.p7s
Description: S/MIME cryptographic signature
