Amaan0907 commented on PR #772:
URL: https://github.com/apache/airavata/pull/772#issuecomment-6999457664

   Thanks @ShivangMishra, both points make sense.
   
   **Stacking:** GitHub only lets a PR into apache/airavata target a branch in 
apache/airavata, so I can't point #773 at my fork's branch for #772. Two 
options: a maintainer creates a feature branch here (e.g. 
`feature/data-storage`) and each PR targets it in turn, or I keep only one PR 
open at a time. For now I've converted #773, #776 and #778 to drafts so only 
#772 needs review.
   
   **Extensibility:** agreed, one S3-specific stack per source won't scale. 
Before I rework anything, does this direction match what you have in mind?
   
   - **One storage resource:** `/api/v1/data-storages` with a `storageType` 
(`SCP`, `S3`, later others) and a type-specific `config` that the type's own 
validator checks. Sharing, ownership and listing are written once.
   - **One backend interface:** a Go `storage.Backend` (`Stat`, `Download`, 
`Upload`, `List`) with a registry keyed by type. Staging and the content 
endpoint call the interface, never branch on type, so a new source is one 
implementation plus one registration. The S3 client in #778 would become the 
first one.
   - **Replicas (#776)** already record a storage type and id, so they fit as 
they are.
   
   Open questions:
   1. Should SCP storages move onto the generic resource in this work 
(migrating `scp_data_storages`, keeping `/scp-data-storages` as an alias), or 
stay as they are with new sources only on the generic model?
   2. Credentials: a generic credential resource with a type (SSH key, S3 
access key, …), or keep typed credentials that a storage references by id and 
type?
   3. `config` as a JSON column validated per type, or a table per type behind 
the one API?
   
   @DImuthuUpe it would help to have your view too, since this changes the plan 
in #730.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to