Amaan0907 commented on PR #772: URL: https://github.com/apache/airavata/pull/772#issuecomment-6999457664
Thanks @ShivangMishra, both points make sense. **Stacking:** GitHub only lets a PR into apache/airavata target a branch in apache/airavata, so I can't point #773 at my fork's branch for #772. Two options: a maintainer creates a feature branch here (e.g. `feature/data-storage`) and each PR targets it in turn, or I keep only one PR open at a time. For now I've converted #773, #776 and #778 to drafts so only #772 needs review. **Extensibility:** agreed, one S3-specific stack per source won't scale. Before I rework anything, does this direction match what you have in mind? - **One storage resource:** `/api/v1/data-storages` with a `storageType` (`SCP`, `S3`, later others) and a type-specific `config` that the type's own validator checks. Sharing, ownership and listing are written once. - **One backend interface:** a Go `storage.Backend` (`Stat`, `Download`, `Upload`, `List`) with a registry keyed by type. Staging and the content endpoint call the interface, never branch on type, so a new source is one implementation plus one registration. The S3 client in #778 would become the first one. - **Replicas (#776)** already record a storage type and id, so they fit as they are. Open questions: 1. Should SCP storages move onto the generic resource in this work (migrating `scp_data_storages`, keeping `/scp-data-storages` as an alias), or stay as they are with new sources only on the generic model? 2. Credentials: a generic credential resource with a type (SSH key, S3 access key, …), or keep typed credentials that a storage references by id and type? 3. `config` as a JSON column validated per type, or a table per type behind the one API? @DImuthuUpe it would help to have your view too, since this changes the plan in #730. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
