Amaan0907 commented on issue #730: URL: https://github.com/apache/airavata/issues/730#issuecomment-6394389723
Hi @DImuthuUpe, I'd like to work on this. Could you assign it to me? A few questions before I put together a design: 1. **AWS only, or any S3-compatible store?** Supporting a custom endpoint (MinIO, Ceph, etc.) would cover on-prem storage at research sites and make testing easy with MinIO in CI. 2. **Credentials:** Where should S3 access keys live? Should they go through the existing credential store, the same way SSH keys are handled for SCP? And should they be per user or per gateway? 3. **Data model:** Should an S3 object be a new replica location type on the existing data product (e.g. `s3://bucket/key`), so one product can have both SCP and S3 replicas? Or should it be a separate product type? 4. **Virtual data catalog:** S3 has prefixes, not real directories. Is listing by prefix/delimiter acceptable for browsing? 5. **Staging to clusters:** When a job needs an S3 input, should the server download it and push it over SCP, or should the cluster pull it directly using a presigned URL? Since the long-term goal is multiple protocols, I'd also suggest a small storage interface that both SCP and S3 implement, so future protocols plug in the same way. Happy to adjust based on what you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
