Hello everyone, For years, we've relied on specific contributors to run and verify the integration tests for object store integrations like S3A, because the tests require credentials for specific cloud providers. I'd like to explore if we have any path forward today to bringing those tests into the pre-submit automation. If successful, I'd like to apply that strategy to the GCS integration tests, which are part of HADOOP-19343.
To make this work, we'd need to either 1) run tests in a VM hosted in the cloud provider, where credentials are vended natively from an adjacent metadata server, or 2) export credentials so that the tests can run in any VM outside the cloud provider (and be really, really, really careful to secure the access to those exported credentials). Has anyone else already explored this recently? If not, I was thinking of filing an INFRA ticket to discuss if they already have established patterns for this. This is potentially relevant to other projects. (It was the code review for FLINK-37247 that prompted me to start this conversation.) I think it makes sense to solve it in Hadoop first and then extend it to other projects. Chris Nauroth
