Hi, I'm looking to have spark jobs access S3 with temporary credentials. I've seen some examples around AssumeRole, but I have a scenario where the temp credentials are provided by GetFederationToken. Is there anything that can help, or do I need to use boto to execute GetFederationToken, and then pass the temp credentials as config params?
Also, for both GetFederationToken and AssumeRole, is there a valid way of refreshing the tokens once the job executes? Temp credentials from AssumeRole are quite limited in lifetime, and even with GetFederationToken, the maximum a set of temp credentials are valid is limited to 36 hours. If there a callback or similar thing we can give to spark that will be called when credentials are about to (have) expire (expired)? Thanks, Ashic.
