Hi all! Has anyone been through this already?
I have a spark docker images that are used in 2 different environments and each one requires a different credentials provider for s3a. That parameter is the only difference between them. When passing via --conf, it works as expected. When --conf is omitted and we pass it via .option() in the SparkSession builder, it gets ignored and we see just the default providers. Any hint is appreciated. Thanks a lot! Leandro