einavh opened a new pull request, #4861: URL: https://github.com/apache/hadoop/pull/4861
I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is running spark application using pyspark 3.2.1 EMR is using Hadoop distribution:Amazon 3.2.1 my spark application is reading from one bucket in us-west-2 and writing to a bucket in us-east-1. since I'm processing a large amount of data I'm paying a lot of money for the network transport . in order to reduce the cost I have create a vpc interface to s3 endpoint in us-west-2. inside the spark application I'm using aws cli for reading the file names from us-west-2 bucket and it is working through the s3 interface endpoint but when I use pyspark to read the data it is using the us-east-1 s3 endpoint instead of the us-west-2 endpoint. I tried to use per bucket configuration but it is being ignored although I added it to the defualt configuration and to spark submit call. I tried to set the following configuration but they are ignored: '--conf', "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", '--conf', "spark.hadoop.fs.s3a.bucket.<us-west-2-bucket -name>.endpoint=<us-west-2 endpoint >", '--conf', "spark.hadoop.fs.s3a.bucket.<us-west-2-bucket -name>.endpoint.region=us-west-2", '--conf', "spark.hadoop.fs.s3a.bucket.<us-east-1-bucket -name>.endpoint=<us-east-1end point >", '--conf', "spark.hadoop.fs.s3a.bucket.<us-east-1-bucket -name>.endpoint.region=us-east-1", '--conf', "spark.hadoop.fs.s3a.path.style.access=false", '--conf', "spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", '--conf', "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", '--conf', "Dfs.s3a.bucket.<us-east-1-bucket -name>.endpoint=<us-east-1 end point >", '--conf', "Dfs.s3a.bucket.<us-west-2-bucket -name>.endpoint=<us-west-2 end point >", '--conf', "spark.eventLog.enabled=false", -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org