subject:"UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes"

Re: UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes

2020-09-26 Thread mykidong

Sorry, I have missed setting path style access of s3 property to submit. If I have added --conf spark.hadoop.fs.s3a.path.style.access=true in the spark submit, it works fine! - Kidong. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes

2020-09-20 Thread Hitesh Tiwari

Hi, Not sure if this would be useful but you can take a look. https://www.google.com/url?sa=t=web=j=https://github.com/fabric8io/kubernetes-client/issues/2168=2ahUKEwiole7bvvjrAhXWFXcKHSzvBp8QFjAAegQIBxAB=AOvVaw2-mLNJpzDlCCME-eHAXe1u Thanks & Regards, Hitesh Tiwari On Sun, 20 Sep 2020, 10:48

UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes

2020-09-19 Thread mykidong

Hi, I have already succeeded submitting spark job to kubernetes with accessing s3 object storage in the below env. At that time, I was in the following env.: Spark: 3.0.0. S3 Object Storage: Hadoop Ozone S3 Object Storage accessed by HTTP Endpoint containging IP Address, Not Host Name.