If are using kerberized HDFS the spark principal (or whoever is running the
cluster) has to be declared as a proxy user.
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html
Once done, you call the
val ugi = UserGroupInformation.createProxyUser("joe",
Hi Kwangsun,
You may use `—proxy-user` to impersonate.
For example,
bin/spark-shell --proxy-user kent
21/04/12 23:31:34 WARN Utils: Your hostname, hulk.local resolves to a
loopback address: 127.0.0.1; using 192.168.1.14 instead (on interface en0)
21/04/12 23:31:34 WARN Utils: Set SPARK_LOCAL_IP
Hi, Spark users.
I wanted to make unknown users create HDFS files, not the OS user who
executes the spark application.
And I thought it would be possible using
UserGroupInformation.createRemoteUser(“other”).doAS(…)
However, the files are created by the OS user who launched the spark
