Hi,
I am new to Spark and would like to have a quick question about the end user
impersonation on Spark executor process.
Basically I am running SQL queries through Spark thrift server with doAs set to
true to enable end user impersonation. In my experiment, I was able to start
session for multiple end users at the same time and all queries look fine. For
example, user A can query table 1, which is accessible to A exclusively
(according to HDFS permission). At the same time, user B can query table 2,
which is accessible to B exclusively. Looks like the end user UGI has been
flowed to the executor process successfully. I checked SparkContext code and
looks like the end user info is flowed to executor by specifying “SPARK_USER”
env variable. Correct me if I am wrong.
I only see 1 executor process running for all the queries from multiple users
in my experiment. The question is why the single process can impersonate
multiple end users at the same time. I assume the value of “SPARK_USER” env
variable should be either user A or B in the executor. Then there has to be
HDFS permission errors for the other user. But I did not see any error for any
user.
Can someone give some insights on that question? Thanks so much.
