[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549351#comment-16549351 ]
Sean Owen commented on SPARK-22151: ----------------------------------- It wasn't marked as Resolved, but had a Fix version. I think you just Resolved it. My Jira client reports just flag stuff like this. Usually it's because a reporter added Fix version improperly but here looks like the PR had been resolved. > PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly > ------------------------------------------------------------------ > > Key: SPARK-22151 > URL: https://issues.apache.org/jira/browse/SPARK-22151 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 2.1.1 > Reporter: Thomas Graves > Assignee: Parth Gandhi > Priority: Major > Fix For: 2.4.0 > > > Running in yarn cluster mode and trying to set pythonpath via > spark.yarn.appMasterEnv.PYTHONPATH doesn't work. > the yarn Client code looks at the env variables: > val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath) > But when you set spark.yarn.appMasterEnv it puts it into the local env. > So the python path set in spark.yarn.appMasterEnv isn't properly set. > You can work around if you are running in cluster mode by setting it on the > client like: > PYTHONPATH=./addon/python/ spark-submit -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org