[ https://issues.apache.org/jira/browse/SPARK-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-19925: ------------------------------------ Assignee: Apache Spark > SparkR spark.getSparkFiles fails on executor > -------------------------------------------- > > Key: SPARK-19925 > URL: https://issues.apache.org/jira/browse/SPARK-19925 > Project: Spark > Issue Type: Bug > Components: SparkR > Affects Versions: 2.1.0 > Reporter: Yanbo Liang > Assignee: Apache Spark > Priority: Critical > Attachments: error-log > > > SparkR function {{spark.getSparkFiles}} fails when it was called on > executors. For examples, the following R code will fail. (See error logs in > attachment.) > {code} > spark.addFile("./README.md") > seq <- seq(from = 1, to = 10, length.out = 5) > train <- function(seq) { > path <- spark.getSparkFiles("README.md") > print(path) > } > spark.lapply(seq, train) > {code} > However, we can run successfully with Scala API: > {code} > import org.apache.spark.SparkFiles > sc.addFile("./README.md”) > sc.parallelize(Seq(0)).map{ _ => SparkFiles.get("README.md")}.first() > {code} > and also successfully with Python API: > {code} > from pyspark import SparkFiles > sc.addFile("./README.md") > sc.parallelize(range(1)).map(lambda x: SparkFiles.get("README.md")).first() > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org