[ https://issues.apache.org/jira/browse/SPARK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14227385#comment-14227385 ]
Sean Owen commented on SPARK-4170: ---------------------------------- Thanks [~boyork], I will propose a PR that resolves this with a bit of documentation somewhere. > Closure problems when running Scala app that "extends App" > ---------------------------------------------------------- > > Key: SPARK-4170 > URL: https://issues.apache.org/jira/browse/SPARK-4170 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.1.0 > Reporter: Sean Owen > Priority: Minor > > Michael Albert noted this problem on the mailing list > (http://apache-spark-user-list.1001560.n3.nabble.com/BUG-when-running-as-quot-extends-App-quot-closures-don-t-capture-variables-td17675.html): > {code} > object DemoBug extends App { > val conf = new SparkConf() > val sc = new SparkContext(conf) > val rdd = sc.parallelize(List("A","B","C","D")) > val str1 = "A" > val rslt1 = rdd.filter(x => { x != "A" }).count > val rslt2 = rdd.filter(x => { str1 != null && x != "A" }).count > > println("DemoBug: rslt1 = " + rslt1 + " rslt2 = " + rslt2) > } > {code} > This produces the output: > {code} > DemoBug: rslt1 = 3 rslt2 = 0 > {code} > If instead there is a proper "main()", it works as expected. > I also this week noticed that in a program which "extends App", some values > were inexplicably null in a closure. When changing to use main(), it was fine. > I assume there is a problem with variables not being added to the closure > when main() doesn't appear in the standard way. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org