[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-4423. ------------------------------ Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 4696 [https://github.com/apache/spark/pull/4696] > Improve foreach() documentation to avoid confusion between local- and > cluster-mode behavior > ------------------------------------------------------------------------------------------- > > Key: SPARK-4423 > URL: https://issues.apache.org/jira/browse/SPARK-4423 > Project: Spark > Issue Type: Improvement > Components: Documentation > Reporter: Josh Rosen > Assignee: Ilya Ganelin > Fix For: 1.4.0 > > > {{foreach}} seems to be a common source of confusion for new users: in > {{local}} mode, {{foreach}} can be used to update local variables on the > driver, but programs that do this will not work properly when executed on > clusters, since the {{foreach}} will update per-executor variables (note that > this _will_ work correctly for accumulators, but not for other types of > mutable objects). > Similarly, I've seen users become confused when {{.foreach(println)}} doesn't > print to the driver's standard output. > At a minimum, we should improve the documentation to warn users against > unsafe uses of {{foreach}} that won't work properly when transitioning from > local mode to a real cluster. > We might also consider changes to local mode so that its behavior more > closely matches the cluster modes; this will require some discussion, though, > since any change of behavior here would technically be a user-visible > backwards-incompatible change (I don't think that we made any explicit > guarantees about the current local-mode behavior, but someone might be > relying on the current implicit behavior). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org