[
https://issues.apache.org/jira/browse/SPARK-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen resolved SPARK-5363.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
1.2.2
1.3.0
I've merged https://github.com/apache/spark/pull/4776, which fixes one of our
reproductions of this issue (our job added and removed broadcast variables in a
way that might trigger this bug fixed by that patch). Therefore, I'm going to
mark this issue as "Resolved", but please comment here if you still observe the
issue after this latest patch.
> Spark 1.2 freeze without error notification
> -------------------------------------------
>
> Key: SPARK-5363
> URL: https://issues.apache.org/jira/browse/SPARK-5363
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Reporter: Tassilo Klein
> Assignee: Davies Liu
> Priority: Blocker
> Fix For: 1.3.0, 1.2.2, 1.4.0
>
>
> After a number of calls to a map().collect() statement Spark freezes without
> reporting any error. Within the map a large broadcast variable is used.
> The freezing can be avoided by setting 'spark.python.worker.reuse = false'
> (Spark 1.2) or using an earlier version, however, at the prize of low speed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
