[
https://issues.apache.org/jira/browse/SPARK-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343646#comment-14343646
]
Sean Owen commented on SPARK-3621:
----------------------------------
In that case, it really doesn't involve Spark. The remote workers are reading
from HDFS, and can read the closest source of the data directly. I think that's
the good news; it's not even Spark-specific and is quite possible to write
right now. Yes it should not go through the driver. What is addFile() for
though? that would only work to send a file from the driver to workers, no?
> Provide a way to broadcast an RDD (instead of just a variable made of the
> RDD) so that a job can access
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-3621
> URL: https://issues.apache.org/jira/browse/SPARK-3621
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0, 1.1.0
> Reporter: Xuefu Zhang
>
> In some cases, such as Hive's way of doing map-side join, it would be
> benefcial to allow client program to broadcast RDDs rather than just
> variables made of these RDDs. Broadcasting a variable made of RDDs requires
> all RDD data be collected to the driver and that the variable be shipped to
> the cluster after being made. It would be more performing if driver just
> broadcasts the RDDs and uses the corresponding data in jobs (such building
> hashmaps at executors).
> Tez has a broadcast edge which can ship data from previous stage to the next
> stage, which doesn't require driver side processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
