[ https://issues.apache.org/jira/browse/SPARK-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343646#comment-14343646 ]
Sean Owen commented on SPARK-3621: ---------------------------------- In that case, it really doesn't involve Spark. The remote workers are reading from HDFS, and can read the closest source of the data directly. I think that's the good news; it's not even Spark-specific and is quite possible to write right now. Yes it should not go through the driver. What is addFile() for though? that would only work to send a file from the driver to workers, no? > Provide a way to broadcast an RDD (instead of just a variable made of the > RDD) so that a job can access > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-3621 > URL: https://issues.apache.org/jira/browse/SPARK-3621 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.0.0, 1.1.0 > Reporter: Xuefu Zhang > > In some cases, such as Hive's way of doing map-side join, it would be > benefcial to allow client program to broadcast RDDs rather than just > variables made of these RDDs. Broadcasting a variable made of RDDs requires > all RDD data be collected to the driver and that the variable be shipped to > the cluster after being made. It would be more performing if driver just > broadcasts the RDDs and uses the corresponding data in jobs (such building > hashmaps at executors). > Tez has a broadcast edge which can ship data from previous stage to the next > stage, which doesn't require driver side processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org