[
https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza resolved SPARK-3622.
-------------------------------
Resolution: Not a Problem
> Provide a custom transformation that can output multiple RDDs
> -------------------------------------------------------------
>
> Key: SPARK-3622
> URL: https://issues.apache.org/jira/browse/SPARK-3622
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Xuefu Zhang
>
> All existing transformations return just one RDD at most, even for those
> which takes user-supplied functions such as mapPartitions() . However,
> sometimes a user provided function may need to output multiple RDDs. For
> instance, a filter function that divides the input RDD into serveral RDDs.
> While it's possible to get multiple RDDs by transforming the same RDD
> multiple times, it may be more efficient to do this concurrently in one shot.
> Especially user's existing function is already generating different data sets.
> This the case in Hive on Spark, where Hive's map function and reduce function
> can output different data sets to be consumed by subsequent stages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
