[ https://issues.apache.org/jira/browse/SPARK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-13830: ------------------------------------ Assignee: Apache Spark > Fetch large directly result from executor is very slow > ------------------------------------------------------ > > Key: SPARK-13830 > URL: https://issues.apache.org/jira/browse/SPARK-13830 > Project: Spark > Issue Type: Task > Components: Spark Core > Reporter: Davies Liu > Assignee: Apache Spark > > Given two task with 100+M result on each, it take more than 50 seconds to > fetch the results. > The RPC may be not designed to handle large block, we should use block > manager for that. But currently this is based on spark.rpc.message.maxSize, > which is usually very large (> 128M) for safe, it's too large for handling > results. > We also counting the time to fetch the direct result (also deserialize it) as > schedule delay, it also make sense to only fetch much smaller blocks via > DirectResult. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org