[ 
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119362#comment-14119362
 ] 

Mridul Muralidharan commented on SPARK-3019:
--------------------------------------------

Just went over the proposal in some detail.
[~rxin] did you take a look at the proposal in SPARK-1476 ?
The ManagedBuffer detailed in this document does not satisfy most of the 
interface or functional requirements in 1476 - which would require us to 
redesign this interface when we need to support larger than 2 GB for blocks in 
spark : unless I missed something.

> Pluggable block transfer (data plane communication) interface
> -------------------------------------------------------------
>
>                 Key: SPARK-3019
>                 URL: https://issues.apache.org/jira/browse/SPARK-3019
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, Spark Core
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>         Attachments: PluggableBlockTransferServiceProposalforSpark - draft 
> 1.pdf
>
>
> The attached design doc proposes a standard interface for block transferring, 
> which will make future engineering of this functionality easier, allowing the 
> Spark community to provide alternative implementations.
> Block transferring is a critical function in Spark. All of the following 
> depend on it:
> * shuffle
> * torrent broadcast
> * block replication in BlockManager
> * remote block reads for tasks scheduled without locality



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to