[ https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758488#comment-16758488 ]
Jose Torres commented on SPARK-24541: ------------------------------------- I'm not gonna lie, I didn't put a tremendous amount of thought into the title of the Jira ticket. There's a strong argument that using Netty is indeed the right decision here. (Although we have to keep scalability in mind; we'll eventually need to do some kind of multiplexing to support even moderately sized N to N shuffles, so we should probably stay compatible with that.) I'd guess that the RPC framework does carry a performance penalty from things such as extra headers, but I'd argue the major disadvantage is that it's not the right abstraction layer. RPCs normally live exclusively in the control plane. > TCP based shuffle > ----------------- > > Key: SPARK-24541 > URL: https://issues.apache.org/jira/browse/SPARK-24541 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: Jose Torres > Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org